Programmatic login using cognito hosted ui

We have an application that uses cognito user pool to authenticate its users. In the development environment, our users provide their username and password, they are simple users created in cognito user pool. The application is sending requests that are handled by API gateway. There is a cognito client set up, no client secret. (Maybe relevant: after authenticating with cognito, the user lands on our application via a cloudfront distribution. The redirect uri is set to that.)

Everything works fine when I log in using the cognito login ui from my browser. The flow goes like this:

• Browser application logs in using https://<HOSTEDDOMAIN>.auth.eu-central-1.amazoncognito.com/oauth2/authorize, sends client_id, redirect_uri, response_type: "code", code_challenge_method: "S256", code_challenge: some random string generated. (I am not the Frontend developer, I don't really understand everything yet, our function is called getPkceChallengeForVerifier :D).
• After successful login, the app gets tokens posting the /ouath2/token endpoint. We set grant_type: "authorization_code", same client_id, redirect_uri, and the code we have acquired just logging in.
• For each request towards the API gateway we set the header Bearer <ACCESS TOKEN>

What I am now trying to achieve is some automated smoke-tests against our api, NOT bypassing the authentication part.

To do this, first I tried to use the InitiateAuth endpoint calling

response = boto3.client("cognito-idp").initiate_auth(
        ClientId=cognito_client_id,
        AuthFlow="USER_PASSWORD_AUTH",
        AuthParameters={
            "USERNAME": os.environ["COGNITO_USERNAME"],
            "PASSWORD": os.environ["COGNITO_PASSWORD"]
        }
    )

Authenticating with the access token or id token I got both resulted in 401 errors. I digged into the troubleshooting docs https://aws.amazon.com/premiumsupport/knowledge-center/api-gateway-cognito-401-unauthorized/ and I have realized, that we have a custom scope and our api gateway is specifying that, but there is no such scope in the token I got from the initiate auth endpoint. When I decode the token payload, it is only containing the scope 'aws.cognito.signin.user.admin'.

I did some research to find out that there is no way to obtain tokens containing custom scopes. Based on the documentation, my understanding is that the app works because it does not specify any scopes so it gets every custom scope in the token. Based on what I have found, it seems there is no other way to obtain proper access tokens than using the hosted ui. https://stackoverflow.com/questions/59916001/aws-cognito-admininitiateauth-all-custom-scopes-are-missing https://stackoverflow.com/questions/55116702/cognito-authorization-code-grant-flow-for-custom-ui/55120568#55120568

Now in my test, I first GET https://<HOSTEDDOMAIN>.auth.eu-central-1.amazoncognito.com/oauth2/authorize using the same client_id, redirect_uri, response_type: "code". I get 200, and the login url prepared with params: 'https://<HOSTEDDOMAIN>.auth.eu-central-1.amazoncognito.com/login?client_id=<CLIENT_ID>&redirect_uri=<URLENCODED_REDIRECT_URI login endpoint>&response_type=code&state=<MYSTATE>

But when I try to POST this url with data={"username": username, "password": password}, (using python requests.post), I get response 400. Part of the content says An error was encountered with the requested page.

The second Stack Overflow answer refers to an archived post so I am bringing this question to repost now: https://forums.aws.amazon.com/thread.jspa?messageID=832982&#832982

When I inspect our application doing a successful login via cognito, I have caught a request with a form data, where not only username and password are set, but also _csrf and some special fields cognitoAsfData that is a jwt token encoding some data about my browser client and a field calledsigningSubmitButton that has a value of Sign+in.

How could I reliably nicely emulate these fields from my test code? Alternatively, is there any way I might have missed to leverage the id token or access token we get using the boto3.initiate_auth method to obtain a code grant with more scopes? Or is there another api?