How to investigate Aurora Postgres IAM authentication errors from rdsauthproxy

0

I have been using IAM database authentication on an Aurora for Postgres for many months now and everything worked well. A few days ago I started getting login errors until now it is impossible to login at all. I am not sure about the timeline as we only use these accounts for individual user connections. Only accounts not using IAM can login now. I am not aware of any change but I cannot pinpoint the root cause of the error.

The error I am getting in Postgres clients is this:

Unable to connect to server:
FATAL: PAM authentication failed for user "<REDACTED_USERNAME>"
FATAL: pg_hba.conf rejects connection for host "<REDACTED_IP>", user "<REDACTED_USERNAME>", database "postgres", SSL off

If I look into the Postgres logs I get a little more details:

* Trying <REDACTED_IP>:1108...
* Connected to rdsauthproxy (<REDACTED_IP>) port 1108 (#0)
> POST /authenticateRequest HTTP/1.1
Host: rdsauthproxy:1108
Accept: */*
Content-Length: 753
Content-Type: multipart/form-data; boundary=------------------------1f9a4da08078f511

* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Content-Type: text/html;charset=utf-8
< Content-Length: 0
<
* Connection #0 to host rdsauthproxy left intact
2021-12-05 14:42:43 UTC:10.4.2.137(32029):<REDACTED_USERNAME>@postgres:[7487]:LOG: pam_authenticate failed: Permission denied
2021-12-05 14:42:43 UTC:10.4.2.137(32029):<REDACTED_USERNAME>@postgres:[7487]:FATAL: PAM authentication failed for user "<REDACTED_USERNAME>"
2021-12-05 14:42:43 UTC:10.4.2.137(32029):<REDACTED_USERNAME>@postgres:[7487]:DETAIL: Connection matched pg_hba.conf line 13: "hostssl all +rds_iam all pam"
2021-12-05 14:42:43 UTC:10.4.2.137(13615):<REDACTED_USERNAME>@postgres:[7488]:FATAL: pg_hba.conf rejects connection for host "<REDACTED_IP>", user "<REDACTED_USERNAME>", database "postgres", SSL off

So it seems to be "rdsauthproxy" that rejects the authentication. My understanding is that this proxy is part of the Aurora instance and I did not find a way to get its logs where hopefully I could find any information on why the authentication is rejected.

I checked the IAM configuration in case something changed but it seems fine. The users have a policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "rds-db:connect",
            "Resource": "arn:aws:rds-db:eu-west-3:<REDACTED_ACCOUNT_ID>:dbuser:*/<REDACTED_USERNAME>"
        }
    ]
}

The usernames match exactly between IAM and Postgres. In Postgres they all have the "rds_iam" role.

Is there anything I could be missing? At least is there a way to retrieve logs of an Aurora rdsauthproxy instance that maybe could point me in the right direction?

1 Answer
0

Thank you for reaching out. I understand you are facing issues with IAM Authentication in the DB instance.

Ideally, the "PAM authentication failed" errors can occur in scenarios like If the database instance is under heavy load, due to expired tokens, connection or user name typo or SCP policy which specifically denies the required permissions.

To answer your question, we require details that are non-public information. Please open a support case with AWS using the following link.

AWS
SUPPORT ENGINEER
Arnab_S
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions