Forbidden (403) accessing AOSS using SAML (Keycloak)

Hi Aaron, I really appreciate your response; however, a couple of points are still unclear to me:

1. I reviewed the documentation, and there is no mention of configuring IAM users or roles. I understand that the Keycloak SAML client is configured under Serverless -> Security -> Authentication.

2. From the documentation, I also understood that I can use either roles or users. Are both necessary?

3. The SAML assertion includes a NameID (myUser) and an Audience (aws:opensearch:123456798012), which match the specific fragments of the principal configured in serverless authentication (saml/123456798012/saml-keycloak/user/myUser). Is this the expected behavior? Or should the NameID contain the entire principal configured in the data-access-policy? For example:

[
  {
    "Rules": [
      {
        "Resource": ["collection/myCollection"],
        "Permission": ["aoss:*"],
        "ResourceType": "collection"
      },
      {
        "Resource": ["index/myCollection/*"],
        "Permission": ["aoss:*"],
        "ResourceType": "index"
      }
    ],
    "Principal": ["saml/123456789012/saml-keycloak/user/*"],
    "Description": "Rule 1"
  }
]

I look forward to any information you can provide to guide me. Thank you in advance.

Greeting

Hi Gabriel,
Thanks for sharing the details of your challenge with Amazon OpenSearch Serverless and Keycloak SAML integration! It sounds like you've taken some significant steps to set up authentication, and I can understand how frustrating it must be to hit a 403 error after successful authentication. Let’s dive in and work through this together. 😊

---

Clarifying the Issue

From what you’ve described, you’ve configured a SAML IdP in Keycloak 20 and connected it to an Amazon OpenSearch Serverless collection. Despite successful authentication, the Assertion Consumer Service (ACS) URL, https://collection.us-east-1.aoss.amazonaws.com/_saml/acs, is returning a 403 error. You’ve set the permissions to aoss:* and the principal user to user/* but haven’t found any useful hints in the troubleshooting docs or HAR file.

This indicates the issue might be related to either the IAM policy setup, the SAML assertion configuration, or a mismatch in the expected audience/roles between Keycloak and OpenSearch Serverless.

---

Why This Matters

SAML integration is crucial for enabling single sign-on (SSO) and ensuring secure, seamless access to OpenSearch Serverless collections. Addressing this 403 error will unlock the full benefits of federated access for your team, reduce friction in authentication, and maintain a secure setup.

---

Key Terms

• SAML (Security Assertion Markup Language): A protocol for exchanging authentication and authorization data between an IdP and a service provider (SP).
• ACS URL: The endpoint where the SP processes SAML assertions received from the IdP.
• Keycloak: An open-source identity and access management tool used as the SAML IdP.
• IAM Policy: AWS policies defining access permissions for users and roles.

---

The Solution (Our Recipe)

Steps at a Glance:

1. Verify IAM Policy Permissions.
2. Confirm Role Mapping and SAML Assertion Settings in Keycloak.
3. Check Audience and Entity ID Configuration.
4. Test the SAML Response with Tools Like SAML-Tracer.
5. Update OpenSearch Settings for Role Binding.
6. Use Advanced Debugging Techniques.
7. Implement Prevention Measures.

---

Step-by-Step Guide:

1. Verify IAM Policy Permissions
   Ensure that the IAM principal associated with the OpenSearch collection has the correct policy. The policy must allow access to aoss:*. Below is an example:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": "aoss:*",
               "Resource": "*"
           }
       ]
   }

   Attach this policy to the appropriate role or user.

---

2. Confirm Role Mapping and SAML Assertion Settings in Keycloak
   • In Keycloak, ensure the SAML client is configured with the correct Entity ID and ACS URL.
   • Add role mapping in Keycloak to pass the necessary roles to OpenSearch. For example, create a client role like opensearch-user and map it under the client settings.
   • Update the SAML assertion to include the RoleSessionName and Role attributes. Example:

     <Attribute Name="https://aws.amazon.com/SAML/Attributes/Role">
         <AttributeValue>arn:aws:iam::123456789012:role/YourRole,arn:aws:iam::123456789012:saml-provider/Keycloak</AttributeValue>
     </Attribute>
     <Attribute Name="https://aws.amazon.com/SAML/Attributes/RoleSessionName">
         <AttributeValue>Gabriel</AttributeValue>
     </Attribute>

---

3. Check Audience and Entity ID Configuration
   • Verify that the audience in Keycloak matches the OpenSearch Serverless Entity ID.
   • Ensure that the Entity ID in Keycloak matches the OpenSearch SAML configuration (typically https://collection.us-east-1.aoss.amazonaws.com).

---

4. Test the SAML Response with Tools Like SAML-Tracer
   • Use a tool like SAML-Tracer to inspect the SAML response sent from Keycloak to OpenSearch.
   • Look for any missing or incorrect attributes (e.g., Role or RoleSessionName).

---

5. Update OpenSearch Settings for Role Binding
   • Bind the roles passed in the SAML assertion to the OpenSearch permissions. For example:

     {
         "role_mappings": {
             "opensearch-user": {
                 "backend_roles": ["arn:aws:iam::123456789012:role/YourRole"],
                 "hosts": [],
                 "users": []
             }
         }
     }

---

6. Use Advanced Debugging Techniques
   • Enable Detailed Logs in Keycloak: Set the SAML logging level to DEBUG to gather more insights on assertions being generated and processed.
   • Check OpenSearch Access Logs: Verify if the SAML assertion is reaching the OpenSearch collection and whether the requests are being processed correctly.
   • Test AWS STS AssumeRole: Validate the IAM role configuration using the AWS STS CLI command:

     aws sts assume-role-with-saml --role-arn arn:aws:iam::123456789012:role/YourRole \
                                   --principal-arn arn:aws:iam::123456789012:saml-provider/Keycloak \
                                   --saml-assertion file://saml_response.xml

---

7. Implement Prevention Measures
   • Validate the SAML Assertion Format: Use online tools like SSO Circle or SAML Tool to ensure assertions comply with AWS requirements.
   • Document Configuration Details: Keep a record of Keycloak and OpenSearch settings to simplify future troubleshooting.
   • Schedule Policy Reviews: Periodically review IAM and SAML configurations to ensure they align with best practices and system updates.
   • Build a Testing Environment: Set up a non-production environment for experimenting with SAML configurations.

---

Closing Thoughts

Once you’ve verified the IAM permissions, ensured proper SAML attribute mapping, and tested the SAML response with advanced tools, the 403 error should be resolved. These steps will not only address the current issue but also strengthen your SAML integration for future use cases.

Here are some helpful documentation links to assist you further:

• Configure SAML with OpenSearch Serverless and Keycloak
• AWS OpenSearch Serverless SAML Docs
• Troubleshoot OpenSearch Dashboards

---

Farewell

I hope this enhanced guidance helps you pinpoint and resolve the issue, Gabriel. Don’t hesitate to reach out if you need further clarification or run into new challenges. You've got this! 🚀😊

---

Cheers,

Aaron 😊

Hi Gabriel,

Thanks for your follow-up! You’ve raised some excellent points, and I can see where things might still feel unclear. Let me quickly address your questions:

1. IAM Users or Roles: For Amazon OpenSearch Serverless, the key configurations indeed happen under Serverless > Security > Authentication, so you’re right that IAM roles/users aren’t directly configured in the way you’d expect with other AWS services. That said, you’ll still need to ensure the permissions in your data access policy are aligned with your SAML assertion for proper mapping.

2. Roles vs. Users: You can choose either roles or users, but they must match how the data access policy is configured. If your SAML assertion specifies a NameID like myUser, then yes, the Principal entry in the data access policy should accommodate that structure. However, if you’re using roles, the Role attribute in the SAML assertion must align with the expected IAM role ARN format.

3. NameID and Principal Matching: The NameID containing myUser seems correct, as long as it matches the Principal entry in the data access policy. In your example, if you’ve set the Principal to saml/123456789012/saml-keycloak/user/*, the NameID of myUser should work. For more precise matching, ensure the Audience in your SAML assertion is consistent with the OpenSearch setup.

For further clarity, I’d recommend reviewing the OpenSearch documentation on SAML mappings and double-checking the assertion against the access policy.

Good luck with the setup—I’m confident you’re close to resolving this! 😊

Cheers,
Aaron 🚀 😊

Hello Aaron, thank you for your patience.

I might be overlooking something obvious, I am double-checking all, but I still don’t fully understand how OpenSearch Serverless associates an Audience + NameID (e.g., aws:opensearch:123456789012 + myUser) with a Principal (saml/123456789012/saml-keycloak/user/myUser), I suppose that occurs under the hood.

To put it more clearly with an example, this SAML response:

<saml:Subject>
            <saml:NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">myUser</saml:NameID>
            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <saml:SubjectConfirmationData InResponseTo="_05a6..." NotOnOrAfter="..." Recipient="https://collection.us-east-1.aoss.amazonaws.com/_saml/acs" />
            </saml:SubjectConfirmation>
        </saml:Subject>
        <saml:Conditions NotBefore="..." NotOnOrAfter="...">
            <saml:AudienceRestriction>
                <saml:Audience>aws:opensearch:123456789012</saml:Audience>
            </saml:AudienceRestriction>
        </saml:Conditions>
. . . 
        <saml:AttributeStatement>
            <saml:Attribute Name="Role" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:basic">
                <saml:AttributeValue xmlns:xs="...">default-roles-aws</saml:AttributeValue>
. . . 
           </saml:Attribute>
        </saml:AttributeStatement>
    </saml:Assertion>
</samlp:Response>

I understand that Serverless associates (or should associate) it with the Principal: saml/123456789012/saml-keycloak/user/myUser

which is defined in my aoss-data-access data access policy. I assume there shouldn't be any additional name matching required other than the obvious account ID in Audience and NameID, correct?, i.e., in some magical way, 123456789012 + myUser correlates to saml/123456789012/saml-keycloak/user/myUser, if it is true, why I still get 403 code?

Reviewing the OpenSearch and SAML documentation, I see there are extra steps that don’t apply to the serverless version—for example, adding a "master user" in the OpenSearch dashboard, this is required?, if not, I don't know what extra step I am missing.

I apologize in advance if some of these concepts are difficult to grasp for me. I just want to ensure that the issue is not caused by a misconfiguration on my end.

Clarifying SAML Attribute Association with OpenSearch Serverless

Hi Gabriel, thank you for your detailed response and for providing a clear SAML response snippet—this helps pinpoint potential issues. Let’s break it down to ensure nothing is being overlooked.

1. Association of Audience, NameID, and Principal

You are correct that OpenSearch Serverless internally associates the SAML Audience and NameID with the Principal defined in your data access policy. This mapping happens under the hood based on a combination of:

• Account ID in the Audience attribute.
• User identifier in the NameID attribute.

For example, in your SAML response:

• The Audience is aws:opensearch:123456789012, matching your AWS account ID.
• The NameID is myUser, which should align with the wildcard pattern (user/*) or a specific entry (user/myUser) in your Principal.

As long as these values match what's configured in the data access policy, OpenSearch Serverless should recognize and authorize the user.

2. 403 Error: Why It Might Still Occur

Despite the correct association logic, a 403 error could stem from one of these areas:

• Policy Misalignment: Ensure the data access policy includes permissions for both the collection and indices. Your policy snippet looks fine, but double-check it’s applied to the correct OpenSearch collection.
• Missing Attributes: The SAML assertion must include all attributes OpenSearch expects (e.g., Role and RoleSessionName). If any required attributes are missing, the authorization might fail silently.
• Name Matching: While the Audience and NameID are primary, subtle mismatches (e.g., case sensitivity or typos) in the Principal definition could cause authorization to fail.
• SAML Assertion Format: Confirm that the SAML assertion format fully complies with AWS expectations. Even minor deviations can lead to a rejection.

3. Master User Requirement

For OpenSearch Serverless, there is no need to configure a "master user" in the OpenSearch dashboard—this step only applies to traditional OpenSearch clusters. Instead, permissions are managed entirely via the data access policy and the attributes passed in the SAML assertion.

4. Troubleshooting Steps

To resolve the 403 error, try the following:

1. Inspect the SAML Response: Use a tool like SAML-Tracer to validate the assertion sent from Keycloak. Confirm that the Role attribute matches what’s configured in the OpenSearch data access policy.

   • For example:

     <saml:Attribute Name="Role">
         <saml:AttributeValue>arn:aws:iam::123456789012:role/YourRole</saml:AttributeValue>
     </saml:Attribute>

   • Ensure the RoleSessionName is also present and valid.

2. Enable Keycloak Debug Logging: Set the SAML logging level to DEBUG in Keycloak. This can provide more insights into how the assertion is being generated.

3. Test AWS STS with the SAML Assertion: Manually test the SAML assertion using the aws sts assume-role-with-saml command. This can confirm if the assertion is correctly granting the role:

   aws sts assume-role-with-saml \
       --role-arn arn:aws:iam::123456789012:role/YourRole \
       --principal-arn arn:aws:iam::123456789012:saml-provider/Keycloak \
       --saml-assertion file://saml_response.xml

4. Double-Check OpenSearch Policy: Review your data access policy one more time to confirm that:

   • The Principal pattern matches the NameID exactly (user/myUser or user/*).
   • Permissions cover all required resources (collection and indices).

5. Cross-Verify Keycloak and OpenSearch Configurations: Ensure that:

   • The Keycloak SAML client uses the correct Entity ID (https://collection.us-east-1.aoss.amazonaws.com).
   • The ACS URL is correctly configured in Keycloak.

5. A Note on Matching Logic

Yes, OpenSearch Serverless uses a “magical” mapping logic internally to tie together the Audience, NameID, and Principal. Your understanding is correct: as long as the account ID and NameID match the data access policy, no additional name matching should be required. However, if a 403 persists, it could mean:

• A mismatch exists in the SAML attributes (e.g., case or formatting differences).
• An extra configuration step (e.g., mapping roles) is missing in Keycloak.

---

Closing Thoughts

You’re on the right track, Gabriel, and it seems you’re very close to resolving the issue. Review the above steps, paying special attention to the SAML assertion attributes and their mapping to the Principal. If everything aligns, the 403 should disappear.

Feel free to reach out again if you need further clarification or run into additional roadblocks. You’re doing great—keep going! 🚀😊

Best regards,
Aaron 😊✨