EKS Deployments ACCESS DENIED

I see, based on the additional information you provided, it seems like the issue might be related to the Kubernetes RBAC configuration in the "aws-auth" ConfigMap.

Even though the IAM role in the stage account has the necessary permissions to access the EKS resources, the Kubernetes RBAC configuration might not be granting the appropriate permissions to the "system:node:codebuild-kubectl" user.

Here are a few things you can try:

Verify the Kubernetes RBAC configuration: Double-check the "aws-auth" ConfigMap in the stage EKS cluster to ensure that the IAM role is correctly mapped to the appropriate Kubernetes role or ClusterRole with the necessary permissions to access the "deployments" resource in the "namespace" namespace. [1]

Try using a different Kubernetes role: Instead of mapping the IAM role to the "system:masters" group, try mapping it to a more specific Kubernetes role or ClusterRole that has the necessary permissions to access the "deployments" resource. You can create a custom Kubernetes role or ClusterRole with the required permissions and then map the IAM role to it. [2]

Check the Kubernetes version: Ensure that the Kubernetes version of your stage EKS cluster is compatible with the version of Kubernetes used by the CodeBuild project. If there's a mismatch, it could cause issues with the permissions and access.

Troubleshoot the CodeBuild project: Review the CodeBuild project logs to see if there are any additional clues or error messages that could help identify the root cause of the issue.

Try using the AWS CLI directly: Instead of using the "aws sts assume-role" command in the CodeBuild project, try using the AWS CLI directly to interact with the EKS cluster and see if you encounter the same issue. This can help you isolate the problem and determine if it's specific to the CodeBuild project or the IAM role assumption. [3]

If you're still having trouble, you can refer to the AWS documentation for more information on Kubernetes RBAC, troubleshooting EKS access, and troubleshooting CodeBuild issues:

Enabling IAM principal access to your cluster - Amazon EKS

Granting access to an IAM principal to view Kubernetes resources on a cluster - Amazon EKS

Troubleshooting CodeBuild

Let me know if you have any other questions!

Sources [1] AWS.Compute.EKS.AuthRole - AWS Telco Network Builder https://docs.aws.amazon.com/tnb/latest/ug/node-eks-authrole.html [2] Amazon EKS node IAM role - Amazon EKS https://docs.aws.amazon.comekslatestcreate-node-role.html [3] Adding the account connection and IAM roles to your deploy environment - Amazon CodeCatalyst https://docs.aws.amazon.comcodecatalystlatestipa-connect-account-addroles-env.html

Based on the additional information you provided, it seems like the issue is related to the Kubernetes RBAC configuration in the "aws-auth" ConfigMap. [1]

First, let's address the version mismatch between the client and server versions. The client is using Kubernetes version 1.14.7, while the server is using version 1.29.3. This version mismatch could be causing compatibility issues and permission problems.

To resolve this, you should ensure that the CodeBuild project is using the same Kubernetes version as the EKS cluster. You can do this by either:

Updating the CodeBuild project to use the same Kubernetes version as the EKS cluster (1.29.3).

Updating the EKS cluster to use the same Kubernetes version as the CodeBuild project (1.14.7).

Next, let's look at the "aws-auth" ConfigMap configuration:

Verify the "mapRoles" section: Ensure that the IAM role assumed by the CodeBuild project (EKScicdIAMStack-CodeBuildServiceRole-crossAccountDeployment) is correctly mapped to the appropriate Kubernetes role or ClusterRole with the necessary permissions to access the "deployments" resource in the "namespace" namespace.

Check the "mapUsers" section: Verify that the "system:node:{{SessionName}}" user is correctly mapped to the appropriate Kubernetes role or ClusterRole with the necessary permissions.

Consider using a custom Kubernetes role: Instead of mapping the IAM role to the "system:masters" group, try creating a custom Kubernetes role or ClusterRole with the specific permissions required to access the "deployments" resource in the "namespace" namespace, and then map the IAM role to this custom role.

As for the AWS CLI command to get deployments, you can use the following command:

Insert at cursor ** aws eks get-deployments --cluster-name <cluster-name> --namespace <namespace>** This command will retrieve the deployments in the specified namespace for the given EKS cluster.

If you're still having trouble, you can refer to the following AWS documentation for more information:

Enabling IAM principal access to your cluster - Amazon EKS [2]

Granting access to an IAM principal to view Kubernetes resources on a cluster - Amazon EKS

Troubleshooting CodeBuild

Sources [1] Troubleshooting IAM - Amazon EKS https://docs.aws.amazon.comekslatestsecurity_iam_troubleshoot.html [2] Amazon EKS cluster IAM role - Amazon EKS https://docs.aws.amazon.comekslatestservice_IAM_role.html

the issue might be related to the Kubernetes RBAC configuration in the "aws-auth" ConfigMap.

Even though the IAM role in the stage account has the necessary permissions to access the EKS resources, the Kubernetes RBAC configuration might not be granting the appropriate permissions to the "system:node:codebuild-kubectl" user.

Here are a few things you can try:

Verify the Kubernetes RBAC configuration: Double-check the "aws-auth" ConfigMap in the stage EKS cluster to ensure that the IAM role is correctly mapped to the appropriate Kubernetes role or ClusterRole with the necessary permissions to access the "deployments" resource in the "namespace" namespace.

Try using a different Kubernetes role: Instead of mapping the IAM role to the "system:masters" group, try mapping it to a more specific Kubernetes role or ClusterRole that has the necessary permissions to access the "deployments" resource. You can create a custom Kubernetes role or ClusterRole with the required permissions and then map the IAM role to it. [1]

Check the Kubernetes version: Ensure that the Kubernetes version of your stage EKS cluster is compatible with the version of Kubernetes used by the CodeBuild project. If there's a mismatch, it could cause issues with the permissions and access.

Troubleshoot the CodeBuild project: Review the CodeBuild project logs to see if there are any additional clues or error messages that could help identify the root cause of the issue.

Try using the AWS CLI directly: Instead of using the "aws sts assume-role" command in the CodeBuild project, try using the AWS CLI directly to interact with the EKS cluster and see if you encounter the same issue. This can help you isolate the problem and determine if it's specific to the CodeBuild project or the IAM role assumption.

If you're still having trouble, you can refer to the AWS documentation for more information on Kubernetes RBAC, troubleshooting EKS access, and troubleshooting CodeBuild issues:

Enabling IAM principal access to your cluster - Amazon EKS

Granting access to an IAM principal to view Kubernetes resources on a cluster - Amazon EKS

Troubleshooting CodeBuild

Sources [1] Cross-account IAM permissions - Amazon EKS https://docs.aws.amazon.comekslatestcross-account-access.html

The issue you're facing with the kubectl get deployments command is related to the permissions granted to the IAM role assumed by the CodeBuild job. Even though you've mapped the IAM roles in the aws-auth ConfigMap and granted the system:masters group access, it seems that the specific user "system:node:codebuild-kubectl" does not have the necessary permissions to access the "deployments" resource in the "namespace" namespace.

Here are a few things you can try to resolve this issue:

Verify the IAM role permissions: Ensure that the IAM role assumed by the CodeBuild job has the necessary permissions to access the Kubernetes resources in the "namespace" namespace. You can try adding the following permissions to the IAM role:

json

Insert at cursor

Copy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "eks:DescribeCluster", "eks:ListNodegroups", "eks:DescribeNodegroup", "eks:AccessKubernetesApi", "eks:ListFargateProfiles", "eks:DescribeFargateProfile" ], "Resource": [ "arn:aws:eks:::cluster/*" ] }, { ] } ] } Verify the aws-auth ConfigMap: Double-check the aws-auth ConfigMap to ensure that the IAM role is correctly mapped to the Kubernetes user "system:node:codebuild-kubectl". You can use the following command to view the aws-auth ConfigMap:

Insert at cursor

Copy kubectl get configmap -n kube-system aws-auth -o yaml Check for any additional RBAC rules: Ensure that there are no additional RBAC rules or policies that might be restricting the access of the "system:node:codebuild-kubectl" user to the "deployments" resource in the "namespace" namespace.

Verify the EKS cluster version: Make sure that the EKS cluster version is compatible with the Kubernetes resources you're trying to access. If the cluster is running an older version, you may need to update the cluster or adjust your code accordingly.

If you're still facing issues after trying these steps, please provide more information about your setup, such as the EKS cluster version, the IAM role permissions, and the contents of the aws-auth ConfigMap, and I'll be happy to assist you further.