AWS EKS Pods are unable to access any AWS Services SQS SNS S3 Redis

0

Created a new EKS cluster v 1.29 and everything started failing. The last one week, i'm trying to make this work but things were failing one by one.. Finally made the app running but none of the apps able to connect with other AWS services. Gone through all the documentation and tutorials but still no luck. The App primarily uses the below AWS services, S3, SQS, SNS, SES and Redis (through ElastiCache). I believe, i'm missing something for a big time but unable to figure it out. Below are my configuration..

  1. Creating Cluster
eksctl create cluster --name=eks-cluster-1 --region=us-east-2 --version=1.29 --nodegroup-name=ng-101 --nodes=1 --nodes-max=1 --nodes-min=1 --instance-types=t2.large --asg-access --external-dns-access --full-ecr-access --alb-ingress-access --with-oidc --node-private-networking
  1. Edit the configmap to add the current root user
kubectl edit configmap aws-auth -n kube-system

mapUsers: |
  - userarn: arn:aws:iam::xxxxxxxx:root
    groups:
    - system:masXXs
  1. Create a namespace kubectl create namespace dev
  2. Create ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin-dev
  namespace: dev
  1. Create RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks-admin-dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin-dev
subjects:
- kind: ServiceAccount
  name: eks-admin-dev
  namespace: dev
  1. Created an IAM policy to attach with the ServiceAccount
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sns:*",
                "cloudfront:*",
                "s3:*",
                "ses:*",
                "sqs:*",
                "elasticache:*"
            ],
            "Resource": "*"
        }
    ]
}
  1. Executed the below statement to avoid the app unable to read from configMap.
kubectl create rolebinding default-edit --clusterrole=edit --serviceaccount=dev:default --namespace=dev
  1. Add the Policy to the ServiceAccount Role
eksctl create iamserviceaccount --name eks-admin-dev --namespace dev --cluster eks-cluster-1 --role-name eksServiceAccountForDevNamespace \
    --attach-policy-arn arn:aws:iam::XXXXXX:policy/IamPolicyForEksServiceAccount-EksAdmin --override-existing-serviceaccounts --approve
  1. Created the IAM identify Mapping for the Service Account
eksctl create iamidentitymapping --cluster eks-cluster-1 --region=us-east-2 --arn arn:aws:iam::XXXXX:role/eksServiceAccountForDevNamespace --group system:masXXs --username iwowdowd
  1. Annotate the created IAM role to the ServiceAccount
kubectl annotate serviceaccount -n dev eks-admin-dev eks.amazonaws.com/role-arn=arn:aws:iam::XXXXXXXX:role/eksServiceAccountForDevNamespace
  1. When i tried to see whether the service account has the Role execute this: kubectl describe sa eks-admin-dev
Name:                eks-admin-dev
Namespace:           dev
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXX:role/eksServiceAccountForDevNamespace
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>
  1. At this point i have validated the OIDC and looked good to me.
oidc.eks.us-east-2.amazonaws.com/id/XXXXXXX
  1. Then started deploying my app. The app service file
apiVersion: v1
kind: Service
metadata:
  name: demo-service
  labels:
    app.kubernetes.io/name: demo-service
    app.kubernetes.io/instance: demo
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: http
    name: http
  selector:
    app.kubernetes.io/name: demo-service
    app.kubernetes.io/instance: demo
  1. App Deployment Config
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-service
  labels:
    app.kubernetes.io/name: demo-service
    app.kubernetes.io/instance: demo
    app.kubernetes.io/managed-by: helm
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: demo-service
      app.kubernetes.io/instance: demo
  template:
    metadata:
      labels:
        app.kubernetes.io/name: demo-service
        app.kubernetes.io/instance: demo
    spec:
      serviceAccountName: eks-admin-dev
      containers:
      - name: demo-service
        image: ecr.XXXXXX
        imagePullPolicy: "Always"
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: dev,kubernetes
        - name: APP_ENVIRONMENT
          value: dev
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        livenessProbe:
          initialDelaySeconds: 60
          periodSeconds: 15
          failureThreshold: 6
          httpGet:
            path: /info
            port: http
        readinessProbe:
          initialDelaySeconds: 60
          periodSeconds: 15
          failureThreshold: 12
          httpGet:
            path: /info
            port: http

Any help will be appreciated! TIA

1 Answer
0

HI Check these steps to resolve issue:

Security Context and Permissions

Root User in ConfigMap: Using the root user in the aws-auth ConfigMap is a security risk. It's highly recommended to create a dedicated IAM role with specific permissions for your application instead. Refer to the AWS documentation on Kubernetes authentication with IAM roles for service accounts

IAM Policy for Service Account  

Network Connectivity

EKS Node Networking: Ensure your EKS cluster nodes have proper network configuration to reach the desired AWS services. Verify security groups and VPC configuration allow communication between the nodes and the services.

Troubleshooting Steps

Here's a step-by-step approach to troubleshoot further:

  • Review IAM Policy: Revise the Service Account's IAM policy to grant least privilege access.
  • Verify IAM Role Mapping: Double-check the Service Account is annotated with the correct IAM role ARN.
  • Test Connectivity: Try using aws CLI commands directly on the EKS nodes to validate connectivity to S3, SQS, etc., using the IAM role attached to the Service Account.
  • Check Pod Logs: Inspect logs of your application pods to identify any errors related to AWS service interaction.

EKS Troubleshooting Guide: https://docs.aws.amazon.com/eks/latest/APIReference/CommonErrors.html

Kubernetes Network Policies: https://kodekloud.com/blog/kubernetes-networking-explained/ (if you're using Network Policies)

profile picture
EXPERT
Sandeep
answered 6 days ago
  • Thanks. Unfortunately this doesn't help.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions