Questions tagged with Amazon Elastic Kubernetes Service
Content language: English
Sort by most recent
Hello,
Brand new EKS cluster latest version.
Followed the first example in this guide: https://docs.aws.amazon.com/eks/latest/userguide/cross-account-access.html
Created an OIDC Identity provider on Account1 accepting requests from the EKS cluster on account 2.
In the EKS cluster, my k8s ServiceAccount resource have an annotation eks.amazonaws.com/role-arn pointing to an IAM role in account1.
Application running in the pod is a .NET6 app with the AWSSDK.DynamoDBv2 nuget package making DynamoDB queries.
It worked for a while, until at some point I got this exception:
```
Amazon.Runtime.AmazonClientException: Error calling AssumeRole for role arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
---> Amazon.SecurityToken.Model.ExpiredTokenException: Token expired: current date/time 1680295159 must be before the expiration date/time1680281898
---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
```
I do see doing a kubectl describe on my pod these information:
```
Environment:
AWS_ACCESS_KEY_ID:
AWS_SECRET_KEY:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: us-east-1
AWS_REGION: us-east-1
AWS_ROLE_ARN: arn:aws:iam::AcccountNumber:role/EKS-ServiceAccount
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mq27b (ro)
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
```
I also found [this page](https://docs.aws.amazon.com/eks/latest/userguide/pod-configuration.html) mentioning it should renew at 80% expiration time and [this page](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html) with the minimum required SDK version. I can confirm I use AWSSDK.DynamoDBv2, AWSSDK.SecurityToken and AWSSDK.Core all version later than that (3.7.100.14).
I was expecting the EKS cluster to automatically renew the token from the OIDC provider.
Why isn't it doing it?
I need to find network flows between pods, both pods from different nodes and pods from same node as well. I tried VPC Flow logs, but it only provides pod IPs and no identifier of the pod (e.g. pod id, labels). Another limitation VPC Flow logs doesn't provide flows within the same node. Also, I tried Container Insight as well, but it does not give flow information. Is there any way we can find network flows with src and destination pod information? Is there a way I can use Amazon VPC CNI for getting this information? I do not wish to use Cilium or Calico or any eBPF-based agent.
I have the IAM role and policy setup per https://eksctl.io/usage/minimum-iam-policies/. When I create the cluster all the CF stacks complete with no errors at all. But I am getting this on the screen. The acutal error is at the very bottom.
```
2023-03-30 11:51:22 [▶] completed task: create IAM role for serviceaccount "kube-system/aws-node"
2023-03-30 11:51:22 [▶] started task: create serviceaccount "kube-system/aws-node"
2023-03-30 11:51:22 [ℹ] waiting for CloudFormation stack "eksctl-tmdev-us1-pipe-prod-addon-iamserviceaccount-kube-system-AS-cluster-autoscaler"
2023-03-30 11:51:52 [▶] failed task: create serviceaccount "kube-system/aws-node" (will not run other sequential tasks)
2023-03-30 11:51:52 [▶] failed task:
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system/aws-node",
create serviceaccount "kube-system/aws-node",
}
(will continue until other parallel tasks are completed)
2023-03-30 11:51:52 [▶] failed task:
4 parallel sub-tasks: {
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system-LB/aws-lb-controller",
create serviceaccount "kube-system-LB/aws-lb-controller",
},
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system-DNS/external-dns",
create serviceaccount "kube-system-DNS/external-dns",
},
create IAM role for serviceaccount "kube-system-AS/cluster-autoscaler",
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system/aws-node",
create serviceaccount "kube-system/aws-node",
},
}
(will not run other sequential tasks)
2023-03-30 11:51:52 [▶] failed task:
2 sequential sub-tasks: {
4 sequential sub-tasks: {
wait for control plane to become ready,
associate IAM OIDC provider,
4 parallel sub-tasks: {
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system-LB/aws-lb-controller",
create serviceaccount "kube-system-LB/aws-lb-controller",
},
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system-DNS/external-dns",
create serviceaccount "kube-system-DNS/external-dns",
},
create IAM role for serviceaccount "kube-system-AS/cluster-autoscaler",
2 sequential sub-tasks: {
create IAM role for serviceaccount "kube-system/aws-node",
create serviceaccount "kube-system/aws-node",
},
},
restart daemonset "kube-system/aws-node",
},
create managed nodegroup "jenkins-pipeline-nodegroup",
}
(will not run other sequential tasks)
2023-03-30 11:51:52 [!] 1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2023-03-30 11:51:52 [ℹ] to cleanup resources, run 'eksctl delete cluster --region=us-east-1 --name=tmdev-us1-pipe-prod'
2023-03-30 11:51:52 [✖] failed to create service account kube-system/aws-node: checking whether namespace "kube-system" exists: Get "https://XXXXXXXXXXB2A140B1DB492834D6A69A.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system": dial tcp 172.11.111.111:443: i/o timeout
Error: failed to create cluster "tmdev-us1-pipe-prod"
```
When I go to EKS in the AWS Console and click on this cluster I do see a strange error at the top. Not sure if its related or not.
```
Error loading GenericResourceCollection/namespaces
```
I read this page https://docs.aws.amazon.com/eks/latest/userguide/enable-kms.html, it says that
> If you enable secrets encryption for an existing cluster and the KMS key that you use is ever deleted, then there's no way to recover the cluster. If you delete the KMS key, you permanently put the cluster in a degraded state.
My question is what does degraded state means ?
We will enable Kubernetes secret encryption in Production environment. However, our dev environment already set Kubernetes encryption to enabled. So we cannot test if there's downtime when enabling Kubernetes secret encryption.
This is for preparation and comms to user.
Hi
I'm having a problem when I install ebs-csi-driver for AWS EKS
I used the account with administratoraccess to install via AWS console but had an error "namespaces "kube-system" is forbidden: User "eks:addon-manager" cannot patch resource "namespaces" in API group "" in the namespace "kube-system""
I don't know why the administrator permission can not install add-on.
I also tried to create IAM role following Link[https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html]() but received error : Error from server (NotFound): serviceaccounts "ebs-csi-controller-sa" not found
I just started with EKS so don't know how to resolve,
Anyone can help?
Thank you
I created EKS resources via Terraform. I now want to get temporary credentials for a new role (new_dev has eks:DescribeCluster permission). It throws below error, user xxxxx has AdminitratorAccess policy. Should I add an assume role policy to the user xxxxx?
aws sts assume-role --role-arn arn:aws:iam::---:role/new_dev --role-session-name dev
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::---:user/xxxxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::---:role/new_dev
I am creating an EKS cluster from scratch but every time I do I get the following error:
2023-03-28 15:08:05 [✖] creating OIDC provider: operation error IAM:
```
CreateOpenIDConnectProvider, https response error StatusCode: 403, RequestID: bacf7543-bfe0-4b1c-982e-a81e61cef1c7, api error AccessDenied: User: arn:aws:sts::*:assumed-role/DEV-EC2-JenkinsMaster-Instance/i-09f8b9ad4eb5hhh09 is not authorized to perform: iam:TagOpenIDConnectProvider on resource: arn:aws:iam::*:oidc-provider/oidc.eks.us-east-1.amazonaws.com because no identity-based policy allows the iam:TagOpenIDConnectProvider action
```
After much effort and looking I found the following policy which I have in place.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:DeleteInternetGateway",
"Resource": "arn:aws:ec2:*:*:internet-gateway/*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"elasticloadbalancing:ModifyListener",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DescribeInstances",
"ec2:AttachInternetGateway",
"ec2:DeleteRouteTable",
"ec2:RevokeSecurityGroupEgress",
"ec2:CreateRoute",
"ec2:CreateInternetGateway",
"ec2:DescribeVolumes",
"ec2:DeleteInternetGateway",
"ec2:DescribeKeyPairs",
"iam:GetRole",
"elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
"ec2:ImportKeyPair",
"ec2:CreateTags",
"elasticloadbalancing:CreateTargetGroup",
"ecr:GetAuthorizationToken",
"ec2:RunInstances",
"ec2:DisassociateRouteTable",
"ec2:CreateVolume",
"ec2:RevokeSecurityGroupIngress",
"elasticloadbalancing:AddTags",
"ec2:DescribeImageAttribute",
"elasticloadbalancing:DeleteLoadBalancerListeners",
"ec2:DeleteNatGateway",
"autoscaling:DeleteAutoScalingGroup",
"ec2:CreateSubnet",
"ec2:DescribeSubnets",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"ecr:InitiateLayerUpload",
"ec2:AttachVolume",
"ec2:CreateNatGateway",
"ec2:CreateVpc",
"ecr:ListImages",
"ec2:DescribeVpcAttribute",
"ec2:ModifySubnetAttribute",
"autoscaling:DescribeScalingActivities",
"ec2:DescribeAvailabilityZones",
"ssm:GetParametersByPath",
"elasticloadbalancing:CreateLoadBalancerPolicy",
"ec2:ReleaseAddress",
"ec2:DeleteLaunchTemplate",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:DeleteTargetGroup",
"ec2:DescribeSecurityGroups",
"autoscaling:CreateLaunchConfiguration",
"ec2:CreateLaunchTemplate",
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
"ec2:DescribeVpcs",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DetachLoadBalancerFromSubnets",
"ec2:DeleteSubnet",
"elasticloadbalancing:RegisterTargets",
"ec2:DescribeVolumesModifications",
"ssm:GetParameter",
"ec2:AssociateRouteTable",
"elasticloadbalancing:DeleteLoadBalancer",
"ec2:DescribeInternetGateways",
"elasticloadbalancing:DescribeLoadBalancers",
"ec2:DeleteVolume",
"ssm:DeleteParameter",
"ssm:DescribeParameters",
"autoscaling:DescribeAutoScalingGroups",
"elasticloadbalancing:DescribeLoadBalancerPolicies",
"autoscaling:UpdateAutoScalingGroup",
"ec2:DescribeAccountAttributes",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
"elasticloadbalancing:RegisterInstancesWithLoadBalancer",
"ec2:DescribeRouteTables",
"ecr:BatchCheckLayerAvailability",
"ec2:DetachVolume",
"ec2:ModifyVolume",
"ec2:DescribeLaunchTemplates",
"ecr:GetDownloadUrlForLayer",
"ec2:CreateRouteTable",
"cloudformation:*",
"elasticloadbalancing:DeregisterTargets",
"ec2:DetachInternetGateway",
"ssm:GetParameters",
"ssm:DeleteParameters",
"ecr:PutImage",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"ssm:PutParameter",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"ecr:BatchGetImage",
"ecr:DescribeImages",
"ec2:DeleteVpc",
"eks:*",
"autoscaling:CreateAutoScalingGroup",
"ec2:DescribeAddresses",
"ec2:DeleteTags",
"elasticloadbalancing:ConfigureHealthCheck",
"autoscaling:DescribeLaunchConfigurations",
"ec2:DescribeDhcpOptions",
"ecr:UploadLayerPart",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:DescribeListeners",
"ec2:DescribeNetworkInterfaces",
"ec2:CreateSecurityGroup",
"ecr:CompleteLayerUpload",
"elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
"kms:DescribeKey",
"ecr:DescribeRepositories",
"ec2:ModifyVpcAttribute",
"ec2:ModifyInstanceAttribute",
"ec2:AuthorizeSecurityGroupEgress",
"elasticloadbalancing:AttachLoadBalancerToSubnets",
"ec2:DescribeTags",
"ssm:GetParameterHistory",
"ec2:DeleteRoute",
"ec2:DescribeLaunchTemplateVersions",
"ec2:DescribeNatGateways",
"elasticloadbalancing:CreateLoadBalancerListeners",
"ec2:AllocateAddress",
"ec2:DescribeImages",
"autoscaling:DeleteLaunchConfiguration",
"ec2:DeleteSecurityGroup",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyTargetGroup"
],
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
}
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"iam:CreateInstanceProfile",
"iam:DeleteInstanceProfile",
"iam:GetRole",
"iam:GetInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:ListInstanceProfiles",
"iam:AddRoleToInstanceProfile",
"iam:CreateOpenIDConnectProvider",
"iam:ListInstanceProfilesForRole",
"iam:PassRole",
"iam:CreateServiceLinkedRole",
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:DeleteRolePolicy",
"iam:DeleteServiceLinkedRole",
"iam:GetRolePolicy"
],
"Resource": [
"arn:aws:iam::*:instance-profile/eksctl-*",
"arn:aws:iam::*:role/eksctl-*",
"arn:aws:iam::*:role/aws-service-role/eks.amazonaws.com/*",
"arn:aws:iam::*:role/aws-service-role/eks-nodegroup.amazonaws.com/*",
"arn:aws:iam::*:oidc-provider/*"
]
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": "iam:GetOpenIDConnectProvider",
"Resource": "arn:aws:iam::*:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/*"
}
]
}
```
So what am I mising?
I am working on IaC EKS using terraform.
[https://www.ahead.com/resources/automate-iam-role-mapping-on-amazon-eks]()
I receive below error.
Error: creating IAM Role (eks_admin): MalformedPolicyDocument: Invalid principal in policy: "AWS":"arn:aws:iam::xxxxxxxxxxx:user/eks-test-usr"
status code: 400
```
resource "aws_iam_role" "eks_admin" {
name = "eks_admin"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal: { "AWS" : "${var.assume_role}" }
},
]
})
inline_policy {
name = "eks_admin_policy"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = ["eks:DescribeCluster"]
Effect = "Allow"
Resource = "*"
},
]
})
}
}
I pass the variable as
assum_role=["eks-test-dev","eks-test-admin"]
```
Hello Everyone,
I have a Private EKS cluster. I want to access my cluster from a new Ec2 instance having kubectl and aws cli installed. Previously, everything is fine means i am able to access my Eks cluster and performing kubectl commands. But accidentally, i deleted aws-auth-cm.yml file. Then after It gives error : "You must be logged in to the Cluster (Unauthorised)".
After that, i created a new eks cluster, with the same name, configuration and roles. And deleted previous one. Kindly, requesting or guide me how to access my eks cluster now step by step.
I studied lot of articles and posts. But problem not solved.
Hi,
I ran into an issue with our logging from Kubernetes pods. We store logging in JSON format, but saw that with big JSON logs it was breaking. So I investigated the isssue.
Found out that when you are sending big logs from the pods with for example
`cat log/test.log > /proc/1/fd/1`
It will ends in multiple parts into the node log folder (/var/log/pods/pod)
`2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
2023-03-24T11:53:33.107458625Z stdout P YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY`
This breaks the JSON. How can we avoid that sending big log files to /proc/1/fd/1, will be splitted into multiple parts?
Thank you!
Hi,
I'm currently have my cluster upgraded to v1.24, and i have already installed pod-security-admission [webhook](https://github.com/kubernetes/pod-security-admission/tree/master/webhook)
It also worked as i could see there are some warnings. But if i remove the default `eks.privileged` policy, pod will be failed to create with error `no providers available to validate pod request`.
Anything else I need to do to completely disconnect that PSP please ? If I upgrade my cluster to v1.25 now, will it be disrupted because the PSP is removed from kubernetes 1.25 ?
Thanks !
asked 9 days ago