Questions tagged with Amazon Elastic Kubernetes Service

When using Step Functions with a container service like EKS or ECS how does output work for those steps? From the below two linked questions it seems as if input is done by either passing the input to the command override or by passing it as an environment variable. How do these jobs return output to be consumed by the following step? Input questions: * https://repost.aws/questions/QUQqIMl4s6QFKWtGeABPhgVA/step-functions-pass-input-into-fargate-instance * https://repost.aws/questions/QUCWMj_HPIQi6nmpUrbS3UTQ/step-functions-lambda-and-ecs **EDIT** According to this blog there isn't a way to output from ECS or Fargate tasks. So the outputs need to be deterministic based on the previous inputs. Can anyone confirm this and does this go for EKS as well? https://nuvalence.io/blog/aws-step-function-integration-with-ecs-or-fargate-tasks-data-in-and-out

``` apiVersion: apiextensions.k8s.io/v1 kind: TargetGroupBinding metadata: annotations: controller-gen.kubebuilder.io/version: v0.5.0 name: target-group spec: serviceRef: name: name port: 80 targetGroupARN: arn:aws:elasticloadbalancing:us-east-1:XXXXXX:targetgroup/name/ZZZZZZZZ ``` I am trying to create target group binding with service called `name`. Running `kubectl apply -n default -f target-group.yml` keeps on throwing error `error: unable to recognize "target-group.yml": no matches for kind "TargetGroupBinding" in version "apiextensions.k8s.io/v1"`. Have tried changing the `apiVersion` to multiple things but none of them is working. Can anyone help here what is wrong?

We are running EKS cluster with m5.12xlarge as worker nodes. Looking at cloudwatch logs we can see that "PLEG is not healthy: pleg was last seen active 3.xxm ago" error happens very frequently causing instability in our cluster. What could be the possible causes or any other underlying issue? Added details of our cluster below EKS version: v1.21.5-eks-9017834 Container Runtime: Docker Docker Version ``` Docker version 20.10.7, build f0df350 ``` Containerd Version ``` container d - 1.4.6 d71fcd7d8303cbf684402823e425e9dd2e99285d ``` runc version : ``` 1.0.0 commit: 84113eef6fc27af1b01b3181f31bbaf708715301 spec: 1.0.2-dev go: go1.15.14 libseccomp: 2.4.1 ```

I deploy AWS Load Balancer Controller and it creates and configures an application load balancer when I deploy an ingress, which is very convenient. However, the application load balancer is only created when an ingress.yaml is deployed, not when the controller is deployed. Is there a way of forcing the load balancer to be created on deployment of the Controller, before any ingresses have been deployed? I have several spring boot projects and an OAuth server deployed on my cluster - the spring boot projects' deployment require the OAuth2 server's URL. This requirement forces the deployment of the OAuth2 server separately, in order to configure/discover its hostname, before the other projects deploy. Ideally, I would like to deploy everything at once..

I am trying to get an EMR Studio running on an existing EKS cluster by following the instructions here: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-studio-create-eks-cluster.html Most of those steps went smoothly, but the issue occurred when I tried to create a managed endpoint with the following command: ``` aws emr-containers create-managed-endpoint --name [ENDPOINT_NAME] --virtual-cluster-id [VIRTUAL_CLUSTER_ID] --type JUPYTER_ENTERPRISE_GATEWAY --release-label emr-6.2.0-latest --execution-role-arn [EXECUTION_ROLE_ARN] --certificate-arn [CERTIFICATE_ARN] ``` The endpoint has been created, it stayed in CREATING status for a few minutes, then it terminated, without giving any clues what might be the problem. The result of listing the endpoints is the following: ``` { "id": "2w5v7j307hut1", "name": "ENDPOINT_NAME", "arn": "ENDPOINT_ARN", "virtualClusterId": "VIRTUAL_CLUSTER_ID", "type": "JUPYTER_ENTERPRISE_GATEWAY", "state": "TERMINATED_WITH_ERRORS", "releaseLabel": "emr-6.2.0-latest", "executionRoleArn": "EXECUTION_ROLE_ARN", "certificateArn": "CERTIFICATE_ARN", "createdAt": "2022-05-09T19:58:21+00:00", "stateDetails": "Unknown", "failureReason": "INTERNAL_ERROR", "tags": {} }, ``` Could anyone give some suggestions how to troubleshoot this? Is there a way to access logs to figure out what could be wrong? Thanks a lot! Simon

Hi - I'm looking for more information regarding the support model of EKS Anywhere. I know that infrastructure and application is managed by the customer (obviously) and the EKS Distro and the tools provided by AWS are supported by AWS. What about Kubernetes itself? For example, a customer will benefit of Kubernetes support? Is "Kubernetes Support" part of the EKS Distro support? Thanks!

Hello, Newbie to EKS here. I've just provisioned a new EKS cluster on a private subnet. I could get a pod running on the cluster, however when I try to exec into its bash or get its log, I always get: ```bash $ kubectl logs shell Error from server: Get "https://10.0.0.96:10250/containerLogs/external/shell/shell": remote error: tls: internal error $ kubectl exec -it shell -- bash Error from server: error dialing backend: remote error: tls: internal error ``` (the pod is called `shell` under namespace `external`) In the EKS console I see add-ons are all degraded...see https://files.slack.com/files-pri/T09NY5SBT-F03DZNVDEEP/image.png for screenshot. Some findings: ```bash $kubectl get certificatesigningrequests.certificates.k8s.io NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION csr-chlmr 121m kubernetes.io/kubelet-serving system:node:i-02a4c9ad44d2e5520.ec2.internal <none> Approved $ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE external shell 1/1 Running 0 125m kube-system aws-node-jqt5z 1/1 Running 0 122m kube-system coredns-7f5998f4c-mskbm 1/1 Running 0 132m kube-system coredns-7f5998f4c-ntbgt 1/1 Running 0 132m kube-system kube-proxy-mrnmp 1/1 Running 0 122m ``` Can someone please give me some hints on where I should look at? Thanks!

I may be missing something, but it seems our fargate nodes aren't updating their AMI and Kubelet version. Fargate notes are 4.14.275-207.503.amzn2.x86_64 / v1.20.7-eks-135321, but our EC2 nodes are on 5.4.188-104.359.amzn2.x86_64 / v1.20.11-eks-f17b81. We are able to update EC2 nodes as the new versions become available and we are notified. The fargate nodes are recycled often as we deploy fairly regularly. It does seem we must have received SOME update as these clusters have been deployed since 2021, and 1.20.11 was released in March? Does anyone know why our fargate nodes won't update?

Hello, I'm new to EKS and I have setup a cluster with a NodeGroup from the AWS Console. Now, I want to do this setup by code, by eksctl using the yaml file. Currently I can create the cluster properly through it, but I can't find how to create the NodeGroup. My doubt is about where and how I specify the "Node IAM Role" field that appears in the AWS Console, but in the yaml file. Thanks!

I have a use case where i'd like a nodegroup to be zero and to scale up automatically when we deploy a workload. The kubectl deployment scales correctly if the desiredcount is 1 or above but its stuck on a loop when the initial count 0. I see this issue https://github.com/aws/containers-roadmap/issues/724 still open. Is this still a shortcoming? Are there any potential workarounds for this?

Hello, Please advise does "eks:AccessKubernetesApi" allows only to view the workloads or it allows to update any kubernetes resources. If you could point me to security implications of adding "eks:AccessKubernetesApi" to a role in production, that would be really helpful. Thanks

Dears, I am not able to create a EKS service using my account. [https://eu-central-1.console.aws.amazon.com/eks/home?region=eu-central-1#/cluster-create] (EKS Create cluster) As there is a role needed for EKS Management, I created the role using the documentation beneath. [https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html#create-service-role] (Create EKS Role) After the role creation, I am navigating back to the Cluster creation page, but the role which was created using this guide is not visible and not selectable in the drop down menu. BR

I have an EKS cluster that is currently on v1.17 with a set of managed nodegroups on v1.16. I'm trying to upgrade both to v1.18, but I've run into a catch-22 problem. To upgrade the control plane version to 1.18 the nodegroups must match the current cluster version (1.17), and upgrading the nodegroups to 1.17 results in an error ("Cluster's kubernetes version 1.17 is not supported for nodegroup. Minimum supported kubernetes version is 1.18"). Trying to create a new nodegroup results in the same error. My question is: is there a way to get out of this situation and potentially upgrade both components syncronously or is this cluster simply stuck at its current version indefinitely? It seems problematic that clusters can get into this state in the first place simply because they weren't mnaually upgraded for a period of time.

Hello, we'd like to run our FPGA machines inside our EKS cluster. Does anyone know of a working AMI/ECR image combination that would allow this to happen? We've tried rolling our own in several ways with no success. Thanks in advance.

# FailedMount on emr eks virtual cluster while executing a job. I am facing issue while submitting a sample job (pi.py). Its giving following warning. Could you please suggest the reason for this. Warning FailedMount 7 minutes kubelet MountVolume.SetUp failed for volume "config-volume" : object "my-eks-namespace"/"fluentd-qcpzy7oglu651b70thts0u0tf-00000003035s3k842vo" not registered Warning FailedMount 7 minutes kubelet MountVolume.SetUp failed for volume "podtemplate-00000003035s3k842vo" : object "my-eks-namespace"/"podtemplate-00000003035s3k842vo" not registered Warning FailedMount 7 minutes kubelet MountVolume.SetUp failed for volume "00000003035s3k842vo-spark-defaults" : object "my-eks-namespace"/"00000003035s3k842vo-spark-defaults" not registered Warning FailedMount 7 minutes kubelet MountVolume.SetUp failed for volume "kube-api-access-dbwg6" : object "my-eks-namespace"/"kube-root-ca.crt" not registered I have followed below DevelopmentGuide for setting up cluster. https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-eks-cluster.html https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/virtual-cluster.html Thanks

I've been trying to pull the container runtime programmatically with the AWS CLI or Boto3, but haven't been able to find a specific class or method that provides the information I'm looking for. The information I'm looking for is documented here on line item number 4.: https://docs.aws.amazon.com/eks/latest/userguide/view-nodes.html I've searched through the EKS client class in boto3 documentation and haven't found anything that would get more granular than using describe_cluster() and into the node details: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/eks.html As far as infrastructure goes, I followed the exact example from the AWS documentation for setting up EKS with fargate nodes: https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html I can see on the nodes page that it includes all the information I'm looking for, but I'm not sure what class would provide me that information programmatically. The intent is to pull container runtimes across all EKS clusters from multiple AWS accounts. Thanks!

Hi, I have this ingress configuration: ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: "oidc-ingress" annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}' alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300 external-dns.alpha.kubernetes.io/hostname: example.com !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! alb.ingress.kubernetes.io/auth-type: oidc alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate alb.ingress.kubernetes.io/auth-idp-oidc: '{"issuer":"https://login.microsoftonline.com/some-id/v2.0","authorizationEndpoint":"https://login.microsoftonline.com/some-id/oauth2/v2.0/authorize","tokenEndpoint":"https://login.microsoftonline.com/some-id/oauth2/v2.0/token","userInfoEndpoint":"https://graph.microsoft.com/oidc/userinfo","secretName":"aws-alb-secret"}' !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! spec: rules: - http: paths: - pathType: Prefix path: / backend: service: name: ssl-redirect port: name: use-annotation - pathType: Prefix path: /jenkins backend: service: name: jenkins port: number: 8080 - pathType: Prefix path: / backend: service: name: apache port: number: 80 ``` If I `kubectl appy` this `Ingress` config it will apply `annotations` to all routing rules, which means: ``` /* /jenkins /jenkins/* ``` I would like to apply `OIDC annotations` only for the `Jenkins rules`, it means: 1. If I open `https://example.com` it will be available to everyone. 2. If I open `https://example.com/jenkins`, it will redirect me to `OIDC auth` page. I can do this manually through `AWS console` when I remove `authenticate rule` from `/*` and leave it for `/jenkins/*` only. However I would like to achieve this through `Ingress annotations` to be able to automate this process. Please how can I do this? Thanks for your help.

I have 3 eks clusters, and on all of them the: MutatingWebhookConfiguration pod-identity-webhook is created with apiversion: admissionregistration.k8s.io/v1beta1 which will be removed in 1.22 Is it my responsability to re-create the whole pod-identity setup with something supported in 1.22+, or will the upgrade managed this and the admission webhook will still be running on the managed api servers ?

Hello I am having difficulties in bringing an EKS cluster back into compliance **Cluster:** I have an eks cluster with : - 6 EKS Plane Control Networks (network 1-6) i. Network 1/2/3 are in a RA routing table with a 0.0.0.0/0 which refers to an Internet Gateway ii. Network 4/5/6 are in an RB routing table with a 0.0.0.0/0 that refers to a NAT Gateway (+ other routes to my company network) - 4 cluster nodegroupe with networks 4/5/6 used for worker nodes - My EKS cluster has a Public and Private API ( => From a node, when I do a DNS resolution I do see a private IP) **Target:** EKS cluster with : - 6 EKS Plane Control Networks (network 1-6) i. Network 1/2/3 in a RA routing table with a 0.0.0.0/0 that refers to an Internet Gateway ii. Network 4/5/6 also in the RA routing table - 4 cluster nodegroupe i. Nodegroupe 1 : Use networks 10 and should be in the RC routing table with 0.0.0.0/0 which refers to a new NAT Gateway (+ other routes to my company network) ii. Nodegroupe 2 : Use networks 11 and should be in the RC routing table with 0.0.0.0/0 which refers to a new NAT Gateway (+ other routes to my company network) iii. Nodegroupe 3 : Use networks 12 and should be in the RC routing table with 0.0.0.0/0 which refers to a new NAT Gateway (+ other routes to my company network) iiii. Nodegroupe 4 : Use networks 13 and should be in the RC routing table with 0.0.0.0/0 which refers to a new NAT Gateway (+ other routes to my company network) **Problem** When creating a new nodegroup to replace an existing one, I indicate network 10/11/12 or 13 The RC routing table is OK with the NAT Gateway Problem: the node can't join the cluster (error message: **Instances failed to join the kubernetes cluster**) I can see the EC2 instance being created in the right network 10/11/12 or 13 I don't understand the problem, why the nodes in this network 10/11/12 or 13 can't join the API cluster through the ENI in network 1-6? When I create a new nodegroup and I indicate a network 1-6 (network on route table RA or RB) it works without problem Sincerely

I am trying to setup the [AWS node termination handler](https://github.com/aws/aws-node-termination-handler), and am running into an issue where the EventBridge rule is invoked, but no messages are showing up in the sqs queue. I have tested and the termination handler is able to communicate with the SQS queue. I have also tested spinning instances up and down, and see the rule invocations for the EventBridge rules. However, there are no messages appearing in the queue... NOTE: I tried adding a photo here from cloudwatch showing rule invocations but no messages appearing in the queue, it seems like pictures are not supported here yet... Below are my configs for this: SQS policy: ```hcl resource "aws_sqs_queue_policy" "termination_handler_queue_policy" { queue_url = module.termination_handler_queue.sqs_queue_id policy = jsonencode({ "Version" : "2012-10-17", "Id" : "sqspolicy", "Statement" : [ { "Sid" : "TermEventsToHandlerQueue", "Effect" : "Allow", "Principal" : { "Service" : ["events.amazonaws.com", "sqs.amazonaws.com"] }, "Action" : "sqs:*", "Resource" : "${module.termination_handler_queue.sqs_queue_name}", "Condition" : { "ArnEquals" : { "aws:SourceArn" : ["arn:aws:events:us-east-2:${local.account_id}:rule/node-termination-asg-lifecycle-rule", "arn:aws:events:us-east-2:${local.account_id}:rule/node-termination-ec2-status-rule", "arn:aws:events:us-east-2:${local.account_id}:rule/node-termination-ec2-spot-interruption-rule", "arn:aws:events:us-east-2:${local.account_id}:rule/node-termination-ec2-rebalance-rule" ] } } } ] }) } ``` EventBridge Config: ```hcl module "termination_handler_eventbridge" { source = "terraform-aws-modules/eventbridge/aws" version = "~> 1.14.0" create_bus = false rules = { node-termination-asg-lifecycle = { description = "Capture eks asg lifecycle events." event_pattern = jsonencode({ "source" : ["aws.autoscaling"], "detail-type" : ["EC2 Instance Launch Successful", "EC2 Instance Terminate Successful", "EC2 Instance Launch Unsuccessful", "EC2 Instance Terminate Unsuccessful", "EC2 Instance-launch Lifecycle Action", "EC2 Instance-terminate Lifecycle Action"], "detail" : { "AutoScalingGroupName" : ["eks-Group_A", "eks-Group_B"] } }) enabled = true } node-termination-ec2-status = { description = "Capture ec2 status events" event_pattern = jsonencode({ "source" : ["aws.ec2"], "detail-type" : ["EC2 Instance State-change Notification"] }) enabled = true } node-termination-ec2-spot-interruption = { description = "Capture spot interruption events" event_pattern = jsonencode({ "source" : ["aws.ec2"], "detail-type" : ["EC2 Spot Instance Interruption Warning"] }) enabled = true } node-termination-ec2-rebalance = { description = "Capture ec2 rebalance events" event_pattern = jsonencode({ "source" : ["aws.ec2"], "detail-type" : ["EC2 Instance Rebalance Recommendation"] }) enabled = true } } targets = { node-termination-asg-lifecycle = [ { name = "termination_handler-sqs-life" arn = module.termination_handler_queue.sqs_queue_arn }, ] node-termination-ec2-status = [ { name = "termination_handler-sqs-status" arn = module.termination_handler_queue.sqs_queue_arn }, ] node-termination-ec2-spot-interruption = [ { name = "termination_handler-sqs-int" arn = module.termination_handler_queue.sqs_queue_arn }, ] node-termination-ec2-rebalance = [ { name = "termination_handler-sqs-rebalance" arn = module.termination_handler_queue.sqs_queue_arn }, ] } tags = { Name = "node-termination-handler-bus" Service = "aws-node-termination-handler" } } ```

Hi all, Brand new created EKS cluster (using eksctl), created... working... leave for night, next morning, Nodes in NotReady state ? Not sure, but know that most/all next mornings I would have to renew/update my .aws/credentials with new keys, don't know if thats related/relevant.

Hello, in order to take advantage of higher pod density along with network policies we removed AWS VPC CNI add-on (aws-node) and rely on Calico CNI add-on (calico-node) for pod network management. After removing aws-node daemonset and terminating EKS worker nodes, new nodes were created by ASG and all pods scheduled. My question, while most pods scheduled on the new nodes have been assigned IP address from Calico IP pool, as expected, a number of pods, mostly daemonset pods, have been assigned the same IP as the node IP. Any help with this will be highly appreciated. JS

I have setup the following workflow: - private REST API with sources `/POST/event` and `/POST/process` - a `VPCLink` to an `NLB` (which points to an `ALB` pointing to a microservice running on `EKS`) - a `VPC endpoint` with DNS name `vpce-<id>-<id>.execute-api.eu-central-1.vpce.amazonaws.com` with `Private DNS enabled` - an EventBridge `EventBus` with a rule that has two targets: 1 `API Destination` for debugging/testing and 1 `AWS Service` which points to my private REST Api on the source `/POST/process` - all required `Resource Policies` and `Roles` - all resources are defined within the same AWS Account The **designed** worflow is as follows: - invoke `POST/event` on the VPC endpoint (any other invocation is prohibited by the `Resource Policy`) with an `event` payload - the API puts the `event` payload to the `EventBus` - the `EventBusRule` is triggered and sends the `event` payload to the `POST/process` endpoint on the private REST API - the `POST/process` endpoint proxies the payload to a microservice running on EKS (via `VPCLink` > `NLB` > `ALB`> `k8s Service`) **What does work** so far: - invoking `POST/event` on the VPC endpoint - putting the `event` payload to the `EventBus` - forwarding the `event` payload to the `API Destination` set up for testing/debugging (it's a temporary endpoint on https://webhook.site) - testing the `POST/event` and `POST/process` integration in the AWS Console (the latter is verified by checking that the `event` payload reaches the microservice on EKS successfully) That is all single steps in the workflow seem to work, and all permissions seem to be set properly. **Whad does not work **is invoking the `POST/process` endpoint from the `EventBusRule`, i.e. invoking `POST/event` does not invoke `POST/process` via the `EventBus`, _although_ the `EventBusRule` was triggered. So my **question** is: **How to invoke a private REST API endpoint from an EventBusRule?** **What I have already tried:** - change the order of the `EventBusRule targets` - create a Route 53 record pointing to the `VPC endpoint` and treat it as an (external) `API Destination` - allow access from _anywhere_ by _anyone_ to the REST API (temporarily only, of course) **Remark on the design:** I create _two_ endpoints (one for receiving an `event`, one for processing it) with an EventBus in between because - I have to expect a delay of several minutes between the `Event Creation/Notification` and the successful `Event Processing` - I expect several hundred `event sources`, which are different AWS and Azure accounts - I want to keep track of all events that _reach_ our API and of their successful _processing_ in one central EventBus and _not_ inside each AWS account where the event stems from - I want to keep track each _failed_ event processing in the same central EventBus with only one central DeadLetterQueue

The context is enabling authenticated and authorized access to Kubernetes Dashboard in an AWS EKS instance via an AWS ALB configured with OIDC authenticate. The Kubernetes Dashboard is being protected by an AWS ELBv2 load balancer. It is created and configured by Ingress resource and the ALB Controller v2 (v2.2.3). This works. FWIW: We are Azure AD as the IdP. We are also using the OIDC Provider configuration on the Kubernetes (EKS) API. We are successfully using OIDC authenticated access from kubectl to access the API and apply RBAC. We are using ClusterRoleBinding to test with Cluster Admin users. Authentication works. However, the Kubernetes Dashboard still presents its internal token challenge page, because it does not get the Access Token, because the ALB removed it and put it in the ```X-AMZN-OIDC-*``` header. We had some success with: ``` alb.ingress.kubernetes.io/configuration-snippet: auth_request_set $token $upstream_http_authorization; proxy_set_header Authorization $token; proxy_pass_header Authorization; ``` Is this the best way to do this? Is there a better way to configure the ALB to attach the Access Token it obtained from the IdP's token endpoint as an ```Authorization: Bearer <token>```, rather than in a separate header?

Hello aws re:Post I am currently designing a solution that would allow my users to be able to provision workloads on demand through a click-to-deploy interface. In order to minimize costs, I thought of an architecture as follows : 2 K8S clusters (1 on-premise and 1 on AWS) - On-premise : Developer Platform management cluster - nodePool mgmt - Crossplane - Customer cluster definition - Catalog definition - nodePool tools - APIs - Prometheus - Grafana - ... - AWS : Customer resources cluster - nodePool customerName (one per customer) - ns mgmt (created on customer register) - ns tenant (one per tenant : production, development, ..) If you don't know Crossplane, you can look at the documentation here : https://crossplane.io/docs/v1.7/ but it is a bit like AWS Controllers for Kubernetes. It provides infrastructure abstraction as Kubernetes CRDs. First, I want to get feedback on such an architecture and know if you have better ideas/suggestions. Secondly, I would like to get an opinion on how to manage clients on the AWS side please. Thanks in advance for your answers.

We have an EKS Cluster in ap-southeast-2 region. The Kubernetes version is 1.20 and the Platform version is eks.3. We want to restrict access to service to only certain Pods. For example, we have serviceA and serviceB, how would we configure a Pod so that Pod can only access to serviceA not serviceB? I know the `NetworkPolicy` can do the job, but as far as I know the `NetworkPolicy` resource requires Calico CNI which is not supported by EKS Fargate. EKS Fargate seems to offer `SecurityGroupPolicy` but I don't think there's a way to attach `SecurityGroupPolicy` to a Service.

I'm attempting to startup an EKS pod following the getting started guide. When on the step add a Fargate Profile I'm getting the following error. > "Misconfigured PodExecutionRole Trust Policy; Please add the eks-fargate-pods.amazonaws.com Service Principal" However it's already added to my trust policy. So not sure what to do `{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "eks-fargate-pods.amazonaws.com" }, "Action": "sts:AssumeRole", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:<region>:111122223333:fargateprofile/<pod name>/*" } } } ] }` I'm currently following this step. https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html

I am creating a Managed Nodegroup for EKS using CloudFormation. I have an EC2 Launch Template with a `CapacityReservationSpecification` defined. The Launch Template is linked to the Managed Nodegroup using CloudFormation. When the Managed Node Group is initialised the Launch Template is copied with an `eks-***` prefix in the name. The `CapacityReservationSpecification` is not copied to the newly generated Launch Template. Cloud Formation script Example: LaunchTemplate: ``` Resources: LaunchTemplateAux: Type: 'AWS::EC2::LaunchTemplate' Properties: LaunchTemplateData: InstanceType: t3.medium CapacityReservationSpecification: CapacityReservationTarget: CapacityReservationResourceGroupArn: {{reservation_group_arn}} MetadataOptions: HttpPutResponseHopLimit: 2 HttpTokens: optional SecurityGroupIds: - xxxxx LaunchTemplateName: !Sub '${AWS::StackName}Aux' ``` NodeGroup: ``` ManagedNodeGroupAux: Type: 'AWS::EKS::Nodegroup' Properties: AmiType: AL2_x86_64 ClusterName: test-cluster Labels: alpha.eksctl.io/cluster-name: test-cluster alpha.eksctl.io/nodegroup-name: test-ng-aux LaunchTemplate: Id: !Ref LaunchTemplateAux NodeRole: node-instance-role::NodeInstanceRole' NodegroupName: test-nodegroup ScalingConfig: DesiredSize: 1 MaxSize: 2 MinSize: 1 Subnets: - xxx ``` The resulting launch templates are as follows. Obtained using the following command `aws ec2 describe-launch-template-versions --launch-template-id <template-id>` My Launch template Output: ``` { "LaunchTemplateVersions": [ { "LaunchTemplateId": "lt-xx", "LaunchTemplateName": "test-cluster-ngAux", "VersionNumber": 1, "CreateTime": "2022-03-24T12:35:05+00:00", "CreatedBy": "xxx:user/xxx", "DefaultVersion": true, "LaunchTemplateData": { "InstanceType": "t3.medium", "SecurityGroupIds": [ "sg-xxx" ], "CapacityReservationSpecification": { "CapacityReservationTarget": { "CapacityReservationResourceGroupArn": "arn:aws:resource-groups:xxxxx:group/my-group" } }, "MetadataOptions": { "HttpTokens": "optional", "HttpPutResponseHopLimit": 2 } } } ] } ``` Launch template copied by EKS API: ``` { "LaunchTemplateVersions": [ { "LaunchTemplateId": "lt-xxx", "LaunchTemplateName": "eks-xxx", "VersionNumber": 1, "CreateTime": "2022-03-24T12:35:46+00:00", "CreatedBy": "xxx:assumed-role/AWSServiceRoleForAmazonEKSNodegroup/EKS", "DefaultVersion": true, "LaunchTemplateData": { "IamInstanceProfile": { "Name": "xxx" }, "ImageId": "ami-0c37e3f6cdf6a9007", "InstanceType": "t3.medium", "UserData": "xxx", "TagSpecifications": [ { "ResourceType": "volume", "Tags": [ { "Key": "eks:cluster-name", "Value": "test-cluster" }, { "Key": "eks:nodegroup-name", "Value": "test-cluster-ng-aux" } ] }, { "ResourceType": "instance", "SecurityGroupIds": [ "xxx" ], "MetadataOptions": { "HttpTokens": "optional", "HttpPutResponseHopLimit": 2 } } } ] } ```

Hi Experts, I need to to change an EFS filesystem from General Performance to Max I/O. The only way to do this is to create a new FS and use DataSync to copy the data. EFS is the underlying storage for my EKS cluster. My approach is to: 1. create the new FS. 2. copy the data with DataSync. 3. Stop services on the EKS cluster. 4. Change the mount points on the CSI Driver. 5. Restart application services. Is the best approach? Or are there other documented steps to change EFS filesystems with an EKS cluster. Thanks in advance.

while deploying from the helm chart there are 3 pods working which is stateless, however, there are some other pods unable to mount the volumes to the corresponding Postgres & Redis pod. and used the same chart to deploy in other eks cluster and its working as expected. **error logs is below:** mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2b/vol-0b8a7301623f89c7f --scope -- mount -t ext4 -o defaults /dev/xvdbo /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2b/vol-0b8a7301623f89c7f Output: Running scope as unit run-21761.scope. mount: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2b/vol-0b8a7301623f89c7f: wrong fs type, bad option, bad superblock on /dev/xvdbo, missing codepage or helper program, or other error. Warning FailedMount 2s kubelet MountVolume.MountDevice failed for volume "pvc-cec820c5-e53a-41c1-9a8b-c8b14218f990" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2b/vol-0b8a7301623f89c7f --scope -- mount -t ext4 -o defaults /dev/xvdbo /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/eu-west-2b/vol-0b8a7301623f89c7f Output: Running scope as unit run-21972.scope. **pod status:** saleor-588b7c95cd-j8k4k 0/1 Running 12 26m saleor-dashboard-78bdcf6ff9-tlfgg 1/1 Running 0 26m saleor-postgresql-0 0/1 ContainerCreating 0 26m saleor-redis-master-0 0/1 ContainerCreating 0 26m saleor-redis-slave-0 0/1 ContainerCreating 0 26m saleor-storefront-84ff9f4967-l9l4m 1/1 Running 0 26m saleor-worker-6b9887fc47-4rtch 1/1 Running 0 26m

Below is quoted from the document about Amazon EKS platform versions > When new Amazon EKS platform versions become available for a minor version: > * The Amazon EKS platform version number is incremented (eks.<n+1>). * Amazon EKS automatically upgrades all existing clusters to the latest Amazon EKS platform version for their corresponding Kubernetes minor version. * Amazon EKS might publish a new node AMI with a corresponding patch version. However, all patch versions are compatible between the EKS control plane and node AMIs for a given Kubernetes minor version. I have a no. of questions about the second bullet above. 1. When an Amazon EKS platform is updated, will all the underlying pods in each k8s cluster be drained and updated in a rolling manner? 2. Will any notifications be sent prior to a EKS platform update ? 3. Is EKS platform version update a scheduled update as done in RDS ?

I have a springboot microservice which runs on AWS EKS. * I am generating a .tfvars (Terraform runtime parameters) file inside this microservice * I would like to checkin this file on Bitbucket I have a couple of queries here: * If I create a .tfvars file (or any file), where would that file be saved by Springboot? 1. Will the file be stored in the file system of the container? 2. Or is it mandatory to use an EFS to store this file? * Would I be able to clone a Bitbucket repository to checkin my .tfvars file? 1. Will the clone be stored in the file system of the container? 2. Or is it mandatory to use an EFS to store cloned repos?

CoreDNS has a ETCD plugin https://coredns.io/plugins/etcd/, which essentially allows for dynamic DNS by reading the values from ETCD. Since EKS is managed, it means we can't access the etcd instance on master node, that's fine as I can create my own etcd cluster (and I did). Below is my coredns deployment ``` apiVersion: v1 data: Corefile: | .:53 { errors health log kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } etcd { path /skydns endpoint http://etcd-cluster-ip.default.svc.cluster.local:2379 fallthrough } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } kind: ConfigMap metadata: annotations: {} labels: eks.amazonaws.com/component: coredns k8s-app: kube-dns name: coredns namespace: kube-system ``` The issue I face now is that the master node is not able to resolve the ClusterIP DNS `etcd-cluster-ip.default.svc.cluster.local`, which is the ClusterIP of my etcd cluster. If I change that DNS with the actual ClusterIP, name resolution works as expected and CoreDNS is able to access ETCD How can the master node resolve the DNS of my cluster ? I see below line in coredns logs ``` {"level":"warn","ts":"2022-03-16T20:44:42.352Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-fd406ba0-cc21-4132-bfef-ca14e3fd4eb3/etcd-cluster-ip.default.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp: lookup etcd-cluster-ip.default.svc.cluster.local on 10.0.0.2:53: no such host\""} ```

there is requirement to create application load balancer with ingress or external traffic going to 2 different port for example port 80 and 81 into EKS Cluster, any sample yaml config file to setup this requirement ? we are trying to create the config using annotation with alb.ingress.kubernetes.io/group.name with two separate rules service port 80 and service port 81, but when trying to run the yaml file the second rules service is overwriting on the ALB Rules. trying to create something similar with this post https://aws.amazon.com/blogs/containers/how-to-expose-multiple-applications-on-amazon-eks-using-a-single-application-load-balancer/ , but the differences is the requirement to have single services with 2 port [ingress] Thanks

Hi, We have an EKS Cluster wich is stuck in Updating stage for multiple days now. We no longer need the cluster but are not able to delete is because it is in an "updating" status. And in the mean time we are paying for this unusable cluster. How can I resolve this issue and delete the cluster? Kind regards,

Hello, I have an application deployed in an EKS cluster in `Account A` that is trying to read an item from a DynamoDB table in `Account B`. I have done the following : * I have created a role in `Account B` called `DynDBReadAccess` with a policy that allows someone to perform the `dynamodb:GetItem` action on the table `arn:aws:dynamodb:us-east-1:<Account B>:table/myTable`. * I then created the role `CrossAccountDynDBAccess` in `Account A` with permissions to perform the `sts:AssumeRole` action and assume the role `arn:aws:iam::<Account B>:role/DynDBReadAccess`. * I updated the trust policy in the `DynDBReadAccess` role to trust the Principal `arn:aws:iam::<Account A>:role/CrossAccountDynDBAccess`. I have a pod with aws cli deployed in the cluster for debugging purposes. Now, I do the following. * I exec into the pod and run `aws sts get-caller-identity`, I see the correct assumed role `CrossAccountDynDBAccess`. * Then I run `aws sts assume-role --role-arn "arn:aws:iam::<Account B>:role/DynDBReadAccess" --role-session-name crossAccountSession` and I get the temporary credentials. * I set the environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` with the temporary credentials I just received. * I run the following command `aws dynamodb get-item --table-name myTable --key 'somekey'` and I get the following error `An error occurred (AccessDeniedException) when calling the GetItem operation: User: arn:aws:sts::<Account B>:assumed-role/DynDBReadAccess/crossAccountSession is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:ap-southeast-1:<Account B>:table/myTable` I thought that once the roles, permissions and trust policies were set, cross account access should be possible. **Can someone tell me what is missing?** Some other points to note * The OIDC endpoint and IRSA role has been enabled in the EKS cluster and the service account for the cluster has been created. * The aws cli pod that I deployed has been deployed with the service account mapped to the IRSA role. * I tried doing the same thing with a lambda function where I created a lambda function in `Account A` that will read an item from the same DynamoDB table. The execution role of the lambda function assumes the `arn:aws:iam::<Account B>:role/DynDBReadAccess` and reads an item from the same DynamoDB table. **This works**.

When I deploy my HPA's I am choosing ``apiVersion: autoscaling/v2beta2`` but kubernetes is making them autoscaling/v2beta1 For example: If I deploy this. ``` apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: surething-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: surething minReplicas: 2 maxReplicas: 4 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 100 periodSeconds: 15 scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 selectPolicy: Max metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 100 ``` I will get this ``` apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: surething-hpa namespace: dispatch-dev uid: 189cee35-c000-410b-954e-c164a08809e1 resourceVersion: '404150989' creationTimestamp: '2021-04-04T17:30:48Z' labels: app: dispatch deployment: dev microservice: surething annotations:... selfLink:... status:... spec: scaleTargetRef: kind: Deployment name: surething apiVersion: apps/v1 minReplicas: 2 maxReplicas: 4 metrics: - type: Resource resource: name: cpu targetAverageUtilization: 100 ``` All the documentation I can find on EKS and HPA's says that I should be able to use ``apiVersion: autoscaling/v2beta2``. My cluster is version 1.21 and my nodegroup is as well. When I run ``kubectl api-versions`` I can find ``autoscaling/v2beta2`` in the list. I'm at wits end on this one. Can someone tell me what I am doing wrong?

The help page in the console describes it as > The IP address range from which cluster services will receive IP addresses. So what are "cluster services"? Is that the control plane or something else?

I'm trying to use parameter store and secret manager in my EKS cluster but i keep getting this error: MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to get secretproviderclass fastcode/helloworld-secrets, error: SecretProviderClass.secrets-store.csi.x-k8s.io "helloworld-secrets" not found and inside secret store provider logs: secretproviderclasspodstatus_controller.go:99] "failed to patch secret owner ref" err="failed to get spc helloworld-secrets, err: SecretProviderClass.secrets-store.csi.x-k8s.io \"helloworld-secrets\" not found" Both pod and SecretProviderClass are created with helm. SecretProviderClass and pods are in the same namespace ``` apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: helloworld-secrets spec: provider: aws parameters: objects: | - objectName: "/password/db" objectType: "ssmparameter" objectAlias: "dbpassword" - objectName: "/password/instance" objectType: "ssmparameter" objectAlias: "dbinstancepassword" ``` ``` volumes: - name: secrets-store-inline csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "helloworld-secrets" ``` What should i do? Thanks

we created and manage our EKS cluster with eksctl for quite a while. Now we are faced with a use-case where we need multiple interfaces in a pod. EKS supports multus to do exactly this. All documentations I'm able to find start with a cloudformation script to setup new nodegroups. Im not exactly sure how to integrate this into our existsing workflow with eksctl. We already have running nodes. So I just skipped that step and installed the multus daemonset with the existing nodes and tried to assign a second interface to a container which didnt work. I checked the cloudformation script and it does multus spefic stuff (Im not familiar with cloudformation scripts) . Q: Can I somehow patch existing nodes in order to support Multus?

**Summary**: 1. I have successfully deployed EKS via AWS Cloudformation template in the past (about an year ago). 2. Now when I am trying to deploy EKS via AWS Cloudformation its failing. 3. The error message is NOT clear enough for me to go and fix the reason of the crash, any tips on how to go about this error message? **Documentation and Steps Used** 1. Page: https://aws.amazon.com/quickstart/architecture/amazon-eks/ 2. Deploy using AWS CloudFormation with new VPC **Error Message** | Stack name | Status | | --- | --- | | eks-quickstart-RegionalSharedResources | DELETE_FAILED | | eks-quickstart-AccountSharedResources | CREATE_COMPLETE | | Amazon-EKS | ROLLBACK_COMPLETE | Amazon EKS (ROLLBACK_COMPLETE) has the following events that Failed * AutoDetectSharedResources > CREATE_FAILED with log Embedded stack arn:aws:cloudformation:us-east-2:SOME_ID : stack/Amazon-EKS-AutoDetectSharedResources-SOME_UUID was not successfully created: The following resource(s) failed to create: [ PreReqs ]. * Amazon-EKS > ROLLBACK_IN_PROGRESS with log The following resource(s) failed to create: [AutoDetectSharedResources]. Rollback requested by user. **One more log seems to be important (BUT the Cloudformation Script is from AWS so I doubt it might be a root cause)** ``` RegisterHelmType CREATE_FAILED CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [SOME_UUID]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version. ```

I'm in the process of setting up a prototype mesh with mTLS. I've gotten to the point where I have my services coupled with envoy sidecars and the sidecars are receiving certificates from SPIRE. I've been following along with this [article ](https://aws.amazon.com/blogs/containers/using-mtls-with-spiffe-spire-in-app-mesh-on-eks/) and am now running into an issue. In their steps, they perform a curl command from a container outside of the mesh and get some TLS negotiation messages. When I try to do the same thing, I get the following: ``` bash-4.2# curl -v -k https://grpc-client-service.grpc.svc.cluster.local:80/ * Trying 10.100.152.100:80... * Connected to grpc-client-service.grpc.svc.cluster.local (10.100.152.100) port 80 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt * CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol * Closing connection 0 curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol ``` Any advice on where I should start to troubleshoot this issue? Here's a rough overview of my setup: There are two pods that represent a client and a server service. The client has a web interface that allows the user to input text. The client takes the input text and submits that to the server service. The server service responds with an echo message that has some extra formatting so you know it came from the server. I've got both pods wrapped in virtual services that connect directly to virtual nodes. I was able to successfully test this with a basic mesh setup prior to adding the SPIRE workload parameters to the services. Within the envoy sidecars, I can see that the SPIRE server is indeed issuing certificates.

Hi, where can we find the IPs of the Amazon EKS service? I want to correctly identify a CloudTrail event with the name GetCallerIdentity that is made by the EKS. [EKS docs](https://docs.aws.amazon.com/eks/latest/userguide/cluster-auth.html) specifies the existence of such event, but on [AWS IP ranges](https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html) there is no EKS service. Insted the IP is in AMAZON and EC2 CIDR like any other EC2 ip. Thank you!

Hi, I tried to install calico following this url on my EKS (k8s version 1.21) https://docs.aws.amazon.com/eks/latest/userguide/calico.html resources in namespace tigera-operator deployed successfuly. but in calico-system, deployment.apps/calico-kube-controllers is failed on deployment and log shows, NAME READY STATUS RESTARTS AGE calico-kube-controllers-6b5d45f4dc-n9tjn 0/1 CreateContainerConfigError 0 3m42s kubectl logs calico-kube-controllers-6b5d45f4dc-n9tjn -n calico-system >>> Error from server (BadRequest): container "calico-kube-controllers" in pod "calico-kube-controllers-6b5d45f4dc-n9tjn" is waiting to start: CreateContainerConfigError kubectl describe deployment.app/calico-kube-controllers -n calico-system >>> Name: calico-kube-controllers Namespace: calico-system CreationTimestamp: Fri, 04 Mar 2022 11:45:52 +0900 Labels: k8s-app=calico-kube-controllers Annotations: deployment.kubernetes.io/revision: 1 Selector: k8s-app=calico-kube-controllers Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable StrategyType: Recreate MinReadySeconds: 0 Pod Template: Labels: k8s-app=calico-kube-controllers Service Account: calico-kube-controllers Containers: calico-kube-controllers: Image: docker.io/calico/kube-controllers:v3.21.4 Port: <none> Host Port: <none> Liveness: exec [/usr/bin/check-status -l] delay=10s timeout=10s period=10s #success=1 #failure=6 Readiness: exec [/usr/bin/check-status -r] delay=0s timeout=10s period=10s #success=1 #failure=3 Environment: KUBE_CONTROLLERS_CONFIG_NAME: default DATASTORE_TYPE: kubernetes ENABLED_CONTROLLERS: node KUBERNETES_SERVICE_HOST: ######## KUBERNETES_SERVICE_PORT: 443 Mounts: <none> Volumes: <none> Priority Class Name: system-cluster-critical Conditions: Type Status Reason ---- ------ ------ Available False MinimumReplicasUnavailable Progressing True ReplicaSetUpdated OldReplicaSets: <none> NewReplicaSet: calico-kube-controllers-6b5d45f4dc (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 5m41s deployment-controller Scaled up replica set calico-kube-controllers-6b5d45f4dc to 1 please help me to finish the installation>?

How can I view logs for the EKS `fargate-scheduler` ? I can see the 'default-scheduler' logs in CloudWatch (named 'kube-scheduler' in cloudwatch), but cannot find documentation regarding logging the eks `fargate-scheduler`. I have read the documentation regard [Fargate logging](https://docs.aws.amazon.com/eks/latest/userguide/fargate-logging.html) and this is not what I want. This doc describes logging for pods running in eks fargate, but I want logs from the `fargate-scheduler` itself. I have read this [EKS Fargate troubleshooting](https://aws.amazon.com/premiumsupport/knowledge-center/eks-resolve-pending-fargate-pods/) doc as well, and this is also not what I want. While I appreciate AWS putting out troubleshooting guides, the `fargate-scheduler` knows already knows why my pods cannot be scheduled, and I'd like to view those logs. Thanks in advance for you help!

I've been able to create a local EKS Anywhere cluster in vCenter. I'd like to register it with EKS (using the webUI) but when I complete the form the 'Register' button doesn't react at all. There are no indications of missing fields, no error, and the button is not 'greyed out' it just doesn't react at all when clicked. We have created the AmazonEKSConnectorAgentRole role as defined in the prereqs and I have the required permissions. eks:RegisterCluster ssm:CreateActivation ssm:DeleteActivation iam:PassRole

I have deployed an EKS cluster with a node group using Terraform a little bit ago but now when I want to delete it, I get the error "Cannot delete because cluster my-cluster currently has an update in progress" There is no update in progress from what I can tell, meaning that there is no update to stop and no way to delete my cluster now. Both Terraform and the AWS console show the same error and I have no idea how to delete it. There are no more nodes, autoscalers, load balancers, network interfaces or even a VPC associated with the cluster anymore. The cluster shows up as "Active". I did perform updates on the cluster without syncing Terraform though. The VPC-CNI needed updating and the node group version as well. Terraform had that synced before deleting, so that shouldn't've been an issue. How do I delete my cluster?

Hi, I'm trying to access to secrets in Secrets Manager from a pod deployed in EKS cluster. This cluster was created with *eksctl* command. * I attached a iam policy with grants to iam role attached to EC2 nodes: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds" ], "Resource": "arn:aws:secretsmanager:eu-west-1:[masked]:*" }, { "Effect": "Allow", "Action": "secretsmanager:ListSecrets", "Resource": "*" } ] } ``` * This iam role was created by *eksctl* command, and I see that it has this trust relationship: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` When I try from awscli, to retrieve a secret from a running pod in EKS cluster, I have this error: ``` # aws secretsmanager get-secret-value --secret-id arn:aws:secretsmanager:eu-west-1:[masked] An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::[masked] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[masked] ``` awscli has configured in config file: ``` [default] region = eu-west-1 output = json role_arn = arn:aws:iam::[masked] credential_source = Ec2InstanceMetadata ``` What's wrong? Kind regards

I'm using the `eksctl` utility to build and destroy Kubernetes clusters on Amazon EKS. When I delete the CloudFormation stacks, most of the resources are properly destroyed, including the EKS cluster itself. *However*, the VPC and Internet Gateway are getting hung up, and will not clean up properly. This is leaving my account with a bunch of orphaned VPC resources that I don't want, and contributing to my resource limits unnecessarily. The CloudFormation stacks get stuck in the `DELETE_FAILED` state. ``` The vpc 'vpc-096bd7c39859b6afb' has dependencies and cannot be deleted. (Service: AmazonEC2; Status Code: 400; Error Code: DependencyViolation; Request ID: 28dda82e-9aa9-4d05-b870-9371d77cee23; Proxy: null) ``` ``` The internetGateway 'igw-0c214b63b8b795b81' has dependencies and cannot be deleted. (Service: Ec2, Status Code: 400, Request ID: 71c41fe9-1247-4a29-ac47-d5c32d29837d, Extended Request ID: null) ``` **Question**: Can the AWS CloudFormation team fix this, so that the resources are properly cleaned up?

I'm running into an unusual issue where, on one specific new AWS account, I cannot create any nodegroups whatsoever. I've tried on two other AWS accounts and I can make nodegroups without any problems. The nodegroup creation always fails with something like the following: NodeCreationFailure Instances failed to join the kubernetes cluster DUMMY_2f2298a2-f492-439a-b7bb-ff931c539d78 DUMMY_5651ecbb-690e-4f3e-bc28-c52dc0d95bca DUMMY_6db1e73c-a1c7-4258-b10d-f6994864c3ef DUMMY_93f8d481-afd5-4811-ae28-aa2c50bd3ef5 DUMMY_950c3c89-d7ef-489d-8023-bc88a3b8a99c DUMMY_a5e09b94-4c86-4d0b-bb12-b9630ee544de DUMMY_bab43e87-11f8-4747-908a-06ae3741c612 DUMMY_c3f7c48a-4138-48d4-ba15-894a33f2d90a DUMMY_cccca0c7-98ae-4bf7-8441-8124971e8a78 DUMMY_d9909a43-ebf5-4340-99f0-47281499b2e2 DUMMY_daa1703a-8032-4fa5-9eae-c8a0b04fc1dd DUMMY_f3d0c7e8-b265-4927-98d4-33f7d4cd5ace This occurs whether I'm using eksctl to create a new cluster from scratch with nodegroups (both when I specify how the nodegroup should be configured, or if I let it use the defaults for the initial nodegroup), if I use eksctl to create a nodegroup on an existing cluster, or if I try to create a nodegroup on an existing cluster using the AWS web client. I've tried all of these things on different accounts and had success every time. I've tried both us-west-1 and us-west-2 and had no success on the affected account, and nothing but success on the other accounts. I looked up common sources of this issue (https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting.html) and I haven't had success with trying the suggested solutions. The IAM roles that are created with each nodegroup (before they're deleted when the creation fails) look identical to ones on working accounts, and they have the permissions AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly, and AmazonEKS_CNI_Policy. I even tried making an IAM role with those three permissions and using that to make a nodegroup through the web client, and it still failed. The VPCs that these clusters are on are configured for IPV4, not IPV6. The VPCs's main security groups have all outbound traffic allowed, and since they've been set up via eksctl, they have two public and two private subnets, with the public subnets having IP addresses auto-assigned, so they should have public internet access. The managed nodegroups created when I spin up a new cluster with eksctl seem to only be trying to use the public subnets, so they should definitely have public access. The account I'm using has AdministratorAccess permission to the account. I'm running out of ideas as to how to solve this. It really seems to be tied to this account, but I can't figure out what's causing this very specific problem.

Hey there, When using AWS EKS 1.21, currently 1 node running, (incl. cluster-autoscaler with correctly functioning asg) and creating a deployment with 5 replicas and specifying 'topologySpreadConstraints' like this: ``` topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: xxx ``` the scheduler ignores this completly and schedules all pods on the same node. (even though there is the 'DoNotSchedule') How can this be fixed? I want the cluster-autoscaler to see, that 2 nodes can not be scheduled. Thanks in advance.

Hi all, I have setup EKSA loca cluster as per https://anywhere.eks.amazonaws.com/docs/getting-started/local-environment/ which is working fine. I have further setup kube-vip LB as per https://anywhere.eks.amazonaws.com/docs/tasks/workload/loadbalance/kubevip/arp/ I am not able to connect using the service's EXTERNAL-IP. Able to connect using that IP from inside the pod though - screenschot here https://paste.pics/FW5HD

Hi all, I was doing a Udemy course to work with EKS. After I finished, I deleted EC2 instances created by EKS, removed EKS cluster. Deleted elastic IP and load balancer. Now I want to clean up everything else. However, I can't delete Security groups, VPSc, Subnets, Network ACLs. Most of them say that I have associated networks interfaces. I try to delete the interfaces with ui and console but it says that they are in use. Detach also doesn't work and I have already released elastic IPs. I also went to ColoudFormation and tried to remove the stack there. What else I'm missing??

Hello. via terraform I've created a stack with EKS. On EKS I did setup aws-load-balancer-ingress-conotroller and traefik. Before to destroy everything via terraform I didn't delete the 2 pods for my alb and traefik service. Doing so I'm not able to delete the eni created by these process. I'm root in my account and I run also the following command ``` aws ec2 detach-network-interface --attachment-id eni-attach-xxxxxxxx --force An error occurred (AuthFailure) when calling the DetachNetworkInterface operation: You do not have permission to access the specified resource. ``` Also I've a basic account, so I cannot contact the support in order to ask them to delete for me. Can please someone shed some light ? Thanks

Hello. via terraform I've created a stack with EKS. On EKS I did setup aws-load-balancer-ingress-conotroller and traefik. Before to destroy everything via terraform I didn't delete the 2 pods for my alb and traefik service. Doing so I'm not able to delete the eni created by these process. I'm root in my account and I run also the following command ``` aws ec2 detach-network-interface --attachment-id eni-attach-xxxxxxxx --force An error occurred (AuthFailure) when calling the DetachNetworkInterface operation: You do not have permission to access the specified resource. ``` Also I've a basic account, so I cannot contact the support in order to ask them to delete for me. Can please someone shed some light ? Thanks

``` public class Example { public static void main(String[] args) { // print statement at the start of the program System.out.println("Start..."); System.out.print("Number of available processors are: "); System.out.println( Runtime.getRuntime().availableProcessors()); } } ``` Ref: https://www.tutorialspoint.com/get-number-of-available-processors-in-java When I compile and run this code on AWS ECS Fargate getting "1" as the output even though I have specified more than 1 cpu in task def(4098 for 4 vCPU). **Tested Java versions:** - Corretto-17.0.0.35.1 - Corretto-8.275.01.1 But when I do the same on AWS EC2 inside docker by allocating CPU with --cpus for "docker run" I get the exact number what I pass to --cpus. Same results with AWS EC2 with Containerd as well. Also tested this in EKS 1.18 by setting CPU limit. It returns the number of CPUs specified in the CPU limit. So, wondering why "1" only in AWS ECS.

I am trying to create an eks cluster using the eksctl command. However, an error occurs as below: AWS::EKS::Cluster/ControlPlane: CREATE_FAILED – "Resource handler returned message: \"Duplicate key <<cluster_name>/ControlPlane\" If you try to delete it using cloudformation, it fails because of ControlPlane. Currently, there is no EKS Cluster.

Docker has changed it's license terms August 31, 2021 with the grace period ending January 31, 2022 where Docker Desktop is no longer free except under certain circumstances. They say that Docker Desktop includes the docker engine, but there is some mention of "Do-it-yourself with Docker Engine" being separate from Docker Desktop. * Do the Docker licensing changes impact services like ECR or EKS ? Do companies need to seek licensing directly with Docker to be in compliance?

While Adding Nodes in EKS Cluster getting >>nodecreationfailure - Nodes instances failed to join the kubernetes cluster

I have an EKS cluster. Version is 1.21. I started with Self Managed windows nodes I added self managed linux node group. I deployed an application on Linux node group. I am running MSSQL on Linux. I have a windows container running in Windows node group. Windows container is trying to access mssql service. It is not able to find this service. DNS resolution is working fine on Linux nodes but not working on Windows nodes. Let me know if you have any suggestions.

I have an eks cluster in the latest version (1.21). I am trying to use a HorizontalPodAutoscaler to scale with memory usage but there is no scaling happening. Utilization is currently at 164% but the hpa is not scaling up. I am using the following hpa config. ``` apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: "xxx-service-autoscaler" spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: xxx-service minReplicas: 1 maxReplicas: 3 metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 100 ```

**Cluster information:** **Kubernetes version: 1.19** **Cloud being used: AWS EKS** So here is my configuration. I have a private VPC on AWS within which is hosted an AWS EKS cluster. Now this VPC has public facing load balancers which are only accessible from only specific IP addresses. On this EKS cluster are hosted a number of micro services running in their own pods. Each of these pods exposes a REST endpoint. Now here is my requirement. Out of all the REST endpoints that we have, i would like to make only one REST endpoint publicly available from the internet. The remainder of our REST endpoints should remain private accessible only from certain IP addresses. What would be the best approach to achieve this? So far,from what i have researched, here are my options: 1)Have another instance of Ingress controller which deploys a public facing load balancer to handle requests to this public facing REST endpoint. This will work. However, i am concerned with the security aspects here. An attacker might just get into our VPC and create havoc. 2)Have a completely new EKS cluster which is public facing where i deploy this single REST endpoint. This is something i would like to avoid. 3)Use something like AWS API gateway to achieve this. I am not sure if this is possible as i have to research more about it. Anyone has any ideas on how this could be achieved securely? Any advice would be very much appreciated. Regards, Kiran Hegde

I'm working on some POC to migrate applications from on-premise to AWS. What can I automate using Lambda if I deploy and manage workloads on the Amazon EKS cluster (ELB/ECR/RDS)? I'm thinking of different use cases. Thanks.

Hello, We got an issue when we tried to add a node group in EKS with t4g class instance AutoScalingGroupInvalidConfiguration - Amazon EC2 Autoscaling does not support the requested instanceType t4g.medium. I check and t4g are available in my region (eu-west-3) I successfully set a t4g class instance by manually edition a launch configuration in an other node group. Maybe you have forgot a condition somewhere when you release this new type of instance ?

Hi, I am trying to install open edX (Tutor) on AWS EKS by this [guide](https://rajputvaibhav.medium.com/open-edx-on-aws-eks-382419ca0865). I already have a DNS zone that is defined in Cloudflare and I want to use it for external access to the new services. Did anyone implement such a solution or can point me to documentation? Thanks, Yaniv

I have created an EKS cluster using `eksctl`. I am following these steps to establish connectivity to AWS services like S3, cloudwatch using spring-boot. 1. Create EKS using `eksctl` - This has my service account details and OIDC enabled. 2. List the service accounts to see if they were created fine 3. Create a deployment using the account name 4. Create a service I am seeing a 403 in the logs: ``` User: arn:aws:sts:account_id/nodegroup_rule_created_by_eks is not authorized to perform: cloudformation:DescribeStackResources because no identity-based policy allows the cloudformation:DescribeStackResources action (Service: AmazonCloudFormation; Status Code: 403; Error Code: AccessDenied; Request ID: xxxx) ``` Can I get some help here to troubleshoot this issue, please? --- What I have figured out after posting this issue is my node which is provisioned by `eksctl`, has been applied with rules. This is the rule which my app is picking up due to the default CredentialChain. What I haven't still figured out is how do I enable the apps in the pod to assume a service account role. --- Here are relevant snippets from the yaml. #### cluster-config.yaml file: ``` iam: withOIDC: true serviceAccounts: - metadata: name: backend-stage-iam-role namespace: backend-stage labels: { aws-usage: "all-backend-allow" } attachPolicyARNs: - "arn:aws:iam::MY_CUSTOM_RULE_WHICH_ALLOWS_S3_LIST_GET_PUT" ``` #### deployment.yaml ``` spec: replicas: 8 selector: matchLabels: app: my-app strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: my-app spec: serviceAccountName: backend-stage-iam-role ``` When describing the pod, I see that there exists an environment variable : ```AWS_ROLE_ARN: arn:aws:iam::MY_CUSTOM_RULE_WHICH_ALLOWS_S3_LIST_GET_PUT``` I am still to figure out how can I apply this role to the pod?

I see an increase of `Data Transfer $0.010 per GB - regional data transfer - in/out/between EC2 AZs or using elastic IPs or ELB` (approximately 119,626.889 GB) and I would like to determine the root cause. I am using EKS distributed in 3 AZs in one region, RDS, EC2s. The cost doubled from previous months with no apparent reason. How to determine what is causing this? and what are the best practices to reduce this cost?

Hello Everyone, I am new to EKS.I am was going through the docs,it say that when during eks creation it create a ENI in the subnet which we mentioned.In the docs i went through the Type of Cluster Endpoint Access mentioned ie Public and Private.In docs it also mentioned the we can launch Worker Nodes in the subnets which are different than the one mentioned during EKS Creation. When the node group is created,I dont see the ENI that are created are attached to EC2. Then how the communication is taking place.I am not able to understand this. What is use of ENI if not being attached to any EC2. If anyone could please explain it would be really helpful

Hello, I have an EKS cluster (terraform code see below) and follow the guide to set up the Load Balancer Controller (https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html). But when I deploy the service (terraform code see below) and want to expose it via "LoadBalancer" it keeps in a pending state and no external adr. is available. The Load Balancer controller gives the following error: Log Error from eksckubectl logs pod/aws-load-balancer-controller-5b57cdc6cc-dtjbg -n kube-system {"level":"error","ts":1640857282.2362676,"logger":"controller-runtime.manager.controller.service","msg":"Reconciler error","name":"terraform-example","namespace":"default","error":"AccessDenied: User: arn:aws:sts::009661972061:assumed-role/my-cluster2021123008214425030000000b/i-0a40de3c4e8541004 is not authorized to perform: elasticloadbalancing:CreateTargetGroup on resource: arn:aws:elasticloadbalancing:eu-central-1:009661972061:targetgroup/k8s-default-terrafor-630f67813d/* because no identity-based policy allows the elasticloadbalancing:CreateTargetGroup action\n\tstatus code: 403, request id: 2491099a-a6fd-4e6f-bab8-3c758eda0d0b"} If I add the AWSLoadBalancerControllerIAMPolicy to the my-cluster2021123008214425030000000b role manually it works. But as far as I read the documentation the AWSLoadBalancerControllerIAMPolicy is for the controller in the kube-system namespace and not the worker nodes. Is there anything missing from the documentation? Or what is the intended way of solving this? best regards rene Terraform EKS: ``` terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 3.27" } } required_version = ">= 0.14.9" } provider "aws" { profile = "default" region = "eu-central-1" } data "aws_eks_cluster" "eks" { name = module.eks.cluster_id } data "aws_eks_cluster_auth" "eks" { name = module.eks.cluster_id } provider "kubernetes" { host = data.aws_eks_cluster.eks.endpoint cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data) token = data.aws_eks_cluster_auth.eks.token } module "eks" { source = "terraform-aws-modules/eks/aws" cluster_version = "1.21" cluster_name = "my-cluster" vpc_id = "vpc-xx" subnets = ["subnet-xx", "subnet-xx", "subnet-xx"] worker_groups = [ { instance_type = "t3.medium" asg_max_size = 5 role_arn = "arn:aws:iam::xxx:role/worker-node-example" } ] } ``` Terraform service: ``` terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 3.27" } kubernetes = { source = "hashicorp/kubernetes" version = ">= 2.0.1" } } required_version = ">= 0.14.9" } provider "kubernetes" { host = "xxx" cluster_ca_certificate = base64decode("xxx") exec { api_version = "client.authentication.k8s.io/v1alpha1" command = "aws" args = [ "eks", "get-token", "--cluster-name", "my-cluster" ] } } provider "aws" { profile = "default" region = "eu-central-1" } resource "aws_sqs_queue" "gdpr_queue" { name = "terraform-example-queue.fifo" fifo_queue = true content_based_deduplication = true sqs_managed_sse_enabled = true } resource "aws_sqs_queue" "private_data_queue" { name = "terraform-example-queue.fifo" fifo_queue = true content_based_deduplication = true sqs_managed_sse_enabled = true } resource "aws_db_instance" "database" { allocated_storage = 10 engine = "postgres" engine_version = "13.3" instance_class = "db.t3.micro" name = "mydb" username = "foo" password = "foobarbaz" skip_final_snapshot = true vpc_security_group_ids = [aws_security_group.basic_security_group.id] } resource "aws_security_group" "basic_security_group" { name = "allow rds connection" description = "Allow rds traffic" vpc_id = "vpc-xxx" ingress { description = "postgres" from_port = 5432 to_port = 5432 protocol = "all" cidr_blocks = ["0.0.0.0/0"] ipv6_cidr_blocks = ["::/0"] } } resource "kubernetes_service" "gdpr-hub-service" { metadata { name = "terraform-example" annotations = { "service.beta.kubernetes.io/aws-load-balancer-type" = "external" "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type" = "ip" "service.beta.kubernetes.io/aws-load-balancer-scheme" : "internet-facing" } } spec { selector = { App = kubernetes_deployment.gdpr-hub-service-deployment.spec.0.template.0.metadata.0.labels.App } session_affinity = "ClientIP" port { port = 80 target_port = 8080 } type = "LoadBalancer" } } resource "kubernetes_deployment" "gdpr-hub-service-deployment" { depends_on = [ aws_db_instance.database, aws_sqs_queue.gdpr_queue, aws_sqs_queue.private_data_queue ] metadata { name = "gdpr-hub-service" labels = { App = "gdpr-hub-service" } } spec { replicas = 2 selector { match_labels = { App = "gdpr-hub-service" } } template { metadata { labels = { App = "gdpr-hub-service" } } spec { container { image = "xxxx" name = "gdpr-hub-service" port { container_port = 8080 } resources { limits = { cpu = "2" memory = "1024Mi" } requests = { cpu = "250m" memory = "50Mi" } } } } } } } ```

Hi folks - k8s version: 1.21 Platform version: eks.4 I'm trying to get an autoscaling nodegroup using a1.metal instances working. Nodes that are spawned never become ready due to the following error: ``` container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized ``` The aws-node pod on the node fails with the following error: ``` Readiness probe failed: {"level":"info","ts":"2021-12-29T04:56:34.384Z","caller":"/usr/local/go/src/runtime/proc.go:225","msg":"timeout: failed to connect service \":50051\" within 5s"} ``` The kube-proxy pod on the node fails to start with the following log: ``` standard_init_linux.go:228: exec user process caused: exec format error ``` Which leads me to believe that at least the kube-proxy pod is likely pulling an x86 image instead of arm64. I appreciate any assistance; please let me know which other information I can provide. thanks, /-Will

Hi, I have a EKS cluster with version 1.16 and try to update to 1.17 but failed with the following error due to missing default security group (deleted) 2021-11-13 09:19:19 [ℹ] will upgrade cluster "xxx" control plane from current version "1.16" to "1.17" Error: InvalidRequestException: The security group 'sg-0fdxxx' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidGroup.NotFound; Request ID: xxx-xxx-xxx-xxx-xxx; Proxy: null) There seems to be no other way to update it so I wait for the force updating from AWS when it comes to the end of support date. However, nothing has happened. So I would like to ask is there anyway to restore the old security group or remove the default one in the current cluster to allow updating or not Thanks

Hello, I am learning to use terraform and aws. I created a cluster and a installed a flask application with helm. Everything was working fine but then i deleted the eks cluster from the dashboard. Now terraform destroy wont work and i cannot stop the instances that have been created. Every time i stop them they keep coming up again. I just need to stop everything would be grateful for any ideas on how to do that.

Resource pool not found. Create simple Yaml config. Setup is being validated up to Resource pool validation. after setup displays error resource pool not found. I used official guid https://anywhere.eks.amazonaws.com/docs/reference/clusterspec/vsphere/ even I picked: *If there is no resource pool: /<datacenter>/host/<cluster-name>/Resources* Setup displays same error (resource pool no found)

Dear Team, We are building a service which will be exposed as Rest API through API Gateway. The service will perform 3 things. 1. Validate the inputs 2. Upload a document to S3(rather move the document-approx 40kb in size from Staging S3 location to Target S3) 3. Insert the metadata to Dynamo DB We expect this service to be called approx 2 lakh times within 8-10 hours on daily basis through AWS Batch. Afterwards, it may not be used through out the day. We were planning to have a lambda function for each of the above steps which will be called from API Gateway. If required, we may use Step functions to orchestrate above 3 steps. Would Lambda functions be the appropriate way to process these steps in the service or should we use EC2/EKS and ASG to scale in and Scale out. I assume the service will respond back within second but bearing Lambda concurrency limits in mind and any other limits we may face, let me know the best way to process the batch. Regards, Dhaval Mehta

I've seen the following solutions, but they don't seem that great to me: 1 Native integration with CSI driver (Complex to intall and you can inject all values in one environment variable using a local disk, you need parser variables when docker run) https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/ 2 Execute an script when docker run, get the variables, parse and export (Easy to use, not elegant) ``` ENV_VAR=$(aws secretsmanager get-secret-value --secret-id $VARIABLE_ID --region us-west-2 | jq --raw-output '.SecretString') # parse ENV_VAR with code in this line # Export variables ``` 3 Integrate AWS Secret Manager with the framework for example with Rails (Easy to use, some elegant but it is at the code level ) https://anonoz.github.io/tech/2018/12/29/aws-secrets-in-rails.html What do you recommend to carry out this integration and why? Do you know another more elegant and easy way?

Currently we use EKS with managed node groups. In the node group configuration you have to configure the min, max and desired capacity of the nodes. With the use of the classic Cluster Autoscaler the desired node capacity of the autoscaling group will be updated. Now we switched to the AWS developed autoscaler Karpenter (https://karpenter.sh/). Karpenter has a different handling and does not work with the autoscaling group or node group. So my question is, with the use of Karpenter, which is the best setting for the EKS min, max and desired capacity? Should we set all these to "0" for example?

I have created a launch template that has the tenancy: Dedicated with the respective host ID. I am using the launch template to create a node group for an EKS cluster but I see an internal error: " EKS couldn't process your request". I can roll up an ec2 instance using the launch template. How can I create a node group using a Dedicated host?

A customer using EKS on AWS would like to isolate worker nodes on different racks. I can define `labels` manually as per: https://www.eksworkshop.com/beginner/140_assigning_pods/node_selector/ but I was wondering if automatic labels are perhaps available when K8s detects AWS rack-level placement groups. My question ultimately is: are `labels` always user-defined or are there automated labels ? Is there any other strategy to create anti-affinity rules (e.g per-AZ nodes) ?

Dear AWS, I was starting a POD on a Bottlerocket (x86, latest 1.4.0-42360701) Node. Yes, the AWS NFS driver is installed in my cluster. The POD could not start because it could not mount the nfs (it is missing /sbin/mount.nfs). Using the latest Amazon Linux (x87, latest 1.21.5-20211109) did work as expected. I hope this helps. Wanted to give Bottlrocket a try - no success yet ;-). Regards, thorsten

Hi all, I've recently updated one of our clusters from version 1.15 to 1.16 and then to 1.17. Before this one I updated 8 other clusters with no issues whatsoever. However, for some reason, when I update _kube-proxy_ to bring it in line with the new Kubernetes version the pods fail. The logs are empty and the reason why the pods are terminated is "Error", which isn't informative at all. In short: - Server Version: v1.17.17-eks-087e67 - Nodes version: v1.17.17-eks-ac51f2 - Working kube-proxy version: v1.15.11-eksbuild.1 - kube-proxy versions that cause the problem: 1.16.13-eksbuild.1, v1.17.9-eksbuild.1 At first I thought it could just be that particular version of kube-proxy so I decided to keep updating. Now I assume it's something else. Running _kubectl logs -f podName_ doesn't help. It doesn't return anything. State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 18 Oct 2021 12:32:56 +0200 Finished: Mon, 18 Oct 2021 12:32:56 +0200 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 66s default-scheduler Successfully assigned kube-system/kube-proxy-bh7r4 to ip-xxxxxx.eu-west-1.compute.internal Normal Pulling 65s kubelet, ip-xxxxxx.eu-west-1.compute.internal Pulling image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1" Normal Pulled 63s kubelet, ip-xxxxxx.eu-west-1.compute.internal Successfully pulled image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1" Normal Created 23s (x4 over 62s) kubelet, ip-xxxxxx.eu-west-1.compute.internal Created container kube-proxy Normal Started 23s (x4 over 62s) kubelet, ip-xxxxxx.eu-west-1.compute.internal Started container kube-proxy Normal Pulled 23s (x3 over 62s) kubelet, ip-xxxxxx.eu-west-1.compute.internal Container image "602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1" already present on machine Warning BackOff 8s (x6 over 61s) kubelet, ip-xxxxxx.eu-west-1.compute.internal Back-off restarting failed container Can you please advise? I'm quite confused here. I'm comparing what I did with our other clusters and I took the exact same steps. Not sure why this is not working. Thanks a lot! Edited by: twgdavef on Oct 18, 2021 4:24 AM

I'm using AWS EKS 1.21 with service account discovery enabled. Created an OIDC provider, the `.well-known/openid-configuration` endpoint returns a correct configuration: _{_ _"issuer": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",_ _"jwks_uri": "https://ip-***.eu-west-1.compute.internal:443/openid/v1/jwks",_ _"response_types_supported": \[_ _"id_token"_ _],_ _"subject_types_supported": \[_ _"public"_ _],_ _"id_token_signing_alg_values_supported": \[_ _"RS256"_ _]_ _}_ Created a ServiceAccount for one of my deployments and the pod gets this as projected volume: _volumes:_ _- name: kube-api-access-b4xt9_ _projected:_ _defaultMode: 420_ _sources:_ _- serviceAccountToken:_ _expirationSeconds: 3607_ _path: token_ _- configMap:_ _items:_ _- key: ca.crt_ _path: ca.crt_ _name: kube-root-ca.crt_ _- downwardAPI:_ _items:_ _- fieldRef:_ _apiVersion: v1_ _fieldPath: metadata.namespace_ _path: namespace_ The secret created for the ServiceAccount contains this token: _{_ _"iss": "kubernetes/serviceaccount",_ _"kubernetes.io/serviceaccount/namespace": "sbx",_ _"kubernetes.io/serviceaccount/secret.name": "dliver-site-config-service-token-kz874",_ _"kubernetes.io/serviceaccount/service-account.name": "dliver-site-config-service",_ _"kubernetes.io/serviceaccount/service-account.uid": "c26ad760-9067-4d90-a327-b3d6e32bce42",_ _"sub": "system:serviceaccount:sbx:dliver-site-config-service"_ _}_ The projected token mounted in to the pod contains this: _{_ _"aud": \[_ _"https://kubernetes.default.svc"_ _],_ _"exp": 1664448004,_ _"iat": 1632912004,_ _"iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",_ _"kubernetes.io": {_ _"namespace": "sbx",_ _"pod": {_ _"name": "dliver-site-config-service-77494b8fdd-45pxw",_ _"uid": "0dd440a6-1213-4faa-a69e-398b83d2dd6b"_ _},_ _"serviceaccount": {_ _"name": "dliver-site-config-service",_ _"uid": "c26ad760-9067-4d90-a327-b3d6e32bce42"_ _},_ _"warnafter": 1632915611_ _},_ _"nbf": 1632912004,_ _"sub": "system:serviceaccount:sbx:dliver-site-config-service"_ _}_ Kubernetes renew the projected token every hour, so everything looks fine. Except the projected token "exp" field: _"iat": 1632912004_ which is _Wednesday, September 29, 2021 10:40:04 AM_ _"exp": 1664448004_ which is _Thursday, September 29, 2022 10:40:04 AM_ So the problem is, that the projected token expiry time is 1 year, instead of around 1 hour, which makes Kubernetes effort to renew the token basically useless. I searched for hours but was simply unable to figure out where this is coming from. The expiration flag is passed to the kube-api server: _--service-account-max-token-expiration="24h0m0s"_, so my assumption is that this should be configured on the OIDC provider somehow, but unable to find any related documentation. Any idea how to make the projected token expiry date around the same as the _expirationSeconds_ in the pod projected volume?

Hi, We would like to give users permissions to use EKS using their AWS SSO usernames. I'm aware of the aws-iam-authenticator and eksctl, but not quite sure how to make them all work together with SSO rather than an IAM username. Thanks

I can see the below logs describing PVC: Warning ProvisioningFailed 93s (x2 over 93s) persistent volume-controller storageclass.storage.k8s.io "ebs-sc" not found Normal Provisioning 9s (x6 over 90s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 External provisioner is provisioning volume for claim "default/ebs-claim" Normal ExternalProvisioning 3s (x8 over 90s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator Warning ProvisioningFailed 0s (x6 over 80s) ebs.csi.aws.com_ebs-csi-controller-f5d9c9475-wh2t9_e2eea260-a1f6-4b74-9250-baf43ba03780 failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded ------------------------------------------------------------------------------------- kubectl logs ebs-csi-controller-f5d9c9475-wh2t9 -c csi-provisioner -n kube-system: CreateVolume failed, supports topology = true, node selected true => may reschedule = true => state = Background: rpc error: code = DeadlineExceeded desc = context deadline exceeded I0624 09:55:16.287191 1 controller.go:1106] Temporary error received, adding PVC 3f6ec7b5-2a25-4f42-babb-d808dd464535 to claims in progress W0624 09:55:16.287200 1 controller.go:958] Retrying syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535", failure 8 E0624 09:55:16.287222 1 controller.go:981] error syncing claim "3f6ec7b5-2a25-4f42-babb-d808dd464535": failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded I0624 09:55:16.287251 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"3f6ec7b5-2a25-4f42-babb-d808dd464535", APIVersion:"v1", ResourceVersion:"6828149", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ebs-sc": rpc error: code = DeadlineExceeded desc = context deadline exceeded ------------------------------------------- I can't see the right error for this issue to troubleshoot.

A customer needs to generate X509 certificates in Kubernetes for their extensions (validating/mutating/conversion webhooks). [Standard way][1] is to use CertificateSigningRequest for this purpose, but EKS [does not have][2] CertificateSigning admission controller installed, so the CSR is not getting signed. Is there either a way to enable CertificateSigning admission controller on EKS or any other best practice for generating and renewing X509 certificates for EKS cluster internal usage (i.e. kube-apiserver <-> custom-developped-webhook)? [1]: https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/ [2]: https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html#platform-versions-1.19

Hi! I created a simple cluster and a simple deployment with service (see below). --- eksctl create cluster --name artem-cluster --region eu-west-1 --fargate --profile mfa --- apiVersion: apps/v1 kind: Deployment metadata: name: sample-app spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: public.ecr.aws/nginx/nginx:1.19.6 ports: - name: http containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: sample-service annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip spec: ports: - port: 80 targetPort: 80 protocol: TCP type: LoadBalancer selector: app: nginx --- But service is stuck in <pending> state - sample-service LoadBalancer 10.100.5.193 <pending> 80:31505/TCP 5h57m In "describe" I see the only string: Type Reason Age From Message ---- ------ ---- ---- ------- Normal EnsuringLoadBalancer 29m service-controller Ensuring load balancer --- By the way, if I create service without "service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip", it created successfully, but curl <external_ip>:80 returns empty response, so I found in docs that I should use "nlb-ip" and now stuck with it.

Hi there, I've got an issue with an EKS node group where I cannot delete it. I've provisioned the EKS cluster with Terraform and configured aws-auth as per documentation below: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html I've admin access to the cluster, I also have deployments running and worker nodes connected. My ConfigMap can be seen below: _$ kubectl describe configmap -n kube-system aws-auth_ _Name: aws-auth_ _Namespace: kube-system_ _Labels:_ _Annotations:_ _Data_ _====_ _mapRoles:_ _----_ _- rolearn: arn:aws:iam::000000000000:role/cluster-role_ _username: system:node:{{EC2PrivateDNSName}}_ _groups:_ _- system:bootstrappers_ _- system:nodes_ _- rolearn: arn:aws:iam::000000000000:role/node-group-role_ _username: system:node:{{EC2PrivateDNSName}}_ _groups:_ _- system:bootstrappers_ _- system:nodes_ _- rolearn: arn:aws:iam::000000000000:role/aws-reserved/sso.amazonaws.com/eu-west-2/AWSReservedSSO_AdministratorAccess_ _username: {{SessionName}}_ _groups:_ _- system:masters_ _Events: <none>_ When I attempt to delete the node group, either via Terraform or using AWS Console, I get the following error listed under Health Issues: **"AccessDenied The aws-auth ConfigMap in your cluster is invalid."** I did not get this error when I created the node group, and I can't work out what exactly is wrong with my ConfigMap. Any suggestions?

Hi, I created an EKS cluster with Terraform. After destroying it with Terraform, I still have an orphaned nodegroup that I cannot see in the dashboard nor with awscli. And this nodegroup keeps creating an EC2 instance. If I delete the instance, the nodegroup creates a new one. This is generating billing costs for resources I am not using. Any adivice? Thanks

--------------------------- Cluster information: Kubernetes version: 1.18 AWS EKS We have a small (3-node) AWS cluster that we’ve been working on for a few months. It’s been running ArgoCD 1.7.10, and we recently tried to upgrade that to 1.8.7. That upgrade failed - the command didn’t return, it just hung. Since then attempting to apply CRDs has just caused a timeout. To try to work this out We’ve extracted the application CRD from https://raw.githubusercontent.com/argoproj/argo-cd/v1.8.7/manifests/install.yaml and trying to apply that goes like this: $ kubectl -n argocd apply -f application-crd.yaml Error from server (Timeout): error when creating "application-crd.yaml": the server was unable to return a response in the time allotted, but may still be processing the request (post customresourcedefinitions.apiextensions.k8s.io) We can’t find anything in any logs and we suspect that the problem lies in etcd which is invisible to us but cannot be sure. We have tried applying other CRD's, some more basic to prove the point and some just as complex. We have trimmed down the argoo CRD yaml and removed the schema and other parts to see which parts are failing but there appears to be no rhyme or reason to what is causing the problem. Sometimes it fails and sometimes it succeeeds. We have also tried the sample CRD from here: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/ And seen it both work and fail. In short, our kube is behaving erratically and we suspect a bug or a fundamental problem that we cannot diagnose. We are stuck and this is going to be used in production but we are losing confidence in our AWS EKS kube. Where do we go from here? Edited by: kophones77 on Mar 25, 2021 2:31 AM

For non-gpu EKS nodes we can programmatically retrieve the recommended AMI id with a call to GetParameter: > aws ssm get-parameter --name /aws/service/eks/optimized-ami/<1.19>/<amazon-linux-2>/recommended/image_id --region <region-code> --query "Parameter.Value" --output text How can we programmatically retrieve the recommended EKS optimized accelerated Amazon Linux AMI (for GPU nodes)?

Hi there, when deploying new pods in my EKS cluster, the DNS server specified in /etc/resolv.conf is not the one configured in "service/kube-dns" in the kube-system namespace. See here: $ kubectl -n kube-system get svc kube-dns NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 42d But this pod receives a different DNS server: $ kubectl exec -it busybox -- cat /etc/resolv.conf nameserver 10.100.0.10 search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal options ndots:5 I've tried to debug this issue but so far no luck. Any help or hints would be greatly appreciated! Regards, Khaled