By using AWS re:Post, you agree to the Terms of Use
Questions in Management & Governance
Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

ResourceLimitExceeded exception but I have Quota

I am not able to to start a SageMaker notebook neither a SageMaker training job with ml.c5.xlarge (or any other instance type). I checked on "Quota Services", and I clearly have quotes for both tasks. - 1 in "applied quota value" for "ml.c5.xlarge for notebook instance usage". - 15 in "applied quota value" for "ml.c5.xlarge for training job usage". Of course I am checking in the same region I try to work: "us-east-1". I have researched for several days, and all forum suggests to ask for a limit increase. Nevertheless, I already have quota (limits) available. Nevertheless, when I try to start the Jupyter notebook, it raise the exception `The account-level service limit 'ml.c5.xlarge for notebook instance usage' is 0 Instances, with current utilization of 0 Instances and a request delta of 1 Instances. Please contact AWS support to request an increase for this limit.` It is strange because the exception says that I have a limit of 0 instances, while the quota list services says I have 1. Here's the output of the command `service-quotas list-service-quotas` ``` { "ServiceCode": "sagemaker", "ServiceName": "Amazon SageMaker", "QuotaArn": "arn:aws:servicequotas:us-east-1:631720213551:sagemaker/L-E2BB44FE", "QuotaCode": "L-E2BB44FE", "QuotaName": "ml.c5.xlarge for training job usage", "Value": 15.0, "Unit": "None", "Adjustable": true, "GlobalQuota": false }, { "ServiceCode": "sagemaker", "ServiceName": "Amazon SageMaker", "QuotaArn": "arn:aws:servicequotas:us-east-1:631720213551:sagemaker/L-39F5FD98", "QuotaCode": "L-39F5FD98", "QuotaName": "ml.c5.xlarge for notebook instance usage", "Value": 1.0, "Unit": "None", "Adjustable": true, "GlobalQuota": false, "UsageMetric": { "MetricNamespace": "AWS/Usage", "MetricName": "ResourceCount", "MetricDimensions": { "Class": "None", "Resource": "notebook-instance/ml.c5.xlarge", "Service": "SageMaker", "Type": "Resource" }, "MetricStatisticRecommendation": "Maximum" } }, ``` I strongly appreciate your help, because I have no way to open a SageMaker training job for several days. Thanks.
0
answers
0
votes
2
views
asked 16 minutes ago

Cannot add environment variable through Ebextensions

I'm using .ebextensions to create VPCEndpoints so in the **Resources** section I've addded the needed section for the VPCEndpoint. Then after that in the **option_settings** section I'm trying to add an environment variable in my elastic beanstalk application referencing the created VPCEndpoint, but when i check the environment variables from the elastic beanstalk console the value is added as a plain text not the Ref of the VPCEndpoint (Check the screenshot) So how can i make it interpret the Ref of the endpoint ? ![Enter image description here](/media/postImages/original/IMkQEkAlsLRyCG5pYs1i8hkA) ``` Resources: NewsonarVPCEndpoint: Type: AWS::EC2::VPCEndpoint Properties: PrivateDnsEnabled: false SecurityGroupIds: - {"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "ALLOW_INBOUND_FROM_VPC_SECURITY_GROUP", "DefaultValue": "default_value"}} ServiceName: { "Fn::Join": [ "", [ "com.amazonaws.vpce.",{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "AWS_REGION", "DefaultValue": "us-east-1"}},".",{"Ref": "sonarVPCEndpointService"}]] } SubnetIds: - { "Ref": "Subnet1Id" } - { "Ref": "Subnet2Id" } - { "Ref": "Subnet3Id" } VpcEndpointType: Interface VpcId: { "Ref": "VpcId" } option_settings: aws:elasticbeanstalk:application:environment: VPC_ENDPOINT: '`{"Ref" : "NewsonarVPCEndpoint"}`' ```
0
answers
0
votes
4
views
asked 11 hours ago

Ubuntu Managed Nodes creation failure in Fully-Private cluster

Hi, For some reason I am not able to create Ubuntu managed nodes in fully private cluster. Though, managed Amazon-Linux nodes and all other self-managed nodes are joining the cluster successfully. I have followed all the guides and troubleshooting aws websites already still I am not successful. I have also run the troubleshooting script. Below is the result ``` HERE IS A SUMMARY OF THE ITEMS THAT REQUIRE YOUR ATTENTION: [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ . [WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html [WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>". ============================================================================================================================================ Here are the detailed steps of the document execution: [X] Checking EKS cluster test-cluster: EKS Cluster: test-cluster is in Active state. 1. Checking if the cluster Security Group is allowing traffic from the worker node: Passed: The cluster Security Group sg-068d83e5a68aff9c7 is allowing traffic from the worker node. 2. Checking DHCP options of the cluster VPC: Passed: AmazonProvidedDNS is enabled 3. Checking cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T for the required permissions: Passed: IAM role for cluster test-cluster has the required IAM policies attached. Passed: The cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T has the required trust relationship for the EKS service. 4. Checking control plane Elastic Network Interfaces(ENIs) in the cluster VPC: Passed: The cluster Elastic Network Interfaces(ENIs) exist. 5. Cluster Endpoint Private access is disabled for your cluster, checking if the Public CIDR ranges include worker node i-0e775ca75fe57bb70 outbound IP: Passed: The cluster allows public access from 0.0.0.0/0 6. Checking cluster VPC for required DNS attributes: Passed: Cluster VPC vpc-0cbb4879c588fb52a has the required DNS attributes correctly set. -------------------------------------------------------------------------------------------------------------------------------------------- [X] Checking worker node i-0e775ca75fe57bb70 state: The instance is Running. 1. Checking if the EC2 instance family is supported: Passed: EC2 instance family m5.xlarge is supported. 2. Checking the worker node network configuration: Passed: Worker node is created in a private subnet without a NAT Gateway so VPC endpoints need to be used. Passed: Checking VPC Endpoints setup: Passed: The VPC Endpoint com.amazonaws.eu-west-2.ec2 exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ec2 is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.api exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.api is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.sts exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.sts is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: S3 gateway endpoint ['vpce-0e01febb999cf1735', 'vpce-08ef7ffac589c3d01'] is added to the worker's VPC. Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-0e01febb999cf1735. Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-08ef7ffac589c3d01. 3. Checking the IAM instance Profile of the worker node: Passed: The instance profile arn:aws:iam::782534010321:instance-profile/eks-d0c1cede-94e5-c6bc-75fd-2f86043a90eb is used with the worker node i-0e775ca75fe57bb70 . Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX is attached to Instance Profile. Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX has the required IAM policies attached. Passed: No issues detected with the trust relationship policy of the arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX role. 4. Checking worker node's UserData bootstrap script: Passed: The UserData of the worker node contains the required bootstrap script. 5. Checking the worker node i-0e775ca75fe57bb70 tags: Passed: Worker node i-0e775ca75fe57bb70 has the required cluster tags. 6. Checking the AMI version for EC2 instance i-0e775ca75fe57bb70: [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ . 7. Checking worker node i-0e775ca75fe57bb70 Elastic Network Interfaces(ENIs) and Private IP addresses to check if CNI is running: [WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html 8. Checking the outbound SG rules for worker node i-0e775ca75fe57bb70 Passed: The Outbound security group rules for worker node i-0e775ca75fe57bb70 are sufficient to allow traffic to the EKS cluster endpoint 9. Checking if the worker node is running in AWS Outposts subnet Passed: Worker node's subnet subnet-092b2d53a74ac655f is not running in AWS Outposts 10. Checking basic NACL rules Passed: NACL acl-0507434ac9745b3dc has sufficient rules to allow cluster traffic. 11. Checking STS regional endpoint availability: Passed: STS endpoint is activated within region eu-west-2. 12. Checking if Instance Metadata http endpoint is enabled on the worker node: Passed: Instance metadata endpoint is enabled on the worker node. 13. Checking if SSM agent is running and reachable on worker node: [WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>". ============================================================================================================================================ Here is a list of other possible causes that were NOT checked by this document: [-] Ensure that Instance IAM role is added to aws-auth configmap, please check: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html. [-] If your account is a part of AWS Organizations Service, confirm that no Service Control Policy (SCP) is denying required permissions, please check: https://aws.amazon.com/premiumsupport/knowledge-center/eks-node-status-ready/. ``` One strange thing is that in point 5 it says private endpoint access is disabled but I have already created the cluster fully private. ``` { "update": { "id": "4c429db1-80bb-48d2-bf98-3f84524c0b83", "status": "Successful", "type": "EndpointAccessUpdate", "params": [ { "type": "EndpointPublicAccess", "value": "false" }, { "type": "EndpointPrivateAccess", "value": "true" }, { "type": "PublicAccessCidrs", "value": "[\"0.0.0.0/0\"]" } ], "createdAt": "2022-10-03T09:35:42.092000+00:00", "errors": [] } } ``` Also, when I run the command `sudo systemctl status kubelet` I get the result. `Unit kubelet.service could not be found.` My cluster config is as below ``` apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: test-cluster region: eu-west-2 version: "1.23" privateCluster: enabled: true vpc: id: vpc-ID subnets: private: hscn-private1-subnet: id: subnet-1 hscn-private2-subnet: id: subnet-2 managedNodeGroups: - name: serv-test-1 ami: ami-0ebb49de26355a371 instanceType: m5.xlarge desiredCapacity: 1 volumeType: gp2 volumeSize: 50 privateNetworking: true disableIMDSv1: true subnets: - hscn-private2-subnet ssh: allow: true tags: kubernetes.io/cluster/test-cluster: owned overrideBootstrapCommand: | #!/bin/bash /etc/eks/bootstrap.sh test-cluster --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=serv-test-1,eks.amazonaws.com/nodegroup-image=ami-0ebb49de26355a371' --dns-cluster-ip 10.100.0.10 --apiserver-endpoint {My endpoint} --b64-cluster-ca {My-CA} ```
0
answers
0
votes
19
views
asked a day ago