By using AWS re:Post, you agree to the Terms of Use

Networking & Content Delivery

AWS provides the broadest and deepest set of networking services with the highest reliability, most security features, and highest performance in the world. This helps ensure you can run any kind of workload you have in the cloud.

Recent questions

see all
1/18

Https call to API Gateway via VPC Endpoint fails to make connection intermittently

I have a private API gateway in its own account. It is used by clients having VPC Endpoint interfaces to execute-api service, and until now these have had Private DNS enabled, and there have been no issues. A new client uses some existing public APIs, so Private DNS is disabled. However, they have had intermittent connectivity to the gateway during their testing. I tried reproducing this from a second account with a test Lambda (node.js, v16, arm) in a VPC, using a VPC Endpoint with Private DNS disabled. I was able to reproduce the intermittent connectivity, but I can't understand why this happens. [Edit: The subnets attached to the VPC use the same security group, and this allows htttps ingress from 10.57.150.0/24] I found that when using the generic endpoint DNS Name (no AZ marker in the name) the intermittent issue could be reproduced. If I switch to using the Endpoint DNS Names that include the AZ marker, then 1 of the DNS Names connected every time, but the other 2 (we use 3 AZs and 1 subnet per AZ) fail to connect with a timeout error. I added a call to resolve the hostname passed in, and all three hosts resolve to what I would expect (10.57.150.x), so I think this is a routing issue rather than DNS. The route tables for all three subnets are the same, 2 routes for the s3 and DynamoDB prefix lists, a route for 10.57.150.0/24 and the remaining 0.0.0.0/0 going via a transit gateway instance. I'm not sure what other information I would need to add here. Has anyone seen anything like this before?
0
answers
0
votes
5
views
asked an hour ago
1
answers
0
votes
6
views
asked 3 hours ago

Cannot add environment variable through Ebextensions

I'm using .ebextensions to create VPCEndpoints so in the **Resources** section I've addded the needed section for the VPCEndpoint. Then after that in the **option_settings** section I'm trying to add an environment variable in my elastic beanstalk application referencing the created VPCEndpoint, but when i check the environment variables from the elastic beanstalk console the value is added as a plain text not the Ref of the VPCEndpoint (Check the screenshot) So how can i make it interpret the Ref of the endpoint ? ![Enter image description here](/media/postImages/original/IMkQEkAlsLRyCG5pYs1i8hkA) ``` Resources: NewsonarVPCEndpoint: Type: AWS::EC2::VPCEndpoint Properties: PrivateDnsEnabled: false SecurityGroupIds: - {"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "ALLOW_INBOUND_FROM_VPC_SECURITY_GROUP", "DefaultValue": "default_value"}} ServiceName: { "Fn::Join": [ "", [ "com.amazonaws.vpce.",{"Fn::GetOptionSetting": {"Namespace": "aws:elasticbeanstalk:application:environment", "OptionName": "AWS_REGION", "DefaultValue": "us-east-1"}},".",{"Ref": "sonarVPCEndpointService"}]] } SubnetIds: - { "Ref": "Subnet1Id" } - { "Ref": "Subnet2Id" } - { "Ref": "Subnet3Id" } VpcEndpointType: Interface VpcId: { "Ref": "VpcId" } option_settings: aws:elasticbeanstalk:application:environment: VPC_ENDPOINT: '`{"Ref" : "NewsonarVPCEndpoint"}`' ```
0
answers
0
votes
4
views
asked 6 hours ago

Ubuntu Managed Nodes creation failure in Fully-Private cluster

Hi, For some reason I am not able to create Ubuntu managed nodes in fully private cluster. Though, managed Amazon-Linux nodes and all other self-managed nodes are joining the cluster successfully. I have followed all the guides and troubleshooting aws websites already still I am not successful. I have also run the troubleshooting script. Below is the result ``` HERE IS A SUMMARY OF THE ITEMS THAT REQUIRE YOUR ATTENTION: [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ . [WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html [WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>". ============================================================================================================================================ Here are the detailed steps of the document execution: [X] Checking EKS cluster test-cluster: EKS Cluster: test-cluster is in Active state. 1. Checking if the cluster Security Group is allowing traffic from the worker node: Passed: The cluster Security Group sg-068d83e5a68aff9c7 is allowing traffic from the worker node. 2. Checking DHCP options of the cluster VPC: Passed: AmazonProvidedDNS is enabled 3. Checking cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T for the required permissions: Passed: IAM role for cluster test-cluster has the required IAM policies attached. Passed: The cluster IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-cluster-ServiceRole-9WPNFE2Q3N2T has the required trust relationship for the EKS service. 4. Checking control plane Elastic Network Interfaces(ENIs) in the cluster VPC: Passed: The cluster Elastic Network Interfaces(ENIs) exist. 5. Cluster Endpoint Private access is disabled for your cluster, checking if the Public CIDR ranges include worker node i-0e775ca75fe57bb70 outbound IP: Passed: The cluster allows public access from 0.0.0.0/0 6. Checking cluster VPC for required DNS attributes: Passed: Cluster VPC vpc-0cbb4879c588fb52a has the required DNS attributes correctly set. -------------------------------------------------------------------------------------------------------------------------------------------- [X] Checking worker node i-0e775ca75fe57bb70 state: The instance is Running. 1. Checking if the EC2 instance family is supported: Passed: EC2 instance family m5.xlarge is supported. 2. Checking the worker node network configuration: Passed: Worker node is created in a private subnet without a NAT Gateway so VPC endpoints need to be used. Passed: Checking VPC Endpoints setup: Passed: The VPC Endpoint com.amazonaws.eu-west-2.ec2 exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ec2 is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.api exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.api is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.ecr.dkr is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: The VPC Endpoint com.amazonaws.eu-west-2.sts exists. Checking its configuration: Passed: Security groups [{'GroupId': 'sg-01294c96494e79aaa', 'GroupName': 'eksctl-test-cluster-cluster-ClusterSharedNodeSecurityGroup-1TIY3P78RYQOZ'}] applied to VPC Endpoint com.amazonaws.eu-west-2.sts is allowing the worker node to reach the endpoint. Passed: The default VPC Endpoint Policy is being used. Passed: S3 gateway endpoint ['vpce-0e01febb999cf1735', 'vpce-08ef7ffac589c3d01'] is added to the worker's VPC. Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-0e01febb999cf1735. Passed: Worker node's route table rtb-0d26fefe5a613e64f has the required route for the S3 endpoint vpce-08ef7ffac589c3d01. 3. Checking the IAM instance Profile of the worker node: Passed: The instance profile arn:aws:iam::782534010321:instance-profile/eks-d0c1cede-94e5-c6bc-75fd-2f86043a90eb is used with the worker node i-0e775ca75fe57bb70 . Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX is attached to Instance Profile. Passed: IAM role arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX has the required IAM policies attached. Passed: No issues detected with the trust relationship policy of the arn:aws:iam::782534010321:role/eksctl-test-cluster-nodegroup-ser-NodeInstanceRole-12LZWOL4EJAJX role. 4. Checking worker node's UserData bootstrap script: Passed: The UserData of the worker node contains the required bootstrap script. 5. Checking the worker node i-0e775ca75fe57bb70 tags: Passed: Worker node i-0e775ca75fe57bb70 has the required cluster tags. 6. Checking the AMI version for EC2 instance i-0e775ca75fe57bb70: [WARNING]: Worker node's AMI ami-0ebb49de26355a371 differs from the public EKS Optimized AMIs. Ensure that the Kubelet daemon is at the same version as your cluster's version 1.23 or only one minor version behind. Please review this URL for further details: https://kubernetes.io/releases/version-skew-policy/ . 7. Checking worker node i-0e775ca75fe57bb70 Elastic Network Interfaces(ENIs) and Private IP addresses to check if CNI is running: [WARNING]: No secondary private IP addresses are assigned to worker node i-0e775ca75fe57bb70, ensure that the CNI plugin is running properly. Please review this URL for further details: https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html 8. Checking the outbound SG rules for worker node i-0e775ca75fe57bb70 Passed: The Outbound security group rules for worker node i-0e775ca75fe57bb70 are sufficient to allow traffic to the EKS cluster endpoint 9. Checking if the worker node is running in AWS Outposts subnet Passed: Worker node's subnet subnet-092b2d53a74ac655f is not running in AWS Outposts 10. Checking basic NACL rules Passed: NACL acl-0507434ac9745b3dc has sufficient rules to allow cluster traffic. 11. Checking STS regional endpoint availability: Passed: STS endpoint is activated within region eu-west-2. 12. Checking if Instance Metadata http endpoint is enabled on the worker node: Passed: Instance metadata endpoint is enabled on the worker node. 13. Checking if SSM agent is running and reachable on worker node: [WARNING]: As SSM agent is not reachable on worker node, this document did not check the status of Containerd, Docker and Kubelet daemons. Ensure that required daemons (containerd, docker, kubelet) are running on the worker node using command "systemctl status <daemon-name>". ============================================================================================================================================ Here is a list of other possible causes that were NOT checked by this document: [-] Ensure that Instance IAM role is added to aws-auth configmap, please check: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html. [-] If your account is a part of AWS Organizations Service, confirm that no Service Control Policy (SCP) is denying required permissions, please check: https://aws.amazon.com/premiumsupport/knowledge-center/eks-node-status-ready/. ``` One strange thing is that in point 5 it says private endpoint access is disabled but I have already created the cluster fully private. ``` { "update": { "id": "4c429db1-80bb-48d2-bf98-3f84524c0b83", "status": "Successful", "type": "EndpointAccessUpdate", "params": [ { "type": "EndpointPublicAccess", "value": "false" }, { "type": "EndpointPrivateAccess", "value": "true" }, { "type": "PublicAccessCidrs", "value": "[\"0.0.0.0/0\"]" } ], "createdAt": "2022-10-03T09:35:42.092000+00:00", "errors": [] } } ``` Also, when I run the command `sudo systemctl status kubelet` I get the result. `Unit kubelet.service could not be found.` My cluster config is as below ``` apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: test-cluster region: eu-west-2 version: "1.23" privateCluster: enabled: true vpc: id: vpc-ID subnets: private: hscn-private1-subnet: id: subnet-1 hscn-private2-subnet: id: subnet-2 managedNodeGroups: - name: serv-test-1 ami: ami-0ebb49de26355a371 instanceType: m5.xlarge desiredCapacity: 1 volumeType: gp2 volumeSize: 50 privateNetworking: true disableIMDSv1: true subnets: - hscn-private2-subnet ssh: allow: true tags: kubernetes.io/cluster/test-cluster: owned overrideBootstrapCommand: | #!/bin/bash /etc/eks/bootstrap.sh test-cluster --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=serv-test-1,eks.amazonaws.com/nodegroup-image=ami-0ebb49de26355a371' --dns-cluster-ip 10.100.0.10 --apiserver-endpoint {My endpoint} --b64-cluster-ca {My-CA} ```
0
answers
0
votes
18
views
asked a day ago

Problem on Application load balancer with rule: Health check only responds on the default rule

Hi everyone I have 3 microservices running on an **ECS cluster**. Each microservice is launched by a **Fargate task**. Each microservice runs in its own Docker container. * *Microservice A* responds on port 8083. * *Microservice B* responds on port 8084. * *Microservice C* responds on port 8085. My configuration consists of two public subnets, two private, an internet gateway and a NAT, as well as two security groups, one for fargate services and one for ALB. On the security groups I have enabled inbound traffic on all ports. I have defined a listner for the ALB that responds on port 80 and wrote some path-based rules to route requests to the appropriate target group (*every target group is a Target type*) :![Enter image description here](/media/postImages/original/IM8oFOWQXjQEuDjdKe3PeGgw) Only the health check of the target group that responds to the default rule responds ( but I suspect it all happens randomly) , and consequently only the service reachable on port 8083 works ![Enter image description here](/media/postImages/original/IMtOk5-EqJRrmxLa49ium6hg) The remaining target groups are **unreachable**. What you notice is that in the "*Registered Target"* section the assigned IP addresses change continuously. For example: ![![Enter image description here](/media/postImages/original/IMkdJ_RNqsTJazJ3J8j4foqw) Enter image description here](/media/postImages/original/IMCm7LLgy1QJKk0JsLC3XlGg) But every time IP assigned it generates a timeout. It can happen quite randomly that a certain IP address is registered correctly. These are the ECS configurations of one of the unresponsive services: ![Enter image description here](/media/postImages/original/IMOdt86JdpS_2paN_elspK5g) What is the problem and how can I solve it? Thank you. **UPDATE1** I tried to add a new instance for microservice A. For the new IP (10.0.0.137) the health check is not responding. After a few minutes, the provisioning of a new IP (10.0.0.151) appears and it is registered correctly: ![Enter image description here](/media/postImages/original/IMUcZubrfCRrGo-fpqYAvSJQ) **UPDATE2** It is really strange behavior. **All services are now connected correctly**, after several hours of failed attempts. It looks like an IP address assignment problem. Before finding the correct address, AWS makes several attempts with different IP addresses until it randomly finds the correct one. These are the CIDRs of my PRIVATE subnets * private_subnets = ["10.0.0.128/28", "10.0.0.144/28"] * public_subnets = ["10.0.0.0/28", "10.0.0.16/28"] While these are the IPs that connected successfully: 1. 10.0.0.136 (micorservice A istance1) 2. 10.0.0.151 (micorservice A istance2) 3. 10.0.0.153 (micorservice A istance3) 4. 10.0.0.152 (micorservice B) 5. 10.0.0.142 (Microservice C)
2
answers
0
votes
34
views
asked a day ago

Recent articles

see all
1/2

Popular users

see all
1/18

Learn AWS faster by following popular topics

1/2