By using AWS re:Post, you agree to the Terms of Use

Questions tagged with AWS PrivateLink

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Cannot access Timestream via PrivateLink without explicitly passing endpoint_url

Hi, I am trying to access Timestream from EC2/Lambda instances that run within a VPC so that I can speak to a RDS instance from those EC2 instances/Lambda functions. I have spent many hours trying to get access to Timestream via PrivateLink/a VPC instance endpoint to work and think I may have found an issue. When I provision a VPC endpoint for the Timestream ingest service, the Private DNS name is specific to the cell endpoint, e.g. *ingest-cell2.timestream.us-east-1.amazonaws.com* NOT the general endpoint URL that boto3 uses, i.e. *ingest.timestream.us-east-1.com*. When I run a nslookup on *ingest-cell2.timestream.us-east-1.amazonaws.com* it properly resolves to the private IP of the VPC endpoint ENI, but if I lookup the more general endpoint URL of *ingest.timestream.us-east-1.com* it continues to resolve to public AWS IPs. The result of this is that if I initialize the timestream write client normally and perform any actions, it hangs because it is trying to communicate with a public IP from a private subnet, ``` import boto3 ts = boto3.client('timestream-write') ts.meta.endpoint_url # https://ingest.timestream.us-east-1.amazonaws.com ts.describe_endpoints() # hangs ts.describe_database(DatabaseName='dbName') # hangs ``` If I explicitly give it the cell specific endpoint URL, the describe_endpoints() function throws an error but seemingly normal functions work (haven't tested writes or reads yet, just describing databses) ``` import boto3 ts = boto3.client('timestream-write', endpoint_url='https://ingest-cell2.timestream.us-east-1.amazonaws.com') ts.describe_endpoints() # throws UnknwonOperationException error ts.describe_databse(DatabaseName='dbName') # Succeeds ``` If I provision a NAT gateway in the private subnet rather than a VPC endpoint everything works normally as expected. Furthermore for fun, I tried adding the VPC endpoint private IP to the /etc/hosts file with *ingest.timestream.us-east-1.com* to force proper resolution and even then I get the same hanging behavior when running the above block of code This seems pretty broken to me. The whole point of the VPC endpoint is to enable the SDK to operate normally. Maybe I am missing something?
0
answers
0
votes
29
views
asked a day ago

Unable to run kubectl & eks commands in a fully private cluster

I have created a VPC fully private (no direct internet access), let's call it VPC-A. This vpc is peer connected to another VPC, let's call it VPC-B. This VPC-B has internet connection and is being used as a gateway for VPC-A. I have deployed a fully private cluster noly (not any node) in the private subnet of the VPC-A using the [guide](https://eksctl.io/usage/eks-private-cluster/). The problem is I am not able to run any kubectl and eks command just like mentioned in the [guide](https://eksctl.io/usage/eks-private-cluster/). After digging a lot on the internet and I found few things to access the cluster. One thing is that I must create a machine in that private VPC and try to access the cluster from there. I also created many issues on github but did not get proper answer. Below are some experts' answers > You can communicate with the K8s API by deploying EC2 instance inside that VPC and defining the EKS K8s API to your kubectl. Well, I have deployed an instance within the vpc of my cluster but whenever I run the kubectl command from the instance inside the private vpc, I get the following error message `Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)` Also in the [EKS fully private cluster guide](https://eksctl.io/usage/eks-private-cluster/) it is mentioned that > If your setup can reach the EKS API server endpoint via its private address, and has outbound internet access (for EKS:DescribeCluster), all eksctl commands should work. Can please someone guide me properly that how can I create such setup? I ran a number of commands to check if anything is wrong with accessing the server address. ``` nmap -p 443 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com Starting Nmap 7.80 ( https://nmap.org ) at 2022-09-09 11:11 UTC Nmap scan report for 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com (192.168.*.*) Host is up (0.00031s latency). Other addresses for 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com (not scanned): 192.168.*.* rDNS record for 192.168.*.*: ip-192-168-*-*.eu-west-*.compute.internal PORT STATE SERVICE 443/tcp open https Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds ``` Another command is ``` nslookup 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: Name: 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com Address: 192.168.*.* Name: 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com Address: 192.168.*.* ``` And another is ``` telnet 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com 443 Trying 192.168.*.*... Connected to 1E9057EC8C316E£D"@JY$J&G%1C94A.gr7.eu-west-*.eks.amazonaws.com Escape character is '^]'. ^CConnection closed by foreign hos ``` It is clear that I can access the api server endpoints from my machine which is in the same vpc as the api server. But still when I run the kubectl command I am getting this output `Unable to connect to the server: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)` When I ran the below command `kubectl cluster-info dump` I got the following error message `Unable to connect to the server: proxyconnect tcp: dial tcp: lookup socks5h on 127.0.0.53:53: server misbehaving` Thanks
1
answers
0
votes
72
views
asked 25 days ago

ECR Cross Account Private Link

Hi All, Struggling for a couple of days already with the following: I've followed this guide: https://aws.amazon.com/blogs/networking-and-content-delivery/centralize-access-using-vpc-interface-endpoints/ I have setup AWS Organisations with all the separate Accounts like nonprd, prd, .... AND the Shared resources account.... CIDR for Shared: 10.40.0.0/16 CIDR for nonprd: 10.0.0.0/16 CIDR for prd: 10.1.0.0/16 In this shared resources account, I've created the 4 vpc endoints for ECR (Shared resources account holds our ecr docker repos for other accounts). logs,dkr,api and S3. I've setup VPC peering with my nonprd and prd account. I've created the route table entries so that all traffic is flowing from shared to the vpc-peering connections cidr and visa versa. The private dns option for the VPC Endpoints are disabled and manually created as private Route53 records. Exactly as the ecr domain. So I have 3 extra private records IN the SHARED resources account: * api.ecr.eu-west-1.amazonaws.com * dkr.ecr.eu-west-1.amazonaws.com * logs.eu-west-1.amazonaws.com I've create the alias record pointing to the private hosted zones. I've done the Associations for Route53 from all the VPC's in nonprd and prd. I CAN resolve the dns records. BUT... And now the problem arises... When I try to run the containers in my nonprd account in any of the vpc's there, my tasks are given one of the following errors: * ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::${AWS::AccountId}:assumed-rol... * ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post https://api.ecr.... The policy on the VPC endpoints (complete snippet from my cfn-template): ``` EcrApiEndpoint: Type: AWS::EC2::VPCEndpoint Properties: PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: AWS: - "arn:aws:iam::51*******0:root" # NonPrd - "arn:aws:iam::1******3:root" # Prd - !Sub arn:aws:iam::${AWS::AccountId}:root Action: - ecr:BatchGetImage - ecr:GetAuthorizationToken - ecr:GetDownloadUrlForLayer - ecr:BatchCheckLayerAvailability - ecr:PutImage - ecr:InitiateLayerUpload - ecr:UploadLayerPart - ecr:CompleteLayerUpload Resource: - !Sub "arn:aws:ecr:${AWS::Region}:${AWS::AccountId}:repository/*" VpcId: !FindInMap [Environments, !Ref Environment, VPC] VpcEndpointType: Interface PrivateDnsEnabled: false SecurityGroupIds: - !GetAtt VPCESecurityGroup.GroupId SubnetIds: - !Select [ 0, !FindInMap [Environments, !Ref Environment, PrivateSubnets], ] - !Select [ 1, !FindInMap [Environments, !Ref Environment, PrivateSubnets], ] - !Select [ 2, !FindInMap [Environments, !Ref Environment, PrivateSubnets], ] ServiceName: Fn::Join: - "" - - "com.amazonaws." - !Ref "AWS::Region" - ".ecr.api" ``` So the VPC endpoints run in the Private subnet of the SHARED resources account. The ecs fargate service/task also has the correct permissions (everything is working fine without the VPC endpoints). Can someone help... Please...
4
answers
0
votes
307
views
asked 6 months ago