My ultimate goal is to have my EMR Serverless application Internet egress access to invoke a specific third party API/REST call. Unfortunately, the EMR Serverless application must then be put into a VPC configured with a NAT Gateway and subnets to allow for such egress access.
I can configure the networking resources to work using CDK - I have validated this to work. However, this would then instantiate a NAT Gateway which has per hour usage costs. My EMR Serverless application runs only a few times a day and for short periods, so I only need the NAT Gateway to be up when the EMR Serverless job is running.
I believe the only way to accomplish this is by adding a Lambda that configures the networking resources before my EMR Serverless job runs in my AWS step function, then decommissions these resources after the EMR Serverless job finishes. Unfortunately, I cannot get the accompanying boto3 code to work.
My boto3 code is below. I get the error:
Unable to push logs, please ensure logging destination is valid and execution role has sufficient permissions. Error: "Connect timeout on endpoint URL: "https://myapplication-logs-useast2-<myaccount>-test.s3.us-east-2.amazonaws.com/MyApplicationNestedStackc-*/applications/<application-id>/jobs/<job_id>
When running my EMR Serverless job. I'm fairly sure my IAM permissions are correct as it works for my CDK (using the same IAM role I'm doing now in my Lambda, CloudTrail doesn't show any errors. That's why I'm guessing the issue is Internet egress access is not happening.
My Lambda code is here. What's wrong with my Lambda code? Am I missing something or misconfigured something that is preventing Internet egress access from my EMR Serverless application/job?
AWS OFFICIALUpdated 8 days ago
Please post your Lambda code - it's missing.
I've added my Lambda code link.
Could you paste the code as text instead of sharing it as a URL? A 403 error occurs and the URL cannot be accessed.
Unfortunately, it's 300 lines of code so it doesn't fit into the original post. I don't see an option that allows me to attach the code file either.
Also, I tried the link from an incognito browser window and it should work. Here's a different link of the same code: here