Lambda access to the MWAA RESTFul API on a private VPC

Question

Hello,

I need a lambda function in python to access the RestFul API of our MWAA on a private VPC.

The lambda runs on the same VPC and subnet as the MWAA environment.

We were able to connect the lambda to the CLI interface following the available [example](https://docs.aws.amazon.com/mwaa/latest/userguide/samples-lambda.html).

But I want to use the RESTFul API of Airflow for programmatic access using `boto3.client('mwaa')..invoke_rest_api()` and we are unable to make that work.

The error is:
`An error occurred (AccessDeniedException) when calling the InvokeRestApi operation: Private webserver environments can only call InvokeRestApi within a VPC.`

As said above the network configuration (VPC, subnets) sounds ok.

We looked more closely at the IAM execution role of the task and followed instructions given [there](https://docs.aws.amazon.com/mwaa/latest/userguide/access-policies.html#rest-api-access).
But that did not work. I am not even seeing a trace of the access in the MWAA web server logs.

My IAM policy is:

{
            "Action": [
                "airflow:InvokeRestApi"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:airflow:us-west-2:{account-redacted}:environment/{airflow-env-redacted}/Admin"
        }

We also tried without the Airflow role, in our case /Admin at the end:

{
            "Action": [
                "airflow:InvokeRestApi"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:airflow:us-west-2:{account-redacted}:environment/{airflow-env-redacted}"
        },

But that did not work either.

I must say the second bullet in the blue notes below the policy example are unclear to me. Do I need both policy entries so that 1) the regular policy control let the lambda request go, and 2) airflow authorizes the access ?

Thanks

Thomas

Tomas · Answer

I stumbled upon this issue as well, and I must say that the answer marked by "rePost-User-5074363" is very generic and incorrect. In particular, the statement “particularly for MWAA and execute-api” is misleading. This is not true—execute-api is simply an endpoint for AWS API Gateway private execution.

In my case, I had an EC2 instance in the same VPC, with no network filters, a security group open to port 443, and private DNS enabled and working. However, what I was missing was a VPC endpoint for env → `com.amazonaws..airflow.env`. After adding this VPC endpoint to my VPC, my Python code was able to successfully connect to the private web server of Airflow.

I also want to point out that the AWS documentation on this topic is unclear. It adds to the confusion by mentioning sourceVPC filters in the policy:

> While configuring a private web server, the InvokeRestApi action cannot be invoked from outside of a Virtual Private Cloud (VPC). You can use the aws:SourceVpc key to apply more granular access control for this operation. For more information, refer to aws:SourceVpc.

re:Post Agent · Answer

The issue you're experiencing with your Lambda function trying to access the MWAA RESTful API in a private VPC is likely related to network configuration. Here are some points to consider:

1. Network Configuration: Ensure that your Lambda function and MWAA environment are indeed in the same VPC and subnet. While you mentioned this is the case, it's worth double-checking.
2. VPC Endpoints: Make sure you have the necessary VPC endpoints set up, particularly for MWAA and execute-api. These endpoints allow your Lambda to communicate with AWS services within your VPC without going through the public internet. 
3. Security Groups: Check that the security groups associated with your Lambda function allow outbound traffic to the MWAA environment, and that the MWAA security group allows inbound traffic from the Lambda security group.
4. DNS Resolution: Ensure that DNS resolution is working correctly within your VPC. You might need to enable private DNS for your VPC endpoints.
5. Error Handling: Implement proper error handling in your Lambda function to catch and log any specific errors, which can provide more insight into what's going wrong.

If you've verified all these points and are still encountering issues, you might need to contact AWS support for further assistance, as there could be a more specific configuration issue within your AWS account or MWAA environment.
Remember, when working with private VPCs and MWAA, network configuration is crucial. The error message suggests that the Lambda is trying to access the MWAA environment from outside the VPC, which shouldn't be the case if your network configuration is correct. Double-check all VPC, subnet, and endpoint configurations to ensure everything is set up to allow internal communication.

Sources:

https://docs.aws.amazon.com/mwaa/latest/userguide/vpc-vpe-create-access.html
https://docs.aws.amazon.com/mwaa/latest/userguide/access-mwaa-apache-airflow-rest-api.html

Lambda access to the MWAA RESTFul API on a private VPC

Add your answer

Relevant content