SFTP Transfer Service with API Gateway and no source IP in Lambda event

To troubleshoot why the source IP address isn't being passed to your Lambda function in the production environment while it works in your non-production account, I recommend the following approach:

1. First, verify that the AWS Transfer Family server is properly configured to pass the source IP. Since you mentioned seeing the source IP in the Transfer server CloudWatch logs, this suggests the Transfer server is correctly capturing it.

2. Check if there are any differences in how API Gateway is configured between your production and non-production environments. The mapping template looks correct, but there might be other configuration differences.

3. One possibility is that you're using AWS WAF with your API Gateway in the production environment. When AWS Transfer Family integrates with API Gateway for identity management, you can enable AWS WAF rules to filter requests based on source IP addresses. If WAF is configured differently between environments, it might affect how the source IP is handled.

4. Enable detailed CloudWatch logging for your API Gateway in the production environment to see what data is being received from the Transfer server and what's being passed to the Lambda function. This can help identify where the information is being lost.

5. Check if there are any differences in the API Gateway stages or deployment configurations between environments.

6. Verify that the Lambda function has the necessary permissions to receive all parameters from the API Gateway.

7. If you're using a NAT Gateway for your Lambda function's VPC configuration, ensure it's properly configured as this could potentially affect how IP information is handled.

Since you've already confirmed the setup works in your non-production environment, a systematic comparison of the configurations between environments should help identify the discrepancy. Focus particularly on any middleware or security layers (like WAF) that might be processing or filtering the requests differently in production.
Sources
SFTP, FTP, FTPs, Protocol Transfers – AWS Transfer Family Resources – Amazon Web Services
How to Log Original IP with Cloudflare in Front of HTTP API Gateway? | AWS re:Post
How to configure internet access for a lambda function with a properly configured NAT Gateway? | AWS re:Post

In case the questions come up, I am responding to the AI agent.

Check if there are any differences in how API Gateway is configured between your production and non-production environments. The mapping template looks correct, but there might be other configuration differences.

I have checked and there are no known differences other than age of the installation.

One possibility is that you're using AWS WAF with your API Gateway in the production environment. When AWS Transfer Family integrates with API Gateway for identity management, you can enable AWS WAF rules to filter requests based on source IP addresses. If WAF is configured differently between environments, it might affect how the source IP is handled.

WAF is not in use.

Enable detailed CloudWatch logging for your API Gateway in the production environment to see what data is being received from the Transfer server and what's being passed to the Lambda function. This can help identify where the information is being lost.

I need to look into this. As a side note, it would be helpful if the AI agent would provide where this information can be found.

Check if there are any differences in the API Gateway stages or deployment configurations between environments.

No differences that I can see.

Verify that the Lambda function has the necessary permissions to receive all parameters from the API Gateway.

I am not sure how this is checked if this is a relevant answer.

If you're using a NAT Gateway for your Lambda function's VPC configuration, ensure it's properly configured as this could potentially affect how IP information is handled.

How would NAT Gateway be part of the equation here?

I have been researching this more and I still do not have a solution but extra logging in the API Gateway is not providing any useful information. When turning on access logs the source IP shown is an AWS owned IP. I would venture a guess it has to do with the Transfer service itself. This is what I am logging.

      requestId         = "$context.requestId"
      extendedRequestId = "$context.extendedRequestId"
      ip                = "$context.identity.sourceIp"
      caller            = "$context.identity.caller"
      user              = "$context.identity.user"
      requestTime       = "$context.requestTime"
      httpMethod        = "$context.httpMethod"
      resourcePath      = "$context.resourcePath"
      status            = "$context.status"
      protocol          = "$context.protocol"
      responseLength    = "$context.responseLength"

Turning on structured logging for Transfer server is not useful as the proper source IP is already showing in other logs. That is not my area of concern. I need to know what API Gateway is sending to the Lambda and what the Lambda is seeing.

I am doing the following in my Lambda logs.

        const redact_event = event
        delete redact_event.password
        console.info(`REDACTED EVENT: ${JSON.stringify(redact_event)}`)

This should show me everything in th event except the password and all I see are username and serverId. The source IP is not in the event.

I am now wondering if this is age related. Did the Transfer server change over time where the source IP did not exist at some point and then support was added after our Transfer servers were built?

The input passthrough for the integration request for the working and non working systems is When no template matches the request content-type header

According to documentation history, the sourceIp field was added to custom IdP calls on June 9, 2020: https://docs.aws.amazon.com/transfer/latest/userguide/doc-history.html. It's indeed a reasonable guess that servers created before then and possibly not restarted since then might not include the field in custom IdP calls, as you were surmising. You could try stopping and restarting the server exhibiting the issue, if it didn't get resolved earlier.