lambda function event trigger when appflow writes object to S3

The issue you're experiencing is likely related to how Amazon AppFlow writes objects to S3 and how S3 event notifications work with prefix-based triggers.

When AppFlow writes output files to S3 using the partitioning feature with Execution ID, it creates a folder structure. The S3 event notification you're receiving is probably for the creation of the Execution ID folder itself, not for the individual objects within that folder.

S3 event notifications can be triggered for both object creations and folder creations. In your case, it seems the notification is being sent when the Execution ID folder is created, which is why you're only seeing the path up to that level.

To resolve this issue, you have a few options:

1. Modify your Lambda function to handle both folder and file creation events. After receiving the folder creation event, you could list the contents of that folder to process the actual objects.

2. Change your S3 trigger configuration to use a more specific prefix that includes the Execution ID pattern. This might require you to use a wildcard or regex pattern if supported by your trigger configuration method.

3. Adjust your AppFlow configuration to write objects directly to the prefix path without creating additional folders, if that aligns with your data organization needs.

4. Use S3 event notifications with suffix filters instead of just prefix filters. This would allow you to trigger on specific file types or naming patterns of the actual objects.

5. Consider using Amazon EventBridge for more granular control over event filtering and routing from S3 to your Lambda function.

Without changing the AppFlow configuration, the most straightforward solution might be to modify your Lambda function to handle the folder creation event and then retrieve the actual objects within that folder for processing.

Remember to test any changes thoroughly to ensure they work as expected with your specific AppFlow and S3 setup.
Sources
Configure the Lambda function to be invoked when new objects are added to your S3 bucket - Moderating Image Content in Slack with Amazon Rekognition and Amazon AppFlow

Hi,

This is very probably due to the S3 object creation process by the application:

1. If it first creates an empty folder containing 'Execution ID' in its name, it will result in a first invocation of the trigger. It seems to be the one you report in your question
2. When the app object(s) (file(s)), the trigger gets activated for each of those files.

So, you have to adapt your code to skip the first invocation: if the object key ends with the execution id, your lambda can return with no further action.

Note: Folders don't really exist in S3, it is just a naming convention. Parts separated by '/' are considered as folders, but it's not true. See https://medium.com/@chamaln/why-s3-folders-dont-really-exist-35f29cf70477 for more details. In that sense, they are different from Linux directories which are materialized by inodes.

Best,

Didier