AWS Macie based on S3 bucket event

Hi there,

This approach may work for low volumes, with a few caveats.

• Macie's create-classification-job API has a very low TPS limit (once per 10 seconds with some bursting). This means that if you have multiple uploads in a short time, you will be throttled which will add additional code complexity.
• S3 has no way of saying "Get me all objects tagged X". In order for Macie to determine which objects to classify based on a tag, it needs to iterate over all objects in a bucket and call S3's get-object-tagging APIs. If your landing zone bucket is large and you are relying on tags, this can cause a lot of extra S3 API calls and delays in job performance.

In general, Macie's jobs are relatively heavyweight, meaning they are optimized for running over large volumes of data and not real time / just-in-time data flows. Some suggestions:

• Run jobs less frequently based on SLA. For instance, if you have a processing SLA of 4 hours, you can call Macie every 4 hours and process all objects that landed in that time. This leverages Macie's batch efficiencies and will avoid throttling.
• Use prefixes or other filters besides tagging to identify which objects to classify. For instance, instead of tagging an object with "date:5/19/2023", which can incur the overhead described above, you can put objects in a 2023/19/5 prefix and then scan, as prefix filtering within S3 is extremely efficient.

Hope that helps!