- Newest
- Most votes
- Most comments
I believe the bot's answer is wrong and that what you're asking is not possible with S3's event notifications alone. With the prefix and suffix filter patterns that the bot suggested, the prefix test would get applied correctly, but the suffix would match all keys, rather than filtering out objects in subfolders. Specifying the wildcard suffix wouldn't filter out anything.
Without writing custom code, one option could be to configure the S3 bucket to send notifications to EventBridge instead of SQS. The events would arrive in the default event bus in the region. An EventBridge rule could match "Object Created" events for the specific bucket and apply the "prefix" operator in EventBridge to the "key" attribute to match subject-area/ (still including the subfolders), and send matching events to a second event bus in the same region.
Another EventBridge rule attached to the second event bus could filter out the subfolders with a combination of the "anything-but" and "wildcard" operators (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-pattern-operators.html) to exclude the wildcard pattern subject-area/*/*, thereby filtering out events for objects arriving in the subfolders, before sending the events to SQS.
The two event buses and rules would be needed because specifying two filters for the same field in EventBridge would imply "or" (union) instead of "and" (intersection) semantics, and the latter is needed first to match the prefix and then to filter out the values that match the same prefix but contain a forward slash in the rest of the string.
In general, complex wildcard patterns don't perform well. If possible, it would be more streamlined and cost-effective to arrange your data flow so that each set of data chosen for processing would be defined by its own prefix that doesn't overlap other data flows.
For example, instead of arranging data in these folders:
- s3://bucket-name/subject-area/ (without subfolders, for production data)
- s3://bucket-name/subject-area/staging/ (for staging data)
They would be better arranged like this, which would work with a single, simple prefix filter for each environment's data in either S3 event notifications or EventBridge:
- s3://bucket-name/subject-area/production/
- s3://bucket-name/subject-area/staging/
To generate event notifications for only the parent prefix in S3 but not under the child prefixes, you can use a combination of prefix and suffix filters when configuring your S3 event notification. Here's how you can achieve this:
-
When setting up the event notification for your S3 bucket, specify the parent prefix you want to monitor, in this case "subject-area/".
-
To exclude notifications from child prefixes, you can use a suffix filter that matches only files directly under the parent prefix.
For example, you could set up your event notification configuration like this:
- Prefix: subject-area/
- Suffix: *
This configuration will trigger notifications for files like: s3://bucket-name/subject-area/file.txt
But it will not trigger notifications for files like: s3://bucket-name/subject-area/staging/file.txt
The asterisk (*) in the suffix ensures that any file directly under the "subject-area/" prefix will trigger a notification, regardless of its name or extension.
Keep in mind that S3 event notifications are designed to be broad, so this approach may have limitations depending on your specific use case. If you need more granular control over event filtering, you might consider using Amazon EventBridge in conjunction with S3 events, which allows for more sophisticated event pattern matching.
Sources
Enabling and configuring event notifications using the Amazon S3 console - Amazon Simple Storage Service
Amazon S3 Event Notifications - Amazon Simple Storage Service
Relevant content
asked 2 years ago
You are correct Leo and indeed the answer from bot is inaccurate. The idea of adding a prefix for production would be a better design over using event bridge. To your solution on using the below prefix, does one have to configure individual events for each subject areas or is there a wildcard pattern that may be used to set triggers for all paths matching s3://bucket-name/*/production/. I assume it may work, but I am yet to test it. s3://bucket-name/subject-area1/production/ s3://bucket-name/subject-area1/staging/
@MK with S3 event notifications, you can only specify a prefix and/or suffix to be applied to the key.
*/production/*is not possible. If you want to use S3 event notifications, you could arrange the data like so:If you want the subject areas as the first level of the hierarchy, you could use EventBridge notifications from S3, because EventBridge rule patterns do support full wildcard syntax. EventBridge supports SQS natively as a target.