By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Storage Gateway eventbridge events not firing on large multi-part uploads

0

Hopefully not an indication of a bug....

I've got my storage gateway sending eventbridge events that then triggers a step function. I also, for debugging, have the same filtered event going to SNS to email me, and a seperate eventbridge rule which does not filter the events, it just emails me. everything the file gateway sends.

This setup is working well for using the upload complete event to trigger my step function and I have verified it is working on a mix of uploaded data

HOWEVER, I used the storage gateway to upload 3 very large files (200GB+ each) to the S3 bucket and noticed these did NOT trigger the upload complete step function and I did NOT receive event emails (ie the storage gateway did not send any events and because I have the 2nd rule that is unfilted and sends every event I know the storage gateway didn't send an event). I'm seeing no further upload activity from the metrics on the storage gateway, and the object appear in S3 as their full size so the uploads have clearly completed.

Its clear the file gateway uploaded these very large files as multi-part uploads, and I'm wondering if this is the reason I didn't receive any event emails (or the step function being triggered). I highly doubt that is the desired behaviour as I expect 'upload complete' to apply to all objects, not just ones uploaded whole. I was hoping the Storage gateway events would be 'more reliable' as AWS's own blog post about file upload events suggested over S3 based event triggering.

Can you confirm if this behaviour is actually the expected desired behavior, or if I have to employ a work-around using s3 eventbridge events (multipart upload complete? )

Thanks in advance,

Owen.

Topics
StorageServerlessApplication Integration
Tags
AWS Storage GatewayAmazon EventBridge
Language
English
Ommps
asked a year ago473 views
2 Answers
0
Accepted Answer

Owen,

Thank you for taking time to post about your experience with the S3 File Gateway. I work on the storage gateway product team and I would be interested in learning more about how you use storage gateway. Would you be interested in connecting?

Jesse

AWS
jesse_aws
answered a year ago
  • Ommps
    a year ago

    TL;dr we are wanting to store long term data, that doesn't yet have a known 'end date' in terms of retention. some data could eventually be removed within a few months, others years... but we don't know that at the point of upload.

    To aid in malware protection, Bucket versioning is enabled, and we want to enable 'legal hold' on the objects as soon as we can after upload. again, we cannot use the default retention rules as we don't have an end date, so a legal hold allows us that 'infinite' hold.

    So we want to trigger a lambda, after a short-ish delay to apply the hold to the objects. However, and fully understandably, the Storage Gateway eventbridge events will fire when folder objects are created. This is 100% understandable. We wanted however, to not trigger the stepfunction we are using to create the time delay every single time as it was, at that time, just a simple start -> wait -> trigger lambda -> end process, so triggering the step function would trigger the lambda, all adding to costs, yes very small, but 1000's of very small become very big very quickly. and with many 10,000's of files, and maybe as many folder objects it gets expensive.

    putting a filter onto the eventbridge rule that simply says 'is the uploaded object larger than 0 bytes' is a no brainer... bar the very odd and quite bizarre limitation of the numeric compare...

    the range given (-5.0e9 and +5.0e9 inclusive ) is what appears to be a 32bit signed int. I'm not sure why that limitation is imposed.

0

I've found the answer, and I still think its a terrible limitation.....

we were applying filtering to the eventbridge rule as such:

{ "detail": { "object-size": [{ "numeric": [">", 0] }] }, "detail-type": ["Storage Gateway Object Upload Event"], "source": ["aws.storagegateway"] }

Turns out, the numeric test can only deal with values in the range -5.0e9 and +5.0e9 inclusive[ 1 ].... so objects over 5GB in size were being filtered out due to the numeric comparison not being able to do a simple 'is the object greater than 0 bytes in size' test. The point of the test is to stop our step functions triggering when folders are created on the storage gateway, and only trigger for actual file objects.

Is there a better way to achieve what we want within the limits of the json filtering - ie trigger on the upload event, but not if the event was for a folder object being created (ie a 0 byte object).

can I also ask why does such a numeric limitation even exist?

Owen.

EDIT: I have found a solution by adding a choice in the step function where I can use the NumericGreaterThan and NumericEquals choices. These seem to not have the odd limitations put on the Eventbridge filters...... but it now means I'm running the step function on every object upload instead of only when I know the lambda is actually needed. So it does have cost implications.

Ommps
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content