Skip to content

two filers with same Firehose & s3 bucket

0

I have two FSx filers that deliver data to the same Firehose and S3 buckets. In the S3 bucket, I can see events only from one of the filers. Is it possible to receive events from both filers in one S3 bucket?

asked 2 years ago217 views

3 Answers
0

Yes, it is possible to receive events from multiple FSx filers in one S3 bucket. However, if you're only seeing events from one filer, there might be a configuration issue or a conflict in the event notifications.

To ensure you receive events from both filers:

  1. Check the event notification configuration on your S3 bucket. Make sure it's set up correctly to capture events from both filers.

  2. Verify that there are no conflicting event notifications. Sometimes, existing event notifications can interfere with new ones. Remove any overlapping event types, particularly those using 's3:ObjectCreated:' and 's3:ObjectRemoved:'.

  3. Ensure that both FSx filers have the correct permissions to send events to the S3 bucket. This includes proper IAM roles and policies.

  4. Review the Kinesis Data Firehose configuration to confirm it's set up to receive data from both filers.

  5. Check the FSx event notification configuration for both filers. Make sure they are both correctly set up to send events to the S3 bucket via Firehose.

  6. If you've recently added the second filer, try refreshing the file systems' AutoImportPolicy using the AWS CLI or API. This can help reapply the FSx event notification configuration.

If you've verified all these points and are still only seeing events from one filer, it may be worth contacting AWS support for further assistance, as there could be an underlying issue not immediately apparent.
Sources
Troubleshooting a misconfigured linked S3 bucket - FSx for Lustre
Multiple triggers on S3 bucket? | AWS re:Post

answered 2 years ago

  • Is it still possible if I use DirectPut as a source for the Firehose? I have - two filers - one firehose stream with DirectPut as source & one s3 bucket.

0

I would say yes - it's entirely possible to receive events from both filers simultaneously. Make sure both FSx file systems to use the same Kinesis Data Firehose delivery stream, which must have a name starting with aws-fsx-. Verify that file access auditing is enabled on both FSx file systems and they're both configured to use the same Firehose delivery stream. Also, check if the audit settings on both filers include the appropriate logging settings for file/folder access and file share access, and then confirm that both successful and failed access attempts are being logged according to your requirements.

Thank you.

AWS

answered 2 years ago

0

Hello,

It is possible to receive events from both the filers. You may once verify the following from your end,

  1. Make sure both FSx file systems to use the same Kinesis Data Firehose delivery stream, which must have a name starting with aws-fsx-.
  2. Verify that file access auditing is enabled on both FSx file systems and they're both configured to use the same Firehose delivery stream.

However, if you still witness issues with this then we will need to analyze account specific details. Please open a support case with AWS using the following link.

[+] https://support.console.aws.amazon.com/support/home#/case/create

Thank you.

AWS
SUPPORT ENGINEER

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.