Files keep disappearing from S3 bucket!

Greetings!

The issue you're facing with disappearing images from your S3 bucket, despite having significant experience with AWS and S3, is indeed unusual and concerning. Here are several potential causes and troubleshooting steps you can consider:

Unauthorized Access or Human Error: Check if there are other users or services with write access to the bucket who might inadvertently or maliciously be deleting the images. Review the IAM (Identity and Access Management) policies and bucket permissions to ensure that only authorized entities have write or delete access.

Bucket Logging and AWS CloudTrail: Enable logging on your S3 bucket to track requests, especially delete operations. This can help identify the source of the deletions. AWS CloudTrail logs can also provide detailed information about API calls made to your AWS resources, including who made the call, from where, and when.

Cross-Region Replication Issues: If you're using cross-region replication, ensure that the replication rules are correctly configured. Incorrect configurations might lead to unintentional deletions or overwrites.

Lifecycle Policies: Double-check lifecycle policies. Although you mentioned that there's no expiration policy set, it's worth reviewing any existing lifecycle rules to ensure they are not inadvertently configured to delete objects.

Third-Party Services or Applications: If you're using any third-party services or applications integrated with your S3 bucket (like CI/CD pipelines, backup services, or image processing services), verify their configurations and logs to ensure they're not causing the deletions.

Versioning Issues: If versioning is enabled on your bucket, it's possible that objects are being overwritten or deleted in a way that's not immediately apparent. Review the versioning status and history of the affected objects.

Application Bugs: There might be a bug in your mobile app or in any automation scripts that interact with the S3 bucket, leading to unintended deletions.

AWS Support: If you have exhausted all other options, consider contacting AWS Support. They can provide more detailed insights, especially if the issue is related to AWS's internal processes or rare bugs.

Remember, the key is to systematically eliminate each potential cause. Start with access and permission reviews, then move to logging and tracing actions on the bucket, and finally, look into any external integrations or services that might be interacting with your S3 bucket.

Please let me know if i answered your question

This is strange, as objects shouldn't just disappear from a bucket. You've checked there are no lifecycle rules in place, might there be anything else, maybe a housekeeping job running on an EC2 instance or an on-prem server, that could be deleting objects over X days old.

That's another thing - is there any pattern to how and when this is happening, is it a particular day of the week, or time of the day, or just random? And is there any pattern to the objects that are going missing, are they in a partcular area of the bucket, are they older than X days, and so on.

Also look at how the bucket is being populated, if the content is update using an S3 sync jo, or bucket replication, then when the source object is deleted the target object in the bucket can be deleted as well.

Do you have versioning enabled on the bucket, as this could tell you when exactly the object is being deleted.

Going forward you may want to enable CloudTrail logging https://docs.amazonaws.cn/en_us/AmazonS3/latest/userguide/enable-cloudtrail-logging-for-s3.html https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html#logging-data-events-examples

This will give you much more information about the circumstances around the deletion of an object. This knowledge document will be helpful too https://repost.aws/knowledge-center/s3-audit-deleted-missing-objects