Files across all S3 buckets unaccessible, even from GUI

Based on the error message and symptoms you've described, it appears that your S3 objects have been encrypted using Server-Side Encryption (SSE), and the system is now requiring the correct encryption parameters to access these files. This situation can occur if there's been a change in your S3 bucket's encryption settings or if the encryption keys used for SSE have been modified or become inaccessible.

Here are a few potential reasons and steps to investigate:

1. Check your S3 bucket encryption settings: It's possible that default encryption has been enabled on your buckets, causing newly uploaded objects to be encrypted while leaving existing objects unaffected. This would explain why new uploads are accessible while older files are not.

2. Review your IAM policies and roles: Ensure that the IAM roles and policies associated with your access to S3 have not been changed and still include the necessary permissions to decrypt objects.

3. Verify KMS (Key Management Service) settings: If you're using AWS KMS for encryption, check if there have been any changes to your KMS keys or their permissions. Ensure that the necessary KMS decrypt permissions are in place.

4. Check for any recent changes in your AWS account: Look for any modifications to S3 bucket policies, IAM roles, or security settings that might have affected your ability to access the files.

5. Investigate cross-account access: If any of these objects were uploaded by users from different AWS accounts, check the Object Ownership settings of your buckets. Ensure they are set to "Bucket owner enforced" to avoid issues with cross-account object ownership.

6. Contact AWS Support: If you cannot identify the cause after checking these areas, it would be advisable to contact AWS Support. They can help investigate the issue more deeply and provide guidance on resolving the encryption-related access problems.

Remember, when dealing with encrypted objects, it's crucial to have the correct decryption parameters and permissions in place. The fact that this issue is affecting multiple buckets and thousands of files suggests a system-wide change or issue rather than a problem with individual objects.
Sources
Working with objects in Amazon S3 - Amazon Simple Storage Service
Why can't my S3 File Gateway access objects uploaded by cross-account users? | AWS re:Post

The error message seems to indicate that the objects are encrypted with S3's server-side encryption, but using customer-provided keys. It's called SSE-C for short. It means that S3 stores the object in encrypted form but doesn't retain a copy of the key. Instead of S3 generating and storing the keys, the keys are provided to S3 by the client, such as your application or the AWS CLI, every time an object is uploaded. It must also be provided when an object's content is to be read, such as when downloading or copying an object. When you try to access an SSE-C-encrypted object without providing the key, the error message you mentioned is returned.

The "Server-side encryption settings" box showing "UnknownError" in the encryption characteristics of an individual object is caused by the console not having the encryption key with which to access the object.

The settings you see configured for the S3 bucket don't determine whether SSE-C is used. It's solely controlled by the client that uploads the object to the bucket, which would typically be done by your application or the AWS CLI, and the choice of encryption method is made for every object uploaded separately.

General documentation for SSE-C is here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html, but for clarity, there's no way to access any SSE-C-encrypted objects either technically or by raising a support ticket, unless you have the encryption key.

It's useful also to consider that if you don't recognise having encrypted data intentionally with your own keys, it's very possible that your AWS account has been compromised, and an outside bad actor who's gained control over your environment has encrypted your data to hold it for a ransom payment.

If this is the case, you'd probably see that the timestamps of the inaccessible objects in your S3 buckets are all very recent, even if the original data had been placed in the buckets over a long stretch of time. If you find this to be the case or otherwise consider it rather certain you or your colleagues didn't encrypt the data intentionally, you should first secure your AWS account and then see if any data is recoverable from backups or previous object version in S3.

Typically, these types of simple S3 compromises occur when static access keys are created for an IAM user that's granted broad, blanket permissions to S3 resources and perhaps even more broadly to other AWS services. You should deactivate all the static access keys that aren't properly secured. Bad actors typically also attempt to create new IAM users and potentially other access resources, such as IAM roles that can be assumed from other accounts, so that they would retain access to your environment after the initial access path (like an individual IAM user's access key) is blocked. You should follow the official instructions in this article to regain control and block the outsider's access: https://repost.aws/knowledge-center/potential-account-compromise