- Newest
- Most votes
- Most comments
The error message says that the object wasn't found when trying to download it, so it's the multipart upload that had failed, and as the bot mentioned, the Amazon S3 Encryption Client for Go doesn't support multipart uploads (https://docs.aws.amazon.com/amazon-s3-encryption-client/latest/developerguide/go-examples.html). However, the size limit for a single-part upload in S3 is 5 GiB, while your multipart upload chunk size was set to just 10 MiB, so if your files are no larger than 5 GiB, you could consider using basic single-part uploads with the PutObject method of the the encryption client without more extensive changes to your code. It wouldn't perform as well as parallel uploads, but functionally it'd work.
The issue you're experiencing with downloading large encrypted files from S3 using client-side encryption is likely due to limitations in the current implementation of the Amazon S3 Encryption Client for Go. Here's an explanation of the problem and some potential solutions:
-
Multipart uploads and downloads: The Amazon S3 Encryption Client for Go does not currently support multipart uploads or downloads. This is why your code works for small files but fails for large files that require multipart operations.
-
Encryption metadata: When using client-side encryption, the encryption metadata is stored in the object's metadata. For multipart uploads, this metadata needs to be handled differently, which the current Go client doesn't support.
To work around this limitation, you have a few options:
-
Use smaller files: If possible, split your large files into smaller chunks that can be uploaded and downloaded without using multipart operations.
-
Use server-side encryption: If client-side encryption is not strictly necessary, you could switch to server-side encryption (SSE-S3, SSE-KMS, or SSE-C) which is natively supported by S3 and works with multipart uploads and downloads.
-
Use a different SDK: The Amazon S3 Encryption Client for Java supports multipart uploads and downloads with client-side encryption. If you can switch to Java, this might be a viable option.
-
Implement custom logic: You could potentially implement your own multipart upload and download logic using the Go SDK, handling the encryption and decryption of individual parts manually. However, this would be complex and require careful implementation to ensure security.
-
Wait for updates: AWS may update the Go SDK in the future to support multipart operations with client-side encryption. You could check for updates or reach out to AWS support for information on planned improvements.
For now, if you need to stick with Go and client-side encryption, the most straightforward solution would be to work with smaller files that don't require multipart operations. Alternatively, consider if server-side encryption could meet your security requirements, as it would allow you to use the existing multipart upload and download functionality.
Sources
Amazon S3 Encryption Client for Go examples - Amazon S3 Encryption Client
Upload or download large files to and from Amazon S3 using an AWS SDK - AWS SDK Code Examples
Using server-side encryption with Amazon S3 managed keys (SSE-S3) - Amazon Simple Storage Service
Relevant content
- asked 6 years ago
- asked 9 months ago
- asked 4 months ago
- asked 8 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 months ago
- AWS OFFICIALUpdated 7 months ago