Upload/download large encrypted files on S3 with client-side-encryption

0

Hello. I am trying to upload and download large encrypted files to S3 using a client-side encryption (see code below). Uploading works but I get this error message when downloading:

operation error S3: GetObject, failed to load objectMetadata: bucket=0xc0002b6850; key=0xc0002b6860; err=operation error S3: GetObject, https response error StatusCode: 404, RequestID: JB2T72TFB82DWP7M, HostID: tpQaEDUb/ZxaxOE7ZI1/akdZFKoXRjs3vYlKma0GtTVTa2Gmn3yXH49axHJmuUJ6oNrLxRDrtGA=, NoSuchKey:

The same code works with small files, but I can't figure out how to make uploading/downloading large files work with client-side-encryption. Do you have any advice on how to solve this? Thank you in advance.

Code

I am using the Amazon S3 encryption client and the Upload Manager from the sdk for go.

func UploadLargeObject(S3EncryptionClient *client.S3EncryptionClientV3, bucketName string, objectKey string, largeFileName string) error {
    largeFile, _ := os.Open(largeFileName)

    uploader := manager.NewUploader(S3EncryptionClient, func(u *manager.Uploader) {
        u.PartSize = 10 * 1024 * 1024
    })
    _, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
        Body:   largeFile,
    })
    if err != nil {
        return fmt.Errorf("error uploading object %q to %q: %w", largeFileName, objectKey, err)
    }
    log.Printf("File %q uploaded successfully to bucket %s.\n", largeFileName, bucketName)
    return nil
}

func DownloadLargeObject(S3EncryptionClient *client.S3EncryptionClientV3, bucketName string, objectKey string, largeFile_dest string) error {
    largeFile, _ := os.Create(largeFile_dest)

    downloader := manager.NewDownloader(S3EncryptionClient, func(d *manager.Downloader) {
        d.PartSize = 10 * 1024 * 1024
    })
    _, err := downloader.Download(context.TODO(), largeFile, &s3.GetObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
    })
    if err != nil {
        return fmt.Errorf("error downloading object %q from bucket %q: %w", objectKey, bucketName, err)
    }
    log.Printf("File %q downloaded successfully.\n", largeFile_dest)
    return nil
}

func main() {
    cfg, _ := config.LoadDefaultConfig(context.TODO())

    cmm, _ := materials.NewCryptographicMaterialsManager(materials.NewKmsKeyring(kms.NewFromConfig(cfg), "arn:aws:kms:eu-west-3:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab", func(options *materials.KeyringOptions) {
        options.EnableLegacyWrappingAlgorithms = false
    }))
    s3EncryptionClient, _ := client.New(s3.NewFromConfig(cfg), cmm)

    err := UploadLargeObject(s3EncryptionClient, "cryptai-bucket", "largeFile", "largeFile")
    if err != nil {
        log.Fatal(err)
    }

    err = DownloadLargeObject(s3EncryptionClient, "cryptai-bucket", "largeFile", "largeFile_downloaded")
    if err != nil {
        log.Fatal(err)
    }
}
2 Answers
1

The error message says that the object wasn't found when trying to download it, so it's the multipart upload that had failed, and as the bot mentioned, the Amazon S3 Encryption Client for Go doesn't support multipart uploads (https://docs.aws.amazon.com/amazon-s3-encryption-client/latest/developerguide/go-examples.html). However, the size limit for a single-part upload in S3 is 5 GiB, while your multipart upload chunk size was set to just 10 MiB, so if your files are no larger than 5 GiB, you could consider using basic single-part uploads with the PutObject method of the the encryption client without more extensive changes to your code. It wouldn't perform as well as parallel uploads, but functionally it'd work.

EXPERT
answered 14 days ago
0

The issue you're experiencing with downloading large encrypted files from S3 using client-side encryption is likely due to limitations in the current implementation of the Amazon S3 Encryption Client for Go. Here's an explanation of the problem and some potential solutions:

  1. Multipart uploads and downloads: The Amazon S3 Encryption Client for Go does not currently support multipart uploads or downloads. This is why your code works for small files but fails for large files that require multipart operations.

  2. Encryption metadata: When using client-side encryption, the encryption metadata is stored in the object's metadata. For multipart uploads, this metadata needs to be handled differently, which the current Go client doesn't support.

To work around this limitation, you have a few options:

  1. Use smaller files: If possible, split your large files into smaller chunks that can be uploaded and downloaded without using multipart operations.

  2. Use server-side encryption: If client-side encryption is not strictly necessary, you could switch to server-side encryption (SSE-S3, SSE-KMS, or SSE-C) which is natively supported by S3 and works with multipart uploads and downloads.

  3. Use a different SDK: The Amazon S3 Encryption Client for Java supports multipart uploads and downloads with client-side encryption. If you can switch to Java, this might be a viable option.

  4. Implement custom logic: You could potentially implement your own multipart upload and download logic using the Go SDK, handling the encryption and decryption of individual parts manually. However, this would be complex and require careful implementation to ensure security.

  5. Wait for updates: AWS may update the Go SDK in the future to support multipart operations with client-side encryption. You could check for updates or reach out to AWS support for information on planned improvements.

For now, if you need to stick with Go and client-side encryption, the most straightforward solution would be to work with smaller files that don't require multipart operations. Alternatively, consider if server-side encryption could meet your security requirements, as it would allow you to use the existing multipart upload and download functionality.
Sources
Amazon S3 Encryption Client for Go examples - Amazon S3 Encryption Client
Upload or download large files to and from Amazon S3 using an AWS SDK - AWS SDK Code Examples
Using server-side encryption with Amazon S3 managed keys (SSE-S3) - Amazon Simple Storage Service

profile picture
answered 15 days ago
EXPERT
reviewed 14 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions