Dotnet SDK TransferUtility doesn't calculate PartSize properly

Question

I am trying to ensure that file downloaded from s3 matches etag.

When uploading file, I am using TransferUtility with specified PageSize (16 * 1024* 1024). 
Once downloaded, I am calculating MD5 for first 16MB, store it into byte list, than calculate MD5 for remaining part and adds to same list.

Whenever I am using MemoryStream as a source everything works perfrectly and etag matches.
However, once I am using CryptoStream to encrypt content on the fly it is not. Debug information shows that Transfer utility incorrectly processes PartSize and adds extra 8k to portion.

Here is an excerpt from log:

As you can see, 16785408 is not equal to 16777216. It is exactly 8K more. Apparently, if on downloading I split file into 16MB chunks to calculate MD5 I do not have a match.

Some code: 
```
// creating a stream which encrypt Stream named input 
Aes aesAlg = Aes.Create();
aesAlg.GenerateKey();
aesAlg.GenerateIV();

ICryptoTransform encryptor = aesAlg.CreateEncryptor(aesAlg.Key, aesAlg.IV);
CryptoStream cryptoStream = new CryptoStream(input, encryptor, CryptoStreamMode.Read);
// now it is possible to read from this stream in a way that content is encrypted on the fly, without putting whole file and encrypted data into a memory

using var transferUtility = new TransferUtility(_s3Client);
var transferRequest = new TransferUtilityUploadRequest()
{
    BucketName = _bucket,
    Key = key,
    InputStream = cryptoStream ,
    ContentType = "application/x-binary",
    CannedACL = S3CannedACL.Private,
    PartSize = 16 * 1024 * 1024,
};
 await transferUtility.UploadAsync(transferRequest, cancellationToken);
```
For now, my workaround is to use PartSize 16MB + 8K whenever I calculate ETag, but I hope you can fix it.

Answer

The issue you're encountering seems to be related to how TransferUtility processes the part size when uploading a stream with encryption using a CryptoStream. TransferUtility may be adding extra padding to the part size, causing a mismatch in the uploaded file's ETag.

To address this issue, you can try the following workaround:

Instead of relying on TransferUtility to calculate the part size, manually specify the part size in the TransferUtilityUploadRequest to ensure consistency. You can set the part size to 16MB + 8KB to match the behavior observed during the upload.

Here's how you can modify your code to implement this workaround:

```

using System.IO;
using System.Security.Cryptography;
using Amazon.S3;
using Amazon.S3.Transfer;

// Create an Aes object for encryption
Aes aesAlg = Aes.Create();
aesAlg.GenerateKey();
aesAlg.GenerateIV();

// Create an ICryptoTransform object for encryption
ICryptoTransform encryptor = aesAlg.CreateEncryptor(aesAlg.Key, aesAlg.IV);

// Create a CryptoStream for encryption
CryptoStream cryptoStream = new CryptoStream(input, encryptor, CryptoStreamMode.Read);

// Create an instance of AmazonS3Client
IAmazonS3 s3Client = new AmazonS3Client();

// Create a TransferUtility object with the AmazonS3Client
using var transferUtility = new TransferUtility(s3Client);

// Create a TransferUtilityUploadRequest object
var transferRequest = new TransferUtilityUploadRequest()
{
    BucketName = _bucket,
    Key = key,
    InputStream = cryptoStream,
    ContentType = "application/x-binary",
    CannedACL = S3CannedACL.Private,
    PartSize = (16 * 1024 * 1024) + (8 * 1024), // Set part size to 16MB + 8KB
};

// Upload the encrypted stream using TransferUtility
await transferUtility.UploadAsync(transferRequest, cancellationToken);
```

Dotnet SDK TransferUtility doesn't calculate PartSize properly

관련 콘텐츠