How to read a gzip file in byte ranges from S3 in .Net

0

I am trying to read gzip file in two stages. The first lambda creates byte ranges and passes it on to second lambda. The second lambda uses the byte ranges and creates GetObjectRequest object and I call GetObjectAsync method of s3Client. I am using SharpZipLib library. Below is my code in .Net where I try to write it in a json stream.

 private static async Task DownloadZippedFileRangeToStreamAsync(
    IAmazonS3 s3Client,
    Stream jsonStream,
    GetObjectRequest objectRequest
)
{
    var getObjectResponse = await s3Client.GetObjectAsync(objectRequest);
    using var memoryStream = new MemoryStream();
    getObjectResponse.ResponseStream.CopyTo(memoryStream);
    memoryStream.Seek(0, SeekOrigin.Begin);
    using var decompressedStream = new GZipInputStream(memoryStream);
    using var streamReader = new StreamReader(decompressedStream);
    using var jsonWriter = new Utf8JsonWriter(jsonStream);
    jsonWriter.WriteStartArray();

    while (!streamReader.EndOfStream)
    {
        // Read each line of the JSONL file
        string jsonLine = streamReader.ReadLine();

        // Parse and write the line as a JSON item
        if(jsonLine != null)
        {
            using JsonDocument doc = JsonDocument.Parse(jsonLine);                
            doc.WriteTo(jsonWriter);
        }
        
    }
    jsonWriter.WriteEndArray();

}


The above code works for reading the file if the byte range starts from 0. It gives an exception "Error GZIP header, first magic byte doesn't match" while reading the gzip file partially in byte ranges specified in the GetObjectRequest model.

I understand there is another object to read gzip file using SelectObjectContentRequest object and specify the CompressionType in inputserialization parameter but it doesnt support scanRange with gzip. Can you please help ?

Santosh
asked 3 months ago200 views
1 Answer
0

Hello,

I understand that you’re facing an issue while reading a GZIP file from S3 in .NET SDK.

When using AWS SDK for .NET there is no direct support for specifying byte ranges in the “GetObjectRequest” class, [1]. This could explain why, you’re receiving "Error GZIP header, first magic byte doesn't match" exception. Similar issue is pointed out here [2].

To read the file in byte ranges try specifying the range of bytes you want to download in your code and add the Range header to the request [3].

=========

[1]https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestSyntax

[2]https://github.com/aws/aws-sdk-java/issues/1551

[3] https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html

AWS
SUPPORT ENGINEER
Zama_R
answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions