AWS S3 Get object not working. Getting corrupt file

1

I have a .zip file on an S3 bucket. When I try to get it and save it as a file it is corrupt and can't be opened. I don't know what to do, is really hard to get that S3 zip file. I am using this code as a guide https://docs.aws.amazon.com/AmazonS3/latest/userguide/download-objects.html. I'm currently on Unity, but the SDK for .NET should work. The resulted file has the first 7.4 kb all NULL. This is the code that gets the zip file, returns the buffer, and saves it with File.WriteAllBytes:

/// <summary>
    /// Get Object from S3 Bucket
    /// </summary>
    public async Task<byte[]> GetZip(string pFile)
    {
        try
        {
            Debug.Log("KEY: " + pFile);
            GetObjectRequest request = new GetObjectRequest
            {
                BucketName = S3Bucket,
                Key = pFile
            };
            using (GetObjectResponse response = await S3Client.GetObjectAsync(request))
            using (Stream responseStream = response.ResponseStream)
            {
                Debug.Log("Response stream");
                if (responseStream != null)
                {
                    byte[] buffer = new byte[(int)response.ResponseStream.Length];
                    int result = await responseStream.ReadAsync(buffer, 0, (int)response.ResponseStream.Length);
                    Debug.Log("Stream result: " + result);
                    File.WriteAllBytes(songPath, buffer);
                    Debug.Log("Readed all bytes: " + buffer.Length + " - " + ((int)response.ResponseStream.Length));
                    return buffer;
                }
                else
                {
                    Debug.Log("Response is null");
                }
            }
            
        }
        catch (AmazonS3Exception e)
        {
            // If bucket or object does not exist
            Debug.Log("Error encountered ***. Message:"+ e.Message + " when reading object");
        }
        catch (Exception e)
        {
            Debug.Log("Unknown encountered on server. Message:"+ e.Message + " when reading object");
        }
        return null;
    }
asked 2 years ago1221 views
3 Answers
1

C# Directly supports writing the stream to a file:

using (var response = await S3Client.GetObjectAsync(request))
using (var responseStream = response.ResponseStream)
using (var outputFileStream = new FileStream(songPath, FileMode.Create))
{
    await responseStream.CopyToAsync(outputFileStream);
}
wparad
answered 2 years ago
  • It worked but I had to use this:

    byte[] buffer = new byte[(int)response.ContentLength]; using (var outputFileStream = new MemoryStream(buffer)) { await responseStream.CopyToAsync(outputFileStream) } File.WriteAllBytes(CustomSongsManager.CustomFolder + pFile, buffer);

    Thanks

0

Ok. Fiddled with your code a bit. And got NoSupported errors on responseStream.Length.

I changed to code to using response.ContentLength and it started working for me.

This is the code I ended up with:

async Task<byte[]> GetZip(string pFile)
{
    try
    {
        Console.WriteLine("KEY: " + pFile);
        Amazon.S3.Model.GetObjectRequest request = new Amazon.S3.Model.GetObjectRequest
        {
            BucketName = S3Bucket,
            Key = pFile
        };
        using (Amazon.S3.Model.GetObjectResponse response = await S3Client.GetObjectAsync(request))
        using (Stream responseStream = response.ResponseStream)
        {
            Console.WriteLine("Response stream");
            if (responseStream != null)
            {
                byte[] buffer = new byte[(int)response.ContentLength];
                int result = await responseStream.ReadAsync(buffer, 0, (int)response.ContentLength);
                Console.WriteLine("Stream result: " + result);
                File.WriteAllBytes(songPath, buffer);
                Console.WriteLine("Readed all bytes: " + buffer.Length + " - " + ((int)response.ContentLength));
                return buffer;
            }
            else
            {
                Console.WriteLine("Response is null");
            }
        }
    }
    catch (Amazon.S3.AmazonS3Exception e)
    {
        // If bucket or object does not exist
        Console.WriteLine("Error encountered ***. Message:"+ e.Message + " when reading object");
    }
    catch (Exception e)
    {
        Console.WriteLine("Unknown encountered on server. Message:"+ e.Message + " when reading object");
    }
    return new byte[0];
}

GetZip(pFile).Wait();
profile picture
JaccoPK
answered 2 years ago
  • I tried it but stills returns a corrupted zip file. I'm using the same sdk but in Unity. The zip file I get has the first 7.4 kb all at NULL.

  • With these versions?

      <ItemGroup>
        <PackageReference Include="AWSSDK.Core" Version="3.7.11.13" />
        <PackageReference Include="AWSSDK.S3" Version="3.7.9.13" />
      </ItemGroup>
  • On that page I still can find many different versions.

    aws-sdk-netstandard2.0.zip
    
    aws-sdk-net45.zip
    
    aws-sdk-net35.zip
    
  • I am not sure. I think you have the right one. Not sure though if I can help any further on my Mac :-)

  • In Unity I just imported the .dll files. I downloaded the files from here: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/unity-special.html. My version for core is 3.3.0.0 and for S3 is also 3.3.0.0. I downloaded the aws-sdk-netstandard2.0.zip. I'm going to try downloading the aws-sdk-net45.zip instead and see if it works

0

Is the file truly corrupt or has it been unzipped and you didn't expect it to be? From memory the Content Encoding metadata on the object in S3 will influence whether a zip file is transparently unzipped on the fly.

EXPERT
answered 2 years ago
  • In the bucket, the file is a zip file. I didn't know it could be unzipped on the fly. If that's the case how can I save the folder with its content? Or how can I change it so it downloads as a zip file? Please can you be more specific, the content metadata of the file in the bucket is application/zip

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions