AWS S3 tar.gz compressed files issue after download from S3

0

Hello,

I would like to check with you guys here in AWS forum the issue that I discovered a while ago. Basically I'm trying to test our backup recovery process. Below is my simple flow of our backing up process.

Backup Flow
ec2 -> uploads backup files compressed using tar.gz to S3 -> S3(backup bucket).
NOTE: I used AWS CLI for moving files from ec2 to s3.

However when I tried to download the tar.gz file from S3, considering that on the AWS Console S3 file list it's has a correct file extension of tar.gz e.g. backup1.tar.gz BUT after the download, I've noticed that my backup file turns to backup1.tar (not tar.gz) so when I tried to decompress the backup its not usable or readable etc.
I'm not sure if it's on my OS level error but overall it will make a huge confusion to the user if the backup process is working fine or not.

asked 6 years ago620 views
6 Answers
0

Why don't you use native AWS tool for creating snapshots? Why you make the process of backup your EC2 instance so complex?)

Moreover you can automate all this by using any backup vendors tools. For example, Cloudberry Backup Server Edition.

AlViKo
answered 6 years ago
0

I'm having the same problems...

I did aws s3 sync some/local/folder s3://my-special-bucket/some/local/folder
and some of the files that got uploaded were .tar.gz files. Well, AWS decided that those should receive the metadata Content-Type: applicaton/x-tar instead of Content-Type: application/x-gzip. Highly aggravating. I had to write a script to list every file in my bucket and then fix the metadata for any file with the .tar.gz extension.

answered 4 years ago
0

I have yet to see anyone explain how to download a tar.gz from an S3 bucket without AWS changing the format to a .tar and changing the config of the files. I need help to figure out how to down load a large (1.7 GB) tar.gz file.

DSimon
answered 3 years ago
0

Hello All,
I have the same issue, anyone has any solution or suggestion?

answered 3 years ago
0

Hi anjalitrambadia, DSimon,

We took a quick look over the sync issues and wanted to get some more information.

If you wouldn't mind creating an issue on the AWS-CLI's Github (https://github.com/aws/aws-cli/), that will help us better track this. Including the version of your aws cli (aws --version) and a brief set up steps for reproduction will be really useful.

Thanks,
AWS-CLI team

Edited by: nateprewitt on May 12, 2020 6:11 PM

answered 3 years ago
0

You should use --content-encoding "application/x-compressed-tar" option during S3 uploading:
aws s3 cp --content-encoding "application/x-compressed-tar".......

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions