AWS S3 tar.gz compressed files issue after download from S3

Question

Hello,  
  
I would like to check with you guys here in AWS forum the issue that I discovered a while ago. Basically I'm trying to test our backup recovery process. Below is my simple flow of our backing up process.  
  
**Backup Flow**  
ec2 -> **uploads backup files compressed using tar.gz to S3** -> S3(backup bucket).  
_NOTE: I used AWS CLI for moving files from ec2 to s3._  
  
However when I tried to download the tar.gz file from S3, considering that on the AWS Console S3 file list it's has a correct file extension of tar.gz e.g. backup1.tar.gz BUT after the download, I've noticed that my backup file turns to backup1.tar (not tar.gz) so when I tried to decompress the backup its not usable or readable etc.  
I'm not sure if it's on my OS level error but overall it will make a huge confusion to the user if the backup process is working fine or not.

Answer

Hi anjalitrambadia, DSimon,  
  
We took a quick look over the sync issues and wanted to get some more information.  
  
If you wouldn't mind creating an issue on the AWS-CLI's Github (https://github.com/aws/aws-cli/), that will help us better track this. Including the version of your aws cli (aws --version) and a brief set up steps for reproduction will be really useful.  
  
Thanks,  
AWS-CLI team  
  
Edited by: nateprewitt on May 12, 2020 6:11 PM

Answer

Hello All,  
I have the same issue, anyone has any solution or suggestion?

Answer

I'm having the same problems...  
  
I did `aws s3 sync some/local/folder s3://my-special-bucket/some/local/folder`  
and some of the files that got uploaded were .tar.gz files. Well, AWS decided that those should receive the metadata `Content-Type: applicaton/x-tar` instead of `Content-Type: application/x-gzip`. Highly aggravating. I had to write  a script to list every file in my bucket and then fix the metadata for any file with the .tar.gz extension.

Answer

I have yet to see anyone explain how to download a tar.gz from an S3 bucket without AWS changing the format to a .tar and changing the config of the files. I need help to figure out how to down load a large (1.7 GB) tar.gz file.

Answer

Why don't you use native AWS tool for creating snapshots? Why you make the process of backup your EC2 instance so complex?)  
  
Moreover you can automate all this by using any backup vendors tools. For example, Cloudberry Backup Server Edition.

Answer

You should use --content-encoding "application/x-compressed-tar" option during S3 uploading:  
aws s3 cp  --content-encoding "application/x-compressed-tar".......

AWS S3 tar.gz compressed files issue after download from S3

相关内容