User Data script not downloading file(s) from S3


I have been trying for days to get a User Data script for a Windows instance to copy files from S3. At first I was trying to use the 'aws s3 sync' command to copy a large number of files, but since that wouldn't work I zipped the files and I'm trying to now copy just that one zipped file. I am trying to perform the copy with both a script command and powershell command. Since other commands work in both blocks I know the user data script is formatted correctly and executing at launch, but this one file copy command using the AWS CLI is simply not working.

It is also worth noting that I am downloading and installing the AWS CLI first, and that the download/install works from either the script block of powershell block. I am also associating an IAM role with sufficient permissions at launch time via an instance profile. And I know both work (CLI and role) as I am able to manually execute the copy command soon as I logon to the instance, it's just not performing the copy from the user data script.

Here's the (cleansed) script block I'm running (formatting is fouled up due to <script>):

<script> c:\windows\system32\msiexec.exe /i /qn aws s3 cp s3://<bucket>/<path/key>/ C:\Temp\ </script>

Here's the (cleansed) powershell block I'm running (formatting is fouled up due to <powershell>):

<powershell> aws s3 cp s3://<bucket>/<path/key>/ C:\Temp\ </powershell>

Obviously it's the exact same CLI command structured the same for both script and powershell, but again I cannot get it to execute the file copy operation from either block. However, the installation of the AWS CLI works fine from either block.

After pulling my hair out for days on this trying to figure it out, searching the Internet and AWS documentation, and not finding any possible solutions I'm posting here to try to get this figured out. Thank you in advance for any assistance.

More info... I can see in the err.tmp file (for the batch file <script> block that, "'#aws' is not recognized as an internal or external command, operable program or batch file."

And in the <powershell> err.tmp file, "aws : The term 'aws' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again."

So it appears that the AWS CLI is not installed by the time I am attempting to copy files from S3 to the instance, despite the fact that the AWS CLI is being installed as the first line in the batch portion of the User Data script, and I've even added "timeout 90" right after the install AWS CLI line in an attempt to pause for 1.5 minutes to give plenty of time to install the CLI prior to the first attempt at copying the file (in the batch block), then the second attempt to copy the file is in the 27th line of the script in the powershell portion, long after several other commands complete successfully.

Again, thank you in advance for any assistance with this issue.

asked 2 years ago948 views
1 Answer

After the installation of AWS CLI, PowerShell or Command prompt is not recognizing the newly installed aws cli setup. I would recommend you to execute AWS CLI commands using its installed folder path as below:

"C:\Program Files\Amazon\AWSCLIV2\aws.exe" s3 cp s3://<bucket>/<path/key>/ C:\Temp\

answered 2 years ago
  • FYI I have made some progress on this.... Not directly in the User Data script, but I have created an AMI using EC2 Image Builder and used User Data there to install the AWS CLI. Then when I launch an instance from the AMI created in Image Builder which has the AWS CLI installed I am able to download files from my S3 bucket.

    However, I'd still like to figure out how to get this to work as per my original post...

  • Ajay, thank you for the answer. However - I neglected to post this in my original post - I did try to use the absolute path to aws.exe and that did not work for me either.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions