aws command installed with awscli library inside a python venv on Windows invokes a python OUTSIDE the venv

0

For awscli 1.25.86, installing on a freshly minted Windows EC2 (Windows Server 2022 Dataserver), I did this:

  1. selected my home directory (cd)
  2. installed pyenv (e.g., via PowerShell using https://github.com/pyenv-win/pyenv-win#power-shell). This said it didn't succeed but seems to have fully installed pyenv. This is not the bug. I had to select a new PowerShell to see the effect of having installed pyenv)
  3. told pyenv to install python 3.8 (pyenv install 3.8.10)
  4. selected python 3.8 globally (pyenv global 3.8.10)
  5. created a virtual environment (pyenv exec python -m venv myvenv)
  6. entered the venv (myvenv\scripts\activate)
  7. installed awscli (pip install awscli)
  8. tried to invoke awscli (aws --version). This gives the message File association not found for extension .py which is an ignorable problem followed by an error that is the bug I'm reporting:
Traceback (most recent call last):
  File "C:\Users\Andrea\GitHub\Submit4DN\s4dn_venv\Scripts\aws.cmd", line 50, in <module>
    import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'

After studying this problem, I believe I know the source of this problem, and am pretty sure it's in the awscli library. The library installs myvenv\scripts\aws.cmd which implements the aws command inside the virtual environment, but that script sniffs around for a python to invoke and finds one outside of the virtual environment. The problem isn't that it tries to get out of the virtual environment, it's just apparently oblivious to the presence of one, and so it isn't picky about which python it finds. it successively seeks python.cmd, python.bat, and python.exe (see line 7 of myenv\scripts\aws.cmd) but finds python.cmd first, and that is not inside the virtual environment. Had it checked python.exe first, it would have found the one in the virtual environment.

If you swap the order of (cmd bat exe) on line 7 of aws.cmd so that it searches (exe bat cmd) it will invoke the python within the virtual env and so will find the awscli that was just installed within the virtual environment. That's not necessarily the right fix. It still feels fragile. But it seems to me that this proves it's the locus of the problem.

Another somewhat workaround is to install awscli outside of the virtual environment by doing deactivate, then pip install awscli, then myvenv/scripts/activate, and then finally trying aws --version and it will work except that if you change to another version of python globally via pyenv, the aws command within the venv will break again unless you again reinstall awscli in each globally selected python.

I don't have a good fix to suggest because I'm not current on writing of Windows shell scripts, but imagine it involves a different way of discovering python that gives strong preference to a venv if one is active, e.g., by noticing there is a %VIRTUAL_ENV% in effect and just invoking python(since virtual envs always have a python) or %VIRTUAL_ENV%\scripts\python if you're wanting to be double-sure.

Note that I was able to reproduce this problem on my professional desktop version of Windows 10 at my home as well, so it's nothing specific to the EC2 itself, that's just a way to show that this problem can be demonstrated in a clean environment. The problem seems pretty definitely in the awscli library.

Whatever solution you pick, I hope this illustrates the issue clearly enough that you can quickly issue some sort of fix to the awscli library because the present situation is just plain broken and this is impacting some instructions we're trying to give some users about how to access our system remotely. I'd rather not be advising users to edit scripts they got from elsewhere, nor do I want to supply alternate scripts for them to use. Things should just work.

1 Answer
1

Hi,

Thank you for your rePost question. My name is Ahmed - an AWS Support Engineer who will be addressing your inquiry today.

I understand you have identified a bug in awscli and would like an update on this.

Investigating this further on your behalf, I too was able to reproduce the issue in my lab and can confirm that this indeed is a bug within awscli.

What I have done on your behalf is liase directly with the AWS CLI backend teams who have taken this on board with the intention of releasing a deployment that does indeed rectify this very bug.

Unfortunately at this stage I am unable to provide an ETA however can assure you that the backend teams are working on this and will provide updates here: https://github.com/aws/aws-cli

Additionally, for future reference you too can liase directly with the backend teams via creating an issue here: https://github.com/aws/aws-cli/issues

I highly appreciate your patience.

AWS
SUPPORT ENGINEER
answered a year ago
  • Thanks very much for working on this. We'll be excited to see a fix when they're able.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions