Questions tagged with AWS Command Line Interface

Content language: English

Sort by most recent

Browse through the questions and answers listed below or filter and sort to narrow down your results.

aws command installed with awscli library inside a python venv on Windows invokes a python OUTSIDE the venv

For awscli 1.25.86, installing on a freshly minted Windows EC2 (Windows Server 2022 Dataserver), I did this: 1. selected my home directory (`cd`) 2. installed `pyenv` (e.g., via PowerShell using https://github.com/pyenv-win/pyenv-win#power-shell). This said it didn't succeed but seems to have fully installed pyenv. This is not the bug. I had to select a new PowerShell to see the effect of having installed `pyenv`) 3. told `pyenv` to install python 3.8 (`pyenv install 3.8.10`) 4. selected python 3.8 globally (`pyenv global 3.8.10`) 5. created a virtual environment (`pyenv exec python -m venv myvenv`) 6. entered the venv (`myvenv\scripts\activate`) 7. installed `awscli` (`pip install awscli`) 8. tried to invoke `awscli` (`aws --version`). This gives the message `File association not found for extension .py` which is an ignorable problem followed by an error that is the bug I'm reporting: ``` Traceback (most recent call last): File "C:\Users\Andrea\GitHub\Submit4DN\s4dn_venv\Scripts\aws.cmd", line 50, in <module> import awscli.clidriver ModuleNotFoundError: No module named 'awscli' ``` After studying this problem, I believe I know the source of this problem, and am pretty sure it's in the `awscli` library. The library installs `myvenv\scripts\aws.cmd` which implements the `aws` command inside the virtual environment, but that script sniffs around for a `python` to invoke and finds one _outside_ of the virtual environment. The problem isn't that it tries to get out of the virtual environment, it's just apparently oblivious to the presence of one, and so it isn't picky about which python it finds. it successively seeks `python.cmd`, `python.bat`, and `python.exe` (see line 7 of `myenv\scripts\aws.cmd`) but finds `python.cmd` first, and that is not inside the virtual environment. Had it checked `python.exe` first, it would have found the one in the virtual environment. If you swap the order of `(cmd bat exe)` on line 7 of `aws.cmd` so that it searches `(exe bat cmd)` it will invoke the python within the virtual env and so will find the `awscli` that was just installed within the virtual environment. That's not necessarily the right fix. It still feels fragile. But it seems to me that this proves it's the locus of the problem. Another somewhat workaround is to install `awscli` outside of the virtual environment by doing `deactivate`, then `pip install awscli`, then `myvenv/scripts/activate`, and then finally trying `aws --version` and it will work _except_ that if you change to another version of python globally via pyenv, the `aws` command within the venv will break again unless you again reinstall `awscli` in each globally selected python. I don't have a good fix to suggest because I'm not current on writing of Windows shell scripts, but imagine it involves a different way of discovering python that gives strong preference to a venv if one is active, e.g., by noticing there is a `%VIRTUAL_ENV%` in effect and just invoking `python`(since virtual envs always have a `python`) or `%VIRTUAL_ENV%\scripts\python` if you're wanting to be double-sure. Note that I was able to reproduce this problem on my professional desktop version of Windows 10 at my home as well, so it's nothing specific to the EC2 itself, that's just a way to show that this problem can be demonstrated in a clean environment. The problem seems pretty definitely in the `awscli` library. Whatever solution you pick, I hope this illustrates the issue clearly enough that you can quickly issue some sort of fix to the `awscli` library because the present situation is just plain broken and this is impacting some instructions we're trying to give some users about how to access our system remotely. I'd rather not be advising users to edit scripts they got from elsewhere, nor do I want to supply alternate scripts for them to use. Things should just work.
1
answers
0
votes
89
views
profile picture
asked 2 months ago

create-export-task | Filter CloudWatch logs using JMESpath

My objective is to create a mechanism for exporting CloudWatch logs to S3 on a case-by-case basis. Given my logs appear in the following format: ``` { "level": "error", "message": "Oops", "errorCode": "MY_ERROR_CODE_1" } { "level": "info", "message": "All good" } { "level": "info", "message": "Something else" } ``` I'd like the export to **only** include the error logs. Using [create-export-task](https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/create-export-task.html), is it possible to use the `query` param to filter the response data given the above log structure? I'm not sure whether the log structure is incorrect for this use or if I have misunderstood the purpose of the query param. My JMESPath attempts so far have been unsuccessful. Some attempts include: ``` aws logs create-export-task \ --log-group-name myGroup \ --log-stream-name-prefix myGroup-test \ --from 1664537580000 \ --to 1664537640000 \ --destination myGroup-archive-ab1 \ --destination-prefix test \ --query '{Message: message, Error: errorCode}' ``` and same command, but with the following query `--query '{Message: .message, Error: .errorCode}'` which produces the following error: *Bad value for --query {Message: .message, Error: .errorCode}: invalid token: Parse error at column 10, token "." (DOT), for expression: "{Message: .message, Error: .errorCode}"*
0
answers
0
votes
26
views
asked 2 months ago

using transformers module with sagemaker studio project: ModuleNotFoundError: No module named 'transformers'

So as mentioned in my [other recent post](https://repost.aws/questions/QUAL9Vn9abQ6KKCs2ASwwmzg/adjusting-sagemaker-xgboost-project-to-tensorflow-or-even-just-different-folder-name), I'm trying to modify the sagemaker example abalone xgboost template to use tensorfow. My current problem is that running the pipeline I get a failure and in the logs I see: ``` ModuleNotFoundError: No module named 'transformers' ``` NOTE: I am importing 'transformers' in `preprocess.py` not in `pipeline.py` Now I have 'transformers' listed in various places as a dependency including: * `setup.py` - `required_packages = ["sagemaker==2.93.0", "sklearn", "transformers", "openpyxl"]` * `pipelines.egg-info/requires.txt` - `transformers` (auto-generated from setup.py?) but so I'm keen to understand, how can I ensure that additional dependencies are available in the pipline itself? Many thanks in advance ------------ ------------ ------------ ADDITIONAL DETAILS ON HOW I ENCOUNTERED THE ERROR From one particular notebook (see [previous post](https://repost.aws/questions/QUAL9Vn9abQ6KKCs2ASwwmzg/adjusting-sagemaker-xgboost-project-to-tensorflow-or-even-just-different-folder-name) for more details) I have succesfully constructed the new topic/tensorflow pipeline and run the following steps: ``` pipeline.upsert(role_arn=role) execution = pipeline.start() execution.describe() ``` the `describe()` method gives this output: ``` {'PipelineArn': 'arn:aws:sagemaker:eu-west-1:398371982844:pipeline/topicpipeline-example', 'PipelineExecutionArn': 'arn:aws:sagemaker:eu-west-1:398371982844:pipeline/topicpipeline-example/execution/0aiczulkjoaw', 'PipelineExecutionDisplayName': 'execution-1664394415255', 'PipelineExecutionStatus': 'Executing', 'PipelineExperimentConfig': {'ExperimentName': 'topicpipeline-example', 'TrialName': '0aiczulkjoaw'}, 'CreationTime': datetime.datetime(2022, 9, 28, 19, 46, 55, 147000, tzinfo=tzlocal()), 'LastModifiedTime': datetime.datetime(2022, 9, 28, 19, 46, 55, 147000, tzinfo=tzlocal()), 'CreatedBy': {'UserProfileArn': 'arn:aws:sagemaker:eu-west-1:398371982844:user-profile/d-5qgy6ubxlbdq/sjoseph-reg-genome-com-273', 'UserProfileName': 'sjoseph-reg-genome-com-273', 'DomainId': 'd-5qgy6ubxlbdq'}, 'LastModifiedBy': {'UserProfileArn': 'arn:aws:sagemaker:eu-west-1:398371982844:user-profile/d-5qgy6ubxlbdq/sjoseph-reg-genome-com-273', 'UserProfileName': 'sjoseph-reg-genome-com-273', 'DomainId': 'd-5qgy6ubxlbdq'}, 'ResponseMetadata': {'RequestId': 'f949d6f4-1865-4a01-b7a2-a96c42304071', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': 'f949d6f4-1865-4a01-b7a2-a96c42304071', 'content-type': 'application/x-amz-json-1.1', 'content-length': '882', 'date': 'Wed, 28 Sep 2022 19:47:02 GMT'}, 'RetryAttempts': 0}} ``` Waiting for the execution I get: ``` --------------------------------------------------------------------------- WaiterError Traceback (most recent call last) <ipython-input-14-72be0c8b7085> in <module> ----> 1 execution.wait() /opt/conda/lib/python3.7/site-packages/sagemaker/workflow/pipeline.py in wait(self, delay, max_attempts) 581 waiter_id, model, self.sagemaker_session.sagemaker_client 582 ) --> 583 waiter.wait(PipelineExecutionArn=self.arn) 584 585 /opt/conda/lib/python3.7/site-packages/botocore/waiter.py in wait(self, **kwargs) 53 # method. 54 def wait(self, **kwargs): ---> 55 Waiter.wait(self, **kwargs) 56 57 wait.__doc__ = WaiterDocstring( /opt/conda/lib/python3.7/site-packages/botocore/waiter.py in wait(self, **kwargs) 376 name=self.name, 377 reason=reason, --> 378 last_response=response, 379 ) 380 if num_attempts >= max_attempts: WaiterError: Waiter PipelineExecutionComplete failed: Waiter encountered a terminal failure state: For expression "PipelineExecutionStatus" we matched expected path: "Failed" ``` Which I assume is corresponding to the failure I see in the logs: ![buildl pipeline error message on preprocessing step](/media/postImages/original/IMMpF6LeI6TgWxp20TnPZbUw) I did also run `python setup.py build` to ensure my build directory was up to date ... here's the terminal output of that command: ``` sagemaker-user@studio$ python setup.py build /opt/conda/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead warnings.warn( /opt/conda/lib/python3.9/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead. warnings.warn(msg, warning_class) running build running build_py copying pipelines/topic/pipeline.py -> build/lib/pipelines/topic running egg_info writing pipelines.egg-info/PKG-INFO writing dependency_links to pipelines.egg-info/dependency_links.txt writing entry points to pipelines.egg-info/entry_points.txt writing requirements to pipelines.egg-info/requires.txt writing top-level names to pipelines.egg-info/top_level.txt reading manifest file 'pipelines.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'pipelines.egg-info/SOURCES.txt' ``` It seems like the dependencies are being written to `pipelines.egg-info/requires.txt` but are these not being picked up by the pipeline?
1
answers
0
votes
80
views
asked 2 months ago