Line magic error in SageMaker Studio

0

Hi AWS, I am trying to create virtual environment in SageMaker Studio but while doing so I am experiencing a line magic function error. Also I am not able to import the libraries. I am attaching the error screenshot for the same.

Error Screenshot

Thanks

  • Hey, someone from the SageMaker community how to resolve this issue?

profile picture
asked 2 years ago411 views
2 Answers
0

As far as I can tell, %virtualenv and %import are not standard line magics in IPython - and from a quick search around, I couldn't see what package implements them?

But I think you might want to consider more fundamental changes to your workflow and will try to make a case for it...

In SageMaker Studio, as the other answer noted already, kernel environments are containerized - rather than conda-based. This is nicely consistent with how SageMaker job environments (training, deployment, processing, etc) are also container-based.

As a result, my usual environment management suggestions would be:

  • For quickly and interactively experimenting with different library installations in the notebook, simply use !pip shell commands to install/edit packages directly in the running container. (e.g. !pip install abc==x.y.z, !pip show abc, and so on). You could even save these dependencies in a requirements.txt and !pip install from that file (which would be nice because SageMaker script-mode jobs can accept a requirements.txt).
  • For formalizing an environment to be standard, repeatable, and shareable between users without them having to re-run the install commands: set up a custom kernel container image.

As outlined here, all your open notebooks with the same kernel and instance type selected will share a running container. When you delete/restart the "app" (container), these changes will be lost. Keeping running/customized kernel environments as disposable as possible, and trying to use SageMaker jobs (training, processing, etc) early in the workflow instead of sticking everything in notebook, helps to prevent reproducibility problems.

Because they're containerized, I find I don't really need to worry about environment management/virtualization technologies: In notebook experimentation, my different kernels are fully separated anyway and I can just restart the container ("app") to reset. In SageMaker jobs there's only one task being run in the container, so it doesn't need to play nice with other workloads. It's different, and actually quite nice, compared to working on my local laptop where I have to carefully manage separate project environments and fix things if they break!


Edit to add: The following works for me... Perhaps your problem was that the code folder didn't exist yet? os.makedirs() or !mkdir or UI actions can fix that:

%%writefile src/main.py

import os
AWS
EXPERT
Alex_T
answered 2 years ago
  • I have created code folder using !mkdir code command and afterthat I have ran that writefile command. I will give it a shot again and let you know.

  • I am still experiencing the same error unfortunately.

  • I think it may be worth raising a separate question (or even a support ticket?) about the %%writefile issue if you're still not able to get that working? Because that one should work, and I know it gets used in some of our official samples. To try and fix it I could maybe suggest to restart your kernel, JupyterServer + KernelGateway apps... And to unblock you could maybe suggest to just create the file through GUI file editor (just create text file and rename to .py, if you don't see a create py file option)... But somebody might be able to help better in a dedicated question for it!

0

Hi,

When running a notebook on SageMaker Studio you can select the default environment you want by clicking on the top right corner on the kernel you currently have enabled (Default should be Data Science) and select the one you want from the dropdown.

In the case you want to install more/custom packages, then you can install those straight from your notebook using one of the valid magic cell commands. These are %conda install and %pip install (see doc: https://docs.aws.amazon.com/sagemaker/latest/dg/studio-notebooks-add-external.html)

Alternatively you can open a terminal and use your tool of choice to manage the python packages installed.

In terms of importing packages, the correct syntax for that is without the %. So please instead of %import <package_name> try import <package_name>.

Let us know if this solved your issue,

have a nice day

AWS
answered 2 years ago
  • Hi, I am still facing the issue while setting up virtual environment in the SageMaker Notebook. If I am running the command virtualenv venv I am getting invalid syntax error and when I ran the command %virtualenv venv the line magic function %virtualenv not found error is coming.

    Please respond as I need to fix it ASAP.

    Thanks

  • When I was trying to create the file using the command %%writefile code/inference.py I am still experiencing the line magic function error.

    Please let me know how to resolve this issue.

    Thanks

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions