Directory Error when running SageMaker backup-ebs lifecycle for Amazon Linux 2 transition


I'm following this guide for transitioning to Amazon Linux 2 provided by AWS

I've set up the two needed lifecycle configurations and created a new S3 Bucket to store the backup. I've also ensured the IAM roles have the required S3 permissions and updated the notebook with the ebs-backup-bucket tag per the instructions.

When I run the notebook with the new configuration I get the following error: "Notebook Instance Lifecycle Config [LIFECYCLE ARN] for Notebook Instance [NOTEBOOK ARN] took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.

Looking at the logs I get the error: /bin/bash: /tmp/OnStart_2022-11-09-01-51ontlqcqt: /bin/bash^M: bad interpreter: No such file or directory

Any thoughts on how to resolve this issue? The code for the backup lifecycle configuration can be found here

1 Answer
Accepted Answer

The extra ^M symbol (i.e. Ctrl-M) stopped the whole scrip from being interpreted properly.

This issue is normally seen in scripts prepared in MSDOS/Windows based system but used in Linux system due to difference of line endings.

In Unix based OS, lines end with \n but MSDOS/Win based system ends with \r\n

In Linux based system, you could show your prepared scripts by running

cat -e 

The results would be something similar to

... ...^M$

$ is normal Unix end-of-line symbol. Windows uses an extra one ^M and this symbol is not recognized by Unix system. That's why, in SageMaker Notebook Lifecycle Configuration, which is running Linux, your script was interpreted as /bin/bash^M

To mitigate the issue, please convert the scripts to Unix based ending and update life cycle configuration. To achieve this, you could use Notepad++ in Windows. You can go to the Edit menu, select the EOL Conversion submenu, and from the options that come up select UNIX/OSX Format. The next time you save the file, its line endings will, all going well, be saved with UNIX-style line endings.

Alternatively, you could put the script in a Linux environment, e.g. EC2 instance with Amazon Linux 2 and install dos2unix via sudo yum install dos2unix. After installation, you could convert your files via

dos2unix -n 

After the conversion, please update LCC with the new scripts. You could verify that ^M has been removed via

cat -e

The output will print all special characters directly without hiding.

answered 2 years ago
  • I am a mortal among gods. Reformatting to Linux/Unix using Notepad++ worked perfectly. Did try it in VScode first but didn't have the same impact. Good to know about the conversion of shell scripts from MS to Unix. Thanks again for the help

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions