By using AWS re:Post, you agree to the Terms of Use

OS in Batch compute environment suddenly changed?

0

Hi,

I have multiple AWS accounts in which I use AWS Batch to run jobs. All Batch compute environments were created with the default EC2 AMI: ECS_AL1 (Amazon Linux). Here's part of the output from aws batch describe-compute-environments for one of my accounts:

        "type": "MANAGED",  
        "state": "ENABLED",  
        "status": "VALID",  
        "statusReason": "ComputeEnvironment Healthy",  
        "computeResources": {  
            "type": "EC2",  
            "minvCpus": 0,  
            "maxvCpus": 64,  
            "desiredvCpus": 0,  
            "instanceTypes": \[  
                "c4.large",  
                "c4.xlarge",  
                "c4.2xlarge",  
                "c4.4xlarge",  
                "c4.8xlarge"  
            ],  
            "subnets": \[  
                "...",  
                "..."  
            ],  
            "securityGroupIds": \[  
                "..."  
            ],  
            "instanceRole": "arn:aws:iam::...",  
            "tags": {},  
            "ec2Configuration": \[  
                {  
                    "imageType": "ECS_AL1"  
                }  
            ]  

In the past I could use yum, as you do in Amazon Linux, inside Batch jobs to install dependencies for my workloads.
Yesterday I noticed that yum doesn't work anymore. I also noticed that the OS in my compute environments changed! Now it's Debian. Here's the output of cat /etc/os-release which I ran inside one of my Batch jobs:

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Why did the OS inside my Batch jobs change and how can I make it Amazon Linux again?

For more context, I don't start Batch jobs directly. Metaflow (https://metaflow.org/) in a SageMaker Notebook instance does that for me.

asked 2 years ago23 views
1 Answer
0

Turns out Metaflow was the culprit. Metaflow's config has the following key which specifies the container image: METAFLOW_BATCH_CONTAINER_IMAGE.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions