What service should I use to run a python script stored in a bucket in AWS S3

if you are going to run file stored in S3 bucket, you need either download it in advance and execute it or you can mount S3 as a file system using tool like https://github.com/s3fs-fuse/s3fs-fuse

S3 is primarily for storage. You can define your script as AWS Lambda Function to perform necessary actions on objects in S3. (assuming a single iteration of your script does not run for 15 min ). Another option for long running jobs would be to run the script in an EC2 instance, or evaluate the usefulness of Step Functions / AWS Batch

Hi, could you review the following link where the documentation guides you in detail (Running scripts from Amazon S3): https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-s3.html

The article details the forms of execution, you can choose between: a) Run a shell script from Amazon S3 (console): or b) Run a shell script from Amazon S3 (command line)

I hope it helps you.

Just for people looking for "an" answer sometime in the future, this is how I solved it:

Documentation: SSM Automation Runbook with S3 Script Execution

Overview

This SSM Automation runbook (Template_Test.ssm.yml) demonstrates a method to execute a Python script stored in an S3 bucket (s3://your-s3-bucket/SSM/test.py) by downloading and running it dynamically within an inline script. The runbook takes a Message parameter, passes it to the script, and outputs the processed result. It relies on an IAM role (SSM_Role) to provide the necessary permissions for S3 access.

Runbook Structure

description: A simple SSM runbook that calls a script from S3 to print and output a message.
schemaVersion: '0.3'
parameters:
  Message:
    type: String
    description: The message to print and output.
    default: Hello from the runbook!
assumeRole: arn:aws-us-gov(Your partition):iam::000000000(Account Number):role/SSM_Role
mainSteps:
  - name: ExecuteS3Script
    action: aws:executeScript
    isEnd: true
    inputs:
      Runtime: python3.10
      Handler: main
      InputPayload:
        Message: '{{ Message }}'
      Script: |
        import boto3
        import traceback

        def main(events, context):
            try:
                s3 = boto3.client('s3')
                bucket = 'your-s3-bucket'
                key = 'SSM/test.py'
                response = s3.get_object(Bucket=bucket, Key=key)
                script_content = response['Body'].read().decode('utf-8')
                namespace = {}
                exec(script_content, namespace)
                if 'process_message' not in namespace:
                    raise AttributeError("process_message function not found in the script")
                result = namespace['process_message'](events)
                return result
            except Exception as e:
                print(f"Error: {str(e)}")
                print(traceback.format_exc())
                raise
    outputs:
      - Name: OutputMessage
        Selector: $.Payload.OutputMessage
        Type: String

Key Components

1. Schema and Parameters:

   • schemaVersion: '0.3': Specifies the SSM document schema version, supporting advanced features like aws:executeScript.
   • parameters.Message: Accepts a string input, defaulting to "Hello from the runbook!", which is passed to the script.

2. assumeRole:

   • assumeRole: arn:aws-us-gov:iam::000000000000:role/SSM_Role: Defines the IAM role that SSM assumes during execution. This role provides the credentials needed for AWS API calls (e.g., S3 access) within the script.

3. Main Step: ExecuteS3Script:

   • action: aws:executeScript: Executes a Python script inline within SSM.
   • Runtime: python3.10: Specifies the Python 3.10 runtime environment.
   • Handler: main: Indicates the main function as the entry point in the inline script.
   • InputPayload.Message: '{{ Message }}': Passes the Message parameter to the script as part of the events dictionary.
   • Script: The inline Python code that:
     • Uses boto3 to download test.py from S3.
     • Executes the downloaded script in a separate namespace.
     • Calls process_message from the downloaded script and returns its result.
   • outputs: Captures the OutputMessage from the script’s return value.

How It Works

1. Execution Initiation:

   • You run aws ssm start-automation-execution --document-name "Template_Test" --parameters Message="Test Message".
   • SSM launches an automation execution in an AWS-managed environment.

2. Role Assumption:

   • SSM assumes the SSM_Role role specified in assumeRole.
   • The trust policy (below) allows ssm.amazonaws.com to assume this role, providing temporary credentials for the execution.

3. Inline Script Execution:

   • The main(events, context) function runs:
     • events contains {"Message": "Test Message"}.
     • context provides runtime metadata (unused here).
   • boto3.client('s3') uses the role’s credentials to download test.py from s3://your-s3-bucket/SSM/test.py.
   • exec(script_content, namespace) executes the downloaded script, making its functions (e.g., process_message) available in namespace.
   • namespace['process_message'](events) processes the input and returns a result (e.g., {"OutputMessage": "Test Message"}).

4. Output Handling:

   • The return value from main is captured by SSM.
   • Selector: $.Payload.OutputMessage extracts "Test Message" as OutputMessage.

Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ssm.amazonaws.com"
        ],
        "AWS": [
          "arn:aws-us-gov(Your partition):iam::000000000(Account Number):role/SSM_Role"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Why the Trust Policy Is Needed

• Purpose: Defines who or what can assume the SSM_Role role via sts:AssumeRole.
• Principals:
  • Service Principals:
    • "ssm.amazonaws.com": Critical here—allows SSM Automation to assume the role during execution, providing credentials for S3 access.
  • AWS Principals: Specific roles that can also assume SSM_Role (e.g., for cross-account or delegated access).
• Action: sts:AssumeRole enables these principals to obtain temporary credentials.
• SSM Context: Without ssm.amazonaws.com, SSM couldn’t assume the role, leading to the NoCredentialsError you saw earlier. The assumeRole field links this policy to the execution.

S3 Script (test.py)

The script in S3 must define process_message to handle the events payload:

def process_message(events):
    message = events.get('Message', 'No message provided')
    print(f"Received: {message}")
    return {"OutputMessage": message}

• Note: This script doesn’t need main(events, context) since the inline script calls process_message directly. It’s simpler and avoids nesting execution issues.

Permissions

The SSM_Role role must have an attached policy allowing S3 access:

{
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::your-s3-bucket/SSM/test.py"
}

How It All Ties Together

1. Runbook Execution:

   • SSM starts the automation, assuming the SSM_Role role via the trust policy.
   • The inline script (main) executes with the role’s credentials.

2. S3 Access:

   • boto3.client('s3') uses the role’s permissions to download test.py.
   • The trust policy ensures these credentials are available.

3. Dynamic Execution:

   • exec runs the downloaded script, making process_message available.
   • The inline script calls it and returns the result, which SSM captures as output.

Troubleshooting Tips

• Credentials Failure: If NoCredentialsError reappears, verify the role ARN (arn:aws-us-gov:iam::000000000000:role/SSM_Role) and trust policy are correct.
• S3 Access Denied: Ensure the role’s policy includes the s3:GetObject permission for the exact bucket and key.
• Function Not Found: If process_message isn’t found, double-check test.py in S3 matches the expected content.

Supporting AWS Documentation

• SSM Automation Overview:
  • Explains how SSM Automation executes workflows and uses IAM roles.
• aws:executeScript Action:
  • Details the Handler, Script, and InputPayload fields. Confirms Python handlers must accept events and context.
• assumeRole Field:
  • Describes how assumeRole specifies the IAM role for execution, assumed by SSM via sts:AssumeRole.
• IAM Trust Relationships:
  • Explains how to configure a trust policy for service principals like ssm.amazonaws.com.
• Boto3 Credentials:
  • Notes that boto3 uses the execution role’s credentials in AWS-managed environments like SSM Automation.