Is it possible to set up a local AWS Batch jobs endpoint for AWS Step functions emulator

Question

Hello

I'm in the process of building a custom data pipeline using AWS Step-functions which would call different aws batches. I would like to run all of this locally in order to iterate fast and make sure each step of the process is working correctly, setting up some unit tests etc...

I came across the docker image to run step functions locally and set that up https://docs.aws.amazon.com/step-functions/latest/dg/sfn-local-config-options.html

The problem comes in the configuration file where they ask for a AWS Batch endpoint, I want to pass a local endpoint with ideally a local AWS Batch emulator that I could run on my computer, this AWS Batch emulator would run some local docker images that I've built and would then be the same docker images uploaded for the production jobs

Is this possible ? If so let me know where I can find some resources on how to do it

Answer

AWS does not provide an official local emulator for AWS Batch. However, you can set up a mock environment to test your Step Functions locally with the following approach:

- you can use LocalStack https://github.com/localstack/localstack to emulate various AWS services locally, including Step Functions, Lambda, and more. LocalStack does not support AWS Batch, but it can help you create a mock environment for other services involved in your data pipeline.
- implement a mock AWS Batch service using AWS Lambda or a simple web server (like Flask in Python or Express in Node.js). This mock service should accept the same API calls as AWS Batch, and instead of running actual batch jobs, it would execute the local Docker images you've built.

Is it possible to set up a local AWS Batch jobs endpoint for AWS Step functions emulator

관련 콘텐츠