Optimizing Python Script Execution on ECS with Apache Airflow: Cost-effective Strategies

"I have multiple Python scripts stored within a Docker image, each with its set of input parameters. My goal is to execute these scripts on Amazon Elastic Container Service (ECS) as required. Inside the Docker image,

I have the following scripts:

• ex-tract_satellite_data.py

• calculates_ndvi.py

These scripts rely on various input parame-ters like 'sensor' (e.g., 'sentinel2') and 'config_file' (a YAML file with start and end dates, etc.).

Currently, we're using ECS and EFS to run these Python scripts. The scripts are executed using ECS tasks, and the required files are mounted in EFS. To further streamline this process, we're considering automating the ECS task runs using Apache Airflow. However, we're facing cost concerns with AWS Managed Apache Airflow (MWAA), as it lacks the option to pause when not in use.

Considering this, we're exploring the most cost-effective way to manage this workflow. We're contemplating whether to set up our own Apache Airflow instance on an EC2 instance or continue with AWS Managed Apache Airflow. Are there any alternative solutions or best practices for managing this process efficiently and economically?

Your insights and recommendations would be greatly appreciated."