Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How to execute parallel dags in Airflow in EC2 Instance

0

I have Installed Airflow as a service in EC2 Instance. It is working fine with Sequential execution. But While I am trying with Parallel execution I am facing Issues. It is not running parallel dags execution. I have installed PostgreSQL inside EC2 instance.

Step1: I have installed Airflow. Step2: Installed the Postgresql Step3: In airflow.cfg, modified to sequential Executor to Local Executor. Step4: Restarted the Airflow and scheduler.

Topics
Application IntegrationCompute
Tags
Amazon Managed Workflows for Apache Airflow (MWAA)Amazon EC2
Language
English
asked 4 years ago299 views
1 Answer
0

If you want parallel dags to run in EC2, you need to use another executor. Sequential executor will only allow single dag to run. You can use celery executor which is used for production level tasks. You can use Local executor also if you are just testing things out.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content