- Newest
- Most votes
- Most comments
Hello,
Please follow below steps
-
Once the docker image is pushed to ECR, you will need to make a Docker Registry connection on Airflow UI. Make sure to choose connection type as "Docker Registry" and Username as "AWS". Password is the string that you get when running "aws ecr get-login-password --region us-east-1"
-
Write your DAG, and specify the docker_conn_id with the name you chose above. Make sure your provided docker URL is correct. Also, make sure that the docker host has access from the MWAA Security group.
Below is the sample Dag
from airflow import DAG from datetime import datetime, timedelta import json, csv, requests from airflow.operators.docker_operator import DockerOperator
default_args = { "owner": "xxxxx", "start_date": datetime(2020, 11, 29), "depends_on_past": False, "email_on_failure": False, "email_on_retry": False, "email": "xxxx@xxx.com", "retries": 1, "retry_delay": timedelta(minutes=5) }
with DAG(dag_id="docker-test", schedule_interval="@hourly", default_args=default_args, catchup=False) as dag:
docker_test_task = DockerOperator(
task_id='docker_task',
image='123456789012.dkr.ecr.us-east-1.amazonaws.com/airflow-docker-rep:latest',
execution_timeout=timedelta(minutes=30),
docker_conn_id='docker',
api_version='auto',
docker_url='tcp://ec2-11-222-333-444.compute-1.amazonaws.com:1111'
)
docker_test_task
Hello, Please clarify the "docker host has access from the MWAA Security group". Also, could you give us information about where to get the docker_url : docker_url='tcp://ec2-11-222-333-444.compute-1.amazonaws.com:1111'
Relevant content
- asked a year ago
- asked a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
Can you clarify what you mean by "Also, make sure that the docker host has access from the MWAA Security group." Does that mean that in order to use the DockerOperator in MWAA, I need to be running an additional ec2 instance running docker?