From your above description, I understand that you are facing issue while trying to connect from your MWAA environment to Redshift cluster and you are using DAG to transfer the data from S3 to Redshift table. Very first, you can troubleshoot why logs are not visible because once logs are visible it would help to understand the issue. Please verify that:
You have enabled task logs at the INFO level for your environment. For more information, see Viewing Airflow logs in Amazon CloudWatch.
Verify that the environment execution role has the correct permission policies. https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-create-role.html
Also check if you are unable to view the logs of any of the others DAG or you are facing this issue with this DAG only. You can also refer the below AWS documentation to troubleshoot: https://docs.aws.amazon.com/mwaa/latest/userguide/t-cloudwatch-cloudtrail-logs.html#t-task-logs
I am sharing the below sample operator example for your reference:
transfer_s3_to_redshift = S3ToRedshiftOperator( task_id="transfer_s3_to_redshift", redshift_conn_id=conn_id_name, s3_bucket=bucket_name, s3_key=S3_KEY_2, schema="PUBLIC", table=REDSHIFT_TABLE, copy_options=["csv"], )
Additionally you can refer this third party documents
I hope you would find above information helpful.
- How do I access the Apache Airflow UI using the private network access mode in my Amazon MWAA environment?AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
- EXPERTpublished a year ago