Questions tagged with Amazon Managed Workflows for Apache Airflow (MWAA)

I keep getting the error message `Task exited with return code Negsignal.SIGKILL` even on tasks not requiring much memory or compute on my airflow DAGs. I have already upgraded from small to medium. Want to know if there's some configuration I can change before upgrading to large or considering another option.lg...

Trying to use Airflow deferral task. It needs triggerer service. Wondering how to start this one.lg...

Encountering the following error ``` 2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED Resource handler returned message: "null (Service: Mwaa, Status Code: 403, Request ID: de70d88f-383a-43bb-91a7-4ebdee31223e)" (RequestToken: fe034e51-e5e2-73e9-3616-0bd928d64a1f, HandlerErrorCode: AccessDenied) ``` Not sure which resource access is denied.lg...

Trying to verify the best practice. Say I have a "folder" structure like so: ``` s3://my-bucket/events/2022/09/01 s3://my-bucket/events/2022/09/02 ... ``` As an example, On 9/3, the client should export into the "folder" `s3://my-bucket/events/2022/09/02` (after the 9/2 day). We have a client who is running their etl on 9/3 and then placing the files on `2022/09/03` folder and we're trying to convince them to put it in the **9/2** folder. Am I correct is saying that this is the best practice?lg...

I have tried creating several buckets in AWS (all public access blocked) which have ` "Action": [ "s3:GetEncryptionConfiguration" ]` included. However, I keep getting the error `Error occurred while trying to read bucket encryption configuration, please ensure that the s3:GetEncryptionConfiguration IAM permission is present` at the airflow environment creation stage. Where should I be adding this IAM permission and how?lg...

How do I choose between MWAA (Managed Workflows for Apache Airflow) and Step Functions? It seems like I can do the same thing with both. Is there a side-by-side code example showing the same use case in both systems so I can see the differences?lg...

Hi all, I am trying to install GDAL in MWAA. library link - https://pypi.org/project/GDAL/ I have already tried it through requirements.txt file. Content of requirements.txt - ``` --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt apache-airflow-providers-ssh==2.3.0 GDAL==3.5.1 ``` But getting some errors - error file link - https://github.com/aws/aws-mwaa-local-runner/files/9478133/error_log.txt As I can see in the pypi.org it depends on the following -  looking forward to any solutions in this Thanks, SKClg...

We have multiple MWAA environments. All stopped sending logs to CloudWatch on the 16th Aug 2022. All environments have logging enabled. I've tried restarting the environments. I've also created a new environment. No environments are sending logs. Is anyone else seeing this?lg...

Can i install/deploy a separate tool like sqlcmd or bcp into mwaa? these are linux tools to help move data around on MS SQL Server. we use this on-prem in our Airflow linux boxes but don't know if this is supported in MWAA. Basically, i need to install non-pip dependency on MWAA https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver16 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-migrate-bcp?view=sql-server-ver16lg...

Hi all! I am new to MWAA service. I would like to use it to manage my dags, which I deployed by myself on ec2 instances with airflow running in a docker. It would be much simpler for me to use managed airflow :) But: I already use Dynamic Task Mapping which is from Airflow 2.3 where MWAA Airflow is 2.2 + my packages are python 3.9 where MWAA python is 3.7. 1) Would you know the roadmap of AWS concerning the updates of Airflow and Python? 2) Would you know how to run my airflow tasks in a 3.9 environment (I read it could be possible using docker, but I don't find any tutorial to do this). Thanks a lot for your help!lg...

I'm using MWAA with Airflow 2.2, private network access mode. I'm trying to log in using a token generated through boto3. My role has Admin privileges in Airflow. The URL I'm using looks like: `https://<HOST>/aws_mwaa/aws-console-sso?login=true#<TOKEN>`, just as recommended in the [docs](https://docs.aws.amazon.com/mwaa/latest/userguide/call-mwaa-apis-web.html). The issue is that the generated URL does not always directs me to the Airflow home page, seemingly randomly it ends up redirecting me to the Login page using the following URL: `https://<HOST>/aws_mwaa/aws-console-sso?next=https%3A%2F%2F<HOST>%2Fhome`. Using the same login URL with the same token it sometimes works, sometimes doesn't. Looking at the Network traffic it shows that the Login page redirects to the /home endpoint as it should, but it receives 302 instead of 200 and directs back to /aws_mwaa/aws-console-sso. Again, this happens randomly, in ~30% of the times when I want to open the Airflow UI this way. I'm using MacOS Monterey 12.4 and Chrome Version 103.0.5060.134 (Official Build) (arm64). I could not reproduce the issue on Safari, apparently the login always works there.lg...

hi all, I have a Dag that I schedule to run every quarterly using: **'schedule_interval='@quarterly'**, but for some reason it runs every day. I don't see the run in managed airflow UI but the operator is been executed (the pipeline delete some file and I can see its been deleted everyday). can you help me find the reason it happen? additional information: the start date is schedule to the begging of the year and the catchup=False. * start_date: datetime(2022, 1, 1), * catchup=False,lg...