Browse through the questions and answers listed below or filter and sort to narrow down your results.
Negsignal.SIGKILL in MWAA
I keep getting the error message `Task exited with return code Negsignal.SIGKILL` even on tasks not requiring much memory or compute on my airflow DAGs. I have already upgraded from small to medium. Want to know if there's some configuration I can change before upgrading to large or considering another option.
Error while deploying MWAA using Cloudformation
Encountering the following error ``` 2022-09-16 15:25:42 UTC+0530 MwaaEnvironment CREATE_FAILED Resource handler returned message: "null (Service: Mwaa, Status Code: 403, Request ID: de70d88f-383a-43bb-91a7-4ebdee31223e)" (RequestToken: fe034e51-e5e2-73e9-3616-0bd928d64a1f, HandlerErrorCode: AccessDenied) ``` Not sure which resource access is denied.
s3 - date folder structure for etl data processes (airflow, etc)
Trying to verify the best practice. Say I have a "folder" structure like so: ``` s3://my-bucket/events/2022/09/01 s3://my-bucket/events/2022/09/02 ... ``` As an example, On 9/3, the client should export into the "folder" `s3://my-bucket/events/2022/09/02` (after the 9/2 day). We have a client who is running their etl on 9/3 and then placing the files on `2022/09/03` folder and we're trying to convince them to put it in the **9/2** folder. Am I correct is saying that this is the best practice?
What should I edit in AWS bucket policy to work with MWAA (Airflow)
I have tried creating several buckets in AWS (all public access blocked) which have ` "Action": [ "s3:GetEncryptionConfiguration" ]` included. However, I keep getting the error `Error occurred while trying to read bucket encryption configuration, please ensure that the s3:GetEncryptionConfiguration IAM permission is present` at the airflow environment creation stage. Where should I be adding this IAM permission and how?
Installing GDAL in MWAA
Hi all, I am trying to install GDAL in MWAA. library link - https://pypi.org/project/GDAL/ I have already tried it through requirements.txt file. Content of requirements.txt - ``` --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt apache-airflow-providers-ssh==2.3.0 GDAL==3.5.1 ``` But getting some errors - error file link - https://github.com/aws/aws-mwaa-local-runner/files/9478133/error_log.txt As I can see in the pypi.org it depends on the following - ![dependencies](/media/postImages/original/IMntrn7tDYS8qwI64sqww07Q) looking forward to any solutions in this Thanks, SKC
MWAA not sending logs to CloudWatch
We have multiple MWAA environments. All stopped sending logs to CloudWatch on the 16th Aug 2022. All environments have logging enabled. I've tried restarting the environments. I've also created a new environment. No environments are sending logs. Is anyone else seeing this?
MWAA - can I use external programs (non-python) - sqlcmd or bcp?
Can i install/deploy a separate tool like sqlcmd or bcp into mwaa? these are linux tools to help move data around on MS SQL Server. we use this on-prem in our Airflow linux boxes but don't know if this is supported in MWAA. Basically, i need to install non-pip dependency on MWAA https://docs.microsoft.com/en-us/sql/tools/sqlcmd-utility?view=sql-server-ver16 https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-migrate-bcp?view=sql-server-ver16
MWAA legacy airflow and python versions
Hi all! I am new to MWAA service. I would like to use it to manage my dags, which I deployed by myself on ec2 instances with airflow running in a docker. It would be much simpler for me to use managed airflow :) But: I already use Dynamic Task Mapping which is from Airflow 2.3 where MWAA Airflow is 2.2 + my packages are python 3.9 where MWAA python is 3.7. 1) Would you know the roadmap of AWS concerning the updates of Airflow and Python? 2) Would you know how to run my airflow tasks in a 3.9 environment (I read it could be possible using docker, but I don't find any tutorial to do this). Thanks a lot for your help!
MWAA WebLoginToken auth randomly fails
I'm using MWAA with Airflow 2.2, private network access mode. I'm trying to log in using a token generated through boto3. My role has Admin privileges in Airflow. The URL I'm using looks like: `https://<HOST>/aws_mwaa/aws-console-sso?login=true#<TOKEN>`, just as recommended in the [docs](https://docs.aws.amazon.com/mwaa/latest/userguide/call-mwaa-apis-web.html). The issue is that the generated URL does not always directs me to the Airflow home page, seemingly randomly it ends up redirecting me to the Login page using the following URL: `https://<HOST>/aws_mwaa/aws-console-sso?next=https%3A%2F%2F<HOST>%2Fhome`. Using the same login URL with the same token it sometimes works, sometimes doesn't. Looking at the Network traffic it shows that the Login page redirects to the /home endpoint as it should, but it receives 302 instead of 200 and directs back to /aws_mwaa/aws-console-sso. Again, this happens randomly, in ~30% of the times when I want to open the Airflow UI this way. I'm using MacOS Monterey 12.4 and Chrome Version 103.0.5060.134 (Official Build) (arm64). I could not reproduce the issue on Safari, apparently the login always works there.
mwaa pipeline scheduling issue
hi all, I have a Dag that I schedule to run every quarterly using: **'schedule_interval='@quarterly'**, but for some reason it runs every day. I don't see the run in managed airflow UI but the operator is been executed (the pipeline delete some file and I can see its been deleted everyday). can you help me find the reason it happen? additional information: the start date is schedule to the begging of the year and the catchup=False. * start_date: datetime(2022, 1, 1), * catchup=False,