Push logs to S3

Question

How can I push Airflow task logs to S3?  
  
I tried adding the following configuration parameters to "Airflow configuration options",  
logging.base_log_folder:	          /usr/local/airflow/logs  
logging.remote_base_log_folder:  s3://my-bucket/logs/  
logging.remote_log_conn_id:	  aws_default  
logging.remote_logging:	          True  
but it seems to have no effect.  
  
Came across this https://blogs.halodoc.io/experience-with-evaluating-managed-workflow-for-apache-airflow-mwaa/ post saying it ain't possible to push logs into S3.

John_J · Answer

Hi,  
  
Logging configurations cannot be changed in MWAA.  All logs are sent exclusively to CloudWatch.  
  
Thanks.

John_J · Answer

Unfortunately Airflow currently does not support multiple logging destinations, and CloudWatch is required for service monitoring.  
  
You can however export from Cloudwatch to S3: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html

bcoznotgcp · Answer

Hi John,  
  
Thanks for your response!  
Would you know if support for other logging destinations is on the roadmap as such?

Push logs to S3

Add your answer

Relevant content