Push logs to S3

0

How can I push Airflow task logs to S3?

I tried adding the following configuration parameters to "Airflow configuration options",
logging.base_log_folder: /usr/local/airflow/logs
logging.remote_base_log_folder: s3://my-bucket/logs/
logging.remote_log_conn_id: aws_default
logging.remote_logging: True
but it seems to have no effect.

Came across this https://blogs.halodoc.io/experience-with-evaluating-managed-workflow-for-apache-airflow-mwaa/ post saying it ain't possible to push logs into S3.

質問済み 3年前947ビュー
3回答
0

Hi,

Logging configurations cannot be changed in MWAA. All logs are sent exclusively to CloudWatch.

Thanks.

AWS
John_J
回答済み 3年前
0

Hi John,

Thanks for your response!
Would you know if support for other logging destinations is on the roadmap as such?

回答済み 3年前
0

Unfortunately Airflow currently does not support multiple logging destinations, and CloudWatch is required for service monitoring.

You can however export from Cloudwatch to S3: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html

AWS
John_J
回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ