Why is my AWS Glue job not writing logs to Amazon CloudWatch?

Short description

If your AWS Glue jobs are not pushing logs to CloudWatch, then check the following:

• Be sure that your AWS Glue job has all the required AWS Identity and Access Management (IAM) permissions.
• Be sure that the AWS Key Management Service (AWS KMS) key allows the CloudWatch Logs service to use the key.
• Be sure that the IAM permission logs:AssociateKmsKey is attached to the AWS Glue role.
• If you haven't turned on continuous logging for your AWS Glue Spark ETL job, then check if the job failed before log aggregation.
• Be sure that you are checking the correct CloudWatch log group.

Resolution

The AWS Glue job role lacks IAM permissions to create and write to the CloudWatch log group

If you aren't using the managed AWSGlueServiceRole policy, then be sure that the IAM role attached to the ETL job has the following required permissions to interact with CloudWatch. If the job uses a custom log group, then the IAM policy must provide access to the custom log group.

{
    "Effect": "Allow",
    "Action": "cloudwatch:PutMetricData",
    "Resource": [
        "*"
    ]
},
{
    "Effect": "Allow",
    "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
    ],
    "Resource": [
        "arn:aws:logs:*:*:/aws-glue/*",
        "arn:aws:logs:*:*:/customlogs/*"
    ]
}

Be sure to replace arn:aws:logs:*:*:/customlogs/* with the ARN of the custom log group.

The AWS KMS key used does not grant the required permissions to the CloudWatch Logs service

If you are using security configurations with your AWS Glue job, then be sure that the AWS KMS key attached to the security configuration allows the CloudWatch Logs service to use the key. Attach the following policy to the AWS KMS key:

{
    "Effect": "Allow",
    "Principal": {
        "Service": "logs.region.amazonaws.com"
    },
    "Action": [
        "kms:Encrypt*",
        "kms:Decrypt*",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:Describe*"
    ],
    "Resource": "*",
    "Condition": {
        "ArnEquals": {
            "kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-west-2:1111222233334444:log-group:log-group-name"
        }
    }
}

Be sure to replace the following in the policy:

• us-west-2 with the AWS Region of your choice
• 1111222233334444 with your AWS account ID.
• log-group-name with the name of your log group.

For more information, see Encrypt log data in CloudWatch Logs using AWS Key Management Service.

Also, be sure that the IAM permission logs:AssociateKmsKey is attached to the AWS Glue role. For more information, see Security configuration with continuous logging.

Continuous logging is not turned on for your AWS Glue Spark ETL job

If you haven't turned on continuous logging for your AWS Glue Spark ETL job, then log aggregation happens after the job run is completed. If the job fails before log aggregation, then the logs might not get pushed to CloudWatch. To make sure that logs are populated irrespective of application failures, you can turn on continuous logging for your AWS Glue jobs.

You are not looking for the logs in the correct log group

Keep the following things in mind when you are looking for the CloudWatch Logs:

• If you turned on continuous logging and use the default log groups, then custom messages, such as those from print statements, are pushed to /aws-glue/jobs/output log group.
• If you turned on continuous logging and use the default log groups, then the messages emitted by loggers are pushed to the driver logs under /aws-glue/jobs/logs-v2.
• If you turned on continuous logging and use the default log groups, jobs using security configurations push custom messages to /aws-glue/jobs/logs-v2-testconfig. Be sure to replace testconfig with the name of the security configuration.
• If you turned on continuous logging and use custom log groups, then you can find the custom log messages along with the driver and executor logs under the custom log group.
• If you didn't turn on continuous logging, then you can find messages, such as print statement outputs, under /aws-glue/jobs/output and all custom messages from logger under /aws-glue/jobs/error.

For more information, see Logging behavior.

Related information

Logging and monitoring in AWS Glue

Running and monitoring in AWS Glue