Glue 5.0 Continuous Logging Removed

The removal of continuous logging in Glue 5.0 was primarily driven by AWS's effort to simplify logging architecture and reduce costs.

Here are your main options for handling this:

CloudWatch Logs Subscription Filters:

• You can create subscription filters on the error log group
• Filter and route logs based on Job name or other patterns to different destinations
• Can send to other log groups, Lambda functions, or Kinesis streams

Custom Logging Solution:

import logging
import watchtower

# Create custom logger
logger = logging.getLogger('custom_logger')
logger.setLevel(logging.INFO)

# Set up CloudWatch handler with specific log group
handler = watchtower.CloudWatchLogHandler(
  log_group_name='/custom/glue/logs',
  stream_name='my-glue-job'
)
logger.addHandler(handler)

# Use in your code
logger.info('Custom log message')

Many users are requesting the ability to specify custom log groups Until AWS provides a better solution, the CloudWatch Logs Subscription Filters approach is probably your best option for production environments, as it requires no code changes and can be managed centrally.