How to increase logging level for spark workers in Glue 5.0?

In AWS Glue 5.0, you can control logging levels for Spark workers through several approaches:

1. Custom Configuration Options: AWS Glue 5.0 provides real-time logging capabilities with customizable configuration options. You can specify the CloudWatch log group name, log stream prefix, and log conversion pattern for messages.

2. Log4j2 Configuration: Glue 5.0 uses Log4j 2, not Log4j 1.x. This is important because the configuration format has changed. Instead of using log4j.properties, you should use log4j2.properties with the appropriate syntax.

3. Job Arguments: You can customize logging behavior by setting specific job arguments when configuring your Glue job.

4. Custom Logger Implementation: You can implement a custom logger in your script to control logging levels more precisely.

For your specific case, since you're using Glue 5.0 Notebooks, try the following approach:

1. Create a log4j2.properties file with appropriate configurations (setting root logger level to WARN or ERROR)
2. Reference this file in your job configuration
3. Use the proper Log4j2 syntax in your properties file

Note that in Glue 5.0, you need to use the appropriate classification for Log4j2 configurations. The methods you've tried are mostly for older versions or different environments.

You can also use the custom script logger provided by AWS Glue to log application-specific messages at your desired level, which gives you more control over what gets logged.

If you need to aggregate logs in custom CloudWatch log groups with different expiration policies, you can do so by specifying custom log group names and log stream prefixes in your job configuration.
Sources
Logging for AWS Glue jobs - AWS Glue
How to correctly provide log4j properties file for pyspark glue jobs? | AWS re:Post
Configure Apache Log4j2 properties for Amazon EMR Serverless - Amazon EMR
Configuring job properties for Spark jobs in AWS Glue - AWS Glue