Log level of external libraries - Kinesis Data Analytics for Apache Flink

Controlling log levels for specific libraries in an application using SLF4J for logging can be achieved by adjusting the logging configuration. In your case, since you're using Amazon CloudWatch for logging, you need to configure the logging properties appropriately to filter out or adjust the log levels for specific libraries.

Here's how you can do it:

1. Create a log4j.properties file:

   Create a log4j.properties file in your application's resources directory (usually the src/main/resources directory). If you're using SLF4J, you can still use a log4j properties file since SLF4J provides a bridge for log4j.

2. Configure Log Levels:

   In the log4j.properties file, you can specify different log levels for various packages or classes. Here's a general structure:

   # Set the root logger level
   log4j.rootLogger=INFO, stdout
   
   # Define the output appender
   log4j.appender.stdout=org.apache.log4j.ConsoleAppender
   log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
   log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p [%c{1}] %m%n
   
   # Set log level for specific packages or classes
   log4j.logger.org.apache.flink=INFO
   log4j.logger.com.jayway.jsonpath=ERROR
   log4j.logger.com.amazonaws=ERROR
   log4j.logger.org.apache.hadoop=ERROR
   
   # Set default log level for all other classes/packages
   log4j.logger=INFO

   In the above example, we're setting the log level to INFO for the org.apache.flink package, and to ERROR for com.jayway.jsonpath, com.amazonaws, and org.apache.hadoop.

3. Place the log4j.properties file in the classpath:

   Make sure the log4j.properties file is included in the classpath of your application. This ensures that the logging configuration is picked up when your application runs.

4. Update Kinesis Data Analytics Configuration:

   If you're running your Flink application through Kinesis Data Analytics, make sure that your application's JAR file containing the log4j.properties file is uploaded to Kinesis and is part of your application's classpath.

5. Restart Application:

   After making these changes, restart your Kinesis Data Analytics application. The new logging configuration should take effect, and you should see the adjusted log levels for the specified libraries.

Remember that logging configuration can vary based on the specific environment and framework you're using, but the general approach of adjusting log levels through configuration files is consistent. Also, double-check the compatibility of the logging framework versions with Flink 1.13 to avoid any compatibility issues.