Log level of external libraries - Kinesis Data Analytics for Apache Flink


I have a Kinesis data analytics application written in Java with Flink 1.13 framework. Currently on Cloudwatch I also get logs from libraries outside my application (e.g. 'org.apache.flink', 'com.jayway.jsonpath', 'com.amazonaws', 'org.apache.hadoop', etc.). I would like to figure out how to decrease the log level precisely for each library. I am using the Amazon CloudWatch logging option and the SLF4J library for logging. I have already read the official documentation of AWS KDA for Apache Flink but I did not find what I was looking for.


1 Answer

Controlling log levels for specific libraries in an application using SLF4J for logging can be achieved by adjusting the logging configuration. In your case, since you're using Amazon CloudWatch for logging, you need to configure the logging properties appropriately to filter out or adjust the log levels for specific libraries.

Here's how you can do it:

  1. Create a log4j.properties file:

    Create a log4j.properties file in your application's resources directory (usually the src/main/resources directory). If you're using SLF4J, you can still use a log4j properties file since SLF4J provides a bridge for log4j.

  2. Configure Log Levels:

    In the log4j.properties file, you can specify different log levels for various packages or classes. Here's a general structure:

    # Set the root logger level
    log4j.rootLogger=INFO, stdout
    # Define the output appender
    log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %-5p [%c{1}] %m%n
    # Set log level for specific packages or classes
    # Set default log level for all other classes/packages

    In the above example, we're setting the log level to INFO for the org.apache.flink package, and to ERROR for com.jayway.jsonpath, com.amazonaws, and org.apache.hadoop.

  3. Place the log4j.properties file in the classpath:

    Make sure the log4j.properties file is included in the classpath of your application. This ensures that the logging configuration is picked up when your application runs.

  4. Update Kinesis Data Analytics Configuration:

    If you're running your Flink application through Kinesis Data Analytics, make sure that your application's JAR file containing the log4j.properties file is uploaded to Kinesis and is part of your application's classpath.

  5. Restart Application:

    After making these changes, restart your Kinesis Data Analytics application. The new logging configuration should take effect, and you should see the adjusted log levels for the specified libraries.

Remember that logging configuration can vary based on the specific environment and framework you're using, but the general approach of adjusting log levels through configuration files is consistent. Also, double-check the compatibility of the logging framework versions with Flink 1.13 to avoid any compatibility issues.

answered 6 months ago
  • Thanks for your answer. Which is the difference between:

    • Log Level setted in the KDA settings
    • log4j.rootLogLevel
    • log4j.logger Moreover, how do I make sure that the file log4j.properties is placed in the classpath of my application? If I go to unpack the jar file of my application, I find the file right in the root folder.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions