Skip to content

How can I set env var or system property in managed flink

0

I have an application running in Amazon Managed Service for Apache Flink (MSF). On startup it logs

The AWS SDK for Java 1.x entered maintenance mode starting July 31, 2024 and will reach end of support on December 31, 2025. For more information, see https://aws.amazon.com/blogs/developer/the-aws-sdk-for-java-1-x-is-in-maintenance-mode-effective-july-31-2024/
You can print where on the file system the AWS SDK for Java 1.x core runtime is located by setting the AWS_JAVA_V1_PRINT_LOCATION environment variable or aws.java.v1.printLocation system property to 'true'.
This message can be disabled by setting the AWS_JAVA_V1_DISABLE_DEPRECATION_ANNOUNCEMENT environment variable or aws.java.v1.disableDeprecationAnnouncement system property to 'true'.

I don't actually think we're using SDK 1.x anywhere, so I'd like to turn on the extra logging and see where it is used. But I don't see any way to set an environment variable or system property in MSF. There is runtime properties, but those don't seem to do it. SO how can I set these?

asked 8 months ago270 views

2 Answers
-1

In Amazon Managed Service for Apache Flink, you cannot directly set environment variables or JVM system properties through the console or API in the same way you might with other compute services. The runtime properties feature you mentioned is specifically designed for application configuration and is accessed programmatically through the Flink application code itself, not as system-level environment variables or JVM properties.

Runtime properties in Managed Service for Apache Flink are accessed within your Java application using the KinesisAnalyticsRuntime.getApplicationProperties() method, which returns a Map<String, Properties> object. These are application-level configuration properties used to configure Flink sources, sinks, and other application components without recompilation - they don't translate to environment variables or system properties that would be visible to the AWS SDK.

For your specific use case of setting AWS_JAVA_V1_PRINT_LOCATION or aws.java.v1.printLocation, you would need to modify your application code to programmatically set the system property before the AWS SDK is initialized. You could do this in your application's main method or initialization code by adding:

System.setProperty("aws.java.v1.printLocation", "true");

Alternatively, you could set:

System.setProperty("aws.java.v1.disableDeprecationAnnouncement", "true");

This would need to be done early in your application startup, before any AWS SDK v1.x classes are loaded. Since you'll need to modify and redeploy your application code anyway, this is the appropriate approach for Managed Service for Apache Flink.
Sources
Use runtime properties in Managed Service for Apache Flink - Managed Service for Apache Flink

answered 8 months ago

  • This does not work. I called this as the very first line in my main() method, but the version check happens before my code is ever called (which I can tell because I also have logging in my main). Looking at the stacktrace I can that my code is never called and the first thing in the stack is org.apache.flink.runtime.entrypoint.StandaloneSessionClusterEntrypoint.main

-1
  • Would you consider migrating to SDK 2.x The proper long-term solution is to ensure your application and all its dependencies use AWS SDK for Java 2.x. AWS provides a migration tool that can help automate this process, refer [1]:

      mvn org.openrewrite.maven:rewrite-maven-plugin:6.17.0:run \
     -Drewrite.recipeArtifactCoordinates=software.amazon.awssdk:v2-migration:2.34.0 \
     -Drewrite.activeRecipes=software.amazon.awssdk.v2migration.AwsSdkJavaV1ToV2
    
  • Configuration Options Available in MSF

For reference, the configuration options you can control in Managed Service for Apache Flink include, refer [2]:

  • Additionally, few more points, FYI:
  1. Runtime properties: Application-specific parameters accessible via your code
  2. Flink configuration properties: Limited set of Apache Flink settings (requires support case for most changes), refer [3]
  3. Parallelism configuration: Application scaling parameters, refer [4]
  4. Monitoring configuration: CloudWatch logging settings, refer [5]

But remember, none of these provide access to JVM system properties or OS environment variables.

[1] https://aws.amazon.com/blogs/developer/general-availability-release-of-the-migration-tool-for-the-aws-sdk-for-java-2-x/

[2] https://docs.aws.amazon.com/managed-flink/latest/apiv2/API_ApplicationConfiguration.html

[3] https://docs.aws.amazon.com/managed-flink/latest/java/reference-flink-settings.title.html

[4] https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesisanalyticsv2/model/ParallelismConfiguration.html

[5] https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesisanalyticsv2/model/FlinkApplicationConfiguration.html

AWS
EXPERT

answered 8 months ago

  • We're already using SDK 2.41.16. Using mvn clean dependency:tree shows only 2.41.16. So I am wondering where it is finding 1.x, and that is why I am trying to turn on logging.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.