AWS Managed Flink: Unable to run the python example of GettingStarted

Question

I am unable to run successfully the "Getting Started"  python example of AWS Managed Flink from the archive https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples.git  on AWS.

Locally on my ubuntu machine, this works fine.  I tried using Flink 1.19.1 and 1.20.0 .  Results are the same with both.

From the same archive, I am able to run the java version of the example application on AWS.

For reference here is the cloud watch log.

{
    "applicationARN": "arn:aws:kinesisanalytics:us-east-1::application/MyApplicationPy",
    "applicationVersionId": "5",
    "locationInformation": "org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:127)",
    "logger": "org.apache.flink.client.python.PythonDriver",
    "message": "Run python process failed",
    "messageSchemaVersion": "1",
    "messageType": "ERROR",
    "threadName": "Flink-DispatcherRestEndpoint-thread-3",
    "throwableInformation": "java.lang.RuntimeException: Python process exits with code: 1
	at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:124)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:356)
	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:223)
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:113)
	at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
	at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
	at org.apache.flink.runtime.webmonitor.handlers.JarRunOverrideHandler.lambda$handleRequest$3(JarRunOverrideHandler.java:239)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
"
}

re:Post Agent · Answer

It appears you're encountering an issue running the Python example of the "Getting Started" application for AWS Managed Service for Apache Flink. The error message indicates that the Python process is exiting with a code of 1, which typically suggests that an error occurred during the execution of the Python script.

There are a few things to consider:

1. Compatibility: AWS Managed Service for Apache Flink supports Python version 3.11 for Flink versions 1.19 and 1.20. Make sure your Python code is compatible with this version.

2. Dependencies: Ensure that all required Python dependencies are properly included in your application package. When running on AWS, you need to bundle your Python code and any dependencies into a zip file and upload it to an S3 bucket.

3. Main method: Verify that your Python script has a 'main' method, as this is required for AWS Managed Service for Apache Flink.

4. Environment differences: Since the application runs successfully on your local Ubuntu machine but fails on AWS, there might be environmental differences causing the issue. This could include differences in available libraries, file paths, or permissions.

5. Logging: To get more detailed information about the error, you may need to add more logging statements in your Python code. This can help pinpoint where exactly the script is failing.

6. Known issues: There are some known issues with Python dependencies on certain hardware configurations. If you're developing on a Mac with Apple Silicon, for instance, you might need to run the Python interpreter in Docker to resolve these issues.

Given that the Java version of the example works, it's likely that the issue is specific to the Python setup or code. You may want to review your Python script, ensure all dependencies are correctly packaged, and add more detailed logging to help identify the exact point of failure.  
**Sources**  
[Use Python with Managed Service for Apache Flink - Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/how-python.html)  
[Amazon Managed Service for Apache Flink 1.19 - Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/flink-1-19.html)

AWS Managed Flink: Unable to run the python example of GettingStarted

Add your answer

Relevant content