Deploy Python Pyflink Code to AWS Managed flink Service

To deploy your Python PyFlink code to AWS Managed Service for Apache Flink (previously known as Amazon Kinesis Data Analytics for Apache Flink), you'll need to follow these steps:

1. Create a zip file containing your Python application:

   • Include your main Python script with a main method.
   • Add any additional Python code files you're using.
   • Include the flink-sql-connector-kafka JAR file you mentioned.
   • Add any other Java libraries or user-defined Java code in JAR files if needed.

2. Upload this zip file to an Amazon S3 bucket:

   • Use the Amazon S3 console or AWS CLI to upload your zip file.

3. Create your Managed Service for Apache Flink application:

   • Specify the S3 location of your uploaded zip file as the code location.
   • Configure the necessary application properties and settings.

4. When creating the application, you'll need to provide the appropriate IAM permissions for your application to access the required AWS resources (like Kafka and OpenSearch).

It's important to note that even though you're writing in Python, the AWS Managed Service for Apache Flink executes Python code by translating it to Java statements, which are then run in the Java VM. This is why you can include Java dependencies (like your Kafka connector JAR) alongside your Python code.

Make sure your Python code is compatible with the version supported by AWS Managed Service for Apache Flink, which is currently Python 3.10 for Apache Flink version 1.18.1.

If you encounter any issues with dependencies, especially if you're developing on a Mac with Apple Silicon, you might need to use Docker to run your Python interpreter for testing before deployment.

Remember to include all necessary dependencies in your zip file, as the Managed Service environment won't have access to external resources not included in your package.
Sources
Program your Managed Service for Apache Flink Python application - Managed Service for Apache Flink
Use Python with Managed Service for Apache Flink - Managed Service for Apache Flink
Create and run a Managed Service for Apache Flink for Python application - Managed Service for Apache Flink