Export AWS Managed Apache Flink savepoints to S3

Question

We'd like to migrate from AWS Managed Apache Flink to our self-hosted Flink cluster on EKS. As part of this, we need to migrate Flink operator states to the new cluster.

From my understanding this can be done by creating a Flink savepoint and start the new Flink job from this savepoint. However, we don't find the options to create and download Flink savepoint in AWS Managed Apache Flink Console or API. It's possible to create and recover from snapshots but it seems that we can't download or export the snapshots to S3.

My question is how we can download/export Flink snapshots or savepoints to S3 in order to do the migration? Thank you!

Brent Everman · Answer

It looks like the Export Snapshot to S3 functionality is no longer available (at least its not documented here, are there alternate options

Brent Everman

answered 2 years ago

BrianDAnderson_AWS · Answer

Hello Luc Tran.  To migrate from AWS Managed Apache Flink to a self-hosted Flink cluster on EKS and transfer the operator states, you'll need to export the Flink snapshots to Amazon S3. Here's an overview of the process and the steps to accomplish this task:

**Enable S3 Export for Snapshots**
        Navigate to the AWS Managed Apache Flink console.
        Select your application and go to the "Configuration" tab.
        Enable the option to export snapshots to S3 if it's not already enabled.
        Specify an S3 bucket where snapshots will be exported.

**Create a Snapshot**
        In the AWS Managed Apache Flink console, select your application.
        Choose the "Snapshots" tab.
        Click on "Create snapshot" and follow the prompts to create a new snapshot.

**Export Snapshot to S3**
        Once the snapshot is created, select it from the list.
        Choose the "Export to S3" option.
        Confirm the S3 bucket destination and initiate the export process.

**Verify Snapshot in S3**
        Open the Amazon S3 console.
        Navigate to the specified bucket and verify that the snapshot files are present.

**Prepare EKS Cluster**
        Ensure your EKS cluster is set up and configured to run Apache Flink.
        Install the Flink Kubernetes operator on your EKS cluster if not already done.

**Create Flink Deployment on EKS**
        Create a FlinkDeployment YAML file specifying the exported snapshot as the initial state.
        Use the **`kubectl`** command to apply the FlinkDeployment to your EKS cluster.

**Verify Migration**
        Access the Flink UI for your new deployment on EKS.
        Confirm that the job has started successfully from the imported snapshot.

By following these steps, you'll be able to export your Flink snapshots from AWS Managed Apache Flink to S3 and use them to initialize your self-hosted Flink cluster on EKS. This process ensures that your operator states are preserved during the migration.

Additional Resources:

https://docs.aws.amazon.com/managed-flink/latest/java/how-snapshots.html

https://docs.aws.amazon.com/managed-flink/latest/java/upgrading-application-new-version.html

https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/jobruns-flink-kubernetes-operator-run-application.html

Best of luck and thank you for using AWS!

Brian

Export AWS Managed Apache Flink savepoints to S3

Add your answer

Relevant content