Managed Service for Apache Flink system rollback not triggering

Question

I have enabled **automatic system rollbacks** for an Apache Flink 1.20 application on Managed Service for Apache Flink, but it doesn't seem to trigger.

I deployed a new version with state changes, so the job encounters state incompatibility exceptions. Non-restored state is not allowed. The job seems to be restarting indefinitely with [exponential delay strategy](https://docs.aws.amazon.com/managed-flink/latest/java/reference-modifiable-settings.html#reference-modifiable-settings-fault-tolerance), so my theory is that the job never fails and the system rollback is not triggered. It doesn't even seem to be possible to define [maximum number of restart attempts](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/ops/state/task_failure_recovery/#restart-strategy-fixed-delay-attempts) even if fixed delay strategy would be used.

Maybe somebody has encountered this issue and could point me to the right direction? Thanks!

re:Post Agent · Accepted Answer

Thank you for describing your issue with the automatic system rollback feature in Amazon Managed Service for Apache Flink. Your observation about the system rollback not triggering is correct, and there are a few important points to consider:

1. System rollback is designed to occur before the application transitions to the RUNNING status. If your application has already successfully transitioned to RUNNING and then encounters runtime issues like state incompatibility exceptions, the automatic rollback will not be triggered.

2. The behavior you're seeing with indefinite restarts using the exponential delay strategy is expected. As you noted, the job never actually fails, it just keeps restarting, which prevents the automatic rollback from triggering.

3. In your situation, where the application is already in the RUNNING state and facing runtime errors, you have the option to manually trigger a rollback using the RollbackApplication API.

4. Regarding the maximum number of restart attempts, you're correct that this isn't directly configurable in Managed Service for Apache Flink as it is in open-source Apache Flink.

To address your issue, you have a few options:

1. Manually trigger a rollback using the RollbackApplication API if you want to revert to the previous working version.

2. Force-stop the application, which will allow you to then update it with a corrected version that resolves the state incompatibility issues.

3. Review your application code and configuration to resolve the state incompatibility exceptions, then update the application with the corrected version.

For troubleshooting, you can use the ListApplicationOperations API to view the history of operations on your application, and the DescribeApplicationOperation API to get more details on specific operations. This can help you understand what's happening with your application and why the rollback isn't triggering as expected.

If you continue to face issues, it may be helpful to check the AWS Health Dashboard for any service-related problems or consider opening a support case for more personalized assistance.  
**Sources**  
[System rollback best practices - Managed Service for Apache Flink](https://docs.aws.amazon.com/managed-flink/latest/java/troubleshooting-system-rollback.html)  
[Improve the resilience of Amazon Managed Service for Apache Flink application with system-rollback feature | AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/improve-the-resilience-of-amazon-managed-service-for-apache-flink-application-with-system-rollback-feature/)

Managed Service for Apache Flink system rollback not triggering

Add your answer

Relevant content