- Newest
- Most votes
- Most comments
The error you're encountering is related to a compatibility issue with batch processing in Amazon Managed Service for Apache Flink. The error message shows a UnsupportedOperationException occurring in the ResultPartition.getAllDataProcessedFuture method, which suggests an issue with how the batch job is being processed.
There are a few potential issues to address:
- Region Configuration: When working with S3 in opt-in AWS regions, you need to explicitly specify the region in your Flink configuration. Make sure you're setting the S3 endpoint region properly in your application configuration:
fs.s3a.endpoint.region: us-east-2
- Execution Environment: Your code appears to be using a batch processing approach with bounded sources (reading from S3 files), but you might be running it in a streaming context. For batch processing with Flink, consider using:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
instead of the streaming environment, or explicitly set the runtime mode to BATCH.
-
Connector Configuration: Ensure you have the proper S3 connector dependencies and configurations. The S3 connector might need additional configuration parameters to work correctly in the managed environment.
-
Application Status: Even when your application shows as "RUNNING" in the AWS console, it doesn't necessarily mean the job is healthy and processing data. The application might be stuck in a fail-and-restart loop, which can happen when a connector is misconfigured or there are networking issues preventing connection to external systems.
Try adding these configurations to your application and ensure you're using the appropriate execution environment for your batch processing job.
Sources
Deep dive into the Amazon Managed Service for Apache Flink application lifecycle – Part 2 | AWS Big Data Blog
Troubleshooting - Amazon EMR
answered a year ago
The same code worked fine when reading the file from the same input S3 Bucket and writing the file as-is to another S3 Bucket. But S3 Data Source to FlatMap appears to be causing the failure.
Relevant content
asked 2 years ago
asked 2 years ago
- AWS OFFICIALUpdated a year ago

A more detailed stacktrace from CloudWatch.
{ "applicationARN": "arn:aws:kinesisanalytics:us-east-2:047472788728:application/datahose-app", "applicationVersionId": "2", "locationInformation": "org.apache.flink.runtime.executiongraph.Execution.transitionState(Execution.java:1506)", "logger": "org.apache.flink.runtime.executiongraph.ExecutionGraph", "message": "Source: S3-Data-Source -> Flat Map -> anonymous_datastream_source$1[1] -> HashAggregate[2] (1/1) (a397e35328e987dd6cbf40d3b3342eea_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from RUNNING to FAILED on 172.21.124.163:6122-161e4b @ ip-172-21-124-163.us-east-2.compute.internal (dataPort=6121).", "messageSchemaVersion": "1", "messageType": "INFO", "threadName": "flink-pekko.actor.default-dispatcher-21", "throwableInformation": "java.lang.UnsupportedOperationException\n\tat org.apache.flink.runtime.io.network.partition.ResultPartition.getAllDataProcessedFuture(ResultPartition.java:231)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:988)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:923)\n\tat org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat java.base/java.lan