- 最新
- 最多得票
- 最多評論
The error you're encountering is related to a compatibility issue with batch processing in Amazon Managed Service for Apache Flink. The error message shows a UnsupportedOperationException occurring in the ResultPartition.getAllDataProcessedFuture method, which suggests an issue with how the batch job is being processed.
There are a few potential issues to address:
- Region Configuration: When working with S3 in opt-in AWS regions, you need to explicitly specify the region in your Flink configuration. Make sure you're setting the S3 endpoint region properly in your application configuration:
fs.s3a.endpoint.region: us-east-2
- Execution Environment: Your code appears to be using a batch processing approach with bounded sources (reading from S3 files), but you might be running it in a streaming context. For batch processing with Flink, consider using:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
instead of the streaming environment, or explicitly set the runtime mode to BATCH.
-
Connector Configuration: Ensure you have the proper S3 connector dependencies and configurations. The S3 connector might need additional configuration parameters to work correctly in the managed environment.
-
Application Status: Even when your application shows as "RUNNING" in the AWS console, it doesn't necessarily mean the job is healthy and processing data. The application might be stuck in a fail-and-restart loop, which can happen when a connector is misconfigured or there are networking issues preventing connection to external systems.
Try adding these configurations to your application and ensure you're using the appropriate execution environment for your batch processing job.
Sources
Deep dive into the Amazon Managed Service for Apache Flink application lifecycle – Part 2 | AWS Big Data Blog
Troubleshooting - Amazon EMR
已回答 1 年前
The same code worked fine when reading the file from the same input S3 Bucket and writing the file as-is to another S3 Bucket. But S3 Data Source to FlatMap appears to be causing the failure.
相關內容
已提問 2 年前
已提問 1 年前

A more detailed stacktrace from CloudWatch.
{ "applicationARN": "arn:aws:kinesisanalytics:us-east-2:047472788728:application/datahose-app", "applicationVersionId": "2", "locationInformation": "org.apache.flink.runtime.executiongraph.Execution.transitionState(Execution.java:1506)", "logger": "org.apache.flink.runtime.executiongraph.ExecutionGraph", "message": "Source: S3-Data-Source -> Flat Map -> anonymous_datastream_source$1[1] -> HashAggregate[2] (1/1) (a397e35328e987dd6cbf40d3b3342eea_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from RUNNING to FAILED on 172.21.124.163:6122-161e4b @ ip-172-21-124-163.us-east-2.compute.internal (dataPort=6121).", "messageSchemaVersion": "1", "messageType": "INFO", "threadName": "flink-pekko.actor.default-dispatcher-21", "throwableInformation": "java.lang.UnsupportedOperationException\n\tat org.apache.flink.runtime.io.network.partition.ResultPartition.getAllDataProcessedFuture(ResultPartition.java:231)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:988)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:923)\n\tat org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat java.base/java.lan