Skip to content

Apache Flink 1.20 Batch Job Failure (S3 Source to S3 Sink)

0

I am relatively new to AMS for Flink. I wrote a simple Flink 1.20 program using Java 11 to analyze a file in a S3 Bucket and write the results to another S3 Bucket. This program worked fine on a single nodelocal Linux filesystem but failed in AMS for Flink with the following error.

{ "applicationARN": "arn:aws:kinesisanalytics:us-east-2:047472788728:application/datahose-app", "applicationVersionId": "2", "locationInformation": "org.apache.flink.runtime.taskmanager.Task.transitionState(Task.java:1131)", "logger": "org.apache.flink.runtime.taskmanager.Task", "message": "Source: S3-Data-Source -> Flat Map (1/1)#0 (013a3701c2a1bb4a97998e5dae2e7b10_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FAILED with failure cause:", "messageSchemaVersion": "1", "messageType": "WARN", "threadName": "Source: S3-Data-Source -> Flat Map (1/1)#0", "throwableInformation": "java.lang.UnsupportedOperationException\n\tat org.apache.flink.runtime.io.network.partition.ResultPartition.getAllDataProcessedFuture(ResultPartition.java:231)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:988)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:923)\n\tat org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n" }

I have pasted the a code snippet below from the main Java Class. Can somebody tell me what am I missing?

   LOG.info("S3 Input Path: {}", s3InputPath);
    LOG.info("S3 Output Path: {}", s3OutputPath);
    LOG.info("Region: {}", region);

    FileSource<String> fileSource = forRecordStreamFormat(new TextLineInputFormat(), new Path(s3InputPath)).build();
    DataStream<String> sourceRecords = env.fromSource(fileSource, WatermarkStrategy.noWatermarks(), "S3-Data-Source");

    // Parse the data to obtain a Tuple of (name,1) for each record
    DataStream<Tuple2<String, Integer>> parsedData = sourceRecords.flatMap(new FlatMapFunction<String, Tuple2<String,Integer>>() {
        @Override
        public void flatMap(String record, Collector<Tuple2<String, Integer>> out) {
               String[] elements = record.trim().split(",");
               if(elements.length ==2) {
                   // Emit (name,1) for each record
                   out.collect(new Tuple2<>(elements[0].trim(), 1));
               }
        }
    });

    // Group by name and sum the counts
    DataStream<Tuple2<String, Integer>> visitsPerPerson = parsedData.keyBy(value ->  value.f0).sum(1);

    // Transform to the Output Map
    DataStream<String> results = visitsPerPerson.map(new MapFunction<Tuple2<String, Integer>, String>() {
        @Override
        public String map(Tuple2<String, Integer> value) {
            return value.f0 + " visited the gym " + value.f1 + " times";
        }
    });

    FileSink<String> s3Sink = FileSink.forRowFormat(new Path(s3OutputPath), new SimpleStringEncoder<String>("UTF-8"))               .withRollingPolicy(DefaultRollingPolicy.builder().withRolloverInterval(Duration.ofSeconds(5)).withInactivityInterval(Duration.ofSeconds(3)).build())
            .build();

    results.sinkTo(s3Sink);
    env.execute("S3 to S3 - Analytical Use Case");

Thanks!!

  • A more detailed stacktrace from CloudWatch.

    { "applicationARN": "arn:aws:kinesisanalytics:us-east-2:047472788728:application/datahose-app", "applicationVersionId": "2", "locationInformation": "org.apache.flink.runtime.executiongraph.Execution.transitionState(Execution.java:1506)", "logger": "org.apache.flink.runtime.executiongraph.ExecutionGraph", "message": "Source: S3-Data-Source -> Flat Map -> anonymous_datastream_source$1[1] -> HashAggregate[2] (1/1) (a397e35328e987dd6cbf40d3b3342eea_cbc357ccb763df2852fee8c4fc7d55f2_0_5) switched from RUNNING to FAILED on 172.21.124.163:6122-161e4b @ ip-172-21-124-163.us-east-2.compute.internal (dataPort=6121).", "messageSchemaVersion": "1", "messageType": "INFO", "threadName": "flink-pekko.actor.default-dispatcher-21", "throwableInformation": "java.lang.UnsupportedOperationException\n\tat org.apache.flink.runtime.io.network.partition.ResultPartition.getAllDataProcessedFuture(ResultPartition.java:231)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.afterInvoke(StreamTask.java:988)\n\tat org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:923)\n\tat org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:972)\n\tat org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:951)\n\tat org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:765)\n\tat org.apache.flink.runtime.taskmanager.Task.run(Task.java:577)\n\tat java.base/java.lan

asked a year ago301 views

1 Answer
0

The error you're encountering is related to a compatibility issue with batch processing in Amazon Managed Service for Apache Flink. The error message shows a UnsupportedOperationException occurring in the ResultPartition.getAllDataProcessedFuture method, which suggests an issue with how the batch job is being processed.

There are a few potential issues to address:

  1. Region Configuration: When working with S3 in opt-in AWS regions, you need to explicitly specify the region in your Flink configuration. Make sure you're setting the S3 endpoint region properly in your application configuration:
fs.s3a.endpoint.region: us-east-2
  1. Execution Environment: Your code appears to be using a batch processing approach with bounded sources (reading from S3 files), but you might be running it in a streaming context. For batch processing with Flink, consider using:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

instead of the streaming environment, or explicitly set the runtime mode to BATCH.

  1. Connector Configuration: Ensure you have the proper S3 connector dependencies and configurations. The S3 connector might need additional configuration parameters to work correctly in the managed environment.

  2. Application Status: Even when your application shows as "RUNNING" in the AWS console, it doesn't necessarily mean the job is healthy and processing data. The application might be stuck in a fail-and-restart loop, which can happen when a connector is misconfigured or there are networking issues preventing connection to external systems.

Try adding these configurations to your application and ensure you're using the appropriate execution environment for your batch processing job.
Sources
Deep dive into the Amazon Managed Service for Apache Flink application lifecycle – Part 2 | AWS Big Data Blog
Troubleshooting - Amazon EMR

answered a year ago

  • The same code worked fine when reading the file from the same input S3 Bucket and writing the file as-is to another S3 Bucket. But S3 Data Source to FlatMap appears to be causing the failure.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.