Apache Flink app throws ClassNotFoundException for com.amazonaws.kinesis.analytics.s3credsproxy.ProxiedCredentialsProvider

Question

I have an Apache Flink application for 1.19 runtime with S3 integration. I use flink-s3-fs-hadoop-1.19.1.jar for sending files to S3.
JVM version 11.
Flink version 1.19

However, on starting this app in AWS runtime, I am receiving the following error:

```
Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class com.amazonaws.kinesis.analytics.s3credsproxy.ProxiedCredentialsProvider not found
```

I have specified `fs.s3a.aws.credentials.provider` property as `com.amazonaws.auth.DefaultAWSCredentialsProvider`

And I am unable to find ANY information about `ProxiedCredentialsProvider` class anywhere.

What could be the source of this issue?

Eric · Accepted Answer

It usually occurs when the code is trying to use a custom S3 connector or Hadoop libraries as it can conflict with libraries that we use for other S3 backend systems. One possible solution is checking the S3 sink example at reference [1] and compare how the dependencies are shaded in that application to yours, to see if there are any discrepancies there.

Reference: [1] - https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/S3Sink

Reference:
[1] - https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/S3Sink

Apache Flink app throws ClassNotFoundException for com.amazonaws.kinesis.analytics.s3credsproxy.ProxiedCredentialsProvider

Add your answer

Relevant content