- Mais recentes
- Mais votos
- Mais comentários
Finally I found the cause. Although its very very deceptive.
Flink internally uses AWS SDK version 1.11.271
.
The class S3ErrorResponseHandler
which is causing NoClassDefFoundError...
has the following static variables.
public class S3ErrorResponseHandler implements
HttpResponseHandler<AmazonServiceException> {
/** Shared logger for profiling information */
private static final Log log = LogFactory
.getLog(S3ErrorResponseHandler.class);
/** Shared factory for creating XML event readers */
private static final XMLInputFactory xmlInputFactory = XMLInputFactory
.newInstance();
private static enum S3ErrorTags {
Error, Message, Code, RequestId, HostId
};
....
...
In AWS SDK 1.11.272
, the initialization of XMLInputFactory has been removed.
That gave me a clue.
I rebuilt flink library 1.7.1 by switching to 1.11.272
. Voila it started working.
This still posed me few unanswered questions.
I did thorough debugging within Flink JVM running in EMR.
The classpath clearly has flink-s3-fs-hadoop-1.7.1.jar
. I wrote a code to read that jar and print all its entries and I did see S3ErrorResponseHandler
.
In my flink operator, I could initialize as below -
XMLInputFactory xmlInputFactory = XMLInputFactory.newInstance()
The classloader clearly had reference to jre libs.
Yet, flink was unable to initialize this specific class. I wonder why!! Is it because of how Flink deals with operators!!!
Flink underneath serializes the operators and transmits to slave nodes. In slave nodes, the operators are deserialized, initialized and run as a task.
Between those vaarious phases, the Flink classloader somehow didnt have access to default implementation of XMLInputFactory from JRE. Thats very wierd!!! Also I wish JRE is more specific about which static variable it could not initialize when loading the class.
Should I call this as a bug in Flink?
No one has reported this error in AWS EMR when using Flink?
Also noted that flink release 1.7.1 has a bug with StreamingFlileSink. If your EMR cluster has 2 mounts, it tries do a read/write operation on both causing it to fail.
It was resolved in 1.8.0 and above. However AWS EMR still uses 1.7.1 release by default. So pls ensure you put 1.8.0 libraries under /usr/lib/flink/lib
Conteúdo relevante
- AWS OFICIALAtualizada há um ano
- AWS OFICIALAtualizada há 2 anos
- AWS OFICIALAtualizada há 2 anos