I keep running into the error java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.<init>() even though I've created TextArrayWritable to implement it

I have a dataset of which each entry is in the form as: ((a,b,c),(d,e,f,g)). They are tuples in pyspark and saved to sequential files by:

output.saveAsSequenceFile(
        path=os.path.join(output_path, 'date=%s' % date),
        compressionCodecClass='org.apache.hadoop.io.compress.SnappyCodec'
    )

Now I want to load them in java by saying

JavaPairRDD<TextArrayWritable,TextArrayWritable> distFile = sc.sequenceFile(s3inputPath.toString(), TextArrayWritable.class, TextArrayWritable.class);

where TextArrayWritable heritages ArrayWritable, see below:

public static class TextArrayWritable extends ArrayWritable {
        public TextArrayWritable() {
            super(Text.class);
        }

        public TextArrayWritable(String[] strings) {
            super(Text.class);
            Text[] texts = new Text[strings.length];
            for (int i = 0; i < strings.length; i++) {
                texts[i] = new Text(strings[i]);
            }
            set(texts);
        }
    }

unfuntunately, I ran into the error saying java.lang.NoSuchMethodException: org.apache.hadoop.io.ArrayWritable.<init>()

can anyone help me out with it?

Thanks!!!

0 Answers