MapReduce Avro Output is Creating Text File Instead

Question

I have a MapReduce job that reads in avro data and then is supposed to output avro data. However, when I check the output files when the job succeeds, they do not have the .avro extension and I am able to view them with simple text editors.

I have my Driver configured to output avro so I'm not sure what the problem is and any help would be greatly appreciated.

Here is my Driver class:

public class Driver extends Configured implements Tool{

public static void main(String[] args) throws Exception {
    int res =
            ToolRunner.run(new Configuration(), new Driver(), args);
    System.exit(res);
}

@Override
public int run(String[] args) throws Exception {
    Job job = new Job(getConf());
    job.setJarByClass(Driver.class);
    job.setJobName("nearestpatient");


    AvroJob.setOutputKeySchema(job, Pair.getPairSchema(Schema.create(Schema.Type.LONG), Schema.create(Schema.Type.STRING)));
    job.setOutputValueClass(NullWritable.class);

    job.setMapperClass(PatientMapper.class);
    job.setReducerClass(PatientReducer.class);

    job.setInputFormatClass(AvroKeyInputFormat.class);
    AvroJob.setInputKeySchema(job, PatientAvro.getClassSchema());

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(LongWritable.class);


    FileInputFormat.setInputPaths(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));

    job.waitForCompletion(true);

    return 0;
}
}

And here is my Reducer class:

public class PatientReducer extends Reducer<LongWritable, LongWritable, AvroWrapper<Pair<Long, String>>, NullWritable> {

    @Override
    public void reduce(LongWritable providerKey, Iterable<LongWritable> patients, Context context) throws IOException, InterruptedException {

        String outputList = "[";
   `enter code here` List<Long> patientList = new ArrayList<>();
    for (LongWritable patientKey : patients) {
        outputList += new LongWritable(patientKey.get()) + ", ";
    }
    outputList = outputList.substring(0, outputList.length() - 2);
    outputList += "]";
    context.write(new AvroWrapper<Pair<Long, String>>(new Pair<Long, String> (providerKey.get(), outputList)), NullWritable.get());
}
}

Answer 1

In your run() method you need to add the following

job.setOutputFormatClass(AvroKeyValueOutputFormat.class);

Answer 2

In your code replace line

FileOutputFormat.setOutputPath(job, new Path(args[1]));

with

job.setOutputFormatClass(AvroKeyOutputFormat.class);
AvroKeyOutputFormat.setOutputPath(job, new Path(args[1]));