I am trying to use Distcp Java API to do copy data from one hadoop cluster to another cluster.
However I am getting the following exception:
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.StringUtils.toLowerCase(Ljava/lang/String;)Ljava/lang/String;
at org.apache.hadoop.tools.util.DistCpUtils.getStrategy(DistCpUtils.java:126)
at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:235)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
at com.monitor.BackupUtil.doBackup(BackupUtil.java:72)
at com.monitor.BackupUtil.main(BackupUtil.java:45)
I am using the following code:
public void doBackup() throws Exception {
System.out.println("Beginning Distcp");
DistCpOptions options = new DistCpOptions(
new Path(prop.getProperty("sourceClusterDirectory") + "/" + prop.getProperty("tablename")
+ "/distcp.txt"),
new Path(prop.getProperty("targetCluster") + prop.getProperty("targetClusterDirectory")));
System.out.println("Disctp between--->" + prop.getProperty("sourceClusterDirectory")+ "/distcp.txt" + "AND" + prop.getProperty("targetCluster")
+ prop.getProperty("targetClusterDirectory"));
DistCp distcp = new DistCp(new Configuration(), options);
Job job = distcp.execute();
job.waitForCompletion(true);
System.out.println("DistCp Completed Successfully");
}
I am using hadoop 2.7.1 and the distcp dependency is this:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-distcp</artifactId>
<version>2.7.1</version>
</dependency>