native-lzo not available error | Windows 10 | Java

Question

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:61)
Caused by: java.lang.RuntimeException: native-lzo library not available
        at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
        at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.<init>(CodecFactory.java:144)
        at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:206)
        at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:189)
        at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:287)
        at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:569)

I am trying to use lzo compression codec in my parquetwriter. I don't need Hadoop, so I only have hadoop.dll and winutils.exe in my Hadoop_Home path. In my windows system, I tried building the lzo2.dll as given in documentation but don't know how to use it. I went through the twitter fork hadoop-lzo but the instructions were not clear for windows. Please tell me how to get this codec running in Windows. I downloaded hadoop-3.2.1, lzo-2.1.0, hadoop-lzo.

Using mvn clean install in hadoop-lzo gives me gplcompression.dll not found error.

[exec] "P:\hadoop\hadoop-lzo\src\main\native\gplcompression.sln" (default target) (1) ->
     [exec] "P:\hadoop\hadoop-lzo\src\main\native\gplcompression.vcxproj.metaproj" (default target) (2) ->
     [exec] (Build target) -> 
     [exec]   P:\hadoop\hadoop-lzo\src\main\native\gplcompression.vcxproj.metaproj : warning MSB4078: The project file "gplcompression.vcxproj" is not supported by MSBuild and cannot be built.
     [exec] 
     [exec]     1 Warning(s)
     [exec]     0 Error(s)
     [exec] 
     [exec] Time Elapsed 00:00:01.57
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.603 s
[INFO] Finished at: 2020-05-28T17:16:24+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (build-native-win) on project hadoop-lzo: An Ant BuildException has occured: Warning: Could not find file P:\hadoop\hadoop-lzo\target\native\Windows_NT-${env.PLATFORM}\gplcompression.dll to copy.
[ERROR] around Ant part ...<copy file="${build.native}/gplcompression.dll" todir="${build.native}/lib"/>... @ 19:80 in P:\hadoop\hadoop-lzo\target\antrun\build-build-native-win.xml
[ERROR] -> [Help 1]

Answer 1

It doesn't work in Windows. You get build errors at every step if you try to build Hadoop-LZO for windows. The issue came because I didn't include the correct build of Hadoop-LZO (twitter). LZO has some native (system-dependent) code and licensing issues so you need to build a specific Hadoop-LZO using LZO from "https://www.oberhumer.com/opensource/lzo/" as per instructions in "https://github.com/twitter/hadoop-lzo". I built it for linux and ran the code in linux and it worked. I was not able to build it successfully for Windows. I face all sorts of errors, I think there is a issue with the Windows native code in Hadoop-LZO.