Mahout 0.13.0 spark-shell examples fails with "no jniViennaCL in java.library.path"

Question

I'm Trying to make Mahout 0.13.0 works with spark 1.6.3,

• I already have spark 1.6.3 and Hadoop 2.7 working.

• I download the last build from the homepage mahout_download.

• Unpackage on /opt/mahout.

• try to execute the example on spark-shell from the web.

Enter on the scala prompt

MASTER=local mahout spark-shell

Execute:

:load /opt/mahout/examples/bin/SparseSparseDrmTimer.mscala
timeSparseDRMMMul(1000,1000,1000,1,.02,1234L)

And I get thw follow error:

18/08/08 15:22:23 ERROR executor.Executor: Exception in task 0.0 in stage 3.0 (TID 3)
java.lang.UnsatisfiedLinkError: no jniViennaCL in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1122)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:726)
    at org.bytedeco.javacpp.Loader.load(Loader.java:501)
    at org.bytedeco.javacpp.Loader.load(Loader.java:434)
    at org.apache.mahout.viennacl.opencl.javacpp.Context$.loadLib(Context.scala:63)
    at org.apache.mahout.viennacl.opencl.javacpp.Context$.<init>(Context.scala:65)
    at org.apache.mahout.viennacl.opencl.javacpp.Context$.<clinit>(Context.scala)
    at org.apache.mahout.viennacl.opencl.GPUMMul$.org$apache$mahout$viennacl$opencl$GPUMMul$$gpuSparseRWRW(GPUMMul.scala:282)
    at org.apache.mahout.viennacl.opencl.GPUMMul$.org$apache$mahout$viennacl$opencl$GPUMMul$$jvmSparseRWCW(GPUMMul.scala:343)
    at org.apache.mahout.viennacl.opencl.GPUMMul$$anonfun$18.apply(GPUMMul.scala:86)
    at org.apache.mahout.viennacl.opencl.GPUMMul$$anonfun$18.apply(GPUMMul.scala:86)
    at org.apache.mahout.viennacl.opencl.GPUMMul$.apply(GPUMMul.scala:127)
    at org.apache.mahout.viennacl.opencl.GPUMMul$.apply(GPUMMul.scala:33)
    at org.apache.mahout.math.scalabindings.RLikeMatrixOps.$percent$times$percent(RLikeMatrixOps.scala:37)
    at org.apache.mahout.sparkbindings.blas.ABt$.org$apache$mahout$sparkbindings$blas$ABt$$mmulFunc$1(ABt.scala:98)
    at org.apache.mahout.sparkbindings.blas.ABt$$anonfun$6.apply(ABt.scala:113)
    at org.apache.mahout.sparkbindings.blas.ABt$$anonfun$6.apply(ABt.scala:113)
    at org.apache.mahout.sparkbindings.blas.ABt$$anonfun$pairwiseApply$1.apply(ABt.scala:209)
    at org.apache.mahout.sparkbindings.blas.ABt$$anonfun$pairwiseApply$1.apply(ABt.scala:209)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsatisfiedLinkError: /tmp/javacpp106197113868261/libjniViennaCL.so: libOpenCL.so.1: cannot open shared object file: No such file or directory
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:709)
    ... 27 more

After a search for the problem, I install the ViennaCL libraries as is showed on the web.

mkdir ~/tmp
cd ~/tmp && git clone https://github.com/viennacl/viennacl-dev.git
cp -r viennacl/ /usr/local/
cp -r CL/ /usr/local/

Also, I try with copying on the path: /usr/local/include/

But nothing change...

Answer 1

When working with ViennaCL, you must build from source. Please try pulling the mahout 0.13.0 tag and build from source (with the -Pviennacl profile activated).