Problem installing MXNet.jl using Julia1.2 in UBUNTU 16.04

Question

I am having some trouble installing "MXNet.jl" package in julia.

1) I first tryed typing:

 (v1.2) pkg> add MXNet

and I get this:

(v1.2) pkg> add MXNet
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/environments/v1.2/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.2/Manifest.toml`
 [no changes]

At this point, nothing wrong was going on. There were no error messages thrown. Nice. Now, I went back to the Julia REPL and tried to use the module, for which I wrote:

julia> using MXNet
[ Info: Precompiling MXNet [a7949054-b901-59c6-b8e3-7238c29bf7f0]
ERROR: LoadError: LoadError: IOError: open: permission denied (EACCES)
Stacktrace:
 [1] uv_error at ./libuv.jl:90 [inlined]
 [2] open(::String, ::UInt16, ::UInt16) at ./filesystem.jl:81
 [3] touch(::String) at ./file.jl:404
 [4] top-level scope at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/base.jl:60
 [5] include at ./boot.jl:328 [inlined]
 [6] include_relative(::Module, ::String) at ./loading.jl:1094
 [7] include at ./Base.jl:31 [inlined]
 [8] include(::String) at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:25
 [9] top-level scope at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:138
 [10] include at ./boot.jl:328 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1094
 [12] include(::Module, ::String) at ./Base.jl:31
 [13] top-level scope at none:2
 [14] eval at ./boot.jl:330 [inlined]
 [15] eval(::Expr) at ./client.jl:432
 [16] top-level scope at ./none:3
in expression starting at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/base.jl:57
in expression starting at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:138
ERROR: Failed to precompile MXNet [a7949054-b901-59c6-b8e3-7238c29bf7f0] to /home/gbrunini/.julia/compiled/v1.2/MXNet/Sc0jU.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1253
 [3] _require(::Base.PkgId) at ./loading.jl:1013
 [4] require(::Base.PkgId) at ./loading.jl:911
 [5] require(::Module, ::Symbol) at ./loading.jl:906

and all this error messages where thrown. Knowing my limitations with regards to all this errors I went to my second attempt, which consisted in what follows...

2) My second approach consisted in using brute force, so I copied and paste all the github scr folder for the julia MXNet.jl package to a given folder in my personal Desktop. This was done subroutine by subroutine, folder by folder, until I copied every file and folder. So it is fair to say that the scr folder in this link: https://github.com/apache/incubator-mxnet/tree/master/julia/src looks excatly like mine now.

Now, I went to my folder and proceeded as if "MXNet.jl" were any other ordinary function I used in the past, so I type (On julia REPL):

julia> include("MXNet.jl")
ERROR: LoadError: LoadError: Cannot find or load libmxnet.so. Please see the document on how to build it.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] top-level scope at /home/gbrunini/Desktop/TRABAJOS/MTI/MTI_Neural_network/MTI_ejemplo_simple_3/base.jl:55
 [3] include at ./boot.jl:328 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1094
 [5] include at ./Base.jl:31 [inlined]
 [6] include(::String) at /home/gbrunini/Desktop/TRABAJOS/MTI/MTI_Neural_network/MTI_ejemplo_simple_3/MXNet.jl:25
 [7] top-level scope at /home/gbrunini/Desktop/TRABAJOS/MTI/MTI_Neural_network/MTI_ejemplo_simple_3/MXNet.jl:150
 [8] include at ./boot.jl:328 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1094
 [10] include(::Module, ::String) at ./Base.jl:31
 [11] include(::String) at ./client.jl:431
 [12] top-level scope at REPL[1]:1
in expression starting at /home/gbrunini/Desktop/TRABAJOS/MTI/MTI_Neural_network/MTI_ejemplo_simple_3/base.jl:51
in expression starting at /home/gbrunini/Desktop/TRABAJOS/MTI/MTI_Neural_network/MTI_ejemplo_simple_3/MXNet.jl:150

Which returned all this error message.

My assumption is that there is something wrong with this libmxnet.so (I don't know what that is...) and I am tempted to follow steps 1-3 on this documentation(MXNet istallation guide), but I dont want to screw things up as I really dont know what any of the following commands do:

sudo apt-get update
sudo apt-get install -y build-essential git ninja-build ccache libopenblas-dev libopencv-dev cmake 
git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
cd mxnet
cp config/linux.cmake config.cmake  # or config/linux_gpu.cmake for build with CUDA
rm -rf build
mkdir -p build && cd build
cmake -GNinja ..
cmake --build .

nor what any of the followings do for the Julia:

mkdir julia-depot
export JULIA_DEPOT_PATH=$HOME/julia/julia-depot

a) whats a depot? b) whats a environment variable (I looked online, and It confusses me...)

export MXNET_HOME=$HOME/incubator-mxnet
export LD_LIBRARY_PATH=$HOME/incubator-mxnet/lib:$LD_LIBRARY_PATH
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so:$LD_PRELOAD
export PATH=$HOME/bin:$HOME/.local/bin:$HOME/julia/julia-1.0.3/bin:$PATH
export JULIA_DEPOT_PATH=$HOME/julia/julia-depot
export MXNET_HOME=$HOME/incubator-mxnet
export LD_LIBRARY_PATH=$HOME/incubator-mxnet/lib:$LD_LIBRARY_PATH
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so:$LD_PRELOAD
julia --color=yes --project=./ -e \
      'using Pkg; \
       Pkg.develop(PackageSpec(name="MXNet", path = joinpath(ENV["MXNET_HOME"], "julia")))'

I really just want to install MXNet.jl and use it. If I understand all the rest as a side-efftect the better, but I settle with having all correctly installed, so for short:

1. What is the problem I am having?,
2. Do I fix it by just "copying and pasteing" the instructions in the documentation steps?
3. If not, how do I install MXNet.jl?,
4. Am I conffusing GPU with CPU(my option) installation? what changes between the both?. I don't think that has anything to do with what happens here, but....

Thanks a lot in advance I hope you are all doing Ok.

• UBUNTU 16.4

julia> versioninfo()
    Julia Version 1.2.0
    Commit c6da87ff4b (2019-08-20 00:03 UTC)
    Platform Info:
      OS: Linux (x86_64-pc-linux-gnu)
      CPU: Intel(R) Core(TM) i5-2310 CPU @ 2.90GHz
      WORD_SIZE: 64
      LIBM: libopenlibm
      LLVM: libLLVM-6.0.1 (ORCJIT, sandybridge)

EDIT:

Following @Przemyslaw I went and installed the lastest julia version, julia1.4. I was hopping that by doing this, the permissions for writing into .julia would be fixed. However, the problem remains. I still get the following messages:

(@v1.4) pkg> add MXNet
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
   Updating `~/.julia/environments/v1.4/Project.toml`
  [a7949054] + MXNet v1.5.0
   Updating `~/.julia/environments/v1.4/Manifest.toml`
  [9e28174c] + BinDeps v1.0.1
  [682c06a0] + JSON v0.21.0
  [a7949054] + MXNet v1.5.0
  [69de0a69] + Parsers v1.0.2
  [30578b45] + URIParser v0.4.1
  [8ba89e20] + Distributed 
  [a63ad114] + Mmap 
  [8dfed614] + Test 

julia> using MXNet
[ Info: Precompiling MXNet [a7949054-b901-59c6-b8e3-7238c29bf7f0]
ERROR: LoadError: LoadError: IOError: open: permission denied (EACCES)
Stacktrace:
 [1] uv_error at ./libuv.jl:97 [inlined]
 [2] open(::String, ::UInt16, ::UInt16) at ./filesystem.jl:87
 [3] touch(::String) at ./file.jl:422
 [4] top-level scope at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/base.jl:60
 [5] include(::Module, ::String) at ./Base.jl:377
 [6] include(::String) at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:25
 [7] top-level scope at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:138
 [8] include(::Module, ::String) at ./Base.jl:377
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:331 [inlined]
 [11] eval(::Expr) at ./client.jl:449
 [12] top-level scope at ./none:3
in expression starting at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/base.jl:57
in expression starting at /home/gbrunini/.julia/packages/MXNet/XoVCW/src/MXNet.jl:138
ERROR: Failed to precompile MXNet [a7949054-b901-59c6-b8e3-7238c29bf7f0] to /home/gbrunini/.julia/compiled/v1.4/MXNet/Sc0jU_ATtl5.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922

julia> versioninfo()
Julia Version 1.4.1
Commit 381693d3df* (2020-04-14 17:20 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-2310 CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, sandybridge)

So,

1. Why is installing a new julia not fixing the issue over .julia?
2. All other pakages I have install works fine, is MXNet.jl the only package that needs to write over .julia?
3. Problem seems related to the lines:

    if isempty(MXNET_LIB)
      # touch this file, so that after the user properly build libmxnet, the precompiled
      # MXNet.ji will be re-compiled to get MXNET_LIB properly.
      touch(@__FILE__)
      error("Cannot find or load libmxnet.$(Libdl.dlext). " *
            "Please see the document on how to build it.")
    else
      include_dependency(MXNET_LIB)
    end

in the base.jl subroutine of MXNet.jl. Any idea what this means?

Answer 1

The output of your add MXNet command shows that it is already installed.

However, it fails to precompile because your Julia process does not have sufficient permissions.

Your Julia needs to be able to write to /home/gbrunini/.julia/, otherwise the modules will not able to precompile and load. Try to set up permissions and rebuild the package with build MXNet

The above folder is your default JULIA_DEPOT_PATH. You can configure that variable to point store all Julia packages in a different location but this does not seem to be your problem (unless you are unable to configure permission for /home/gbrunini/.julia/ due some other reason).

The brute-force aproach is never a good way to go since Julia packages have many external dependencies. At Julia 1.2 they were managed by downloading the files when running package build (that is why you should rebuild). In the current Julia there is a more elegant solution with Julia artifacts.