While running following command on GPU server:
$HOROVOD_WITH_PYTORCH=1 HOROVOD_WITH_TENSORFLOW=1 python --no-cache-dir setup.py install --user
It's showing the following error:
running install
running bdist_egg
running egg_info
writing horovod.egg-info/PKG-INFO
writing dependency_links to horovod.egg-info/dependency_links.txt
writing requirements to horovod.egg-info/requires.txt
writing top-level names to horovod.egg-info/top_level.txt
reading manifest file 'horovod.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'third_party/eigen/Eigen/src/IterativeSolvers/*'
writing manifest file 'horovod.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
gcc -pthread -B /export/application/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -std=c++11 -fPIC -O2 -Wall -I/export/application/anaconda3/include/python3.7m -c build/temp.linux-x86_64-3.7/test_compile/test_cpp_flags.cc -o build/temp.linux-x86_64-3.7/test_compile/test_cpp_flags.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -shared -B /export/application/anaconda3/compiler_compat -L/export/application/anaconda3/lib -Wl,-rpath=/export/application/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/test_compile/test_cpp_flags.o -o build/temp.linux-x86_64-3.7/test_compile/test_cpp_flags.so
gcc -pthread -B /export/application/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/export/application/anaconda3/include/python3.7m -c build/temp.linux-x86_64-3.7/test_compile/test_link_flags.cc -o build/temp.linux-x86_64-3.7/test_compile/test_link_flags.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -shared -B /export/application/anaconda3/compiler_compat -L/export/application/anaconda3/lib -Wl,-rpath=/export/application/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ -Wl,--version-script=horovod.lds build/temp.linux-x86_64-3.7/test_compile/test_link_flags.o -o build/temp.linux-x86_64-3.7/test_compile/test_link_flags.so
error: import tensorflow failed, is it installed?
Traceback (most recent call last):
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/export/application/anaconda3/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/export/application/anaconda3/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 72, in check_tf_version
import tensorflow as tf
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/__init__.py", line 27, in <module>
from tensorflow._api.v2 import audio
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/_api/v2/audio/__init__.py", line 8, in <module>
from tensorflow.python.ops.gen_audio_ops import decode_wav
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/export/application/anaconda3/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/export/application/anaconda3/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /home/pg2019/cse/19071010/.local/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
I have tried installing tensorflow using following command: pip install --user tensorflow==1.14.0, and pip install --user tensorflow, and pip install --user tensorflow-gpu
I following this tutorial: https://github.com/horovod/horovod
I have tried the command below and it is running but while running
$pip install --no-cache-dir --user horovod
But if I run following program
$mpirun -np 4 python tensorflow2_mnist.py
as said on https://github.com/horovod/horovod/tree/master/examples it's showing as similar error:
Failed to load the native TensorFlow runtime.