MacOS High Sierra.
After compilation, syntaxnet works properly on this sentence:
echo "This took a long time" | syntaxnet/demo.sh
But it crashes python on this sentence:
echo "This took a long time" | syntaxnet/demo.sh --conll
The crash log file shows that one thread was processing the conll converter and died. The most relevant lines from the apple crash report are:
Thread 11 Crashed:
0 libsystem_kernel.dylib 0x00007fff7d102fce __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fff7d240150 pthread_kill + 333
2 libsystem_c.dylib 0x00007fff7d05f30a abort + 127
3 _pywrap_tensorflow_internal.so 0x000000011267edc0 tensorflow::internal::LogMessageFatal::~LogMessageFatal() + 32
4 _pywrap_tensorflow_internal.so 0x000000011267edd0 tensorflow::internal::LogMessageFatal::~LogMessageFatal() + 16
5 parser_ops.so 0x0000000118d87468 syntaxnet::CoNLLSyntaxFormat::ConvertFromString(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<syntaxnet::Sentence*, std::__1::allocator<syntaxnet::Sentence*> >*) + 2776
6 parser_ops.so 0x0000000118d7b740 syntaxnet::TextReader::Read() + 400
7 parser_ops.so 0x0000000118da6cfd syntaxnet::SentenceBatch::AdvanceSentence(int) + 61
8 parser_ops.so 0x0000000118d94a0f syntaxnet::BeamState::AdvanceSentence() + 63
9 parser_ops.so 0x0000000118d94876 syntaxnet::BeamState::Reset() + 38
10 parser_ops.so 0x0000000118d93ebd syntaxnet::BatchState::ResetBeams() + 189
11 parser_ops.so 0x0000000118d8fe55 syntaxnet::BeamParseReader::Compute(tensorflow::OpKernelContext*) + 69
12 _pywrap_tensorflow_internal.so 0x00000001123bf83c tensorflow::ThreadPoolDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) + 348
The entire apple crash log file is below. The script used to try to compile syntaxnet in a regular controllable way on Mac is below. The output of this script is below as well.
Does anyone have any idea how to solve this syntaxnet-python crash?
Compiling and running syntaxnet on Mac continues to be challenging for most people, and impossible for many, outside of a Docker file which limits flexibility. Let me know if anyone can solve this specific problem. Thanks!
--- original question below ---
Many, many bright people have spent many, many hours trying to compile syntaxnet on Mac, and given up. I'll try to start from scratch, and if we can identify bugs, maybe we can get a better set of instructions than the highly incomplete ones given here:
https://github.com/tensorflow/models/tree/master/research/syntaxnet
FRESH START
So these instructions, judging by public comment, are misleadingly incomplete. Let's start a shell script to do these, wiping everything clean. This isn't for general execution—it is cobbled together and very unwisely written—it's just so people can help by looking at the output.
Here is what I'm working with:
#!/usr/bin/env bash
# SYNTAXNET INSTALLER TEST FOR MAC (mac only!)
# DO NOT RUN THIS UNLESS YOU KNOW WHAT YOU ARE DOING.
# DO NOT RUN THIS UNLESS YOU UNDERSTAND EACH LINE OF THE CODE BELOW.
# THIS CAN DELETE IMPORTANT FILES AND CHANGE YOUR SYSTEM CONFIGURATION.
# to run this, put the shell script in the location you want syntaxnet files to go
# make sure your pwd is that directory, and then:
# bash -x ./syxnt.sh
# or, to both view stdout and save to a file:
# bash -x ./syxnt.sh 2>&1 | tee syxnt.out
echo "STOP NOW."
echo "YOU PROBABLY DONT KNOW WHAT YOURE DOING."
echo "DONT RUN THIS."
echo "HIT CTRL-C."
date ; read -t 60 -p "Hit ENTER or wait sixty seconds to begin; Hit Ctrl-C to stop." ; echo ; date
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
echo "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
echo "#########################################################################"
echo "starting ################################################################"
echo "#########################################################################"
pwd
# xcode-select --install
rm -rf models
export PATH="/usr/local/opt/python/libexec/bin:$PATH"
export PATH="/usr/local/bin:/usr/local/lib:$PATH"
echo $PATH
echo $PYTHONPATH
echo "STEP 0 ################################################################"
echo "not listed in instructions; helpful for fresh start"
echo "only include these if you are finding errors with permissions"
echo "these are overbroad deletions, but desperate times..."
sudo rm -rf /usr/local/lib/python2.7/site-packages/tensorboard*
sudo rm -rf /usr/local/lib/python2.7/site-packages/tensorflow*
sudo rm -rf /usr/local/lib/python2.7/site-packages/mock*
sudo rm -rf /usr/local/lib/python2.7/site-packages/funcsigs*
sudo rm -rf /usr/local/lib/python2.7/site-packages/six*
sudo rm -rf /usr/local/lib/python2.7/site-packages/pbr*
pip freeze | grep tensorflow
pip uninstall -y tensorflow
pip uninstall -y tensorflow-tensorboard
pip freeze | grep tensorflow
rm -rf /private/var/tmp/_bazel*
rm -rf /tmp/syntaxnet_pkg*
rm ./bazel-0.5.4-installer-darwin-x86_64.sh
rm -fr ~/.bazel ~/.bazelrc ~/.cache/bazel
echo "brew stuff ################################################################"
brew upgrade
brew cleanup
brew doctor
brew prune
echo "STEP 1 ################################################################"
type python
which python
python --version
echo "STEP 2 ################################################################"
# brew install bazel
# cant use the above line: installs most recent version, not the required 0.5.4
which bazel
rmb="$(which bazel)"
rm $rmb
rmb="$(which bazel)"
rm $rmb
which bazel
brew uninstall bazel --force
which bazel
mdfind -name bazel
curl -L -o ./bazel-0.5.4-installer-darwin-x86_64.sh 'https://github.com/bazelbuild/bazel/releases/download/0.5.4/bazel-0.5.4-installer-darwin-x86_64.sh'
chmod +x bazel-0.5.4-installer-darwin-x86_64.sh
./bazel-0.5.4-installer-darwin-x86_64.sh --user
export PATH="$PATH:$HOME/bin"
type bazel
which bazel
bazel version
echo "STEP 3 ################################################################"
type swig
which swig
brew uninstall swig --force
which swig
brew install swig
type swig
which swig
swig -version
echo "STEP 4 ################################################################"
pip freeze | grep protobuf
pip uninstall -y protobuf
# do we need --upgrade --force-reinstall here?
pip freeze | grep protobuf
pip install -U protobuf==3.3.0
pip freeze | grep protobuf
echo "STEP 5 ################################################################"
pip freeze | grep mock
pip uninstall -y mock
pip freeze | grep mock
pip install --upgrade --force-reinstall mock
pip freeze | grep mock
echo "STEP 6 ################################################################"
pip freeze | grep asciitree
pip uninstall -y asciitree
pip freeze | grep asciitree
pip install --upgrade --force-reinstall asciitree
pip freeze | grep asciitree
echo "STEP 7 ################################################################"
pip freeze | grep numpy
pip uninstall -y numpy
pip freeze | grep numpy
pip install --upgrade --force-reinstall numpy
pip freeze | grep numpy
echo "STEP 8 ################################################################"
pip freeze | grep autograd
pip uninstall -y autograd
pip freeze | grep autograd
# do we need --upgrade --force-reinstall here?
pip install autograd==1.1.13
pip freeze | grep autograd
echo "STEP 9 ################################################################"
# note that this was probably uninstalled above as part of killing autograd
# so dont worry about it not showing up in freeze or the initial error re uninstall
pip freeze | grep pygraphviz
pip uninstall -y pygraphviz
pip freeze | grep pygraphviz
pip install pygraphviz --install-option="--include-path=/usr/include/graphviz" --install-option="--library-path=/usr/lib/graphviz/"
pip freeze | grep pygraphviz
################################################################################
################################################################################
################################################################################
pwd
mdfind -name tensorflow
mdfind -name bazel
echo "STEP A ################################################################"
git clone --recursive https://github.com/tensorflow/models.git
echo "STEP B ################################################################"
cd models/research/syntaxnet/tensorflow
echo "STEP C ################################################################"
# youll need to push RETURN ten or eleven times here, depending on where your files are located
# feed return times 11 to this via echo pipe
# cleaning is not listed in the steps, but it might help
bazel clean
bazel clean --expunge
./configure
echo "STEP D ################################################################"
cd ..
echo "STEP E ################################################################"
echo "not done -- only for windows/linux"
date
echo "STEP F ################################################################"
# should probably verify mutex-thread warnings etc are not of interest here and suppress them
bazel test --linkopt=-headerpad_max_install_names dragnn/... syntaxnet/... util/utf8/...
date
################################################################################
################################################################################
################################################################################
echo "STEP I ################################################################"
mkdir /tmp/syntaxnet_pkg
echo "STEP II ################################################################"
bazel-bin/dragnn/tools/build_pip_package --output-dir=/tmp/syntaxnet_pkg
echo "STEP III ################################################################"
tswhl="$(ls /tmp/syntaxnet_pkg/syntax*.whl)"
# sudo pip install /tmp/syntaxnet_pkg/syntaxnet-x.xx-none-any.whl
sudo pip install $tswhl
################################################################################
################################################################################
################################################################################
echo "STEP TESTING ################################################################"
chmod u+x syntaxnet/demo.sh
echo "This took a long time" | syntaxnet/demo.sh
echo "This took a long time" | syntaxnet/demo.sh --conll
################################################################################
################################################################################
################################################################################
popd
This works, FOR ME ONLY, up to the --conll line (the very very last executable line above popd, above). Remarkably, it does the simple test without conll formatting; then the conll formatting crashes python. The abbreviated and full too-much-information crash report looks like this:
abbreviated: https://pastebin.com/AUvFQiEJ
full TMI: https://pastebin.com/FQrLLKY1
The full stdout file is this; only the last few lines, plus all the setup stuff at the very top, are really of great interest:
If anyone can either (i) identify this python conll crash or (ii) offer any suggestions as to other ways to force clean the system, and reset things, and get a reliable compilation procedure, that would be appreciated.
If there are requests or queries about diagnostics or cleanup (like rerunning this with mutex warnings muted), I am glad to oblige.
[Edited to use pastebin instead of file.io.]