How to compile Apache Arrow Flight C++ example in Windows-10 (VS 2019)

1.9k views Asked by At

I downloaded the Apache Arrow Flight sample code from git (https://github.com/apache/arrow) and I was able to build the cpp example (using CMake and VS2019). I could not build the example "minimal_build" because it gives CMake missing library errors. The package-links given in cmake files are incorrect. When I change them individually then that package compiles and the next-one breaks.

Command Line Output :

D:\Git\Arrow-Master\cpp\examples\minimal_build>SET CMAKE_MODULE_PATH=D:\Git\Arrow-Master\cpp\cmake_modules

D:\Git\Arrow-Master\cpp\examples\minimal_build>SET Arrow_DIR=D:\Git\Arrow-Master\cpp\build\release\Release

D:\Git\Arrow-Master\cpp\examples\minimal_build>run_static.bat

D:\Git\Arrow-Master\cpp\examples\minimal_build>if exist "arrow-build" rd arrow-build /s /q

D:\Git\Arrow-Master\cpp\examples\minimal_build>if exist "dist" rd dist /s /q

D:\Git\Arrow-Master\cpp\examples\minimal_build>if exist "example" rd example /s /q

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "== Building Arrow C++ library"
"== Building Arrow C++ library"

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>set INSTALL_PREFIX=D:\Git\Arrow-Master\cpp\examples\minimal_build\dist

D:\Git\Arrow-Master\cpp\examples\minimal_build>mkdir arrow-build

D:\Git\Arrow-Master\cpp\examples\minimal_build>pushd arrow-build

D:\Git\Arrow-Master\cpp\examples\minimal_build\arrow-build>cmake ..\..\..     -GNinja     -DCMAKE_INSTALL_PREFIX=D:\Git\Arrow-Master\cpp\examples\minimal_build\dist     -DARROW_DEPENDENCY_SOURCE=BUNDLED     -DARROW_BUILD_SHARED=OFF     -DARROW_BUILD_STATIC=ON     -DARROW_COMPUTE=ON     -DARROW_CSV=ON     -DARROW_DATASET=ON     -DARROW_FILESYSTEM=ON     -DARROW_HDFS=ON     -DARROW_JSON=ON     -DARROW_MIMALLOC=ON     -DARROW_ORC=ON     -DARROW_PARQUET=ON     -DARROW_PLASMA=ON     -DARROW_WITH_BROTLI=ON     -DARROW_WITH_BZ2=OFF     -DARROW_WITH_LZ4=ON     -DARROW_WITH_SNAPPY=ON     -DARROW_WITH_ZLIB=ON     -DARROW_WITH_ZSTD=ON
-- Building using CMake version: 3.20.0-rc3
-- The C compiler identification is Clang 11.0.0 with GNU-like command-line
-- The CXX compiler identification is Clang 11.0.0 with GNU-like command-line
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/Llvm/bin/clang.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/Llvm/bin/clang++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Arrow version: 4.0.0 (full: '4.0.0-SNAPSHOT')
-- Arrow SO version: 400 (full: 400.0.0)
-- clang-tidy not found
-- clang-format not found
-- Could NOT find ClangTools (missing: CLANG_FORMAT_BIN CLANG_TIDY_BIN)
-- infer not found
-- Found Python3: C:/Users/amishra2/AppData/Local/Programs/Python/Python39/python.exe (found version "3.9.1") found components: Interpreter 
-- Found cpplint executable at D:/Git/Arrow-Master/cpp/build-support/cpplint.py
-- System processor: AMD64
-- Performing Test CXX_SUPPORTS_SSE4_2
-- Performing Test CXX_SUPPORTS_SSE4_2 - Success
-- Performing Test CXX_SUPPORTS_AVX2
-- Performing Test CXX_SUPPORTS_AVX2 - Success
-- Performing Test CXX_SUPPORTS_AVX512
-- Performing Test CXX_SUPPORTS_AVX512 - Success
-- Performing Test CXX_SUPPORTS_BIG_OBJ
-- Performing Test CXX_SUPPORTS_BIG_OBJ - Success
-- Arrow build warning level: PRODUCTION
Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Using BUNDLED approach to find dependencies
-- ARROW_ABSL_BUILD_VERSION: 0f3bb466b868b523cf1dc9b2aaaed65c77b28862
-- ARROW_AWSSDK_BUILD_VERSION: 1.8.133

**** more and more logs **** 

-- ARROW_ZSTD_BUILD_VERSION: v1.4.8
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Looking for __SIZEOF_INT128__
-- Looking for __SIZEOF_INT128__ - not found
-- Boost include dir: D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/boost_ep-prefix/src/boost_ep
-- Boost libraries: boost_system_static;boost_filesystem_static
-- Building snappy from source
-- Building brotli from source
-- Building without OpenSSL support. Minimum OpenSSL version 1.0.2 required.
-- Building gflags from source
-- Added static library dependency gflags_static: D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/gflags_ep-prefix/src/gflags_ep/lib/libgflags.a
Building Apache Thrift from source
-- Building (vendored) mimalloc from source
-- Building RapidJSON from source
-- Building xsimd from source
-- Building ZLIB from source
-- Building lz4 from source
-- Building zstd from source
-- Building RE2 from source
-- Building utf8proc from source
-- Found hdfs.h at: D:/Git/Arrow-Master/cpp/thirdparty/hadoop/include/hdfs.h
-- All bundled static libraries: Snappy::snappy;Brotli::brotlicommon;Brotli::brotlienc;Brotli::brotlidec;gflags_static;thrift::thrift;mimalloc::mimalloc;ZLIB::ZLIB;LZ4::lz4;zstd::libzstd;re2::re2;utf8proc::utf8proc
-- CMAKE_C_FLAGS:  -Qunused-arguments -O3 -DNDEBUG  -Wa,-mbig-obj -Wall -Wno-unknown-warning-option -Wno-pass-failed -msse4.2
-- CMAKE_CXX_FLAGS:   -Qunused-arguments -fcolor-diagnostics -O3 -DNDEBUG  -Wa,-mbig-obj -Wall -Wno-unknown-warning-option -Wno-pass-failed -msse4.2
-- Looking for backtrace
-- Looking for backtrace - not found
-- Could NOT find Backtrace (missing: Backtrace_LIBRARY Backtrace_INCLUDE_DIR)
-- Creating bundled static library target arrow_bundled_dependencies at D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/release/arrow_bundled_dependencies.lib
-- -------------
-- Arrow version:                                 4.0.0-SNAPSHOT
--
-- Build configuration summary:
--   Generator: Ninja
--   Build type: RELEASE
--   Source directory: D:/Git/Arrow-Master/cpp
--   Install prefix: D:/Git/Arrow-Master/cpp/examples/minimal_build/dist
--
-- Compile and link options:
--
--   ARROW_CXXFLAGS="" [default=""]
--       Compiler flags to append when compiling Arrow
--   ARROW_BUILD_STATIC=ON [default=ON]
--       Build static libraries
--   ARROW_BUILD_SHARED=OFF [default=ON]
--       Build shared libraries
--   ARROW_PACKAGE_KIND="" [default=""]
--       Arbitrary string that identifies the kind of package
--       (for informational purposes)
--   ARROW_GIT_ID=178c7ddbfa015e2418df8c6729a16cad0bf24b9b [default=""]
--       The Arrow git commit id (if any)

**** more and more logs **** 

-- Project component options:
--
--   ARROW_BUILD_UTILITIES=OFF [default=OFF]
--       Build Arrow commandline utilities
--   ARROW_COMPUTE=ON [default=OFF]
--       Build the Arrow Compute Modules
--   ARROW_CSV=ON [default=OFF]
--       Build the Arrow CSV Parser Module
--   ARROW_CUDA=OFF [default=OFF]
--       Build the Arrow CUDA extensions (requires CUDA toolkit)
--   ARROW_DATASET=ON [default=OFF]
--       Build the Arrow Dataset Modules
--   ARROW_FILESYSTEM=ON [default=OFF]
--       Build the Arrow Filesystem Layer
--   ARROW_FLIGHT=OFF [default=OFF]
--       Build the Arrow Flight RPC System (requires GRPC, Protocol Buffers)
--   ARROW_GANDIVA=OFF [default=OFF]
--       Build the Gandiva libraries
--   ARROW_HDFS=ON [default=OFF]
--       Build the Arrow HDFS bridge
--   ARROW_HIVESERVER2=OFF [default=OFF]
--       Build the HiveServer2 client and Arrow adapter
--   ARROW_IPC=ON [default=ON]
--       Build the Arrow IPC extensions
--   ARROW_JEMALLOC=OFF [default=OFF]
--       Build the Arrow jemalloc-based allocator
--   ARROW_JNI=OFF [default=OFF]
--       Build the Arrow JNI lib
--   ARROW_JSON=ON [default=OFF]
--       Build Arrow with JSON support (requires RapidJSON)
--   ARROW_MIMALLOC=ON [default=OFF]
--       Build the Arrow mimalloc-based allocator
--   ARROW_PARQUET=ON [default=OFF]
--       Build the Parquet libraries
--   ARROW_ORC=OFF [default=OFF]
--       Build the Arrow ORC adapter
--   ARROW_PLASMA=ON [default=OFF]
--       Build the plasma object store along with Arrow
--   ARROW_PLASMA_JAVA_CLIENT=OFF [default=OFF]
--       Build the plasma object store java client
--   ARROW_PYTHON=OFF [default=OFF]
--       Build the Arrow CPython extensions
--   ARROW_S3=OFF [default=OFF]
--       Build Arrow with S3 support (requires the AWS SDK for C++)
--   ARROW_TENSORFLOW=OFF [default=OFF]
--       Build Arrow with TensorFlow support enabled
--   ARROW_TESTING=OFF [default=OFF]
--       Build the Arrow testing libraries
--
-- Thirdparty toolchain options:
--
--   ARROW_DEPENDENCY_SOURCE=BUNDLED [default=AUTO|BUNDLED|SYSTEM|CONDA|VCPKG|BREW]
--       Method to use for acquiring arrow's build dependencies
--   ARROW_VERBOSE_THIRDPARTY_BUILD=OFF [default=OFF]
--       Show output from ExternalProjects rather than just logging to files
--   ARROW_DEPENDENCY_USE_SHARED=ON [default=ON]
--       Link to shared libraries
--   ARROW_BOOST_USE_SHARED=ON [default=ON]
--       Rely on boost shared libraries where relevant
--   ARROW_BROTLI_USE_SHARED=ON [default=ON]
--       Rely on Brotli shared libraries where relevant
--   ARROW_BZ2_USE_SHARED=ON [default=ON]
--       Rely on Bz2 shared libraries where relevant
--   ARROW_GFLAGS_USE_SHARED=ON [default=ON]
--       Rely on GFlags shared libraries where relevant
--   ARROW_GRPC_USE_SHARED=ON [default=ON]
--       Rely on gRPC shared libraries where relevant
--   ARROW_LZ4_USE_SHARED=ON [default=ON]
--       Rely on lz4 shared libraries where relevant
--   ARROW_OPENSSL_USE_SHARED=ON [default=ON]
--       Rely on OpenSSL shared libraries where relevant
--   ARROW_PROTOBUF_USE_SHARED=ON [default=ON]
--       Rely on Protocol Buffers shared libraries where relevant
--   ARROW_THRIFT_USE_SHARED=OFF [default=OFF]
--       Rely on thrift shared libraries where relevant
--   ARROW_UTF8PROC_USE_SHARED=ON [default=ON]
--       Rely on utf8proc shared libraries where relevant
--   ARROW_SNAPPY_USE_SHARED=ON [default=ON]
--       Rely on snappy shared libraries where relevant
--   ARROW_UTF8PROC_USE_SHARED=ON [default=ON]
--       Rely on utf8proc shared libraries where relevant
--   ARROW_ZSTD_USE_SHARED=ON [default=ON]
--       Rely on zstd shared libraries where relevant
--   ARROW_USE_GLOG=OFF [default=OFF]
--       Build libraries with glog support for pluggable logging
--   ARROW_WITH_BACKTRACE=ON [default=ON]
--       Build with backtrace support
--   ARROW_WITH_BROTLI=ON [default=OFF]
--       Build with Brotli compression
--   ARROW_WITH_BZ2=OFF [default=OFF]
--       Build with BZ2 compression
--   ARROW_WITH_LZ4=ON [default=OFF]
--       Build with lz4 compression
--   ARROW_WITH_SNAPPY=ON [default=OFF]
--       Build with Snappy compression
--   ARROW_WITH_ZLIB=ON [default=OFF]
--       Build with zlib compression
--   ARROW_WITH_ZSTD=ON [default=OFF]
--       Build with zstd compression
--   ARROW_WITH_UTF8PROC=ON [default=ON]
--       Build with support for Unicode properties using the utf8proc library
--       (only used if ARROW_COMPUTE is ON)
--   ARROW_WITH_RE2=ON [default=ON]
--       Build with support for regular expressions using the re2 library
--       (only used if ARROW_COMPUTE or ARROW_GANDIVA is ON)
--
-- MSVC options:
--
--   MSVC_LINK_VERBOSE=OFF [default=OFF]
--       Pass verbose linking options when linking libraries and executables
--   BROTLI_MSVC_STATIC_LIB_SUFFIX=-static [default="-static"]
--       Brotli static lib suffix used on Windows with MSVC
--   PROTOBUF_MSVC_STATIC_LIB_SUFFIX="" [default=""]
--       Protobuf static lib suffix used on Windows with MSVC
--   RE2_MSVC_STATIC_LIB_SUFFIX=_static [default="_static"]
--       re2 static lib suffix used on Windows with MSVC
--   SNAPPY_MSVC_STATIC_LIB_SUFFIX="" [default=""]
--       Snappy static lib suffix used on Windows with MSVC
--   LZ4_MSVC_STATIC_LIB_SUFFIX=_static [default="_static"]
--       Lz4 static lib suffix used on Windows with MSVC
--   ZSTD_MSVC_STATIC_LIB_SUFFIX=_static [default="_static"]
--       ZStd static lib suffix used on Windows with MSVC
--   ARROW_USE_STATIC_CRT=OFF [default=OFF]
--       Build Arrow with statically linked CRT
--
-- Parquet options:
--
--   PARQUET_MINIMAL_DEPENDENCY=OFF [default=OFF]
--       Depend only on Thirdparty headers to build libparquet.
--       Always OFF if building binaries
--   PARQUET_BUILD_EXECUTABLES=OFF [default=OFF]
--       Build the Parquet executable CLI tools. Requires static libraries to be built.
--   PARQUET_BUILD_EXAMPLES=OFF [default=OFF]
--       Build the Parquet examples. Requires static libraries to be built.
--   PARQUET_REQUIRE_ENCRYPTION=OFF [default=OFF]
--       Build support for encryption. Fail if OpenSSL is not found
--
-- Gandiva options:
--
--   ARROW_GANDIVA_JAVA=OFF [default=OFF]
--       Build the Gandiva JNI wrappers
--   ARROW_GANDIVA_STATIC_LIBSTDCPP=OFF [default=OFF]
--       Include -static-libstdc++ -static-libgcc when linking with
--       Gandiva static libraries
--   ARROW_GANDIVA_PC_CXX_FLAGS="" [default=""]
--       Compiler flags to append when pre-compiling Gandiva operations
--
-- Advanced developer options:
--
--   ARROW_EXTRA_ERROR_CONTEXT=OFF [default=OFF]
--       Compile with extra error context (line numbers, code)
--   ARROW_OPTIONAL_INSTALL=OFF [default=OFF]
--       If enabled install ONLY targets that have already been built. Please be
--       advised that if this is enabled 'install' will fail silently on components
--       that have not been built
--   Outputting build configuration summary to D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/cmake_summary.json
-- Configuring done
-- Generating done
-- Build files have been written to: D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build

D:\Git\Arrow-Master\cpp\examples\minimal_build\arrow-build>ninja install
[20/335] Performing build step for 'lz4_ep'
FAILED: lz4_ep-prefix/src/lz4_ep-stamp/lz4_ep-build lz4_ep-prefix/src/lz4_ep/lib/liblz4.a
cmd.exe /C "cd /D D:\Git\Arrow-Master\cpp\examples\minimal_build\arrow-build\lz4_ep-prefix\src\lz4_ep && "C:\Program Files\CMake\bin\cmake.exe" -P D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/lz4_ep-prefix/tmp/lz4_ep-build-RELEASE.cmake && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/lz4_ep-prefix/src/lz4_ep-stamp/lz4_ep-build"
CMake Error at D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/lz4_ep-prefix/tmp/lz4_ep-build-RELEASE.cmake:37 (message):
  Command failed: %1 is not a valid Win32 application

   'D:/Git/Arrow-Master/cpp/build-support/build-lz4-lib.sh' 'AR=C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/Llvm/bin/llvm-ar.exe' 'OS=Windows'

  See also

    D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/lz4_ep-prefix/src/lz4_ep-stamp/lz4_ep-build-*.log


-- stdout output is:

-- stderr output is:

CMake Error at D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/lz4_ep-prefix/tmp/lz4_ep-build-RELEASE.cmake:47 (message):
  Stopping after outputting logs.


[26/335] Performing download step (download, verify and extract) for 'mimalloc_ep'
-- Downloading...
   dst='D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/mimalloc_ep-prefix/src/v1.6.4.tar.gz'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/microsoft/mimalloc/archive/v1.6.4.tar.gz'
-- [download 100% complete]
-- Downloading... done
-- extracting...
     src='D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/mimalloc_ep-prefix/src/v1.6.4.tar.gz'
     dst='D:/Git/Arrow-Master/cpp/examples/minimal_build/arrow-build/mimalloc_ep-prefix/src/mimalloc_ep'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[31/335] Performing download step (download, verify and extract) for 'boost_ep'
ninja: build stopped: subcommand failed.

D:\Git\Arrow-Master\cpp\examples\minimal_build\arrow-build>popd

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "== Building example project using Arrow C++ library"
"== Building example project using Arrow C++ library"

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>mkdir example

D:\Git\Arrow-Master\cpp\examples\minimal_build>pushd example

D:\Git\Arrow-Master\cpp\examples\minimal_build\example>cmake ..       -GNinja       -DCMAKE_PREFIX_PATH="D:\Git\Arrow-Master\cpp\examples\minimal_build\dist"       -DARROW_LINK_SHARED=OFF
-- The C compiler identification is Clang 11.0.0 with GNU-like command-line
-- The CXX compiler identification is Clang 11.0.0 with GNU-like command-line
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/Llvm/bin/clang.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/Llvm/bin/clang++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:24 (find_package):
  By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Arrow", but
  CMake did not find one.

  Could not find a package configuration file provided by "Arrow" with any of
  the following names:

    ArrowConfig.cmake
    arrow-config.cmake

  Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
  "Arrow_DIR" to a directory containing one of the above files.  If "Arrow"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "D:/Git/Arrow-Master/cpp/examples/minimal_build/example/CMakeFiles/CMakeOutput.log".

D:\Git\Arrow-Master\cpp\examples\minimal_build\example>ninja
ninja: error: loading 'build.ninja': The system cannot find the file specified.


D:\Git\Arrow-Master\cpp\examples\minimal_build\example>popd

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "== Running example project"
"== Running example project"

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo "=="
"=="

D:\Git\Arrow-Master\cpp\examples\minimal_build>echo
ECHO is on.

D:\Git\Arrow-Master\cpp\examples\minimal_build>call example\arrow_example.exe
'example\arrow_example.exe' is not recognized as an internal or external command,
operable program or batch file.

D:\Git\Arrow-Master\cpp\examples\minimal_build>

1

There are 1 answers

0
ianmcook On

See the details on the Developing on Windows docs page. In particular, see the System Setup section, which shows this command you'll need to run first to set up VS 2019:

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64

Try running that before running run_static.bat.