How do I configure portable parallel builds in CMake?

Question

Is it somehow possible to be able to have a parallel build no matter which build tool is used?

Under Unix we can add make -jN where N are the number of threads, and under Windows I added to the CXX_FLAG "/MP" which is then used in Visual Studio to parallel build...(?) How can I make my version such that CMAKE_MAKE_PROGRAM is not always extended when I run CMake?

What is a general solution?

I came up with this:

# Add some multithreaded build support
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
set(MULTITHREADED_BUILD 12 CACHE STRING "How many threads are used to build the project")
if(MULTITHREADED_BUILD)
    if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
            message(STATUS ${CMAKE_BUILD_TOOL})
            set(CMAKE_MAKE_PROGRAM "${CMAKE_MAKE_PROGRAM} -j${MULTITHREADED_BUILD}")
            message(STATUS "Added arguments to CMAKE_BUILD_TOOL: ${CMAKE_MAKE_PROGRAM}")
    elseif(MSVC)
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /MP")
      message(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
    endif()
endif()

Answer 1

With CMake 3.12 this is possible. From the release notes:

The cmake(1) Build a Project (cmake --build) gained --parallel [<jobs>] and -j [<jobs>] options to specify a parallel build level. They map to corresponding options of the native build tool.

As mentioned by dkg, you can also set the environment variable CMAKE_BUILD_PARALLEL_LEVEL.

Links to CMake's documentation:

• Build a Project
• CMAKE_BUILD_PARALLEL_LEVEL

Answer 2

Now it is very simple to parallel build with cmake. You can add "-j jobs_number" when using "cmake --build". For example:

cmake --build . -j 24

More details can be found in CMAKE manual: https://cmake.org/cmake/help/latest/manual/cmake.1.html#build-tool-mode

--parallel [], -j [] The maximum number of concurrent processes to use when building. If is omitted the native build tool’s default number is used.

The CMAKE_BUILD_PARALLEL_LEVEL environment variable, if set, specifies a default parallel level when this option is not given.

Some native build tools always build in parallel. The use of value of 1 can be used to limit to a single job.

Answer 3

As already mention above one can use --parallel [<jobs>] (or -j [<jobs>]) option of CMake to build solution in parallel.

But I would like to note that in Windows a command

cmake.exe --build --parallel <jobs>

starts a root MSBuild process with the following arguments:

msbuild.exe /m:<jobs> /p:CL_MPCount=1 <another-useful-arguments...>

This means the root MSBuild process will start multiple child MSBuild processes due to /m:<jobs> (or /maxcpucount:<jobs>). But each of this child MSBuild runs at most one compiler process due to /p:CL_MPCount=1 argument (for more details find this link). In other words each MSBuild can compile at most one source file at the same time.

To overcome this limitation one may call CMake in the following way:

cmake.exe --build --parallel <n_msbuild> -- /p:CL_MPcount=<n_cl>

This approach sets /MP option thereby allows multiple compiler processes under each MSBuild instance.

---

And for the record. To configure builds ran form Visual Studio GUI one need to go to Settings:

• Option "Projects and Solutions → Build And Run → maximum number of parallel projects build" affects on maxcpucount.
• Option "Projects and Solutions → VC++ Project Settings → Build → Maximum Concurrent C++ Compilations" represents the default boundary to /MP.

Answer 4

You can't do this cross-platform. The -jN option is a parameter to make, and not part of the generated Makefile. However, you could have CMake generate a Bash script that runs make for your project using -jN (where the script looks up the number of cores you have).

Answer 5

I have settled down to writing a parallelmake.sh script for Unix Makefiles-based generators. This is done here: https://github.com/gabyx/ApproxMVBB

And the relevant parts in the the CMake file:

https://github.com/gabyx/ApproxMVBB/blob/master/CMakeLists.txt#L89

# Add some multithreaded build support =====================================================================================================
MARK_AS_ADVANCED(MULTITHREADED_BUILD)
SET(MULTITHREADED_BUILD ON CACHE BOOL "Parallel build with as many threads as possible!")
if(MULTITHREADED_BUILD)
    if(${CMAKE_GENERATOR} MATCHES "Unix Makefiles")
            file(COPY ${ApproxMVBB_ROOT_DIR}/cmake/parallelmake.sh DESTINATION ${PROJECT_BINARY_DIR}
                FILE_PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE
                NO_SOURCE_PERMISSIONS
            )
            SET(CMAKE_MAKE_PROGRAM "${PROJECT_BINARY_DIR}/parallelmake.sh")
            MESSAGE(STATUS "Set make program to ${PROJECT_BINARY_DIR}/parallelmake.sh")
    elseif(MSVC)
      SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}" "/MP")
      MESSAGE(STATUS "Added parallel build arguments to CMAKE_CXX_FLAGS: ${CMAKE_CXX_FLAGS}")
    endif()
endif()
# ========================================================================================================================================

Answer 6

If you have CMake v2.8.8 or higher, you may use Ninja as an alternative of GNU make:

mkdir build
cd    build
cmake -G Ninja ..
ninja              # Parallel build (no need -j12)

or

mkdir build
cd    build
cmake -G Ninja ..
cmake --build .    # Parallel build using Ninja

As you can see, no need to use CMAKE_MAKE_PROGRAM, the build is run in parallel by default, optimizing the number of jobs depending on available CPU cores.

Ninja is based on a low-level JSON configuration to speed up the startup phase. Therefore its JSON configuration is not easy to write by hand, and I always generate it using a high-level tool/IDE:

• CMake v2.8.8 (2012)
• Qt Creator v2.6 (2012)
• KDevelop v4.6 (2013)
• Meson on Linux (2013)
• ... see the generators of Ninja configuration at https://github.com/ninja-build/ninja/wiki/List-of-generators-producing-ninja-build-files

As a C++ build often requires lots of memory, your computer must provide as much memory as the number of CPU cores.

As pointed out by Ruslan, CMake 3.12 (2018) has a new option cmake --build -j <N> to limit build to <N> cores (jobs) thus limiting the memory consumption (see also the documentation). If you use an older CMake version, you can still use cmake --build -- -j <N>. The option -- tells to CMake to pass the rest directly to the underlying builder tool, here it is Ninja.