How to generate source files during the build time using python script in CMake and build them successfully

Question

I have a requirement where I need to auto generate the list of source files from a python script during the build time and build them. Please note that the file names are generated within the python script and the list is huge. Here is the: directory structure

ROOT
|
-- build
   |
   -- CMakeLists.txt
   __ my_python_script.py
|
-- sub_folder
      |
      -- build
           |
           -- CMakeLists.txt - This Cmake file will glob the sources and run target_source command to generate the library
      -- auto_gen_dir
              |
              -- Script will generate the source/header files here
              
              

I tried using add_custom_command and add_custom_target but both are generating the source files but are not compiling them. Due to this the build fails with undefined reference errors. When I run the build command second time then it goes through the unbuilt files [auto generated ones] and build it successfully. This is always a problem with a clean/first time build.

Just FYI. The other method that I tried is to auto generate the source files during the configuration time using execute_process() command and it generates the source files and are compiled in the build time. This has no issues.

This is the code snippet that I tried in my CMake from ROOT/build/CMakeLists.txt:

add_custom_command(TARGET my_target
   POST_BUILD #Please note I have used PRE_LINK too. PRE_BUILD will work with only Visual Studio geneartor
   COMMENT "Auto generating source/header files for 'amdb_core' library"
   COMMAND ${CMAKE_COMMAND} -E env
   ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   COMMENT "Auto generating source/headers is complete"
   VERBATIM)

Answer 1

To expand on @Tsyvarev's comment, CMake needs to know your source list before build-time, which is why it worked when you generated your sources at configure-time. Now, that doesn't mean your sources have to exist at build-time. Try modifying your generator script to have two modes of operation. One outputs a list of files it would generate, the other generates those files. Then run your new list-mode at configure-time to populate your target's source list, and set generator-mode to be a dependency of your target so its sources are created during build-time.

Answer 2

So generate the files:

add_custom_command(
   OUTPUT ${AUTO_GEN_DIR}/file1.c ${AUTO_GEN_DIR}/file2.c ${AUTO_GEN_DIR}/file3.c etc..
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
)

And then compile them:

add_executable(your_executable ${AUTO_GEN_DIR}/file1.c ${AUTO_GEN_DIR}/file2.c ${AUTO_GEN_DIR}/file3.c etc..)

Typically, you would generate everything inside CMAKE_CURRENT_BINARY_DIR. Generating them in source is confusing.

not to list the actual file names

You can execute the python script for cmake beforehand to get only filenames for example:

execute_process(
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py GET_LIST_OF_FILES
   RESULT_VARIABLE FILES
)
add_custom_command(
   OUTPUT ${FILES}
   COMMAND ${PYTHON3} ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my_python_script.py
)

You could just execute the python script from cmake on configuration stage with execute_process and then after it just use glob.

And finally, you can wrap your cmake with top-level build tool, like make.