I'm trying to enable ccache on gitlab cicd. I want to aciviate the ccache shared between all branches
I added this code to the dockerfile:
RUN apt update -y && apt upgrade -y && apt install -y ccache
ENV CCACHE_CPP2=yes
ENV CCACHE_DIR=/PROJECT-NAME/ccache
RUN export PATH="/usr/lib/ccache:$PATH"
I changed code in the .gitlab-ci.yml:
test:
image: ......../runner_x86:9.0.6
stage: test
tags:
- docker-gpu
cache:
key: shared-cache
paths:
- /ccache/
policy: pull-push
script:
- export CCACHE_DEBUG=1
- ccache -s
- ls -l
- du -sh ccache || true
- ls ccache || true
- cat ccache/ccache.conf || true
- bash scripts/unit_test.sh -D CMAKE_CUDA_COMPILER=$(which nvcc) -D ENABLE_BENCHMARK=OFF
- ccache --show-config
except:
- master
coverage: !reference [ .gitlab-coverage-badge ]
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: docker_build_debug/coverage.xml
when: always
untracked: false
expire_in: 3 days
I add this code to main CMakeLists.txt:
find_program(CCACHE "ccache")
if(CCACHE)
set(CMAKE_CXX_COMPILER_LAUNCHER ccache)
endif ()
The ccache folder creates and has some data, but build time is not reducing
JOB:
Running with gitlab-runner 15.10.1 (dcfb4b66)
on runner on ***** with docker-gpu on ubuntu hzdz5qmn, system ID: xxxxx
Preparing the "docker" executor
Using Docker executor with image ........./runner_x86:9.0.0 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image ................./runner_x86:9.0.0 ...
Using docker image sha256:xxxx for ......../runner_x86:9.0.0 with digest ............../runner_x86@sha256:xxxxx ...
Preparing environment 00:04
Running on runner-hzdz5qmn-project-77-concurrent-0 via RUNNER...
Getting source from Git repository 00:02
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in ......./PROJECT-NAME/.git/
Checking out a9089fc0 as detached HEAD (ref is feature/working_ccache)...
Removing 1
Removing UnityMockDataset.zip
Removing datasets/
Removing docker_build_debug/
Removing models/
Removing parseq.zip
Removing PROJECT-NAME/
Removing test_data/merging/img-this-output.bmp
Removing test_data/merging/merged-dataset-frames.bmp
Removing yolox.zip
Skipping Git submodules setup
Restoring cache 00:04
Checking cache for shared-cache-1-non_protected...
cache.zip is up to date
Successfully extracted cache
...
$ export CCACHE_DEBUG=1
$ ccache -s
cache directory /PROJECT-NAME/ccache
primary config /PROJECT-NAME/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 0
cache hit rate 0.00 %
cleanups performed 0
files in cache 0
cache size 0.0 kB
max cache size 5.0 GB
$ ls -l
total 52
-rw-rw-rw- 1 root root 2822 Oct 31 11:12 CMakeLists.txt
-rw-rw-rw- 1 root root 1372 Aug 2 10:26 README.md
drwxrwxrwx 2 root root 4096 Oct 31 11:12 apps
drwxr-xr-x 19 root root 4096 Oct 23 14:11 ccache
drwxrwxrwx 3 root root 4096 Oct 25 16:06 cmake
-rw-rw-rw- 1 root root 812 Oct 27 14:36 conanfile.txt
drwxrwxrwx 6 root root 4096 Oct 28 17:05 configs
drwxrwxrwx 4 root root 4096 Oct 31 11:12 docker
drwxrwxrwx 3 root root 4096 Aug 2 10:26 include
drwxrwxrwx 4 root root 4096 Oct 31 11:12 scripts
drwxrwxrwx 13 root root 4096 Oct 31 11:12 src
drwxrwxrwx 6 root root 4096 Oct 25 16:06 test_data
drwxrwxrwx 6 root root 4096 Oct 30 12:55 tests
$ du -sh ccache || true
741M ccache
$ ls ccache || true
0
1
2
3
4
5
6
7
8
9
a
b
c
ccache.conf
d
e
f
tmp
$ cat ccache/ccache.conf || true
max_size = 5.0G
$ bash scripts/unit_test.sh -D CMAKE_CUDA_COMPILER=$(which nvcc) -D ENABLE_BENCHMARK=OFF
+ trap cleanup EXIT
+ id
uid=0(root) gid=0(root) groups=0(root)
+ cmake --version
cmake version 3.27.6
CMake suite maintained and supported by Kitware (kitware.com/cmake).
+ gcc --version
gcc (Ubuntu 11.4.0-2ubuntu1~20.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0
+ env
+ grep CXX
+ true
+ export CONAN_USER_HOME=/builds/DEV/PROJECT-NAME/PROJECT-NAME/
+ CONAN_USER_HOME=/builds/DEV/PROJECT-NAME/PROJECT-NAME/
+ source ./scripts/setup/ci_download_dependencies.sh
++ export_variables
++ '[' -z [MASKED] ']'
++ JOB_TOKEN=[MASKED]
++ GITLAB_URL=https://xxxxx/api/v4
++ PROJECT_ID=103
+++ python3 scripts/utils/get_gpu.py
++ export GPU=NVIDIA-GeForce-RTX-3090
++ GPU=NVIDIA-GeForce-RTX-3090
++ '[' '!' -z [MASKED] ']'
++ AUTH_HEADER='JOB-TOKEN: [MASKED]'
++ download_dataset
++ '[' '!' -d datasets/UnityMockDataset ']'
++ echo 'Downloading UnityMockDataset ...'
++ mkdir -p datasets/UnityMockDataset
Downloading UnityMockDataset ...
.. BUILD A FEW MINUTES (THE SAME TIME WITHOUT CCACHE) for example:
[ 75%] Built target camera_viewer
[ 76%] Building CXX object tests/unit/CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ImageCropperTest.cpp.o
cd /builds/DEV/PROJECT-NAME/docker_build_debug/tests/unit && ccache /usr/bin/c++ -DJPEG_FOUND -DPNG_FOUND -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -DWITH_CUDA -I/builds/DEV/PROJECT-NAME/tests/unit/test_util -I/builds/DEV/PROJECT-NAME/src/../include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/range-v3/0.12.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/nanobench/4.3.11/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/commons/0.1/develop/1c4e1906/package/fdfc1f616204390f0c0bb480a1786a53d2bb439e/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/taskflow/3.6.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mio/cci.20230303/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/gtest/1.12.1/_/_/package/e54178ffb73389748e7bf45e64441da62db44ded/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mp-units/0.8.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/boost/1.76.0/_/_/package/e2f1ab856c267df5e6bffc45064137eb4200952e/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/di/1.2.0/_/_/package/964905e09401b68ee5f6b695b187d4fa9eb760c2/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/asio/1.24.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/magic_enum/0.8.2/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/tl-expected/1.1.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/eventpp/0.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mavlink-cpp11/0.3/chimera+3rd_party_open_source+mavlink-cpp11-library/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/TE/latest/boost/te/package/2a448472971d7718b4207a1ee198ae4f78f2995d/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/nlohmann_json/3.11.2/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/spdlog/1.11.0/_/_/package/8adac865734b8b9f6eacb065ed45c095ce372c5b/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/fixed_string/1.0/_/_/package/2a448472971d7718b4207a1ee198ae4f78f2995d/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/gsl-lite/0.40.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/fmt/9.1.0/_/_/package/20da98908a15523bbe9f086a5c57a4bde424a9c0/include -isystem /builds/DEV/PROJECT-NAME/tests/unit/../include -isystem /usr/include/opencv4 -isystem /opt/xxxx/embree/include -isystem /usr/include/eigen3 -isystem /usr/local/include/torchvision -isystem /opt/xxxxx/onnx-112/include -isystem /opt/xxxxx/OC_SORT/include -isystem /opt/xxxxxxxxxx/libtorch/include -isystem /opt/xxxx/libtorch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -isystem /opt/pylon/include -g -fprofile-arcs -ftest-coverage -g -std=c++20 -DGSL_TERMINATE_ON_CONTRACT_VIOLATION -DSPDLOG_FMT_EXTERNAL -DSPDLOG_COMPILED_LIB -DASIO_STANDALONE -DBOOST_DI_CFG_DIAGNOSTICS_LEVEL=1 -D_GLIBCXX_USE_CXX11_ABI=1 -Werror -Wall -Wextra -Wshadow -Wnon-virtual-dtor -Wold-style-cast -Wcast-align -Wunused -Woverloaded-virtual -Wpedantic -Wconversion -Wsign-conversion -Wnull-dereference -Wdouble-promotion -Wformat=2 -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wlogical-op -Wuseless-cast -Wno-psabi -MD -MT tests/unit/CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ImageCropperTest.cpp.o -MF CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ImageCropperTest.cpp.o.d -o CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ImageCropperTest.cpp.o -c /builds/DEV/PROJECT-NAME/tests/unit/plate_recognition/ImageCropperTest.cpp
[ 77%] Building CXX object tests/unit/CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ParseqONNXTest.cpp.o
cd /builds/DEV/PROJECT-NAME/docker_build_debug/tests/unit && ccache /usr/bin/c++ -DJPEG_FOUND -DPNG_FOUND -DUSE_C10D_GLOO -DUSE_C10D_MPI -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -DWITH_CUDA -I/builds/DEV/PROJECT-NAME/tests/unit/test_util -I/builds/DEV/PROJECT-NAME/src/../include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/range-v3/0.12.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/nanobench/4.3.11/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/commons/0.1/develop/1c4e1906/package/fdfc1f616204390f0c0bb480a1786a53d2bb439e/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/taskflow/3.6.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mio/cci.20230303/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/gtest/1.12.1/_/_/package/e54178ffb73389748e7bf45e64441da62db44ded/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mp-units/0.8.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/boost/1.76.0/_/_/package/e2f1ab856c267df5e6bffc45064137eb4200952e/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/di/1.2.0/_/_/package/964905e09401b68ee5f6b695b187d4fa9eb760c2/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/asio/1.24.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/magic_enum/0.8.2/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/tl-expected/1.1.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/eventpp/0.1/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/mavlink-cpp11/0.3/chimera+3rd_party_open_source+mavlink-cpp11-library/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/TE/latest/boost/te/package/2a448472971d7718b4207a1ee198ae4f78f2995d/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/nlohmann_json/3.11.2/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/spdlog/1.11.0/_/_/package/8adac865734b8b9f6eacb065ed45c095ce372c5b/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/fixed_string/1.0/_/_/package/2a448472971d7718b4207a1ee198ae4f78f2995d/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/gsl-lite/0.40.0/_/_/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include -isystem /builds/DEV/PROJECT-NAME/PROJECT-NAME/.conan/data/fmt/9.1.0/_/_/package/20da98908a15523bbe9f086a5c57a4bde424a9c0/include -isystem /builds/DEV/PROJECT-NAME/tests/unit/../include -isystem /usr/include/opencv4 -isystem /opt/xxx/embree/include -isystem /usr/include/eigen3 -isystem /usr/local/include/torchvision -isystem /opt/xxx/onnx-112/include -isystem /opt/xxxx/OC_SORT/include -isystem /opt/xxxx/libtorch/include -isystem /opt/xxxxx/libtorch/include/torch/csrc/api/include -isystem /usr/local/cuda/include -isystem /opt/pylon/include -g -fprofile-arcs -ftest-coverage -g -std=c++20 -DGSL_TERMINATE_ON_CONTRACT_VIOLATION -DSPDLOG_FMT_EXTERNAL -DSPDLOG_COMPILED_LIB -DASIO_STANDALONE -DBOOST_DI_CFG_DIAGNOSTICS_LEVEL=1 -D_GLIBCXX_USE_CXX11_ABI=1 -Werror -Wall -Wextra -Wshadow -Wnon-virtual-dtor -Wold-style-cast -Wcast-align -Wunused -Woverloaded-virtual -Wpedantic -Wconversion -Wsign-conversion -Wnull-dereference -Wdouble-promotion -Wformat=2 -Wmisleading-indentation -Wduplicated-cond -Wduplicated-branches -Wlogical-op -Wuseless-cast -Wno-psabi -MD -MT tests/unit/CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ParseqONNXTest.cpp.o -MF CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ParseqONNXTest.cpp.o.d -o CMakeFiles/test_PROJECT_NAME.dir/plate_recognition/ParseqONNXTest.cpp.o -c /builds/DEV/PROJECT-NAME/tests/unit/plate_recognition/ParseqONNXTest.cpp
THE END
Uploading package 1/1: xxxxx to 'yyyy-conan'
Package is up to date, upload skipped
+ cleanup
++ stat -c %u:%g PROJECT-NAME
+ chown -R 0:0 docker_build_debug
$ ccache --show-config
(default) base_dir =
(environment) cache_dir = /PROJECT-NAME/ccache
(default) cache_dir_levels = 2
(default) compiler =
(default) compiler_check = mtime
(default) compression = false
(default) compression_level = 6
(default) cpp_extension =
(environment) debug = true
(default) depend_mode = false
(default) direct_mode = true
(default) disable = false
(default) extra_files_to_hash =
(default) hard_link = false
(default) hash_dir = true
(default) ignore_headers_in_manifest =
(default) keep_comments_cpp = false
(default) limit_multiple = 0.8
(default) log_file =
(default) max_files = 0
(/PROJECT-NAME/ccache/ccache.conf) max_size = 5.0G
(default) path =
(default) pch_external_checksum = false
(default) prefix_command =
(default) prefix_command_cpp =
(default) read_only = false
(default) read_only_direct = false
(default) recache = false
(environment) run_second_cpp = true
(default) sloppiness =
(default) stats = true
(default) temporary_dir =
(default) umask =
Saving cache for successful job 00:00
Creating cache shared-cache-1-non_protected...
WARNING: processPath: artifact path is not a subpath of project directory: /ccache/
Archive is up to date!
Created cache
Uploading artifacts for successful job 00:01
Uploading artifacts...
docker_build_debug/coverage.xml: found 1 matching artifact files and directories
Uploading artifacts as "cobertura" to coordinator... 201 Created id=xxx responseStatus=201 Created token=xxxx
Cleaning up project directory and file based variables 00:00
Job succeeded
Job succeeded
@Darius
I'm using ccache version 3.7.7
Here is ccache --show-stats -v result:
ccache --show-stats -v
cache directory /PROJECT-NAME/ccache
primary config /PROJECT-NAME/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 0
cache hit rate 0.00 %
cleanups performed 0
files in cache 0
cache size 0.0 kB
max cache size 5.0 GB
ccache: invalid option -- 'v'
Usage:
ccache [options]
ccache compiler [compiler options]
compiler [compiler options] (via symbolic link)
Common options:
-c, --cleanup delete old files and recalculate size counters
(normally not needed as this is done
automatically)
-C, --clear clear the cache completely (except configuration)
-F, --max-files=N set maximum number of files in cache to N (use 0
for no limit)
-M, --max-size=SIZE set maximum size of cache to SIZE (use 0 for no
limit); available suffixes: k, M, G, T (decimal)
and Ki, Mi, Gi, Ti (binary); default suffix: G
-p, --show-config show current configuration options in
human-readable format
-s, --show-stats show summary of configuration and statistics
counters in human-readable format
-z, --zero-stats zero statistics counters
-h, --help print this help text
-V, --version print version and copyright information
Options for scripting or debugging:
--dump-manifest=PATH dump manifest file at PATH in text format
-k, --get-config=K print the value of configuration key K
--hash-file=PATH print the hash (<MD4>-<size>) of the file at PATH
--print-stats print statistics counter IDs and corresponding
values in machine-parsable format
-o, --set-config=K=V set configuration item K to value V
See also <https://ccache.dev>.
Here is ccache --show-stats -V result:
$ ccache --show-stats -V
cache directory /PROJECT-NAME/ccache
primary config /PROJECT-NAME/ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 0
cache hit rate 0.00 %
cleanups performed 0
files in cache 0
cache size 0.0 kB
max cache size 5.0 GB
ccache version 3.7.7
Copyright (C) 2002-2007 Andrew Tridgell
Copyright (C) 2009-2020 Joel Rosdahl
This program is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free Software
Foundation;
@Botej
I removed last line in Dockerfile, thanks
I fix the mismatch between CCACHE_DIR=/PROJECT-NAME/ccache and /ccache.
cache:
key: shared-cache
paths:
- ccache/
policy: pull-push
but it's still not working
UPDATE
It was too long to post it here
https://docs.google.com/document/d/1NcwiaXVkEqxExasXXiIZ8R4wUwtTdp1G9MOnEYu_DFg/edit?usp=sharing
You need a newer version of
ccache, preferably4.5+.https://github.com/rapidsai/raft/pull/729 explains that fixes have been backported to
ccache3.7.9+, which is above what you use (3.7.7).https://github.com/ccache/ccache/issues/772#issuecomment-760462748 details
cudaandcusupport inccache:Note that there are still some issues with
cudasupport but this should get you somewhere.