Getting llvm-cov to talk to codecov.io

Question

I'm in the process of (finally!) setting up code coverage monitoring for my brand new C++ project. Due to the fact that I need some advanced C++20 features (read, coroutines), I am using clang 6 as compiler.

Now, I followed this guide on how to do basic code coverage for your project, and everything worked like magic. If I do:

clang++ -fprofile-instr-generate -fcoverage-mapping test.cpp -o test.out
LLVM_PROFILE_FILE="coverage/test.profraw" ./test.out
llvm-profdata merge -sparse coverage/test.profraw -o coverage/test.profdata
llvm-cov show ./test.out -instr-profile=coverage/test.profdata

I get a nice, colored report on my terminal that tells me what is covered and what is not.

So far so good! I thought I was close to what I wanted, but then the pain started when I tried to get the report uploaded to codecov.io.

I have tried a few things, including:

• Running their https://codecov.io/bash script on my coverage folder in the hope that maybe it would catch on my test.profdata. No dice, and it makes sense, since even llvm-cov needs the path to the executable file to run.

• Using the export functionality: when running llvm-cov export --instr-profile=coverage/test.profdata ./test.out I get a good-looking JSON file (via terminal). I tried throwing the output in a coverage.json file, which actually got uploaded, but then codecov just says that there was an error parsing it, with no further information.

I'm feeling completely lost. Everything seems so black-box-ish on their website that I just don't understand how to get anything done that doesn't by chance perfectly fit the cases that they can manage.

How can I get this working with codecov? If codecov can't handle my reports, is there any other equivalent online code coverage that I can use to get this to work?

Answer 1

It looks like the bash script codecov uses to upload coverage data to their site looks for files matching a wide range of patterns associated with formats that it understands. These are poorly documented, but you can at least see which patterns are viable by looking at the script on Github. Of course, this doesn't tell you what expectations codecov has about the format of files matching a given pattern, as you discovered when your coverage.json file was rejected.

Through trial and error I have found that the following produces a file that codecov will interpret correctly when you run the bash script:

llvm-cov show ./test.out -instr-profile=default.profdata > coverage.txt

I haven't extensively tested what file names are allowed, but it seems that you can put whatever additional characters you want between coverage and .txt in the name of the file that you're piping the coverage data to (e.g. you could call it coverage_my_file_name.txt).

EDIT: Just in case this is helpful to anyone, it turns out that an important corollary to the above is that it's critical that you avoid naming anything that isn't a coverage report something that matches this pattern. I just dealt with a scenario where I had a bunch of executables named coverage_[more_text_here].out that were getting uploaded with the reports. It turns out that attempting to parse assembly code as a coverage report can cause codecov to mysteriously fail without any useful errors.

Answer 2

Another option is to use GCOV profiling, which is a little less precise than source-based, but it is supported by codecov.io. You need the "--coverage" compiler flag to enable it.

You can use grcov (which you can also download from https://github.com/mozilla/grcov/releases) to parse the gcno/gcda files and upload them via the codecov.io bash uploader:

grcov OBJ_DIR -s SRC_DIR -t lcov --branch > lcov.info
bash codecov.sh -f "lcov.info"

I'm planning to add support for source-based reports to grcov, which will make it easier to support the format on codecov too.