Measure CPU time spent on each thread separately in C++

Question

I know that this question sounds like an easy question and a duplicate of former ones, in which boost.timer and the chrono facility of C++11 are given as answers.

But, what I have in mind is a bit different and I found no answer to it either on StackOverflow or elsewhere:

In my (C++11) program on Ubuntu Linux, I start several threads with std::async and the std::future mechanism.

Inside every thread I measure CPU-Time with boost.timer(). If I start only one thread I get a CPU time of (in my example) ~0.39 sec and an equal WC time of ~0.39 sec.

If I start several threads I get a longer WC time for each, say 0.8 sec for 16 threads and now the CPU-time for each is about 6.4 sec, that is 8 * 0.8 sec (I have a quad-core Xeon CPU).

So the CPU-Time of each thread is seemingly multiplied by the (number of CPU cores) * 2.

Of course(?) I would like to see a CPU-time near 0.39 sec for each thread, as this is probably still the time the thread uses the CPU for its purposes. The longer CPU time shown (multiplied with the "CPU number factor") is not so much of help in gauging the true CPU consumption of each thread separately.

For illustration I append my test program and its output, first for one thread, then for 16 threads.

So my question is: What can I do, which library, function or programming technique can I use, to get the true CPU usage of each thread which should not change much with the number of threads started?

#include <iostream>
#include <fstream>

#include <vector>
#include <cmath>

#include <future>
#include <mutex>
#include <chrono>

#include <boost/timer/timer.hpp>

std::mutex mtx;


class XTimer
{
public:
    XTimer() {};

    void start();
    void stop();

    double cpu_time();
    double boost_cpu_time();
    double wc_time();

    std::chrono::time_point<std::chrono::system_clock> timestamp_wc;
    std::chrono::time_point<std::chrono::steady_clock> timestamp_cpu;

    boost::timer::cpu_timer timer_cpu;

    double wc_time_val;
    double cpu_time_val;

    double boost_cpu_time_val;

};

void XTimer::start()
{
    timestamp_wc = std::chrono::system_clock::now();
    timestamp_cpu = std::chrono::steady_clock::now();

    timer_cpu.start();

    cpu_time_val = 0;
    wc_time_val = 0;

    boost_cpu_time_val = 0;


}

void XTimer::stop()
{
    const auto ns_wc = std::chrono::system_clock::now() - timestamp_wc;
    const auto ns_cpu = std::chrono::steady_clock::now() - timestamp_cpu;

    auto elapsed_times(timer_cpu.elapsed());

    auto cpu_elapsed(elapsed_times.system + elapsed_times.user);

    //std::cout << "boost: cpu elapsed = " << cpu_elapsed << std::endl;

    wc_time_val = double(ns_wc.count())/1e9;
    cpu_time_val = double(ns_cpu.count())/1e9;

    boost_cpu_time_val = double(cpu_elapsed)/1e9;

}


double XTimer::cpu_time()
{
    return cpu_time_val;
}

double XTimer::boost_cpu_time()
{
    return boost_cpu_time_val;
}

double XTimer::wc_time()
{
    return wc_time_val;
}




template<class T>
int wait_for_all(std::vector<std::future<T>> & fuvec)
{
    std::vector<T> res;

    for(auto & fu: fuvec) {
        res.push_back(fu.get());
    }
    return res.size();

}


int test_thread(int a)
{
    const int N = 10000000;

    double x = 0;

    XTimer tt;

    do {
        std::lock_guard<std::mutex> lck {mtx}; 
        std::cout << "start thread: " << a << std::endl;
    } while (0);

    tt.start(); 

    for(int i = 0; i < N; ++i) {

        if (i % 10000 == 0) {
            //std::cout << (char((int('A') + a)));
        }

        x += sin(i);
    }

    tt.stop();

    do {
        std::lock_guard<std::mutex> lck {mtx};

        std::cout << "end thread: " << a << std::endl;
        std::cout << "boost cpu = " << tt.boost_cpu_time() << " wc = " << tt.wc_time() << std::endl;
    } while (0);

    return 0;
}


int test_threads_start(int num_threads)
{

    std::vector<std::future<int>> fivec;

    XTimer tt;

    tt.start();

    for(int i = 0; i < num_threads; ++i) {
        fivec.push_back(std::async(test_thread, i));
    }

    int sz = wait_for_all(fivec);

    tt.stop();

    std::cout << std::endl << std::endl;

    std::cout << "all threads finished: total wc time = " << tt.wc_time() << std::endl;
    std::cout << "all threads finished: total boost cpu time = " << tt.boost_cpu_time() << std::endl;

}


int main(int argc, char** argv)
{
    const int num_threads_default = 1;
    int num_threads = num_threads_default;

    //boost::timer::auto_cpu_timer ac;

    if (argc > 1) {
        num_threads = atoi(argv[1]);
    }

    std::cout << "starting " << num_threads << " threads." << std::endl;

    test_threads_start(num_threads);

    std::cout << "end." << std::endl;

    return 0;
}

I can be compiled by

g++ -o testit testit.cpp -L/usr/lib/x86_64-linux-gnu -pthread -lboost_timer -lboost_system -lboost_thread 

Sample output with 1 thread

starting 1 threads.
start thread: 0
end thread: 0
boost cpu = 0.37 wc = 0.374107


all threads finished: total wc time = 0.374374
all threads finished: total boost cpu time = 0.37

Sample output with 16 threads

starting 16 threads.
start thread: 0
start thread: 1
start thread: 2
start thread: 3
start thread: 4
start thread: 10
start thread: 5
start thread: 7
start thread: 6
start thread: 11
start thread: 8
start thread: 9
start thread: 13
start thread: 12
start thread: 14
start thread: 15
end thread: 1
boost cpu = 4.67 wc = 0.588818
end thread: 2
boost cpu = 5.29 wc = 0.66638
end thread: 0
boost cpu = 5.72 wc = 0.7206
end thread: 13
boost cpu = 5.82 wc = 0.728717
end thread: 11
boost cpu = 6.18 wc = 0.774979
end thread: 12
boost cpu = 6.17 wc = 0.773298
end thread: 6
boost cpu = 6.32 wc = 0.793143
end thread: 15
boost cpu = 6.12 wc = 0.767049
end thread: 4
boost cpu = 6.7 wc = 0.843377
end thread: 14
boost cpu = 6.74 wc = 0.84842
end thread: 3
boost cpu = 6.91 wc = 0.874065
end thread: 9
boost cpu = 6.83 wc = 0.86342
end thread: 5
boost cpu = 7 wc = 0.896873
end thread: 7
boost cpu = 7.05 wc = 0.917324
end thread: 10
boost cpu = 7.11 wc = 0.930335
end thread: 8
boost cpu = 7.03 wc = 0.940374


all threads finished: total wc time = 0.957748
all threads finished: total boost cpu time = 7.14
end.

Answer 1

Documentation of boost::timer does not mention anything about per thread measurements. Fortunately boost::chrono contains thread_clock which gives per thread CPU usage on platforms which support it. It uses the same interface as the std::chrono clocks and measures thread wall clock.

After adding following lines to your example code:

// Includes section
#include <boost/chrono.hpp>

// XTimer
boost::chrono::thread_clock::time_point timestamp_thread_wc;
double thread_wc_time_val;

// XTimer::start()
timestamp_thread_wc = boost::chrono::thread_clock::now();

// XTimer::stop()
const auto ns_thread_wc = boost::chrono::thread_clock::now() - timestamp_thread_wc;
thread_wc_time_val = double(ns_thread_wc.count())/1e9;

// test_thread() just after for loop
sleep(1);

// test_thread() in bottom do -> while(0) loop
std::cout << "thread cpu = " << tt.thread_wc_time_val << std::endl;

and compiling with additional -lboost_chrono option I get:

starting 1 threads.
start thread: 0
end thread: 0
boost cpu = 0.16 wc = 1.16715
thread cpu = 0.166943


all threads finished: total wc time = 1.16754
all threads finished: total boost cpu time = 0.16
end.

and:

starting 2 threads.
start thread: 0
start thread: 1
end thread: 1
boost cpu = 0.28 wc = 1.14168
thread cpu = 0.141524
end thread: 0
boost cpu = 0.28 wc = 1.14417
thread cpu = 0.14401


all threads finished: total wc time = 1.14442
all threads finished: total boost cpu time = 0.28
end.