Boost threads and non-existant speedups on Linux SMPs

Question

I have written a small example C++ program, using boost::thread. Since it's 215 lines, I've posted it on pastebin instead

http://pastebin.com/LRZ24W7D

The program creates a large number of floats (currently 1gb) and adds them up, first sequentially, and then using a number of threads (hosted inside the device_matrix class). Assuming the machine is a SMP, I'd expect to see a speedup from the code. And on my Windows machine, I see a four-fold speedup, when using 4 device_matrix instances (giving 4 threads, on my dual-core hyperthreaded Intel Core2 CPU). The output on Windows is the following:

starting computation
device_matrix count       4
elements                  268435456
UINT_MAX                  4294967295
data size total           1024 mb
size per device_matrix    256 mb
reference                 134224128.00000
result                    134224128.00000
time taken (init)         12.015 secs
time taken (single)       3.422 secs
time taken (device)       0.859 secs

However, when I compile the same code on an Ubuntu machine I have available, I see the following output:

starting computation
device_matrix count       8
elements                  268435456
UINT_MAX                  4294967295
data size total           1024 mb
size per device_matrix    128 mb
reference                 134215408.00000
result                    134215400.00000
time taken (init)         3.670 secs
time taken (single)       3.030 secs
time taken (threaded)     3.950 secs

Here, no speed up is seen (in fact, it's slower by quite alot).

The Ubuntu machine I'm using has the following uname -a output

Linux gpulab03 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 08:03:28 UTC 2010 x86_64 GNU/Linux

And hwinfo -short gives the following output:

cpu:
                       Intel(R) Core(TM) i7 CPU         930  @ 2.80GHz, 1600 MHz
                       ... 7 more times

Which I read as the machine having eight cores (well, quad core with HT)

I'm using the following line to compile my program on Windows:

cl /Fe"boost.exe" /EHsc -I. boost.cpp /link /LIBPATH:"C:\boost\boost_1_45_0\stage\lib"

And on Ubuntu, I use the following line:

g++ -O0 -v -o boost -I$HOME/Code/boost -L$HOME/Code/boost/stage/lib boost.cpp -lboost_thread-gcc44-mt

The output when running the above line is here http://pastebin.com/Gj6W3pcs in case it can tell anyone anything.

Since I'm not used to developing on Linux, I'm just not sure what to look for. Is there some flag I need to pass to GCC or some setting I need to enable somewhere, to get actual concurrent threads?

I've looked around the net for an example program using boost::thread, that could give me something to benchmark against, but I'm only finding small producer-consumer examples that don't need to crunch anything "heavy".

As an extra thing, using the time command, with one thread gives the following times (just in case boost::timer is wonky):

real    0m9.788s
user    0m9.500s
sys     0m0.280s

And when using 8 threads, I see the following:

real    0m7.292s
user    0m10.340s
sys     0m0.340s

Which doesn't seem to indicate any faster run anyway.

I should also mention that I'm on a normal user account, and I've built boost myself (and so, linking against it outside of the "normal" folders for this purpose on Linux.) This also means I've severely limited in what I can install, etc. Are there similar limitations that applies to threads somehow?

Answer 1

I believe the problem is with boost::timer. I get different timing results if I use gettimeofday and subtract instead.

It looks as if clock(), which is what boost::timer uses, is returning the amount of CPU time used by the entire program, not just one thread. This looks like a Boost bug to me.

I made a new version of your code that was compatible with the Boost on a CentOS 5 machine. I modified your do_sum operation into a free function so I was guaranteed sum was computed exactly the same way for single and multithreaded. I added a non-Windows header so I could use gettimeofday.

The code is here.