Is there a way to profil an MPI Application during Runtime? - Using the time spent in MPI Calls in runtime

Question

I am wondering if there is a way of measuring the time which is spent in MPI calls during runtime. Thus, I can use it for calculating a new load balancing.

I know how to profile and trace the program using some tools from OpenMPI or Intel, but those are all use after runtime. Furthermore, I have tried FPMPI, which was not a success because the last release is not able to be built.

Measuring "by hand" does not make any sense in my application because it is way to big :/

Answer 1

First of all, do you really need to profile low-level communication such as MPI? Can't you simply time your high-level routines instead?

Anyway, it is pretty easy to write your own MPI profiler. Practically all MPI libraries (Open MPI included) export their functions (e.g., MPI_Send) as weak aliases of the same function symbols with prefix P (e.g., PMPI_Send). All you need to do is define your own functions with the same prototypes as the ones in the MPI library. Inside, update your call counters, start the timers, then call the original MPI function with a P prefix, and upon return stop the timers:

extern "C"
int MPI_Send(void *buf, int count, MPI_Datatype dtype, int dest, int tag, MPI_Comm comm) {
   // Update call counters and start the timer
   calls["MPI_Send"]++;
   timers["MPI_Send"].start();

   // Call the original MPI function
   int result = PMPI_Send(buf, count, dtype, dest, tag, comm);

   // Stop the timer
   timers["MPI_Send"].stop();

   return result;
}

The extern "C" part is important, otherwise you won't override the correct weak symbol if you write in C++.

This ability to override symbols from the MPI library is standardised - see #14.2 Profiling Interface in the current version of the standard.