How can I measure the memory occupancy of Python MPI or multiprocessing program?

Question

I am doing this on Cray XE6 machine where I can't log in on compute nodes and there is no possibility for interactive session, therefore I would need to somehow use top command: run top in the background and have it take snapshot at regular times and send it to a log file... Can somebody make an example of how to do this?

Thanks

Answer 1

Use the resource module to query for current memory usage. Then, somewhere in your MPI program, dump the usage to a log at certain intervals.

For example, the following measures maximum resident memory size of the current process, and appends the value to a file. What you would presumably do is insert the snippet somewhere at each timestep of your simulation.

import datetime
import resource
import os
from mpi4py import MPI

mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
rank = MPI.COMM_WORLD.Get_rank()
fname = 'r{}.log'.format(rank)
with open(fname, 'a') as f:
    # Dump timestamp, PID and amount of RAM.
    f.write('{} {} {}\n'.format(datetime.datetime.now(), os.getpid(), mem))

Note the naming of the file, using the rank of the running MPI process (as per @Hristo's recommendation).