I am very new to LSF. I have 4 nodes with with 2 sockets per node. Each node is having 8 cores. I have developed hybrid MPI+OpenMP code. I am submitting the job like the following which asks each core to perform one MPI task. So I loose the power of OpenMP.
##BSUB -n 64
I wish to submit the job so that each socket runs one MPI task rather than each core so that the cores inside the socket can be used for OpenMP. How can I build up job submit scripts to optimize the power of the Hybridization in my code.
First of all, the
BSUB
sentinels have to be preceded by a single#
sign, otherwise they are skipped over as a regular comments.The correct way to start a hybrid job with older LSF versions is to pass the
span
resource request and request nodes exclusively. To start a job with 8 MPI processes and 8 OpenMP threads each, you should use the following:The parameters are as following:
-n 8
- requests 8 slots for MPI processes-x
- requests nodes exclusively-R "span[ptile=2]"
- instructs LSF to span the job over two slots per nodeYou should request nodes exclusively, otherwise LSF will schedule other jobs to the same nodes since only two slots per node will be used.
Then you have to set the
OMP_NUM_THREADS
environment variable to4
(the number of cores per socket), tell the MPI library to pass the variable to the MPI processes, and make the library limit each MPI process to its own CPU socket. This is unfortunately very implementation-specific, e.g.:Open MPI 1.6.x or older:
Open MPI 1.7.x or newer:
Intel MPI (not sure about this one as I don't use IMPI very often):