How do I compile VASP 5.3.5 on Cray XC?

Question

I need to compile VASP 5.3.5 on a Cray XC MPP machine. The system has the Gnu, Intel and Cray compiling environments available. There is also a choice of Intel MKL or Cray LibSci for BLAS, LAPACK, ScaLAPCK.

Which is the best compiler to use, the best numerical library to use and the Makefile options to use?

Answer 1

Tests on ARCHER (http://www.archer.ac.uk) have shown that using the Intel compiler with Intel MKL and FFTW produce the best performance and the most stable build of VASP 5.3.5 on the Cray XC30 system.

Full compilation instructions can be found at:

http://www.archer.ac.uk/documentation/software/vasp/compiling_5-3-5-phase2.php

Briefly, the procedure is:

module swap PrgEnv-cray PrgEnv-intel
module load fftw
module load cray-pe-hugepages2M

Modify the library makefile to have the following options:

CPP     = gcc -E -P -C $*.F >$*.f
FC=ftn 
CFLAGS = -O3
FFLAGS = -O3 -unroll -ip -no-prec-div -xAVX
FREE   = -free

Build the library (assuming makefile is called "makefile.cray_xc_intel.lib"):

cd vasp.5.lib
make -f makefile.cray_xc_intel.lib

Move to the main source code directory:

cd ../vasp.5.3

Setup the preprocessor options in the Makefile (this is for the multiple K-points version):

CPP    = $(CPP_) -DMPI -DHOST=\"CrayXC-Intel\" \
         -DNGZhalf \
         -DLONGCHAR \
         -Dkind8 \
         -DCACHE_SIZE=2000 \
         -Davoidalloc \
         -DRPROMU_DGEMV  \
         -DMPI_BLOCK=100000 \
         -Duse_collective \
         -Drandom_array \
         -DscaLAPACK

Set the makefile compilation options:

FC=ftn
FCL=$(FC)

CPP_ = ./preprocess <$*.F | cpp -P -C -traditional >$*$(SUFFIX)

FFLAGS = -free -march=corei7-avx -assume byterecl  -m64

OFLAG  = -O3 -ip -fno-alias -unroll-aggressive -opt-prefetch -use-intel-optimized-headers -no-prec-div

OFLAG_LOW = -O1 -g -ftz
OBJ_LOW = broydon.o

Set the makefile linear algebra library options for Intel MKL:

MKL_PATH = $(MKLROOT)/lib/intel64
BLAS=
LAPACK=
BLACS=
SCA=

LIB  = ../vasp.5.lib/linpack_double.o -L../vasp.5.lib -ldmy \
        ${MKL_PATH}/libmkl_blas95_lp64.a ${MKL_PATH}/libmkl_lapack95_lp64.a \
        ${MKL_PATH}/libmkl_scalapack_lp64.a  \
        -Wl,--start-group ${MKL_PATH}/libmkl_intel_lp64.a \
        ${MKL_PATH}/libmkl_sequential.a ${MKL_PATH}/libmkl_core.a \
        ${MKL_PATH}/libmkl_blacs_intelmpi_lp64.a  -Wl,--end-group -lpthread -lm

Finally, set the makefile options for linking FFTW:

FFT3D   = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

Now build the code (assuming makefile is called "makefile.cray_xc_intel"):

make -f makefile.cray_xc_intel