parallel make with pgi compiler

Question

I have a set of c++ files that I compile on different machines using make -j command. An issue appears only when I use the pgi compiler with parallel make, I get the following error for some files (this particular error is for the atprop.cpp file):

    /opt/share/gcc/4.6.0/el6/bin/ld: error in atprop.o(.eh_frame); no .eh_frame_hdr table will be created.
    /opt/share/gcc/4.6.0/el6/bin/ld: atprop.o: invalid string offset 615811912 >= 1421 for section `.strtab'
    /opt/share/gcc/4.6.0/el6/bin/ld: final link failed: Nonrepresentable section on output

I realized then that this error occure for source files that are related to many executable targets. For example atprop.cpp is compiled three times with different macros (no macros, -DUBC, -DVBC) which I use within the code. This is the part of the Makefile that is related to atprop.cpp

    $(B)/atprop: $(S)/atprop.cpp  $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o
            $(CPPC) $(S)/atprop.cpp $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o -o $@ $(INC)  $(FLAGSET2)

    $(B)/atprop_ubc: $(S)/atprop.cpp  $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o
            $(CPPC) $(S)/atprop.cpp $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o -o $@ $(INC)  $(FLAGSET2) -DUBC

    $(B)/atprop_vbc: $(S)/atprop.cpp  $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o
            $(CPPC) $(S)/atprop.cpp $(O)/atlib.o $(O)/atwave.o $(O)/atvecop.o -o $@ $(INC)  $(FLAGSET2) -DVBC

The variable $(CPPC)=mpic++ which is equal to the path for MPI wrapper for the PGI compiler.

    $ mpic++ --version

    pgcpp 12.10-0 64-bit target on x86-64 Linux -tp bulldozer 
    Copyright 1989-2000, The Portland Group, Inc.  All Rights Reserved.
    Copyright 2000-2012, STMicroelectronics, Inc.  All Rights Reserved.

Note that the compilation runs smoothly with no problems if I use serial make (i.e. make without the -j option).

I think this is the problem: PGI compiler creates a temporary object files with the same name as source file (for the example above atprop.o. If run in parallel, compilation with different macros will all write to the same object file, which causes the problems I mentioned above.

This happens only with PGI compiler, I don't face this issue when I use intel or gnu compilers. So my question is what can I do to alleviate this problem with PGI compiler? Keep in mind that I have a lot of c++ files with the same problem and I am trying to avoid a major modification to the Makefile.

Answer 1

To determine if your guess about what the PGI compiler is doing is correct, you can try running it (by hand from the command line) using strace, perhaps with the -ff and -o flags, and look to see what files it creates.

If you determine that it is this problem, the first thing you should do is look through the PGI documentation and/or ask on their support forums to see if there are compiler options you can add that will change this behavior, or environment variables you can set.

If none of that works, you're in a world of hurt. A simple answer would be to change your make rules to copy the source to a unique name before compiling, but that destroys the ability to debug it (since the source file encoded by the debugger will be some random filename). Any change I can think of would involve very significant changes to your makefiles (for example, adding order-only prerequisites to the rules where multiple files are built from the same source).

It's too bad that your makefiles appear to be simply long lists of explicit rules; if you were using a smaller number of implicit rules instead it would be easier to fix the makefiles.

Answer 2

For a quick workaround, I made fake dependencies between targets in the Makefile. This way I avoid compiling the same source file at the same time (for different targets).