Condor job - running shell script as executable

Question

I’m trying to run a Condor job where the executable is a shell script which invokes certain Java classes.

Universe = vanilla
Executable = /script/testingNew.sh
requirements = (OpSys == "LINUX")
Output =  /locfiles/myfile.out
Log = /locfiles/myfile.log
Error = /locfiles/myfile.err
when_to_transfer_output = ON_EXIT
Notification = Error
Queue

Here’s the content for /script/testingNew.sh file – (Just becaz I’m getting error, I have removed the Java commands for now)

#!/bin/sh
inputfolder=/n/test_avp/test-modules/data/json
srcFolder=/n/test_avp/test-modules
logsFolder=/n/test_avp/test-modules/log
libFolder=/n/test_avp/test-modules/lib
confFolder=/n/test_avp/test-modules/conf
twpath=/n/test_avp/test-modules/normsrc
dataFolder=/n/test_avp/test-modules/data
scriptFolder=/n/test_avp/test-modules/script
locFolder=/n/test_avp/test-modules/locfiles
bakUpFldr=/n/test_avp/test-modules/backupCurrent

cd $inputfolder
filename=`date -u +"%Y%m%d%H%M"`.txt
echo  $filename  $(date -u)

mkdir $bakUpFldr/`date -u +"%Y%m%d"`

dirname=`date -u +"%Y%m%d"`
flnme=current_json_`date -u +"%Y%m%d%H%M%S"`.txt
echo DIRNameis $dirname Filenameis $flnme

cp $dataFolder/current_json.txt  $bakUpFldr/`date -u +"%Y%m%d"`/current_json_$filename
cp $dataFolder/current_json.txt   $filename

mkdir $inputfolder/`date -u +"%Y%m%d"`
echo Creating Directory  $(date -u)

mv $filename $filename.inprocess
echo Created Inprocess file  $(date -u)

Also, here’s the error log from Condor –

000 (424639.000.000) 09/09 16:08:18 Job submitted from host: <135.207.178.237:9582>
...
001 (424639.000.000) 09/09 16:08:35 Job executing on host: <135.207.179.68:9314>
...
007 (424639.000.000) 09/09 16:08:35 Shadow exception!
    Error from slot1@marcus-8: Failed to execute '/n/test_avp/test-modules/script/testingNew.sh': (errno=8: 'Exec format error')
    0  -  Run Bytes Sent By Job
    0  -  Run Bytes Received By Job
...
012 (424639.000.000) 09/09 16:08:35 Job was held.
    Error from slot1@marcus-8: Failed to execute '/n/test_avp/test-modules/script/testingNew.sh': (errno=8: 'Exec format error')
    Code 6 Subcode 8
...

Can anyone explain whats causing this error, also how to resolve this? The testingNew.sh scripts run fine on the Linux box, if executed on a network machine seperately. Thx a lot!! - GR

Answer 1

You need to specify input as:

input   = /dev/null

Source: Submitting a job to Condor

Answer 2

The cause, in our case, was the shell script using DOS line endings instead of Unix ones.

The Linux kernel will happily try to feed the script not to /bin/sh (as you intend) but to /bin/sh . (Do you see that trailing carriage return character? Neither do I, but the Linux kernel does.) That file doesn't exist, so then, as a last resort, it will try to execute the script as a binary executable, which fails with the given error.