Expect Script working manually not via NiFi Processor

Question

I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in an SFTP directory. I am using expect to make this happen.

#!/bin/sh -e


USER=$1
export PASSWORD="${SFTP_PASSWORD}"
REMOTE_DIR=$3
HOST=$4

BATCH_FILE=$(mktemp)
echo "spawn sftp $USER@$HOST" > $BATCH_FILE
echo "expect \"*fingerprint*\"" >> $BATCH_FILE
echo "send \"yes\\r\"" >> $BATCH_FILE
echo "expect \"password:\"" >> $BATCH_FILE
echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"ls -l $REMOTE_DIR \\r\"" >> $BATCH_FILE
echo "expect \"sftp>\"" >> $BATCH_FILE
echo "send \"exit\\r\"" >> $BATCH_FILE

#expect $BATCH_FILE

dirs=$(expect $BATCH_FILE|  sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$0 ~ /^d/ {print $NF}')
rm $BATCH_FILE
total=0

SAVEIFS=$IFS
IFS=$'\n'
directories=($dirs)
IFS=$SAVEIFS


for value in "${directories[@]}"; do
       val=$(echo "$value" | tr -d '[:blank:]' | tr -d '[:space:]')
       if [[ "$val" == "Completed" || "$val" == "Failed" || "$val" == "Rejected" || "$val" == "Error" ]]; then
           dir=$REMOTE_DIR/$val
           BATCH_FILE=$(mktemp)
           echo "spawn sftp $USER@$HOST" > $BATCH_FILE
           echo "expect \"*fingerprint*\"" >> $BATCH_FILE
           echo "send \"yes\\r\"" >> $BATCH_FILE
           echo "expect \"password:\"" >> $BATCH_FILE
           echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE
           echo "expect \"sftp>\"" >> $BATCH_FILE
           echo "send \"ls -l $dir\\r\"" >> $BATCH_FILE
           echo "expect \"sftp>\"" >> $BATCH_FILE
           echo "send \"exit\\r\"" >> $BATCH_FILE
           count=$(expect $BATCH_FILE|  sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| wc -l)
           total=$((total + count))
           rm $BATCH_FILE
       fi
done

echo "$total"

This script is working fine when I run inside this manually with in the container, also is working fine when NiFi runs on standalone mode in 1.23.0

bash-4.4$ ./test_files.sh
4

test_files.sh has the same script but HOST, USER and REMOTE_DIR hardcoded.

When I call the same script from NiFi processor I am getting

execution.error
send: spawn id exp3 not open while executing "send "yes\r"" (file "/tmp/tmp.eatmyc06B6" line 3)

The fingerprint expect had to be added inside the container as it is being asked. when running outside NiFi it is not required.

Configuration for ExecuteStreamCommand

There are times when it works, in this screen grab, one has correct output the other doesn't

correct output:

Error and Incorrect output

I have tried the header with both /bash and /sh.

what could be the reason? Please advise.

Answer 1

Turns out there is a workaround, there is an option to change the first processor to execute in single node, this will execute flowfiles from single node.