How to loop in a Yaml file calling SQLFluff lint for each file that has changed

170 views Asked by At

I am answering my own question bc I couldn't find a str8 answer anywhere and if I can help others then I can pay back where some have helped me.

All of the syntax below has been tested and what I am using now so you should be able to copy/paste it into your CICD provided you have installed SQLfluff in the docker image and can access your git repo within the pipeline (reference credentials).

Script:

- echo "SQL Fluff Job Script section"
- sqlfluff --version

#set this to 0 so if the linter doesn't fail then its a 0 (known value)
- SQLFluffStepStatusFail=0

# iterate over all of the incoming changes (diff between last commit and this feature branch calling the linter with tsql dialect, nofail (for now) and get the output (/1 to check))
- |
    for file in $( git diff --name-only $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA -- *.sql ); 
    do
        sqlfluff lint $file --dialect tsql --annotation-level warning --disable-noqa --nofail && SQLFluffStepStatus=$(echo $?) || export SQLFluffStepStatus=1
    
        if [ "$SQLFluffStepStatus" != 0 ];
        then
            # We just need to know 1 failed but iterate through all so when the dev look at the resulting 
            # output he/she sees all of it rather than having to keep running to see future failures
            SQLFluffStepStatusFail=1
        fi
    done

# now check if there were any failures and send the message if needed and stop the PL
- |
   if [ "$SQLFluffStepStatusFail" != 0 ];
   then
      echo "SQLFluffStepStatusFail $SQLFluffStepStatusFail"
      chmod +x teams-chat-post.sh
      ./teams-chat-post.sh "$teamsurl" "$title" "$failcolor" "$failbody"
       exit 1
   fi
0

There are 0 answers