Script to write .csv using cron job produces empty cells

Question

Background

I am using a raspberrypi zero to test the internet speed with the speedtest --csv command.

I have written a script that is placed in /usr/local/bin/speedtest_script.sh:

#!/bin/bash

# Define the file path for storing results
FILE_PATH="/home/marko/speedtest/speedtest_raspberrypi.csv"

# Run the speedtest and capture the results
SPEEDTEST_RESULT=$(speedtest --csv)

# Check if the CSV file exists; if not, create the header
if [ ! -f "$FILE_PATH" ]; then
   echo "Date,Time,Ping (ms),Download,Upload,Location,Distance" > "$FILE_PATH"
fi

# Extract and format the data from the `speedtest` output
DATE=$(date '+%Y/%m/%d')
TIME=$(date '+%H:%M:%S')
PING=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $6}')
DOWNLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $7}')
UPLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $8}')
LOCATION=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $3}')
DISTANCE=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $5}')

# Append the formatted results to the CSV file
echo "$DATE,$TIME,$PING,$DOWNLOAD,$UPLOAD,$LOCATION,$DISTANCE" >> "$FILE_PATH"

The file speedtest_script.sh has been made into an executable using the commands sudo chmod +x speedtest_script.sh and sudo chmod +w speedtest_script.sh to give it read and write permissions.

My Problem

With this background, here is the problem I am facing:

As seen in the code, the script is supposed to generate a file speedtest_raspberrypi.csv in /home/marko/speedtest/speedtest_raspberrypi.csv. When I manually run the command speedtest_script.sh, it executes perfectly. However, as part of a cron job, I only get the date and time, but not the other relevant information.

A snippet of the speedtest_raspberrypi.csv is given below:

Date,Time,Ping (ms),Download,Upload,Location,Distance
2023/10/29,14:35:10,28.941,33661436.554366775,12456063.667141322,MyCity,196.5378436042561
2023/10/29,14:45:02,,,,,

The first line of results was generated when I manually executed the command. The second line of results was generated as part of the cron job.

Here is what I have written in my crontab -e:

# m h  dom mon dow   command
*/15 * * * * speedtest_script.sh

Question

Why does my script generate the correct output when run manually, but not when part of a cronjob?

Answer 1

Firstly I think you have a misconception -

sudo chmod +w speedtest_script.sh doesn't give the script read and write permissions. Rather it gives write permission on that file for all categories of users. This means that after the command is executed, any user will be able to modify the speedtest_script.sh script itself.

i.e. rather than giving "read and write permissions" to it, you are giving users write permissions on it.

Secondly, to answer your question "Why does my script generate the correct output when run manually, but not when part of a cronjob?" - I would guess your issue is that cron jobs do not have the same PATH environment variable as the user. So if speedtest is not found in the default PATH that the cron job has, it will fail to execute the command. Hence you not seeing any data.

To fix this just ensure your script has the correct path to speedtest explicitly set. e.g.

run the command

which speedtest

It should output something like /usr/bin/speedtest

Then just use that absolute path in your script, e.g.

#!/bin/bash

# Define the file path for storing results
FILE_PATH="/home/marko/speedtest/speedtest_raspberrypi.csv"

# Run the speedtest and capture the results using the absolute path
# for example...
SPEEDTEST_RESULT=$(/usr/bin/speedtest --csv)

# Check if the CSV file exists; if not, create the header
if [ ! -f "$FILE_PATH" ]; then
   echo "Date,Time,Ping (ms),Download,Upload,Location,Distance" > "$FILE_PATH"
fi

# Extract and format the data from the `speedtest` output
DATE=$(date '+%Y/%m/%d')
TIME=$(date '+%H:%M:%S')
PING=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $6}')
DOWNLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $7}')
UPLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $8}')
LOCATION=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $3}')
DISTANCE=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $5}')

# Append the formatted results to the CSV file
echo "$DATE,$TIME,$PING,$DOWNLOAD,$UPLOAD,$LOCATION,$DISTANCE" >> "$FILE_PATH"

Answer 2

A general recommendation for testing errors in scripts where you can't directly access STDERR (such as when it's run via crontab): append 2>> /tmp/stderr.log (or a similar log location) to any lines that you think might be producing errors. Then after the crontab has run, you can go back and review /tmp/stderr.log to see what errors were generated.

For example, you might modify your script like this:

#!/bin/bash

# Define the file path for storing results
FILE_PATH="/home/marko/speedtest/speedtest_raspberrypi.csv"

# Run the speedtest and capture the results
SPEEDTEST_RESULT=$(speedtest --csv 2>> /tmp/stderr.log)

# Check if the CSV file exists; if not, create the header
if [ ! -f "$FILE_PATH" ]; then
   echo "Date,Time,Ping (ms),Download,Upload,Location,Distance" > "$FILE_PATH"
fi

# Extract and format the data from the `speedtest` output
DATE=$(date '+%Y/%m/%d')
TIME=$(date '+%H:%M:%S')
PING=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $6}')
DOWNLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $7}')
UPLOAD=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $8}')
LOCATION=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $3}')
DISTANCE=$(echo "$SPEEDTEST_RESULT" | awk -F ',' '{print $5}')

# Append the formatted results to the CSV file
echo "$DATE,$TIME,$PING,$DOWNLOAD,$UPLOAD,$LOCATION,$DISTANCE" >> "$FILE_PATH"

I suspect that the error is likely that you need the absolute path of speedtest (as @Fraser mentioned), but if the error is due to something else this method will also help you troubleshoot (and is useful in general).