Bash read 1000 lines of a file at a time until complete

Question

I am executing a bash script on a txt file that contains 10,000 lines.

Instead of split a file up into individual files with 1000 lines each and executing my loop over these splits, I would like to read 1000 lines at a time and run a function, then read the next 1000 lines and so on until the file has been read.

I tried head the file and then sed these lines out to no avail.

Any help would be much appreciated

Answer 1

You probably want to pass the individual chunks to your function. use a while loop like this.

i=1
chunk_file="yourfile.txt"
j=$(($i + 1000 - 1 ))
total_lines=`wc -l $yourfile | awk '{print $1}'`
#total_lines=10000
while [[ $i -le $total_lines ]]
do
echo $i,$j
sed -n "$i,$j p"  > ${chunk_file}
yourfunction ${chunk_file}

i=$(( $j + 1))
j=$(($i + 1000 - 1 ))

done