Calculate variance in bash

Question

I want to compute the variance of an input txt file like this one:

1, 5
2, 5
3, 5
4, 10

And I want the output to be like:

1, 0
2, 0 
3, 0
4, 4.6875

I've used this line:

awk '{c[NR]=$2; s=s+c[NR]; avg= s / NR; var=var+(($2 - avg)^2 / (NR )); print var }' inputfile > outputfile

Answer 1

Standard deviation formula is described in http://www.mathsisfun.com/data/standard-deviation.html

So basically you need to say:

for i in items
   sum += [(item - average)^2]/#items

Doing it in your sample input:

5   av=5/1=5       var=(5-5)/1=0
5   av=10/2=5      var=(5-5)^2+(5-5)^2/2=0
5   av=15/3=5      var=3*(5-5)^2/3=0
10  av=25/4=6.25   var=3*(5-6.25)^2+(10-6.25)^2/4=4.6875

So in awk we can say:

$ awk 'BEGIN {FS=OFS=","}      # set comma as field input/output separator
       {a[NR]=$2               # store data in an array
        sum+=a[NR]             # keep track of the sum
        av=sum/NR              # calculate average so far
        v=0                    # reset counter for variance
        for (i=1;i<=NR;i++)    # loop through all the values
             v+=(a[i]-av)^2    # calculate the variance
        print $1, v/NR}        # print the 1st field + result
  ' file

Test

$ awk 'BEGIN {FS=OFS=","} {a[NR]=$2; sum+=a[NR]; av=sum/NR; v=0; for (i=1;i<=NR;i++) v+=(a[i]-av)^2; print $1, v/NR}' a
1,0
2,0
3,0
4,4.6875