prevent duplicate variable and print using awk statement

Question

I am iterating through a file and printing a set of values using awk

echo $value | awk ' {print $4}' >> 'some location'

the command works fine , but I want to prevent the duplicate values being stored in the file

Thanks in advance.

Answer 1

Instead of processing the file line by line, you should use a single awk command for the entire file

For example:

awk '!a[$4]++{print $4}' file >> 'some location'

Will only keep the unique values of the fourth column

Answer 2

Using only one instance of awk as suggested by user000001 is certainly the right thing to do, and since very little detail is given in the question this is pure speculation, but the simplest solution may be a trivial refactor of your loop. For example, if the current code is:

while ...; do
  ...
  echo $value | awk ...
  ...
done

You can simply change it to:

while ...; do
    ...    
    echo $value >&5
    ...
done 5>&1 | awk '!a[$4]++{print $4}' >> /p/a/t/h

Note that although this is a "simple" fix in terms of code to change, it is almost certainly not the correct fix! Removing the while loop completely and just using awk is the right thing to do.