how to append the text from daughter files to a single mother file and deleting headers from daughter files

Question

During a structural simulation, I get the following response in a file.txt:

constant                                             
        date 03/23/2011

{BEGIN LEGEND}
 Entity #        Title
    1     blank                                                          
    2     die                                                            
    3     blank                                                         
{END LEGEND}

 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01   
 ....
 .....
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01


 .... are the thousands of lines

I get this file 1 time per second like file1.txt file2. txt etc...

I have to join all the files to make single file out of that so that if I copy the next file in this file I dont get any of the text from the upcoming files which is given below but the remaining text to be appended at the end

 constant                                             
        date 03/23/2011

{BEGIN LEGEND}
 Entity #        Title
    1     blank                                                          
    2     die                                                            
    3     blank                                                         
{END LEGEND}

This above mentioned text needs to be deleted from the upcoming files, stays only from the first one.

I should get:

 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01 
 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01 
 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01 
 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01 
 slave           1 time 1.12999E+01  x  4.81992E-03  y  1.69202E+01  z  1.94541E+01  
 master          1 time 1.12999E+01  x -4.81991E-03  y -1.69202E+01  z -1.94541E+01 

can anyone guide me with awk, sed and cat? I am ok to use piping >> too.

Answer 1

If the header is constant across the files, so is the line count of the header. Thus, the quickest (and maybe dirtiest) way to achieve this is the tail command:

head -n 10 file1.txt > outputFile
fileCount=$(echo file*.txt | wc -w)
for ((i = 1 ; i <= $fileCount ; i++)) ; do
    tail -n +11 file$i.txt >> outputFile
done

wc -w produces the number of words that the glob expression file*.txt results in, i. e. the count of the input files. The arguments -n +11 tell tail to start output at the 11th line of each file.

Answer 2

Apparently you want to print everything from the moment a line starts with slave.

Hence, you can say:

awk '$1=="slave" {f=1} f' f1

so that this is accomplished. It uses a flag f that is activated when it finds a line having slave as first word. From that moment on, the condition f will evaluate to True so that awk will perform its default action: print.

If you happen to contain all the files within the same directory, just loop through them and output accordingly:

for files in /your/path/*
do
    awk '$1=="slave" {f=1} f' "$file" >> new_file
done

It is not very clear if the first "good" line might also start with master. If so, the awk expression should be like:

awk '$1=="slave" || $1=="master" {f=1} f' file