How to extract specific columns in log file to csv file

Question

I0625 17:25:22.544378  3366 solver.cpp:229] Iteration 7120, loss = 8.79839

expected output :

Iteration 7120 loss = 8.79839

Answer 1

You would probably need to use a regular expression to find lines that contain the values you want and to extract them. These lines can then be written in CSV format using Python's CSV library as follows:

import re
import csv

with open('log.txt') as f_input, open('output.csv', 'w', newline='') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(['Iteration', 'loss'])

    for line in f_input:
        re_values = re.search(r'Iteration (\d+), loss = ([0-9.]+)', line)

        if re_values:
            csv_output.writerow(re_values.groups())

Giving you output.csv in CSV format as follows:

Iteration,loss
7120,1.79839
7120,1.79839
7120,1.79839
7120,1.79839