I have some code that reads in multiple csv files into a pandas dataframe. The problem is that the first two lines of all the files need to be ignored and I cannot figure out how to do this.

import pandas as pd
import glob
import os

path = r'D:\E\Traficc\migration\Zambia-Mining\DATA\24monthimport'                     # use your path
all_files = glob.glob(os.path.join(path, "*.csv"))     # advisable to use os.path.join as this makes concatenation OS independent

df_from_each_file = (pd.read_csv(f) for f in all_files)

data   = pd.concat(df_from_each_file, ignore_index=True)
# doesn't create a list, nor does it append to one


I have tried to use next(df) but I am getting an error that df is not iterable.

Can I do all this within the existing 1 line loop or do I need to break it up? What can I use to accomplish this?

0 Answers