Got this error on an ubuntu server (100GB RAM) in anacondas py3.6 installation after it opened multiple files this way in a for loop (7 files * 4GB):
temp_df = pd.read_csv(datafolder + str(file), encoding="ISO-8859-1", delimiter=';',low_memory=False)
this is the error, it appeared after i set
low_memory=False
Doesnt happen when low_memory = True
* Error in `python': free(): invalid pointer: 0x00007fc3c90dc98e *
Anyone ideas? Thanks
This issue is caused due to multiple threads performed parallelly when opening a file , this causes sometimes segmentation fault as well. As you said your file consist of 4 gb each so multiple threads which are executed parallelly to read up your file is causing the issue.