I am saving a Dataframe as an HDF file in Pandas, using:
store = pandas.HDFStore('myfile.h5', table=True)
store.append('df', df, chunksize=100000)
store.close()
I have this code in a Luigi pipeline, and it works fine - except when there is some error in the writing process (such as a mixed type column), such that myfile.h5 appears on the drive but isn't fully written. Luigi thinks that task is complete, and then the next task fails because the myfile.h5 isn't written fully.
How can I tell Python to delete myfile.h5 if store.append doesn't fully complete?