How to fix SettingWithCopyWarning?

Question

I'm trying to create a new column in my dataframe and get a SettingWithCopyWarning error. I've had a look at the other questions about this on the site but can't seem to fix the issue that I'm having.

This was the original code:

tcep['time_difference'] = (pd.to_datetime(tcep['time_end']) - pd.to_datetime(tcep['time_start'])).dt.days

When that came up with the error saying :

A value is trying to be set on a copy of a slice from a Dataframe

I tried the following:

tcep['time_difference'] = tcep.apply(lambda row: ((pd.to_datetime(row.time_end) - pd.to_datetime(row.time_start)).days, axis=1)

But that's also coming up with the same error. Would anyone know how to fix this?

Answer 1

The SettingWithCopyWarning warning is raised in pandas when you try to set a value on a df that might be a view (or slice) of another df, rather than a copy. This means you could change the original dataframe without realizing it.

To ensure that tcep is not a view on some other dataframe, you can create a copy explicitly and then operate like the following :

tcep = tcep.copy()
tcep['time_difference'] = (pd.to_datetime(tcep['time_end']) - pd.to_datetime(tcep['time_start'])).dt.days

If you use the apply method, there's a typo in your code. You're missing a closing parenthesis for the lambda function :

tcep = tcep.copy()
tcep['time_difference'] = tcep.apply(lambda row: (pd.to_datetime(row.time_end) - pd.to_datetime(row.time_start)).days, axis=1)