How to delete abnormal values from particular variable(col) from dataFrame when you have data in lacs

Question

plt.xticks(np.arange(0, len(x) + 1)[::365], x[::365])

plt.plot(dates, CentreLiftEffectiveCurrent)

plt.title('CentreLiftEffectiveCurrent')

I'm getting 1e^38 abnormal values for my variable CentreLiftEffectiveCurrent. how can i remove them and plot the graph again with the desired values

Answer 1

A usually good way to filter "abnormal" values is to filter the outliers, i.e. values that are outside a range around the median values fo your dataset:

rainfall = df["Rainfall"]
q3 = np.quantile(rainfall, 0.75)
q1 = np.quantile(rainfall, 0.25)

iqr = q3 - q1

upper_bound = q1 + 1.5 * iqr
lower_bound = q3 - 1.5 * iqr

rainfall_wo_outliers = df[(rainfall >= lower_bound) | (rainfall <= upper_bound)]["Rainfall"]

you can plot a box plot to see these outliers:

df.boxplot()

ps: sorry this is q3 and not q2 on the sketch

you can see here also a discussion about this