Trying to figure out why I keep getting the message listed as the headline of this question. I think I already cleaned data, removing NaN's. Can anyone help me out?

Looking into a dataset with 11K lines, I am trying to make the code train data to predict level of students dropping out. Using an ordinary Windows laptop, whilst also exercising in getting better doing data analysis.

# divide the data set into categorial and non categorial features and apply models to get the insight of the data
print("\nDEFINING CATEGORICAL AND NUMERICAL FEATURES")
categorical_features = X.select_dtypes(include=['object']).columns
print(categorical_features)
numerical_features = X.select_dtypes(exclude = ["object"]).columns
print(numerical_features)

print("\nDIVIDE THE DATA SET INTO CATEGORIAL AND NON CATEGORIAL FEATURES AND APPLY MODELS TO GET THE INSIGHT OF THE DATA")
print("Numerical features : " + str(len(numerical_features)))
print("Categorical features : " + str(len(categorical_features)))

print("\nFILLING THE MISSING VALUE OF TEST WITH THEIR MEAN VALUE, FOR BETTER ACCURACY")

test = test.select_dtypes(exclude=[np.object])
test.info()
test = test.fillna(test.mean(), inplace=True)
print("\nAPPLYING MODEL RANDOM FOREST REGRESSOR")

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from warnings import simplefilter
# ignore all future warnings
simplefilter(action='ignore', category=FutureWarning)

# pull data into target (y) and predictors (X)
predictor_cols = ['F18 ECTS på kurser med beståede talkarakter']

# -------------------------------------------

# Create training predictors data
train_X = X[predictor_cols]
my_model = RandomForestRegressor()
my_model.fit(train_X, y)
my_model.score(train_X, y)
print(predictor_cols)
print(my_model.score(train_X, y))

test = pd.read_csv("…_test.csv")

# -------------------------------------------

print("\nPRINT PREDICTED FACTORS")
test_X = test[predictor_cols]

#  model to make predictions

predicted_factor = my_model.predict(test_X)

#  at the predicted prices to ensure something sensible.

print(predicted_factor)

Get most of my code running fine, except:

APPLYING MODEL RANDOM FOREST REGRESSOR
Traceback (most recent call last):
  File "C:/Users/jcst/PycharmProjects/Frafaldsanalyse/DefiningCatAndNumFeatures_4_new.py", line 142, in <module>
    my_model.fit(train_X, y)
  File "C:\Users\jcst\PycharmProjects\Frafaldsanalyse\venv\lib\site-packages\sklearn\ensemble\forest.py", line 250, in fit
    X = check_array(X, accept_sparse="csc", dtype=DTYPE)
  File "C:\Users\jcst\PycharmProjects\Frafaldsanalyse\venv\lib\site-packages\sklearn\utils\validation.py", line 573, in check_array
    allow_nan=force_all_finite == 'allow-nan')
  File "C:\Users\jcst\PycharmProjects\Frafaldsanalyse\venv\lib\site-packages\sklearn\utils\validation.py", line 56, in _assert_all_finite
    raise ValueError(msg_err.format(type_err, X.dtype))
ValueError: Input contains NaN, infinity or a value too large for dtype('float32').

Process finished with exit code 1

1 Answers

1
Ashargin On

As said, your dataset X_train or y must contain nans. Check again to see where that comes from. It typically comes from division by 0 or math functions domain error like log of negative values.

Something else you're gonna run in after :

You're using test = test.fillna(test.mean(), inplace=True)

You should use test = test.fillna(test.mean())

Or test.fillna(test.mean(), inplace=True)

When specifying inplace=True, the function returns None and so test is None.

Also you're doing all this without use since you're overwriting test by reading a DataFrame later. Maybe you have an unintended behavior here.