Keep same dummy variable in training and testing data

Question

I am building a prediction model in python with two separate training and testing sets. The training data contains numerical type categorical variable, e.g., zip code,[91521,23151,12355, ...], and also string categorical variables, e.g., city ['Chicago', 'New York', 'Los Angeles', ...].

To train the data, I first use the 'pd.get_dummies' to get dummy variable of these variable, and then fit the model with the transformed training data.

I do the same transformation on my test data and predict the result using the trained model. However, I got the error

ValueError: Number of features of the model must  match the input. Model n_features is 1487 and  input n_features is 1345

The reason is because there are fewer dummy variables in the test data because it has fewer 'city' and 'zipcode'.

How can I solve this problem? For example, 'OneHotEncoder' will only encode all numerical type categorical variable. 'DictVectorizer()' will only encode all string type categorical variable. I search on line and see a few similar questions but none of them really addresses my question.

Handling categorical features using scikit-learn

https://www.quora.com/If-the-training-dataset-has-more-variables-than-the-test-dataset-what-does-one-do

https://www.quora.com/What-is-the-best-way-to-do-a-binary-one-hot-one-of-K-coding-in-Python

Answer 1

Assume you have identical feature's names in train and test dataset. You can generate concatenated dataset from train and test, get dummies from concatenated dataset and split it to train and test back.

You can do it this way:

import pandas as pd
train = pd.DataFrame(data = [['a', 123, 'ab'], ['b', 234, 'bc']],
                     columns=['col1', 'col2', 'col3'])
test = pd.DataFrame(data = [['c', 345, 'ab'], ['b', 456, 'ab']],
                     columns=['col1', 'col2', 'col3'])
train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset_preprocessed = pd.get_dummies(dataset)
train_preprocessed = dataset_preprocessed[:train_objs_num]
test_preprocessed = dataset_preprocessed[train_objs_num:]

In result, you have equal number of features for train and test dataset.

Answer 2

This is a rather old question, but if you aim at using scikit learn API, you can use the following DummyEncoder class: https://gist.github.com/psinger/ef4592492dc8edf101130f0bf32f5ff9

What it does is that it utilizes the category dtype to specify which dummies to create as also elaborated here: Dummy creation in pipeline with different levels in train and test set

Answer 3

You can also just get the missing columns and add them to the test dataset:

# Get missing columns in the training test
missing_cols = set( train.columns ) - set( test.columns )
# Add a missing column in test set with default value equal to 0
for c in missing_cols:
    test[c] = 0
# Ensure the order of column in the test set is in the same order than in train set
test = test[train.columns]

This code also ensure that column resulting from category in the test dataset but not present in the training dataset will be removed

Answer 4

I have this in the past after having run get_dummies on both train and test sets

X_test = X_test.reindex(columns = X_train.columns, fill_value=0)

Obviously a little tweaking for the individual case. But, it throws away novel values in the test set and values missing from the test are filled in, in this case with all zeros.

Answer 5

For sklearn >= 0.20, OneHotEncoder can now encode string data.

from sklearn.preprocessing import OneHotEncoder
import pandas as pd

X_train = pd.DataFrame({
    'zip' : [23151, 12355],
    'city' : ['New York', 'Los Angeles']
})

X_test = pd.DataFrame({
    'zip' : [91521, 23151],
    'city' : ['Chicago', 'New York']
})

ohe = OneHotEncoder(handle_unknown='ignore', sparse_output=False) # New in version 1.2: sparse was renamed to sparse_output
X_train_ohe = ohe.fit_transform(X_train)
X_test_ohe = ohe.transform(X_test)

To get a clean Dataframe with the corresponding column names (similar to pd.get_dummies) :

cols_ohe = ohe.get_feature_names_out()
X_train_ohe = pd.DataFrame(X_train_ohe, columns=cols_ohe)
X_test_ohe = pd.DataFrame(X_test_ohe, columns=cols_ohe)

>>> X_train_ohe 
zip_12355   zip_23151   city_Los Angeles    city_New York
0.0         1.0         0.0                 1.0
1.0         0.0         1.0                 0.0

>>> X_test_ohe 
zip_12355   zip_23151   city_Los Angeles    city_New York
0.0         0.0         0.0                 0.0
0.0         1.0         0.0                 1.0

Answer 6

Convert zip code to str

use fit_transform() for training data and transform() for testing data in OneHotEncoder

Answer 7

train2,test2 = train.align(test, join='outer', axis=1, fill_value=0)

train2 and test2 have the same columns. Fill_value indicates the value to use for missing columns.