Feature scaling in an incremental analysis

262 views Asked by At

I'm doing an incremental analysis of my data. The data belongs to 4 age groups (day1, day2, day3 and day4). Before I feed my data to the model, I standardize the features using the standard scaler implementation in sklearn. When I think of it, 3 approaches comes to my mind.

Approach (1)standardize the newly added data separately
days = [day1, day2, day3, day4]

data=[]
for day in days:
    standard_scaler = StandardScaler()
    scaled = standard_scaler.fit_transform(day)
    data.append(scaled)
    Y = model.fit_transform(data)

Approach (2)standardize all the data up to the current day together separately
days = [day1, day2, day3, day4]

data=[]
for day in days:
    data.append(day)
    standard_scaler = StandardScaler()
    scaled = standard_scaler.fit_transform(data)
    Y = model.fit_transform(scaled)

Approach (3)partial_fit the same standard scaler on the newly added increments
    days = [day1, day2, day3, day4]
    standard_scaler = StandardScaler()

    data=[]
    for day in days:
        standard_scaler.partial_fit(day)
        data.append(day)
        scaled = standard_scaler.transform(data)
       
        Y = model.fit_transform(scaled)

Please advise on which method would be best suited.

1

There are 1 answers

4
Sandeep Kothari On

approach 1 is the best one and in fact the only correct one