Should we always first perform feature normalization and then the feature reduction?

Question

Sometimes performing feature reduction reduces number of features with methods like PCA and then we could scale only the relevant variables. Is there a rule that we need to do normalization/scaling first and then the feature reduction?

Answer 1

I would suggest first do your normalization/scaling on your feature data and then performing feature selection. This is because most of the feature selection techniques require a meaningful representation of your data. By normalizing your data your features have the same order of magnitude and scatter, which makes it easier to find which one of those is more relevant.

For example, for PCA the computation is based on the standard deviation (SD) of your features to find the relevant axis of a new projection of your data. If you do not normalize your data, features with a high SD will have a higher weight compared to features with a small SD distorting their relevance when computing the PCA.