No account yet ?

When applying the k-nearest-neighbor approach to solve a problem, we may sometimes notice that there is a deterioration in the performance of kNN when the number of predictors, p, is large.

The reason for this may be the high number of dimensions.

This problem is known as the curse of dimensionality.

This means that test error tends to increase as the dimensionality of the problem (number of predictors) increases, unless the additional features are truly associated with the answer.

This goes against the thinking one might have that as the number of predictors used to fit a model increases, the quality of the fitted model also increases.

How to avoid the curse of dimensionality?

We can use various feature selection algorithms like best subset selection or stepwise selection to select a subset of predictors.

Another approach is to use PCA (Principal Component Analysis), a dimension reduction method, which transforms the p predictors into M < p predictors.

PCA is a feature extraction method.

The model is fitted using these M predictors.

PCR (Principal Components Regression), based on PCA, provides a way to perform regression using M < p predictors.

Source : https://kharshit.github.io/blog/2018/01/26/the-curse-of-dimensionality