๐ˆ๐ฆ๐ฉ๐จ๐ซ๐ญ๐š๐ง๐ญ ๐…๐ž๐š๐ญ๐ฎ๐ซ๐ž ๐’๐ž๐ฅ๐ž๐œ๐ญ๐ข๐จ๐ง ๐“๐ž๐œ๐ก๐ง๐ข๐ช๐ฎ๐ž๐ฌ

2 minute read

Published:

In a Machine Learning problem, not all features may contribute to the final prediction. Sometimes, reducing the number of features can improve a modelโ€™s performance or interpretability. Regardless of the reason, itโ€™s often beneficial to remove irrelevant or unwanted features.

Generally, ๐ญ๐ก๐ซ๐ž๐ž methods are widely used for feature selection:

  • ๐…๐ข๐ฅ๐ญ๐ž๐ซ ๐›๐š๐ฌ๐ž๐ ๐ฆ๐ž๐ญ๐ก๐จ๐๐ฌ
  • ๐–๐ซ๐š๐ฉ๐ฉ๐ž๐ซ ๐ฆ๐ž๐ญ๐ก๐จ๐๐ฌ
  • ๐„๐ฆ๐›๐ž๐๐๐ž๐ ๐ฆ๐ž๐ญ๐ก๐จ๐๐ฌ In todayโ€™s post we will be going through the first one - ๐…๐ข๐ฅ๐ญ๐ž๐ซ ๐›๐š๐ฌ๐ž๐ ๐ฆ๐ž๐ญ๐ก๐จ๐๐ฌ. For python implementation of these methods visit my kaggle notebook.

๐…๐ข๐ฅ๐ญ๐ž๐ซ ๐›๐š๐ฌ๐ž๐ ๐ฆ๐ž๐ญ๐ก๐จ๐: Itโ€™s usually best to begin your feature selection using a univariate method, as these are easy to implement and time-efficient.

  1. ๐ƒ๐ž๐ฅ๐ž๐ญ๐ข๐ง๐  ๐๐ฎ๐ฉ๐ฅ๐ข๐œ๐š๐ญ๐ž ๐œ๐จ๐ฅ๐ฎ๐ฆ๐ง๐ฌ: Your dataset may contain duplicate columns, possibly with different column names. Itโ€™s always a good idea to identify and remove these duplicates.

  2. ๐•๐š๐ซ๐ข๐š๐ง๐œ๐ž ๐“๐ก๐ซ๐ž๐ฌ๐ก๐จ๐ฅ๐: Columns with zero or low variability usually contribute little to changes in the target variable (y). Thus, you may want to remove them. You can do this manually, or use the dedicated function in scikit-learn for this task. One major challenge with this method is selecting the right threshold. It often requires personal experience and trial and error to determine the optimal threshold for your data.

  3. ๐‚๐จ๐ซ๐ซ๐ž๐ฅ๐š๐ญ๐ข๐จ๐ง: The Pearson correlation coefficient is utilized to filter out features. Features with high correlation are considered to contribute equally to the final prediction, so it doesnโ€™t hurt to remove one of them.

  4. ๐€๐๐Ž๐•๐€: This method examines the relationship between each individual feature and the target variable (categorical). It determines the strength of the relationship using the p-value (from the F-statistic) for each feature, and retains the k best features. Scikit-learn provides a dedicated function called SelectKBest that can be used with f_classif or f_regression to select the k most important features. Finding the optimal value for k may require some trial and error or hyperparameter tuning.

  5. ๐‚๐ก๐ข-๐ฌ๐ช๐ฎ๐š๐ซ๐ž: The chi-square test can be used to study the dependency between categorical features and a categorical target variable, retaining those that show significant dependency.

While these filter-based methods are a good starting point for feature selection, they have some major flaws: - Being a univariate method, they donโ€™t consider the interaction between features and target variables. - There is a chance that we may delete features bearing low importance individually, but having significant impact when combined with other features.

On my next post I will be going through the wrapper method in details.