๐๐ฆ๐ฉ๐จ๐ซ๐ญ๐๐ง๐ญ ๐ ๐๐๐ญ๐ฎ๐ซ๐ ๐๐๐ฅ๐๐๐ญ๐ข๐จ๐ง ๐๐๐๐ก๐ง๐ข๐ช๐ฎ๐๐ฌ
Published:
In a Machine Learning problem, not all features may contribute to the final prediction. Sometimes, reducing the number of features can improve a modelโs performance or interpretability. Regardless of the reason, itโs often beneficial to remove irrelevant or unwanted features.
Generally, ๐ญ๐ก๐ซ๐๐ methods are widely used for feature selection:
- ๐ ๐ข๐ฅ๐ญ๐๐ซ ๐๐๐ฌ๐๐ ๐ฆ๐๐ญ๐ก๐จ๐๐ฌ
- ๐๐ซ๐๐ฉ๐ฉ๐๐ซ ๐ฆ๐๐ญ๐ก๐จ๐๐ฌ
- ๐๐ฆ๐๐๐๐๐๐ ๐ฆ๐๐ญ๐ก๐จ๐๐ฌ In todayโs post we will be going through the first one - ๐ ๐ข๐ฅ๐ญ๐๐ซ ๐๐๐ฌ๐๐ ๐ฆ๐๐ญ๐ก๐จ๐๐ฌ. For python implementation of these methods visit my kaggle notebook.
๐ ๐ข๐ฅ๐ญ๐๐ซ ๐๐๐ฌ๐๐ ๐ฆ๐๐ญ๐ก๐จ๐: Itโs usually best to begin your feature selection using a univariate method, as these are easy to implement and time-efficient.
๐๐๐ฅ๐๐ญ๐ข๐ง๐ ๐๐ฎ๐ฉ๐ฅ๐ข๐๐๐ญ๐ ๐๐จ๐ฅ๐ฎ๐ฆ๐ง๐ฌ: Your dataset may contain duplicate columns, possibly with different column names. Itโs always a good idea to identify and remove these duplicates.
๐๐๐ซ๐ข๐๐ง๐๐ ๐๐ก๐ซ๐๐ฌ๐ก๐จ๐ฅ๐: Columns with zero or low variability usually contribute little to changes in the target variable (y). Thus, you may want to remove them. You can do this manually, or use the dedicated function in scikit-learn for this task. One major challenge with this method is selecting the right threshold. It often requires personal experience and trial and error to determine the optimal threshold for your data.
๐๐จ๐ซ๐ซ๐๐ฅ๐๐ญ๐ข๐จ๐ง: The Pearson correlation coefficient is utilized to filter out features. Features with high correlation are considered to contribute equally to the final prediction, so it doesnโt hurt to remove one of them.
๐๐๐๐๐: This method examines the relationship between each individual feature and the target variable (categorical). It determines the strength of the relationship using the p-value (from the F-statistic) for each feature, and retains the k best features. Scikit-learn provides a dedicated function called SelectKBest that can be used with f_classif or f_regression to select the k most important features. Finding the optimal value for k may require some trial and error or hyperparameter tuning.
๐๐ก๐ข-๐ฌ๐ช๐ฎ๐๐ซ๐: The chi-square test can be used to study the dependency between categorical features and a categorical target variable, retaining those that show significant dependency.
While these filter-based methods are a good starting point for feature selection, they have some major flaws: - Being a univariate method, they donโt consider the interaction between features and target variables. - There is a chance that we may delete features bearing low importance individually, but having significant impact when combined with other features.
On my next post I will be going through the wrapper method in details.
