Feature selection is a critical step that affects the ML model performance directly. Reducing the number of features has two main benefits in developing machine learning models.
- First, extracting a subset from the whole feature list helps to build a more accurate model by removing the noise caused by unnecessary features.
- Second, model training time significantly decreases with a decreased number of features. The decrease is linear in the worst case depending on the training algorithm. There are primarily two methods
There are primarily two methods in feature selection.
Filter methods measure each feature’s correlation (independent variable) and the target (dependent) variable separately. Creating feature set and check accuracy. Check correlation with each IV and DV.
Second one is Wrapper Method. They have two different approaches; forward selection proposes bottom-up and backward elimination top-down strategy. In both ways, an ML model is built by selecting a learning algorithm. Based on its accuracy, features are either added or subtracted from the model until the best feature subset is generated. 1) Forward Selection Proposes Bottom-up, 2) Backward Elimination Top-down Strategy.
Filter Method
- How much a feature is corelated with target attribute.
- Same we check for all features and check with target attribute.
- There is dependency on attribute from target attribute.
- Relevance of attribute we check in filter method.
Wrapper method
- f = feature, t=target
- We choose f1 and t1 then created a model and checked accuracy.
- We choose f1, f2, and t1, then created a model, and check accuracy.
- we choose f1, f3 and t1, then created a model, and checked accuracy.
- Now we have multiple models, then it will choose most useful model based on accuracy. This model has used which features, those subsets of features will be used.
To identified the threshold in feature selection filtered method, you can generate the heat map of all features. Choose the best features using any Feature Selection Method, e.g. Chi Square, mRMR method.
There are two types of feature selection techniques, Supervised and Unsupervised Method. In supervised method, there is Wrapper Method, Filter Method, and Intrinsic Method.
About The Author
