- class feature_engine.selection.RecursiveFeatureAddition(estimator, scoring='roc_auc', cv=3, threshold=0.01, variables=None)¶
RecursiveFeatureAddition() selects features following a recursive addition process.
The process is as follows:
Train an estimator using all the features.
Rank the features according to their importance derived from the estimator.
Train an estimator with the most important feature and determine performance.
Add the second most important feature and train a new estimator.
Calculate the difference in performance between estimators.
If the performance increases beyond the threshold, the feature is kept.
Repeat steps 4-6 until all features have been evaluated.
Model training and performance calculation are done with cross-validation.
More details in the User Guide.
- estimator: object
A Scikit-learn estimator for regression or classification. The estimator must have either a
coef_attribute after fitting.
- variables: str or list, default=None
The list of variable to be evaluated. If None, the transformer will evaluate all numerical features in the dataset.
- scoring: str, default=’roc_auc’
Desired metric to optimise the performance of the estimator. Comes from sklearn.metrics. See the model evaluation documentation for more options: https://scikit-learn.org/stable/modules/model_evaluation.html
- threshold: float, int, default = 0.01
The value that defines if a feature will be kept or removed. Note that for metrics like roc-auc, r2_score and accuracy, the thresholds will be floats between 0 and 1. For metrics like the mean_square_error and the root_mean_square_error the threshold can be a big number. The threshold must be defined by the user. Bigger thresholds will select less features.
- cv: int, cross-validation generator or an iterable, default=3
Determines the cross-validation splitting strategy. Possible inputs for cv are:
None, to use cross_validate’s default 5-fold cross validation
int, to specify the number of folds in a (Stratified)KFold,
An iterable yielding (train, test) splits as arrays of indices.
For int/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used. These splitters are instantiated with
shuffle=Falseso the splits will be the same across calls. For more details check Scikit-learn’s
- initial_model_performance_ :
Performance of the model trained using the original dataset.
- feature_importances_ :
Pandas Series with the feature importance (comes from step 2)
Dictionary with the performance drift per examined feature (comes from step 5).
List with the features to remove from the dataset.
The variables that will be considered for the feature selection.
The number of features in the train set used in fit.
Find the important features.
Reduce X to the selected features.
Fit to data, then transform it.
- fit(X, y)¶
Find the important features. Note that the selector trains various models at each round of selection, so it might take a while.
- X: pandas dataframe of shape = [n_samples, n_features]
The input dataframe
- y: array-like of shape (n_samples)
Target variable. Required to train the estimator.
- fit_transform(X, y=None, **fit_params)¶
Fit to data, then transform it.
Fits transformer to
ywith optional parameters
fit_paramsand returns a transformed version of
- Xarray-like of shape (n_samples, n_features)
- yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None
Target values (None for unsupervised transformations).
Additional fit parameters.
- X_newndarray array of shape (n_samples, n_features_new)
Get parameters for this estimator.
- deepbool, default=True
If True, will return the parameters for this estimator and contained subobjects that are estimators.
Parameter names mapped to their values.
Set the parameters of this estimator.
The method works on simple estimators as well as on nested objects (such as
Pipeline). The latter have parameters of the form
<component>__<parameter>so that it’s possible to update each component of a nested object.
- selfestimator instance
Return dataframe with selected features.
- X: pandas dataframe of shape = [n_samples, n_features].
The input dataframe.
- X_new: pandas dataframe of shape = [n_samples, n_selected_features]
Pandas dataframe with the selected features.