DecisionTreeEncoder¶
API Reference¶
-
class
feature_engine.encoding.
DecisionTreeEncoder
(encoding_method='arbitrary', cv=3, scoring='neg_mean_squared_error', param_grid=None, regression=True, random_state=None, variables=None)[source]¶ The DecisionTreeEncoder() encodes categorical variables with predictions of a decision tree model.
Each categorical feature is recoded by training a decision tree, typically of limited depth (2, 3 or 4) using that feature alone, and let the tree directly predict the target. The probabilistic predictions of this decision tree are used as the new values of the original categorical feature, that now is linearly (or at least monotonically) correlated with the target.
In practice, the categorical variable will be first encoded into integers with the OrdinalCategoricalEncoder(). The integers can be assigned arbitrarily to the categories or following the mean value of the target in each category. Then a decision tree will fit the resulting numerical variable to predict the target variable. Finally, the original categorical variable values will be replaced by the predictions of the decision tree.
Note that a decision tree is fit per every single categorical variable to encode.
- Parameters
- encoding_methodstr, default=’arbitrary’
The categorical encoding method that will be used to encode the original categories to numerical values.
‘ordered’: the categories are numbered in ascending order according to the target mean value per category.
‘arbitrary’ : categories are numbered arbitrarily.
- cvint, default=3
Desired number of cross-validation fold to be used to fit the decision tree.
- scoringstr, default=’neg_mean_squared_error’
Desired metric to optimise the performance for the decision tree. Comes from sklearn.metrics. See the DecisionTreeRegressor or DecisionTreeClassifier model evaluation documentation for more options: https://scikit-learn.org/stable/modules/model_evaluation.html
- regressionboolean, default=True
Indicates whether the encoder should train a regression or a classification decision tree.
- param_griddictionary, default=None
The list of parameters over which the decision tree should be optimised during the grid search. The param_grid can contain any of the permitted parameters for Scikit-learn’s DecisionTreeRegressor() or DecisionTreeClassifier().
If None, then param_grid = {‘max_depth’: [1, 2, 3, 4]}.
- random_stateint, default=None
The random_state to initialise the training of the decision tree. It is one of the parameters of the Scikit-learn’s DecisionTreeRegressor() or DecisionTreeClassifier(). For reproducibility it is recommended to set the random_state to an integer.
- variableslist, default=None
The list of categorical variables that will be encoded. If None, the encoder will find and select all object type variables.
Attributes
encoder_ :
sklearn Pipeline containing the ordinal encoder and the decision tree.
See also
sklearn.ensemble.DecisionTreeRegressor
sklearn.ensemble.DecisionTreeClassifier
feature_engine.discretisation.DecisionTreeDiscretiser
feature_engine.encoding.RareLabelEncoder
Notes
The authors designed this method originally, to work with numerical variables. We can replace numerical variables by the preditions of a decision tree utilising the DecisionTreeDiscretiser().
NAN are introduced when encoding categories that were not present in the training dataset. If this happens, try grouping infrequent categories using the RareLabelEncoder().
References
- 1
Niculescu-Mizil, et al. “Winning the KDD Cup Orange Challenge with Ensemble Selection”. JMLR: Workshop and Conference Proceedings 7: 23-34. KDD 2009 http://proceedings.mlr.press/v7/niculescu09/niculescu09.pdf
Methods
fit:
Fit a decision tree per variable.
transform:
Replace categorical variable by the predictions of the decision tree.
fit_transform:
Fit to the data, then transform it.
-
fit
(X, y=None)[source]¶ Fit a decision tree per variable.
- Parameters
- Xpandas dataframe of shape = [n_samples, n_features]
The training input samples. Can be the entire dataframe, not just the categorical variables.
- ypandas series.
The target variable. Required to train the decision tree and for ordered ordinal encoding.
- Returns
- self
- Raises
- TypeError
If the input is not a Pandas DataFrame.
If any user provided variable is not categorical
- ValueError
If there are no categorical variables in the df or the df is empty
If the variable(s) contain null values
-
transform
(X)[source]¶ Replace categorical variable by the predictions of the decision tree.
- Parameters
- Xpandas dataframe of shape = [n_samples, n_features]
The input samples.
- Returns
- Xpandas dataframe of shape = [n_samples, n_features].
Dataframe with variables encoded with decision tree predictions.
- rtype
DataFrame
..
- Raises
- TypeError
If the input is not a Pandas DataFrame
- ValueError
If the variable(s) contain null values
If dataframe is not of same size as that used in fit()
- Warning
If after encoding, NAN were introduced.
Example¶
The DecisionTreelEncoder() replaces categories in the variable with the predictions of a decision tree. The transformer first encodes categorical variables into numerical variables using ordinal encoding. You have the option to have the integers assigned to the categories as they appear in the variable, or ordered by the mean value of the target per category. After this, the transformer fits with this numerical variable a decision tree to predict the target variable. Finally, the original categorical variable is replaced by the predictions of the decision tree.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from feature_engine.encoding import DecisionTreeEncoder
# Load dataset
def load_titanic():
data = pd.read_csv('https://www.openml.org/data/get_csv/16826755/phpMYEkMl')
data = data.replace('?', np.nan)
data['cabin'] = data['cabin'].astype(str).str[0]
data['pclass'] = data['pclass'].astype('O')
data['embarked'].fillna('C', inplace=True)
return data
data = load_titanic()
# Separate into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
data.drop(['survived', 'name', 'ticket'], axis=1),
data['survived'], test_size=0.3, random_state=0)
X_train[['cabin', 'pclass', 'embarked']].head(10)
cabin pclass embarked
501 n 2 S
588 n 2 S
402 n 2 C
1193 n 3 Q
686 n 3 Q
971 n 3 Q
117 E 1 C
540 n 2 S
294 C 1 C
261 E 1 S
# set up the encoder
encoder = DecisionTreeEncoder(variables=['cabin', 'pclass', 'embarked'], random_state=0)
# fit the encoder
encoder.fit(X_train, y_train)
# transform the data
train_t = encoder.transform(X_train)
test_t = encoder.transform(X_test)
train_t[['cabin', 'pclass', 'embarked']].head(10)
cabin pclass embarked
501 0.304843 0.307580 0.338957
588 0.304843 0.307580 0.338957
402 0.304843 0.307580 0.558011
1193 0.304843 0.307580 0.373494
686 0.304843 0.307580 0.373494
971 0.304843 0.307580 0.373494
117 0.649533 0.617391 0.558011
540 0.304843 0.307580 0.338957
294 0.649533 0.617391 0.558011
261 0.649533 0.617391 0.338957