RareLabelEncoder#

The RareLabelEncoder() groups infrequent categories into one new category called ‘Rare’ or a different string indicated by the user. We need to specify the minimum percentage of observations a category should have to be preserved and the minimum number of unique categories a variable should have to be re-grouped.

tol

In the parameter tol we indicate the minimum proportion of observations a category should have, not to be grouped. In other words, categories which frequency, or proportion of observations is <= tol will be grouped into a unique term.

n_categories

In the parameter n_categories we indicate the minimum cardinality of the categorical variable in order to group infrequent categories. For example, if n_categories=5, categories will be grouped only in those categorical variables with more than 5 unique categories. The rest of the variables will be ignored.

This parameter is useful when we have big datasets and do not have time to examine all categorical variables individually. This way, we ensure that variables with low cardinality are not reduced any further.

max_n_categories

In the parameter max_n_categories we indicate the maximum number of unique categories that we want in the encoded variable. If max_n_categories=5, then the most popular 5 categories will remain in the variable after the encoding, all other will be grouped into a single category.

This parameter is useful if we are going to perform one hot encoding at the back of it, to control the expansion of the feature space.

Example

Let’s look at an example using the Titanic Dataset.

First, let’s load the data and separate it into train and test:

from sklearn.model_selection import train_test_split
from feature_engine.datasets import load_titanic
from feature_engine.encoding import RareLabelEncoder

X, y = load_titanic(
    return_X_y_frame=True,
    handle_missing=True,
    predictors_only=True,
    cabin="letter_only",
)
X["pclass"] = X["pclass"].astype("O")

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0,
)

print(X_train.head())

We see the resulting data below:

      pclass     sex        age  sibsp  parch     fare cabin embarked
501        2  female  13.000000      0      1  19.5000     M        S
588        2  female   4.000000      1      1  23.0000     M        S
402        2  female  30.000000      1      0  13.8583     M        C
1193       3    male  29.881135      0      0   7.7250     M        Q
686        3  female  22.000000      0      0   7.7250     M        Q

Let’s explore the number of uniue categories in the variable "cabin".

X_train["cabin"].unique()

We see the number of unique categories in the output below:

array(['M', 'E', 'C', 'D', 'B', 'A', 'F', 'T', 'G'], dtype=object)

Now, we set up the RareLabelEncoder() to group categories shown by less than 3% of the observations into a new group or category called ‘Rare’. We will group the categories in the indicated variables if they have more than 2 unique categories each.

encoder = RareLabelEncoder(
    tol=0.03,
    n_categories=2,
    variables=['cabin', 'pclass', 'embarked'],
    replace_with='Rare',
)

# fit the encoder
encoder.fit(X_train)

With fit(), the RareLabelEncoder() finds the categories present in more than 3% of the observations, that is, those that will not be grouped. These categories can be found in the encoder_dict_ attribute.

encoder.encoder_dict_

In the encoder_dict_ we find the most frequent categories per variable to encode. Any category that is not in this dictionary, will be grouped.

{'cabin': ['M', 'C', 'B', 'E', 'D'],
'pclass': [3, 1, 2],
'embarked': ['S', 'C', 'Q']}

Now we can go ahead and transform the variables:

# transform the data
train_t = encoder.transform(X_train)
test_t = encoder.transform(X_test)

Let’s now inspect the number of unique categories in the variable "cabin" after the transformation:

train_t["cabin"].unique()

In the output below, we see that the infrequent categories have been replaced by "Rare".

array(['M', 'E', 'C', 'D', 'B', 'Rare'], dtype=object)

We can also specify the maximum number of categories that can be considered frequent using the max_n_categories parameter.

Let’s begin by creating a toy dataframe and count the values of observations per category:

from feature_engine.encoding import RareLabelEncoder
import pandas as pd
data = {'var_A': ['A'] * 10 + ['B'] * 10 + ['C'] * 2 + ['D'] * 1}
data = pd.DataFrame(data)
data['var_A'].value_counts()
A    10
B    10
C     2
D     1
Name: var_A, dtype: int64

In this block of code, we group the categories only for variables with more than 3 unique categories and then we plot the result:

rare_encoder = RareLabelEncoder(tol=0.05, n_categories=3)
rare_encoder.fit_transform(data)['var_A'].value_counts()
A       10
B       10
C        2
Rare     1
Name: var_A, dtype: int64

Now, we retain the 2 most frequent categories of the variable and group the rest into the ‘Rare’ group:

rare_encoder = RareLabelEncoder(tol=0.05, n_categories=3, max_n_categories=2)
Xt = rare_encoder.fit_transform(data)
Xt['var_A'].value_counts()
A       10
B       10
Rare     3
Name: var_A, dtype: int64

Tips#

The RareLabelEncoder() can be used to group infrequent categories and like this control the expansion of the feature space if using one hot encoding.

Some categorical encodings will also return NAN if a category is present in the test set, but was not seen in the train set. This inconvenient can usually be avoided if we group rare labels before training the encoders.

Some categorical encoders will also return NAN if there is not enough observations for a certain category. For example the WoEEncoder() and the PRatioEncoder(). This behaviour can be also prevented by grouping infrequent labels before the encoding with the RareLabelEncoder().

More details#

In the following notebook, you can find more details into the RareLabelEncoder() functionality and example plots with the encoded variables:

For more details about this and other feature engineering methods check out these resources: