YeoJohnsonTransformer

The Yeo-Johnson transformation is defined as:

../_images/yeojohnsonformula.png

where Y is the response variable and λ is the transformation parameter.

API Reference

class feature_engine.transformation.YeoJohnsonTransformer(variables=None)[source]

The YeoJohnsonTransformer() applies the Yeo-Johnson transformation to the numerical variables.

The Yeo-Johnson transformation implemented by this transformer is that of SciPy.stats: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html

The YeoJohnsonTransformer() works only with numerical variables.

A list of variables can be passed as an argument. Alternatively, the transformer will automatically select and transform all numerical variables.

Parameters
variables: list, default=None

The list of numerical variables to transform. If None, the transformer will automatically find and select all numerical variables.

Attributes

lambda_dict_

Dictionary containing the best lambda for the Yeo-Johnson per variable.

variables_:

The group of variables that will be transformed.

n_features_in_:

The number of features in the train set used in fit.

References

1

Weisberg S. “Yeo-Johnson Power Transformations”. https://www.stat.umn.edu/arc/yjpower.pdf

Methods

fit:

Learn the optimal lambda for the Yeo-Johnson transformation.

transform:

Apply the Yeo-Johnson transformation.

fit_transform:

Fit to data, then transform it.

fit(X, y=None)[source]

Learn the optimal lambda for the Yeo-Johnson transformation.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The training input samples. Can be the entire dataframe, not just the variables to transform.

y: pandas Series, default=None

It is not needed in this transformer. You can pass y or None.

Returns
self
Raises
TypeError
  • If the input is not a Pandas DataFrame

  • If any of the user provided variables are not numerical

ValueError
  • If there are no numerical variables in the df or the df is empty

  • If the variable(s) contain null values

transform(X)[source]

Apply the Yeo-Johnson transformation.

Parameters
X: Pandas DataFrame of shape = [n_samples, n_features]

The data to be transformed.

Returns
X: pandas dataframe

The dataframe with the transformed variables.

rtype

DataFrame ..

Raises
TypeError

If the input is not a Pandas DataFrame

ValueError
  • If the variable(s) contain null values

  • If the df has different number of features than the df used in fit()

Example

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

from feature_engine import transformation as vt

# Load dataset
data = data = pd.read_csv('houseprice.csv')

# Separate into train and test sets
X_train, X_test, y_train, y_test =  train_test_split(
            data.drop(['Id', 'SalePrice'], axis=1),
            data['SalePrice'], test_size=0.3, random_state=0)

# set up the variable transformer
tf = vt.YeoJohnsonTransformer(variables = ['LotArea', 'GrLivArea'])

# fit the transformer
tf.fit(X_train)

# transform the data
train_t= tf.transform(X_train)
test_t= tf.transform(X_test)

# un-transformed variable
X_train['LotArea'].hist(bins=50)
../_images/lotarearaw.png
# transformed variable
train_t['LotArea'].hist(bins=50)
../_images/lotareayeojohnson.png