OutlierTrimmer

class feature_engine.outliers.OutlierTrimmer(capping_method='gaussian', tail='right', fold=3, variables=None, missing_values='raise')

The OutlierTrimmer() removes observations with outliers from the dataset.

The OutlierTrimmer() first calculates the maximum and /or minimum values beyond which a value will be considered an outlier, and thus removed.

The extreme values beyond which an observation is considered an outlier are determined using:

• a Gaussian approximation

• the inter-quantile range proximity rule (IQR)

• MAD-median rule (MAD)

• percentiles

Gaussian limits:

• right tail: mean + 3* std

• left tail: mean - 3* std

IQR limits:

• right tail: 75th quantile + 3* IQR

• left tail: 25th quantile - 3* IQR

where IQR is the inter-quartile range: 75th quantile - 25th quantile.

MAD limits:

• right tail: median + 3* MAD

• left tail: median - 3* MAD

where MAD is the median absoulte deviation from the median.

percentiles:

• right tail: 95th percentile

• left tail: 5th percentile

You can select how far out to cap the maximum or minimum values with the parameter 'fold'.

If capping_method='gaussian' fold gives the value to multiply the std.

If capping_method='iqr' fold is the value to multiply the IQR.

If capping_method='mad' fold is the value to multiply the MAD.

If capping_method='quantiles', fold is the percentile on each tail that should be censored. For example, if fold=0.05, the limits will be the 5th and 95th percentiles. If fold=0.1, the limits will be the 10th and 90th percentiles.

The OutlierTrimmer() works only with numerical variables. A list of variables can be indicated. Alternatively, it will select all numerical variables.

The transformer first finds the values at one or both tails of the distributions (fit). The transformer then removes observations with outliers from the dataframe (transform).

More details in the User Guide.

Parameters

capping_method: str, default=’gaussian’

Desired outlier detection method. Can be ‘gaussian’, ‘iqr’, ‘mad’, ‘quantiles’.

The transformer will find the maximum and / or minimum values beyond which a data point will be considered an outlier using: ‘gaussian’: the Gaussian approximation. ‘iqr’: the IQR proximity rule. ‘quantiles’: the percentiles. ‘mad’: the Gaussian approximation but using robust statistics.

tail: str, default=’right’

Whether to look for outliers on the right, left or both tails of the distribution. Can take ‘left’, ‘right’ or ‘both’.

fold: int or float, default=0.05 if `quantile`, or 3 otherwise.

The factor used to multiply the std, MAD or IQR to calculate the maximum or minimum allowed values. Recommended values are 2 or 3 for the gaussian approximation, 1.5 or 3 for the IQR proximity rule and 3 or 3.5 for MAD rule.

If capping_method='quantile', then 'fold' indicates the percentile. So if fold=0.05, the limits will be the 95th and 5th percentiles.

Note: Outliers will be removed up to a maximum of the 20th percentiles on both sides. Thus, when capping_method='quantile', then 'fold' takes values between 0 and 0.20.

variables: list, default=None

The list of numerical variables to transform. If None, the transformer will automatically find and select all numerical variables.

missing_values: string, default=’raise’

Indicates if missing values should be ignored or raised. If 'raise' the transformer will return an error if the the datasets to fit or transform contain missing values. If 'ignore', missing data will be ignored when learning parameters or performing the transformation.

Attributes

right_tail_caps_:

Dictionary with the maximum values beyond which a value will be considered an outlier.

left_tail_caps_:

Dictionary with the minimum values beyond which a value will be considered an outlier.

variables_:

The group of variables that will be transformed.

feature_names_in_:

List with the names of features seen during fit.

n_features_in_:

The number of features in the train set used in fit.

Examples

>>> import pandas as pd
>>> from feature_engine.outliers import OutlierTrimmer
>>> X = pd.DataFrame(dict(x = [0.49671,
>>>                         -0.1382,
>>>                          0.64768,
>>>                          1.52302,
>>>                         -0.2341,
>>>                         -17.2341,
>>>                          1.57921,
>>>                          0.76743,
>>>                         -0.4694,
>>>                          0.54256]))
>>> ot = OutlierTrimmer(capping_method='gaussian', tail='left', fold=3)
>>> ot.fit(X)
>>> ot.transform(X)
          x
0   0.49671
1  -0.13820
2   0.64768
3   1.52302
4  -0.23410
5 -17.23410
6   1.57921
7   0.76743
8  -0.46940
9   0.54256

>>> import pandas as pd
>>> from feature_engine.outliers import OutlierTrimmer
>>> X = pd.DataFrame(dict(x = [0.49671,
>>>                         -0.1382,
>>>                          0.64768,
>>>                          1.52302,
>>>                         -0.2341,
>>>                         -17.2341,
>>>                          1.57921,
>>>                          0.76743,
>>>                         -0.4694,
>>>                          0.54256]))
>>> ot = OutlierTrimmer(capping_method='mad', tail='left', fold=3)
>>> ot.fit(X)
>>> ot.transform(X)
         x
0  0.49671
1 -0.13820
2  0.64768
3  1.52302
4 -0.23410
6  1.57921
7  0.76743
8 -0.46940
9  0.54256

Methods

fit:	Find maximum and minimum values.
fit_transform:	Fit to data, then transform it.
get_feature_names_out:	Get output feature names for transformation.
get_params:	Get parameters for this estimator.
set_params:	Set the parameters of this estimator.
transform:	Remove outliers.
transform_x_y:	Remove rows with outliers from X set and y.

fit(X, y=None)

Learn the values that should be used to replace outliers.

Parameters

Xpandas dataframe of shape = [n_samples, n_features]

The training input samples.

ypandas Series, default=None

y is not needed in this transformer. You can pass y or None.

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters

Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns

X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_feature_names_out(input_features=None)

Get output feature names for transformation. In other words, returns the variable names of transformed dataframe.

Parameters

input_featuresarray or list, default=None

This parameter exits only for compatibility with the Scikit-learn pipeline.

• If None, then feature_names_in_ is used as feature names in.

• If an array or list, then input_features must match feature_names_in_.

Returns

feature_names_out: list

Transformed feature names.

rtype

List[Union[str, int]] ..

get_metadata_routing()

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns

routingMetadataRequest

A MetadataRequest encapsulating routing information.

get_params(deep=True)

Get parameters for this estimator.

Parameters

deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns

paramsdict

Parameter names mapped to their values.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters

**paramsdict

Estimator parameters.

Returns

selfestimator instance

Estimator instance.

transform(X)

Remove observations with outliers from the dataframe.

Parameters

Xpandas dataframe of shape = [n_samples, n_features]

The data to be transformed.

Returns

X_new: pandas dataframe of shape = [n_samples, n_features]

The dataframe without outlier observations.

rtype

DataFrame ..

transform_x_y(X, y)

Transform, align and adjust both X and y based on the transformations applied to X, ensuring that they correspond to the same set of rows if any were removed from X.

Parameters

X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to transform.

y: pandas Series or Dataframe of length = n_samples

The target variable to transform. Can be multi-output.

Returns

X_new: pandas dataframe

The transformed dataframe of shape [n_samples - n_rows, n_features]. It may contain less rows than the original dataset.

y_new: pandas Series or DataFrame

The transformed target variable of length [n_samples - n_rows]. It contains as many rows as those left in X_new.