DropMissingData#

class feature_engine.imputation.DropMissingData(missing_only=True, threshold=None, variables=None)[source]#

DropMissingData() deletes rows containing missing values. It provides similar functionality to pandas.drop_na(), but within the fit and transform framework.

It works for numerical and categorical variables. You can enter the list of variables for which missing values should be removed. Alternatively, the imputer will find and remove missing data in all dataframe variables.

More details in the User Guide.

Parameters
variables: list, default=None

The list of variables to consider for the imputation. If None, the imputer will check missing data in all variables in the dataframe. Alternatively, the imputer will evaluate missing data only in the variables in the list.

Note that if missing_only=True, missing data will be removed from variables that had missing data in the train set. These might be a subset of the variables indicated in the list.

missing_only: bool, default=True

If True, rows will be dropped when they show missing data in variables that had missing data during fit(). If False, rows will be dropped if there is missing data in any of the variables. This parameter only works when threshold=None, otherwise it is ignored.

threshold: int or float, default=None

Require that percentage of non-NA values in a row to keep it. If threshold=1, all variables need to have data to keep the row. If threshold=0.5, 50% of the variables need to have data to keep the row. If threshold=0.01, 10% of the variables need to have data to keep the row. If thresh=None, rows with NA in any of the variables will be dropped.

Attributes
variables_:

The variables for which missing data will be examined to decide if a row is dropped. The attribute variables_ is different from the parameter variables when the latter is None, or when only a subset of the indicated variables show NA in the train set if missing_only=True.

feature_names_in_:

List with the names of features seen during fit.

n_features_in_:

The number of features in the train set used in fit.

Examples

>>> import pandas as pd
>>> import numpy as np
>>> from feature_engine.imputation import DropMissingData
>>> X = pd.DataFrame(dict(
>>>        x1 = [np.nan,1,1,0,np.nan],
>>>        x2 = ["a", np.nan, "b", np.nan, "a"],
>>>        ))
>>> dmd = DropMissingData()
>>> dmd.fit(X)
>>> dmd.transform(X)
    x1 x2
2  1.0  b

Methods

fit:

Find the variables for which missing data should be evaluated.

fit_transform:

Fit to data, then transform it.

get_feature_names_out:

Get output feature names for transformation.

get_params:

Get parameters for this estimator.

set_params:

Set the parameters of this estimator.

return_na_data:

Returns a dataframe with the rows that contain missing data.

transform:

Remove rows with missing data.

transform_x_y:

Remove rows with missing data from X and y.

fit(X, y=None)[source]#

Find the variables for which missing data should be evaluated to decide if a row should be dropped.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The training data set.

y: pandas Series or dataframe, default=None

y is not needed in this imputation. You can pass None or y.

fit_transform(X, y=None, **fit_params)[source]#

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters
Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns
X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_feature_names_out(input_features=None)[source]#

Get output feature names for transformation. In other words, returns the variable names of transformed dataframe.

Parameters
input_featuresarray or list, default=None

This parameter exits only for compatibility with the Scikit-learn pipeline.

  • If None, then feature_names_in_ is used as feature names in.

  • If an array or list, then input_features must match feature_names_in_.

Returns
feature_names_out: list

Transformed feature names.

rtype

List[Union[str, int]] ..

get_metadata_routing()[source]#

Get metadata routing of this object.

Please check User Guide on how the routing mechanism works.

Returns
routingMetadataRequest

A MetadataRequest encapsulating routing information.

get_params(deep=True)[source]#

Get parameters for this estimator.

Parameters
deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns
paramsdict

Parameter names mapped to their values.

return_na_data(X)[source]#

Returns the subset of the dataframe with the rows with missing values. That is, the subset of the dataframe that would be removed with the transform() method. This method may be useful in production, for example if we want to store or log the removed observations, that is, rows that will not be fed into the model.

Parameters
X_na: pandas dataframe of shape = [n_samples_with_na, features]

The subset of the dataframe with the rows with missing data.

:rtype: :py:class:`~pandas.core.frame.DataFrame`
set_params(**params)[source]#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters
**paramsdict

Estimator parameters.

Returns
selfestimator instance

Estimator instance.

transform(X)[source]#

Remove rows with missing data.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to be transformed.

Returns
X_new: pandas dataframe

The complete case dataframe for the selected variables, of shape [n_samples - n_samples_with_na, n_features]

rtype

DataFrame ..

transform_x_y(X, y)[source]#

Transform, align and adjust both X and y based on the transformations applied to X, ensuring that they correspond to the same set of rows if any were removed from X.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to transform.

y: pandas Series or Dataframe of length = n_samples

The target variable to transform. Can be multi-output.

Returns
X_new: pandas dataframe

The transformed dataframe of shape [n_samples - n_rows, n_features]. It may contain less rows than the original dataset.

y_new: pandas Series or DataFrame

The transformed target variable of length [n_samples - n_rows]. It contains as many rows as those left in X_new.