DropMissingData

class feature_engine.imputation.DropMissingData(missing_only=True, threshold=None, variables=None)[source]

DropMissingData() will delete rows containing missing values. It provides similar functionality to pandas.drop_na().

It works for numerical and categorical variables. You can enter the list of variables for which missing values should be evaluated. Alternatively, the imputer will evaluate missing data in all variables in the dataframe.

More details in the User Guide.

Parameters
variables: list, default=None

The list of variables to consider for the imputation. If None, the imputer will evaluate missing data in all variables in the dataframe. Alternatively, the imputer will evaluate missing data only in the variables in the list.

Note that if missing_only=True only variables with missing data in the train set will be considered to drop a row, which might be a subset of the indicated list.

missing_only: bool, default=True

If True, rows will be dropped when they show missing data in variables that had missing data during fit(). If False, rows will be dropped if there is missing data in any of the variables. This parameter only works when threshold=None, otherwise it is ignored.

threshold: int or float, default=None

Require that percentage of non-NA values in a row to keep it. If threshold=1, all variables need to have data to keep the row. If threshold=0.5, 50% of the variables need to have data to keep the row. If threshold=0.01, 10% of the variables need to have data to keep the row. If thresh=None, rows with NA in any of the variables will be dropped.

Attributes
variables_:

The variables for which missing data will be examined to decide if a row is dropped. The attribute variables_ is different from the parameter variables when the latter is None, or when only a subset of the indicated variables show NA in the train set if missing_only=True.

feature_names_in_:

List with the names of features seen during fit.

n_features_in_:

The number of features in the train set used in fit.

Methods

fit:

Find the variables for which missing data should be evaluated.

fit_transform:

Fit to data, then transform it.

get_feature_names_out:

Get output feature names for transformation.

get_params:

Get parameters for this estimator.

set_params:

Set the parameters of this estimator.

return_na_data:

Returns a dataframe with the rows that contain missing data.

transform:

Remove rows with missing data.

fit(X, y=None)[source]

Find the variables for which missing data should be evaluated to decide if a row should be dropped.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The training data set.

y: pandas Series, default=None

y is not needed in this imputation. You can pass None or y.

fit_transform(X, y=None, **fit_params)[source]

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters
Xarray-like of shape (n_samples, n_features)

Input samples.

yarray-like of shape (n_samples,) or (n_samples, n_outputs), default=None

Target values (None for unsupervised transformations).

**fit_paramsdict

Additional fit parameters.

Returns
X_newndarray array of shape (n_samples, n_features_new)

Transformed array.

get_feature_names_out(input_features=None)[source]

Get output feature names for transformation.

Parameters
input_features: str, list, default=None

If None, then the names of all the variables in the transformed dataset is returned. If list with feature names, the features in the list will be returned. This parameter exists mostly for compatibility with the Scikit-learn Pipeline.

Returns
feature_names_out: list

The feature names.

:rtype:py:class:~typing.List[Union[str, int]]
get_params(deep=True)[source]

Get parameters for this estimator.

Parameters
deepbool, default=True

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns
paramsdict

Parameter names mapped to their values.

return_na_data(X)[source]

Returns the subset of the dataframe with the rows with missing values. That is, the subset of the dataframe that would be removed with the transform() method. This method may be useful in production, for example if we want to store or log the removed observations, that is, rows that will not be fed into the model.

Parameters
X_na: pandas dataframe of shape = [n_samples_with_na, features]

The subset of the dataframe with the rows with missing data.

:rtype:py:class:~pandas.core.frame.DataFrame
set_params(**params)[source]

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters
**paramsdict

Estimator parameters.

Returns
selfestimator instance

Estimator instance.

transform(X)[source]

Remove rows with missing data.

Parameters
X: pandas dataframe of shape = [n_samples, n_features]

The dataframe to be transformed.

Returns
X_new: pandas dataframe

The complete case dataframe for the selected variables, of shape [n_samples - n_samples_with_na, n_features]

:rtype:py:class:~pandas.core.frame.DataFrame