DropSimilar#

class skrub.DropSimilar(threshold=1)[source]#

Drop columns found too redundant to the rest of the dataframe, according to association defined by Cramér’s V.

This is done by computing Cramér’s V between every possible two columns, and sorting these couples in descending order. Then, for every association above the given threshold, one of the two columns is dropped.

Parameters:

thresholdfloat, default=1: The Cramér association score value above which to start dropping columns.

Attributes:

to_drop_list: The names of columns evaluated for removal
all_outputs_list: The names of columns that the transformer keeps
column_associations_dataframe: A dataframe with columns `left_column_name’, ‘right_column_name’ and ‘cramer_v’ listing association scores between every pair of columns.

See also

DropUninformative: Drops a single column if certain criteria indicate that it contains little to no information (amount of nulls, of distinct values…)
Cleaner: Runs several checks to sanitize a dataframe, including converting columns to standard formats or dropping certain columns.

Examples

>>> from skrub import DropSimilar
>>> from skrub.datasets import toy_cities
>>> df = toy_cities(size=5000, n_metrics=0)
>>> df.head()
          uid   cities  encoded_cities               start                 end
0  SHAoqcdajQ  Vilnius            17.0 2004-09-02 03:22:56 2014-06-26 11:36:43
1  HVAFYLGCDW      NaN             NaN 1979-10-22 01:43:56 1987-10-15 06:30:05
2  oQIauSCbNL     Rome            13.0 1986-08-09 19:01:10 2002-04-06 03:56:09
3  SjeSbCepzv  Vilnius            17.0 2008-11-26 15:57:13 2021-11-16 23:16:13
4  ubagaIBHnG   London             8.0 1982-09-13 20:54:54 2000-12-02 07:36:41

>>> ds = DropSimilar(threshold=0.8)
>>> clean_df = ds.fit_transform(df)

ds has now removed a column for each pair with association above 0.8. These associations are stored in the table_associations_ attribute:

>>> ds.table_associations_.head()
  left_column_name right_column_name  cramer_v
         cities    encoded_cities  1.000000
            uid            cities  0.052979
            uid    encoded_cities  0.052979
 encoded_cities               end  0.046455
         cities               end  0.046455

A single pair is above the threshold, cities and encoded_cities, with an association score of 1. Since one is an encoding of the other, this is to be expected. Therefore, one of these two has been marked as dropped by ds:

>>> ds.to_drop_
['encoded_cities']

This leaves us with the shortened dataframe:

>>> clean_df.head()
          uid   cities               start                 end
SHAoqcdajQ  Vilnius 2004-09-02 03:22:56 2014-06-26 11:36:43
HVAFYLGCDW      NaN 1979-10-22 01:43:56 1987-10-15 06:30:05
oQIauSCbNL     Rome 1986-08-09 19:01:10 2002-04-06 03:56:09
SjeSbCepzv  Vilnius 2008-11-26 15:57:13 2021-11-16 23:16:13
ubagaIBHnG   London 1982-09-13 20:54:54 2000-12-02 07:36:41

Methods

`fit_transform`(X[, y])	Fit to data, then transform it.
`get_params`([deep])	Get parameters for this estimator.
`set_output`(*[, transform])	Set output container.
`set_params`(**params)	Set the parameters of this estimator.

fit
get_feature_names_out
transform

fit_transform(X, y=None)[source]#

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters:

Xarray_like of shape (n_samples, n_features): Input samples.
yarray_like of shape (n_samples,) or (n_samples, n_outputs), default=None: Target values (None for unsupervised transformations).
**fit_paramsdict: Additional fit parameters. Pass only if the estimator accepts additional params in its fit method.

Returns:

X_newndarray array of shape (n_samples, n_features_new): Transformed array.

get_params(deep=True)[source]#

Get parameters for this estimator.

Parameters:

deepbool, default=True: If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

paramsdict: Parameter names mapped to their values.

set_output(*, transform=None)[source]#

Set output container.

Refer to the user guide for more details and Introducing the set_output API for an example on how to use the API.

Parameters:

transform{“default”, “pandas”, “polars”}, default=None

Configure output of transform and fit_transform.

"default": Default output format of a transformer
"pandas": DataFrame output
"polars": Polars output
None: Transform configuration is unchanged

Added in version 1.4: "polars" option was added.

Returns:

selfestimator instance: Estimator instance.

set_params(**params)[source]#

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as Pipeline). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Parameters:

**paramsdict: Estimator parameters.

Returns:

selfestimator instance: Estimator instance.