.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/0010_apply_to_cols.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via JupyterLite or Binder. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_0010_apply_to_cols.py: Hands-On with Column Selection and Transformers =============================================== .. |ApplyToCols| replace:: :class:`~skrub.ApplyToCols` .. |StringEncoder| replace:: :class:`~skrub.StringEncoder` .. |SelectCols| replace:: :class:`~skrub.SelectCols` .. |DropCols| replace:: :class:`~skrub.DropCols` .. |TableVectorizer| replace:: :class:`~skrub.TableVectorizer` .. |OrdinalEncoder| replace:: :class:`~sklearn.preprocessing.OrdinalEncoder` .. |PCA| replace:: :class:`~sklearn.decomposition.PCA` .. |Pipeline| replace:: :class:`~sklearn.pipeline.Pipeline` .. |ColumnTransformer| replace:: :class:`~sklearn.compose.ColumnTransformer` In this example, we show how to create flexible pipelines by selecting and transforming dataframe columns using arbitrary logic with |ApplyToCols|. .. GENERATED FROM PYTHON SOURCE LINES 22-24 We begin with loading a dataset with heterogeneous datatypes, and replacing Pandas's display with the TableReport display via :func:`skrub.patch_display`. .. GENERATED FROM PYTHON SOURCE LINES 24-36 .. code-block:: Python import pandas as pd import skrub from skrub.datasets import fetch_employee_salaries skrub.patch_display() file_path = fetch_employee_salaries().path data = pd.read_csv(file_path) X = data.drop(columns="current_annual_salary") y = data["current_annual_salary"] X .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 37-49 Our goal is now to apply a |StringEncoder| to two columns of our choosing: ``division`` and ``employee_position_title``. We can achieve this using |ApplyToCols|, whose job is to apply a transformer to multiple columns in parallel, and let unmatched columns through without changes. This can be seen as a handy drop-in replacement of the |ColumnTransformer|. Since we selected two columns and set the number of components to ``30`` each, |ApplyToCols| will create ``2*30`` embedding columns in the dataframe ``Xt``, which we prefix with ``lsa_``. .. GENERATED FROM PYTHON SOURCE LINES 49-59 .. code-block:: Python from skrub import ApplyToCols, StringEncoder apply_string_encoder = ApplyToCols( StringEncoder(n_components=30), cols=["division", "employee_position_title"], rename_columns="lsa_{}", ) Xt = apply_string_encoder.fit_transform(X) Xt .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 60-64 The |ApplyToCols| class can detect automatically whether the transformer is a ``SingleColumnTransformer`` (i.e., it can only be applied to one column at a time) or not, and apply it accordingly. The |StringEncoder| is a ``SingleColumnTransformer`` and thus applied to each column independently. .. GENERATED FROM PYTHON SOURCE LINES 66-82 The |ApplyToCols| class can also be used with transformers that can be applied to multiple columns at once, such as the |PCA|. Here, we want to use PCA to reduce the number of dimensions of the new ``lsa_`` columns. To select columns without hardcoding their names, we introduce :ref:`selectors`, which allow for flexible matching pattern and composable logic. The regex selector below will match all columns prefixed with ``"lsa"``, and pass them to |ApplyToCols| which will assemble these columns into a dataframe and finally pass it to the PCA Note that |ApplyToCols| will automatically detect that PCA is not a ``SingleColumnTransformer`` and apply it to the whole sub-dataframe of columns chosen by the selector at once. .. GENERATED FROM PYTHON SOURCE LINES 82-91 .. code-block:: Python from sklearn.decomposition import PCA from skrub import selectors as s apply_pca = ApplyToCols(PCA(n_components=8), cols=s.regex("lsa")) Xt = apply_pca.fit_transform(Xt) Xt .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 92-94 These two selectors are scikit-learn transformers and can be chained together within a |Pipeline|. .. GENERATED FROM PYTHON SOURCE LINES 94-101 .. code-block:: Python from sklearn.pipeline import make_pipeline model = make_pipeline( apply_string_encoder, apply_pca, ).fit_transform(X) .. GENERATED FROM PYTHON SOURCE LINES 102-104 Note that selectors also come in handy in a pipeline to select or drop columns, using |SelectCols| and |DropCols|. .. GENERATED FROM PYTHON SOURCE LINES 104-115 .. code-block:: Python from sklearn.preprocessing import StandardScaler from skrub import SelectCols # Select only numerical columns pipeline = make_pipeline( SelectCols(cols=s.numeric()), StandardScaler(), ).set_output(transform="pandas") pipeline.fit_transform(Xt) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 116-123 Let's run through one more example to showcase the expressiveness of the selectors. Suppose we want to apply an |OrdinalEncoder| on categorical columns with low cardinality (e.g., fewer than ``40`` unique values). We define a column filter using skrub selectors with a lambda function. Note that the same effect can be obtained directly by using :func:`~skrub.selectors.cardinality_below`. .. GENERATED FROM PYTHON SOURCE LINES 123-128 .. code-block:: Python from sklearn.preprocessing import OrdinalEncoder low_cardinality = s.filter(lambda col: col.nunique() < 40) ApplyToCols(OrdinalEncoder(), cols=s.string() & low_cardinality).fit_transform(X) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 129-136 Notice how we composed the selector with :func:`~skrub.selectors.string()` using a logical operator. This resulting selector matches string columns with cardinality below ``40``. We can also define the opposite selector ``high_cardinality`` using the negation operator ``~`` and apply a |StringEncoder| to vectorize those columns. .. GENERATED FROM PYTHON SOURCE LINES 136-152 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor high_cardinality = ~low_cardinality pipeline = make_pipeline( ApplyToCols( OrdinalEncoder(), cols=s.string() & low_cardinality, ), ApplyToCols( StringEncoder(), cols=s.string() & high_cardinality, ), HistGradientBoostingRegressor(), ).fit(X, y) pipeline .. raw:: html 
Pipeline(steps=[('applytocols-1',
                     ApplyToCols(cols=(string() & filter(<lambda>)),
                                 transformer=OrdinalEncoder())),
                    ('applytocols-2',
                     ApplyToCols(cols=(string() & (~filter(<lambda>))),
                                 transformer=StringEncoder())),
                    ('histgradientboostingregressor',
                     HistGradientBoostingRegressor())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 153-158 Interestingly, the pipeline above is similar to the datatype dispatching performed by |TableVectorizer|, also used in :func:`~skrub.tabular_pipeline`. Click on the dropdown arrows next to the datatype to see the columns are mapped to the different transformers in |TableVectorizer|. .. GENERATED FROM PYTHON SOURCE LINES 158-161 .. code-block:: Python from skrub import tabular_pipeline tabular_pipeline("regressor").fit(X, y) .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingregressor',
                     HistGradientBoostingRegressor())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 10.198 seconds) **Estimated memory usage:** 609 MB .. _sphx_glr_download_auto_examples_0010_apply_to_cols.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.10.0?urlpath=lab/tree/notebooks/auto_examples/0010_apply_to_cols.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../lite/lab/index.html?path=auto_examples/0010_apply_to_cols.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 0010_apply_to_cols.ipynb <0010_apply_to_cols.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 0010_apply_to_cols.py <0010_apply_to_cols.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 0010_apply_to_cols.zip <0010_apply_to_cols.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_