.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/01_encoding/0010_encodings.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via JupyterLite or Binder. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_01_encoding_0010_encodings.py: .. _example_encodings: ===================================================================== Encoding: from a dataframe to a numerical matrix for machine learning ===================================================================== This example shows how to transform a rich dataframe with columns of various types into a numerical matrix on which machine-learning algorithms can be applied. We study the case of predicting wages using the `employee salaries `_ dataset. .. |TableVectorizer| replace:: :class:`~skrub.TableVectorizer` .. |Pipeline| replace:: :class:`~sklearn.pipeline.Pipeline` .. |OneHotEncoder| replace:: :class:`~sklearn.preprocessing.OneHotEncoder` .. |GapEncoder| replace:: :class:`~skrub.GapEncoder` .. |MinHashEncoder| replace:: :class:`~skrub.MinHashEncoder` .. |DatetimeEncoder| replace:: :class:`~skrub.DatetimeEncoder` .. |HGBR| replace:: :class:`~sklearn.ensemble.HistGradientBoostingRegressor` .. |RandomForestRegressor| replace:: :class:`~sklearn.ensemble.RandomForestRegressor` .. |permutation importances| replace:: :func:`~sklearn.inspection.permutation_importance` .. GENERATED FROM PYTHON SOURCE LINES 42-55 Easy learning on a dataframe ---------------------------- Let's first retrieve the dataset, using one of the downloaders from the :mod:`skrub.datasets` module. As all the downloaders, :func:`~skrub.datasets.fetch_employee_salaries` returns a dataset with a ``path`` field pointing to the dataframe file, which contains both the features and the target. We load the dataframe from the path using pandas. ``X`` is a dataframe which contains the features (aka design matrix, explanatory variables, independent variables). ``y`` is a column (pandas Series) which contains the target (aka dependent, response variable) that we want to learn to predict from ``X``. In this case ``y`` is the annual salary, found in column "current_annual_salary". .. GENERATED FROM PYTHON SOURCE LINES 55-65 .. code-block:: Python import pandas as pd from skrub.datasets import fetch_employee_salaries file_path = fetch_employee_salaries().path employees = pd.read_csv(file_path) X = employees.drop(columns="current_annual_salary") y = employees["current_annual_salary"] .. GENERATED FROM PYTHON SOURCE LINES 66-77 Most machine-learning algorithms work with arrays of numbers. The challenge here is that the ``employees`` dataframe is a heterogeneous set of columns: some are numerical (``'year_first_hired'``), some dates (``'date_first_hired'``), some have a few categorical entries (``'gender'``), some many (``'employee_position_title'``). Therefore our table needs to be "vectorized": processed to extract numeric features. ``skrub`` provides an easy way to build a simple but reliable machine-learning model which includes this step, working well on most tabular data. .. GENERATED FROM PYTHON SOURCE LINES 77-86 .. code-block:: Python from sklearn.model_selection import cross_validate from skrub import tabular_pipeline model = tabular_pipeline("regressor") results = cross_validate(model, X, y) results["test_score"] .. rst-class:: sphx-glr-script-out .. code-block:: none array([0.90842776, 0.87932023, 0.91346596, 0.92178993, 0.9210775 ]) .. GENERATED FROM PYTHON SOURCE LINES 87-91 The estimator returned by :obj:`tabular_pipeline` combines 2 steps: - a |TableVectorizer| to preprocess the dataframe and vectorize the features - a supervised learner (by default a |HGBR|) .. GENERATED FROM PYTHON SOURCE LINES 91-93 .. code-block:: Python model .. raw:: html 
Pipeline(steps=[('tablevectorizer',
                     TableVectorizer(low_cardinality=ToCategorical())),
                    ('histgradientboostingregressor',
                     HistGradientBoostingRegressor())])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 94-98 In the rest of this example, we focus on the first step and explore the capabilities of skrub's |TableVectorizer|. | .. GENERATED FROM PYTHON SOURCE LINES 100-102 More details on encoding tabular data ------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 102-109 .. code-block:: Python from skrub import TableVectorizer vectorizer = TableVectorizer() vectorized_X = vectorizer.fit_transform(X) vectorized_X .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 110-114 From our 8 columns, the |TableVectorizer| has extracted 143 numerical features. Most of them are one-hot encoded representations of the categorical features. For example, we can see that 3 columns ``'gender_F'``, ``'gender_M'``, ``'gender_nan'`` were created to encode the ``'gender'`` column. .. GENERATED FROM PYTHON SOURCE LINES 116-118 By performing appropriate transformations on our complex data, the |TableVectorizer| produced numeric features that we can use for machine-learning: .. GENERATED FROM PYTHON SOURCE LINES 118-123 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor HistGradientBoostingRegressor().fit(vectorized_X, y) .. raw:: html 
HistGradientBoostingRegressor()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 124-128 The |TableVectorizer| bridges the gap between tabular data and machine-learning pipelines. It allows us to apply a machine-learning estimator to our dataframe without manual data wrangling and feature extraction. .. GENERATED FROM PYTHON SOURCE LINES 130-144 Inspecting the TableVectorizer ------------------------------ The |TableVectorizer| distinguishes between 4 basic kinds of columns (more may be added in the future). For each kind, it applies a different transformation, which we can configure. The kinds of columns and the default transformation for each of them are: - numeric columns: simply casting to floating-point - datetime columns: extracting features such as year, day, hour with the |DatetimeEncoder| - low-cardinality categorical columns: one-hot encoding - high-cardinality categorical columns: a simple and effective text representation pipeline provided by the |GapEncoder| .. GENERATED FROM PYTHON SOURCE LINES 144-147 .. code-block:: Python vectorizer .. raw:: html 
TableVectorizer()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 148-151 We can inspect which transformation was chosen for each column and retrieve the fitted transformer. ``vectorizer.kind_to_columns_`` provides an overview of how the vectorizer categorized columns in our input: .. GENERATED FROM PYTHON SOURCE LINES 151-154 .. code-block:: Python vectorizer.kind_to_columns_ .. rst-class:: sphx-glr-script-out .. code-block:: none {'numeric': ['year_first_hired'], 'datetime': ['date_first_hired'], 'low_cardinality': ['gender', 'department', 'department_name', 'assignment_category'], 'high_cardinality': ['division', 'employee_position_title'], 'specific': []} .. GENERATED FROM PYTHON SOURCE LINES 155-156 The reverse mapping is given by: .. GENERATED FROM PYTHON SOURCE LINES 156-159 .. code-block:: Python vectorizer.column_to_kind_ .. rst-class:: sphx-glr-script-out .. code-block:: none {'year_first_hired': 'numeric', 'date_first_hired': 'datetime', 'gender': 'low_cardinality', 'department': 'low_cardinality', 'department_name': 'low_cardinality', 'assignment_category': 'low_cardinality', 'division': 'high_cardinality', 'employee_position_title': 'high_cardinality'} .. GENERATED FROM PYTHON SOURCE LINES 160-162 ``vectorizer.transformers_`` gives us a dictionary which maps column names to the corresponding transformer. .. GENERATED FROM PYTHON SOURCE LINES 162-165 .. code-block:: Python vectorizer.transformers_["date_first_hired"] .. raw:: html 
DatetimeEncoder()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 166-168 We can also see which features in the vectorizer's output were derived from a given input column. .. GENERATED FROM PYTHON SOURCE LINES 168-171 .. code-block:: Python vectorizer.input_to_outputs_["date_first_hired"] .. rst-class:: sphx-glr-script-out .. code-block:: none ['date_first_hired_year', 'date_first_hired_month', 'date_first_hired_day', 'date_first_hired_total_seconds'] .. GENERATED FROM PYTHON SOURCE LINES 172-175 .. code-block:: Python vectorized_X[vectorizer.input_to_outputs_["date_first_hired"]] .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 176-178 Finally, we can go in the opposite direction: given a column in the input, find out from which input column it was derived. .. GENERATED FROM PYTHON SOURCE LINES 178-182 .. code-block:: Python vectorizer.output_to_input_["department_BOA"] .. rst-class:: sphx-glr-script-out .. code-block:: none 'department' .. GENERATED FROM PYTHON SOURCE LINES 183-188 Dataframe preprocessing ~~~~~~~~~~~~~~~~~~~~~~~ Note that ``"date_first_hired"`` has been recognized and processed as a datetime column. .. GENERATED FROM PYTHON SOURCE LINES 188-191 .. code-block:: Python vectorizer.column_to_kind_["date_first_hired"] .. rst-class:: sphx-glr-script-out .. code-block:: none 'datetime' .. GENERATED FROM PYTHON SOURCE LINES 192-193 But looking closer at our original dataframe, it was encoded as a string. .. GENERATED FROM PYTHON SOURCE LINES 193-196 .. code-block:: Python X["date_first_hired"] .. rst-class:: sphx-glr-script-out .. code-block:: none 0 09/22/1986 1 09/12/1988 2 11/19/1989 3 05/05/2014 4 03/05/2007 ... 9223 11/03/2015 9224 11/28/1988 9225 04/30/2001 9226 09/05/2006 9227 01/30/2012 Name: date_first_hired, Length: 9228, dtype: str .. GENERATED FROM PYTHON SOURCE LINES 197-207 Note the ``dtype: object`` in the output above. Before applying the transformers we specify, the |TableVectorizer| performs a few preprocessing steps. For example, strings commonly used to represent missing values such as ``"N/A"`` are replaced with actual ``null``. As we saw above, columns containing strings that represent dates (e.g. ``'2024-05-15'``) are detected and converted to proper datetimes. We can inspect the list of steps that were applied to a given column: .. GENERATED FROM PYTHON SOURCE LINES 207-210 .. code-block:: Python vectorizer.all_processing_steps_["date_first_hired"] .. rst-class:: sphx-glr-script-out .. code-block:: none [CleanNullStrings(), DropUninformative(), ToDatetime(), DatetimeEncoder(), {'date_first_hired_day': ToFloat(), 'date_first_hired_month': ToFloat(), ...}] .. GENERATED FROM PYTHON SOURCE LINES 211-212 These preprocessing steps depend on the column: .. GENERATED FROM PYTHON SOURCE LINES 212-215 .. code-block:: Python vectorizer.all_processing_steps_["department"] .. rst-class:: sphx-glr-script-out .. code-block:: none [CleanNullStrings(), DropUninformative(), ToStr(), OneHotEncoder(drop='if_binary', dtype='float32', handle_unknown='ignore', sparse_output=False), {'department_BOA': ToFloat(), 'department_BOE': ToFloat(), ...}] .. GENERATED FROM PYTHON SOURCE LINES 219-225 A simple Pipeline for tabular data ---------------------------------- The |TableVectorizer| outputs data that can be understood by a scikit-learn estimator. Therefore we can easily build a 2-step scikit-learn ``Pipeline`` that we can fit, test or cross-validate and that works well on tabular data. .. GENERATED FROM PYTHON SOURCE LINES 225-238 .. code-block:: Python import numpy as np from sklearn.ensemble import HistGradientBoostingRegressor from sklearn.model_selection import cross_validate from sklearn.pipeline import make_pipeline pipeline = make_pipeline(TableVectorizer(), HistGradientBoostingRegressor()) results = cross_validate(pipeline, X, y) scores = results["test_score"] print(f"R2 score: mean: {np.mean(scores):.3f}; std: {np.std(scores):.3f}") print(f"mean fit time: {np.mean(results['fit_time']):.3f} seconds") .. rst-class:: sphx-glr-script-out .. code-block:: none R2 score: mean: 0.912; std: 0.015 mean fit time: 1.628 seconds .. GENERATED FROM PYTHON SOURCE LINES 239-272 Specializing the TableVectorizer for HistGradientBoosting --------------------------------------------------------- The encoders used by default by the |TableVectorizer| are safe choices for a wide range of downstream estimators. If we know we want to use it with a |HGBR| (or classifier) model, we can make some different choices that are only well-suited for tree-based models but can yield a faster pipeline. We make 2 changes. The |HGBR| has built-in support for categorical features, so we do not need to one-hot encode them. We do need to tell it which features should be treated as categorical with the ``categorical_features`` parameter. In recent versions of scikit-learn, we can set ``categorical_features='from_dtype'``, and it will treat all columns in the input that have a ``Categorical`` dtype as such. Therefore we change the encoder for low-cardinality columns: instead of ``OneHotEncoder``, we use skrub's ``ToCategorical``. This transformer will simply ensure our columns have an actual ``Categorical`` dtype (as opposed to string for example), so that they can be recognized by the |HGBR|. The second change replaces the |GapEncoder| with a |MinHashEncoder|. The |GapEncoder| is a topic model. It produces interpretable embeddings in a vector space where distances are meaningful, which is great for interpretation and necessary for some downstream supervised learners such as linear models. However fitting the topic model is costly in computation time and memory. The |MinHashEncoder| produces features that are not easy to interpret, but that decision trees can efficiently use to test for the occurrence of particular character n-grams (more details are provided in its documentation). Therefore it can be a faster and very effective alternative, when the supervised learner is built on top of decision trees, which is the case for the |HGBR|. The resulting pipeline is identical to the one produced by default by :obj:`tabular_pipeline`. .. GENERATED FROM PYTHON SOURCE LINES 272-287 .. code-block:: Python from skrub import MinHashEncoder, ToCategorical vectorizer = TableVectorizer( low_cardinality=ToCategorical(), high_cardinality=MinHashEncoder() ) pipeline = make_pipeline( vectorizer, HistGradientBoostingRegressor(categorical_features="from_dtype") ) results = cross_validate(pipeline, X, y) scores = results["test_score"] print(f"R2 score: mean: {np.mean(scores):.3f}; std: {np.std(scores):.3f}") print(f"mean fit time: {np.mean(results['fit_time']):.3f} seconds") .. rst-class:: sphx-glr-script-out .. code-block:: none R2 score: mean: 0.916; std: 0.011 mean fit time: 0.950 seconds .. GENERATED FROM PYTHON SOURCE LINES 288-291 We can see that this new pipeline achieves a similar score but is fitted much faster. This is mostly due to replacing |GapEncoder| with |MinHashEncoder| (however this makes the features less interpretable). .. GENERATED FROM PYTHON SOURCE LINES 293-308 Feature importances in the statistical model -------------------------------------------- As we just saw, we can fit a |MinHashEncoder| faster than a |GapEncoder|. However, the |GapEncoder| has a crucial advantage: each dimension of its output space is associated with a topic which can be inspected and interpreted. In this section, after training a regressor, we will plot the feature importances. .. topic:: Note: To minimize computation time, we use the feature importances computed by the |RandomForestRegressor|, but you should prefer |permutation importances| instead (which are less subject to biases). First, we train another scikit-learn regressor, the |RandomForestRegressor|: .. GENERATED FROM PYTHON SOURCE LINES 308-317 .. code-block:: Python from sklearn.ensemble import RandomForestRegressor vectorizer = TableVectorizer() # now using the default GapEncoder regressor = RandomForestRegressor(n_estimators=50, max_depth=20, random_state=0) pipeline = make_pipeline(vectorizer, regressor) pipeline.fit(X, y) .. raw:: html 
Pipeline(steps=[('tablevectorizer', TableVectorizer()),
                    ('randomforestregressor',
                     RandomForestRegressor(max_depth=20, n_estimators=50,
                                           random_state=0))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 318-319 We are retrieving the feature importances: .. GENERATED FROM PYTHON SOURCE LINES 319-326 .. code-block:: Python avg_importances = regressor.feature_importances_ std_importances = np.std( [tree.feature_importances_ for tree in regressor.estimators_], axis=0 ) indices = np.argsort(avg_importances)[::-1] .. GENERATED FROM PYTHON SOURCE LINES 327-328 And plotting the results: .. GENERATED FROM PYTHON SOURCE LINES 328-348 .. code-block:: Python import matplotlib.pyplot as plt top_indices = indices[:20] labels = vectorizer.get_feature_names_out()[top_indices] plt.figure(figsize=(12, 9)) plt.barh( y=labels, width=avg_importances[top_indices], xerr=std_importances[top_indices], ecolor="k", color="b", alpha=0.5, ) plt.yticks(fontsize=15) plt.title("Feature importances") plt.tight_layout(pad=1) plt.show() .. image-sg:: /auto_examples/01_encoding/images/sphx_glr_0010_encodings_001.png :alt: Feature importances :srcset: /auto_examples/01_encoding/images/sphx_glr_0010_encodings_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 349-354 The |GapEncoder| creates feature names that show the first 3 most important words in the topic associated with each feature. As we can see in the plot above, this helps inspecting the model. If we had used a |MinHashEncoder| instead, the features would be much less helpful, with names such as ``employee_position_title_0``, ``employee_position_title_1``, etc. .. GENERATED FROM PYTHON SOURCE LINES 356-365 We can see that features such the time elapsed since being hired, having a full-time employment, and the position, seem to be the most informative for prediction. However, feature importances must not be over-interpreted -- they capture statistical associations `rather than causal effects `_. Moreover, the fast feature importance method used here suffers from biases favouring features with larger cardinality, as illustrated in a scikit-learn `example `_. In general we should prefer |permutation importances|, but it is a slower method. .. GENERATED FROM PYTHON SOURCE LINES 367-378 Conclusion ---------- In this example, we motivated the need for a simple machine learning pipeline, which we built using the |TableVectorizer| and a |HGBR|. We saw that by default, it works well on a heterogeneous dataset. To better understand our dataset, and without much effort, we were also able to plot the feature importances. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 41.675 seconds) **Estimated memory usage:** 611 MB .. _sphx_glr_download_auto_examples_01_encoding_0010_encodings.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.10.0?urlpath=lab/tree/notebooks/auto_examples/01_encoding/0010_encodings.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../../lite/lab/index.html?path=auto_examples/01_encoding/0010_encodings.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 0010_encodings.ipynb <0010_encodings.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 0010_encodings.py <0010_encodings.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 0010_encodings.zip <0010_encodings.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_