.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/02_data_ops/1140_subsampling.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via JupyterLite or Binder. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_02_data_ops_1140_subsampling.py: .. _example_subsampling: Subsampling for faster development ================================== Here we show how to use :meth:`.skb.subsample() ` to speed up interactive construction of a skrub DataOps plan by computing previews on a subsampled version of the original data. .. currentmodule:: skrub .. GENERATED FROM PYTHON SOURCE LINES 16-28 .. code-block:: Python import pandas as pd import skrub import skrub.datasets file_path = skrub.datasets.fetch_employee_salaries().path dataset = pd.read_csv(file_path) full_data = skrub.var("data", dataset) full_data .. raw:: html
<Var 'data'>
Show/Hide graph Var 'data'

Result:

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 29-39 We are working with a dataset of over 9K rows. As we build up our plan, we see previews of the intermediate results so we can check that it behaves as expected. However, if some estimators are slow, fitting them and computing results on the whole data can slow us down. Lightweight construction of the DataOps plan on a subsample ---------------------------------------------------------- We can tell skrub to subsample the data when computing the previews with :meth:`.skb.subsample() `. .. GENERATED FROM PYTHON SOURCE LINES 41-44 .. code-block:: Python data = full_data.skb.subsample(n=100) data .. raw:: html
<SubsamplePreviews>
Show/Hide graph Var 'data' SubsamplePreviews

Result (on a subsample):

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 45-59 The rest of the plan will now use only 100 points for its previews. .. topic:: Subsampling only applies to previews by default By default subsampling is applied *only for previews*: the results shown when we display the plan, and the output of calling :meth:`.skb.preview() `. For other methods such as :meth:`.skb.get_learner() ` or :meth:`.skb.cross_validate() `, *no subsampling is done by default*. We can explicitly ask for it with ``keep_subsampling=True`` as we will see below. Even when ``keep_subsampling=True``, subsampling is not applied to the ``predict`` method. To continue building our plan, we now define X and y: .. GENERATED FROM PYTHON SOURCE LINES 61-68 .. code-block:: Python employees = data.drop( columns="current_annual_salary", errors="ignore", ).skb.mark_as_X() salaries = data["current_annual_salary"].skb.mark_as_y() .. GENERATED FROM PYTHON SOURCE LINES 69-70 And finally we apply a TableVectorizer then gradient boosting: .. GENERATED FROM PYTHON SOURCE LINES 72-78 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor predictions = employees.skb.apply(skrub.TableVectorizer()).skb.apply( HistGradientBoostingRegressor(), y=salaries ) .. GENERATED FROM PYTHON SOURCE LINES 79-83 All the lines above run very fast, including fitting the predictor above. When we display our ``predictions`` DataOp, we see that the preview is computed on a subsample: the result column has only 100 entries. .. GENERATED FROM PYTHON SOURCE LINES 86-88 .. code-block:: Python predictions .. raw:: html
<Apply HistGradientBoostingRegressor>
Show/Hide graph Var 'data' SubsamplePreviews X: CallMethod 'drop' y: GetItem 'current_annual_salary' Apply TableVectorizer Apply HistGradientBoostingRegressor

Result (on a subsample):

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 89-94 We can also turn on subsampling for other DataOps methods, such as :meth:`.skb.cross_validate() `. Here we run the cross-validation on the small subsample of 100 rows we configured. With such a small subsample the scores will be very low but this might help us quickly detect errors in our cross-validation scheme. .. GENERATED FROM PYTHON SOURCE LINES 96-98 .. code-block:: Python predictions.skb.cross_validate(keep_subsampling=True) .. rst-class:: sphx-glr-script-out .. code-block:: none /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. GENERATED FROM PYTHON SOURCE LINES 99-106 Evaluating the DataOps plan on the full data ------------------------------------------- By default, when we do not explicitly ask for ``keep_subsampling=True``, no subsampling takes place. Here we run the cross-validation **on the full data**. Note the longer ``fit_time`` and much better ``test_score``. .. GENERATED FROM PYTHON SOURCE LINES 108-109 .. code-block:: Python predictions.skb.cross_validate() .. raw:: html

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").



.. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 18.973 seconds) **Estimated memory usage:** 637 MB .. _sphx_glr_download_auto_examples_02_data_ops_1140_subsampling.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.10.0?urlpath=lab/tree/notebooks/auto_examples/02_data_ops/1140_subsampling.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../../lite/lab/index.html?path=auto_examples/02_data_ops/1140_subsampling.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 1140_subsampling.ipynb <1140_subsampling.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 1140_subsampling.py <1140_subsampling.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 1140_subsampling.zip <1140_subsampling.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_