`: predicting the salaries of US Government employees. This dataset is so simple that it can be handled without the DataOps, using a scikit-learn :class:`~sklearn.pipeline.Pipeline`, but we will move on to more challenging datasets in later sections. .. GENERATED FROM PYTHON SOURCE LINES 17-19 Here is the dataset we will work with. The column to predict is ``current_annual_salary``. .. GENERATED FROM PYTHON SOURCE LINES 19-26 .. code-block:: Python import skrub train_dataset = skrub.datasets.fetch_employee_salaries(split="train") skrub.TableReport(train_dataset.employee_salaries) .. raw:: html

	gender	department	department_name	division	assignment_category	employee_position_title	date_first_hired	year_first_hired	current_annual_salary
0	F	POL	Department of Police	MSB Information Mgmt and Tech Division Records Management Section	Fulltime-Regular	Office Services Coordinator	09/22/1986	1,986	6.92e+04
1	M	POL	Department of Police	ISB Major Crimes Division Fugitive Section	Fulltime-Regular	Master Police Officer	09/12/1988	1,988	9.74e+04
2	F	HHS	Department of Health and Human Services	Adult Protective and Case Management Services	Fulltime-Regular	Social Worker IV	11/19/1989	1,989	1.05e+05

7,998	F	HHS	Department of Health and Human Services	School Health Services	Fulltime-Regular	Community Health Nurse II	07/28/2015	2,015	8.00e+04
7,999	M	CAT	County Attorney's Office	Insurance Defense Litigation	Fulltime-Regular	Assistant County Attorney III	05/19/2014	2,014	1.11e+05

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	gender	StringDtype	False	10 (0.1%)	2 (< 0.1%)
1	department	StringDtype	False	0 (0.0%)	37 (0.5%)
2	department_name	StringDtype	False	0 (0.0%)	37 (0.5%)
3	division	StringDtype	False	0 (0.0%)	681 (8.5%)
4	assignment_category	StringDtype	False	0 (0.0%)	2 (< 0.1%)
5	employee_position_title	StringDtype	False	0 (0.0%)	431 (5.4%)
6	date_first_hired	StringDtype	False	0 (0.0%)	2101 (26.3%)
7	year_first_hired	Int64DType	False	0 (0.0%)	51 (0.6%)	2.00e+03	9.33	1,965	2,005	2,016
8	current_annual_salary	Float64DType	False	0 (0.0%)	3063 (38.3%)	7.34e+04	2.93e+04	9.20e+03	6.92e+04	3.03e+05

Column 1	Column 2	Cramér's V	Pearson's Correlation
department	department_name	1.00
assignment_category	current_annual_salary	0.712
division	assignment_category	0.605
assignment_category	employee_position_title	0.498
department	assignment_category	0.423
department_name	assignment_category	0.423
division	employee_position_title	0.412
department_name	employee_position_title	0.407
department	employee_position_title	0.407
gender	department_name	0.377
gender	department	0.377
department_name	division	0.361
department	division	0.361
employee_position_title	date_first_hired	0.340
employee_position_title	current_annual_salary	0.274
gender	assignment_category	0.272
gender	employee_position_title	0.261
gender	division	0.253
division	current_annual_salary	0.221
year_first_hired	current_annual_salary	0.209	-0.458
department	current_annual_salary	0.200
department_name	current_annual_salary	0.200
department_name	date_first_hired	0.157
department	date_first_hired	0.157
date_first_hired	year_first_hired	0.154
employee_position_title	year_first_hired	0.142
gender	current_annual_salary	0.111
date_first_hired	current_annual_salary	0.108
gender	date_first_hired	0.107
division	date_first_hired	0.0950
department	year_first_hired	0.0832
department_name	year_first_hired	0.0832
assignment_category	date_first_hired	0.0814
division	year_first_hired	0.0691
gender	year_first_hired	0.0608
assignment_category	year_first_hired	0.0526

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

.. GENERATED FROM PYTHON SOURCE LINES 32-40 A first simple pipeline ----------------------- We start by defining our predictive pipeline. We will need to encode the features with a :class:`TableVectorizer` then predict with a :class:`~sklearn.ensemble.HistGradientBoostingRegressor`. Inputs to our pipeline are declared with :func:`var`: .. GENERATED FROM PYTHON SOURCE LINES 42-45 .. code-block:: Python employee_data = skrub.var("employee_data") employee_data .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 46-49 Transformation steps are added by calling methods on the intermediate results. An important one is :meth:`DataOp.skb.apply`, which applies a scikit-learn estimator: .. GENERATED FROM PYTHON SOURCE LINES 51-60 .. code-block:: Python from sklearn.ensemble import HistGradientBoostingRegressor salary = skrub.var("salary") pred = employee_data.skb.apply(skrub.TableVectorizer()).skb.apply( HistGradientBoostingRegressor(), y=salary ) pred .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 61-63 Note that the methods are accessed through the special attribute ``.skb``: for example ``.skb.apply``. We will explain why shortly. .. GENERATED FROM PYTHON SOURCE LINES 65-67 Once we have added all the steps, we create a *learner*: an object similar to a scikit-learn estimator with ``fit`` and ``predict`` methods. .. GENERATED FROM PYTHON SOURCE LINES 67-71 .. code-block:: Python learner = pred.skb.make_learner() learner.fit({"employee_data": train_dataset.X, "salary": train_dataset.y}) .. raw:: html

SkrubLearner(data_op=<Apply HistGradientBoostingRegressor>)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

.. GENERATED FROM PYTHON SOURCE LINES 72-79 Regular scikit-learn estimators always take the same fixed inputs: X and y. Skrub learners can process arbitrary data, so the signature of methods like ``fit`` and ``predict`` is different: we pass a dictionary of inputs. The keys correspond to the names of the variables we used to define our learner, here ``"employee_data"`` and ``"salary"``. Finally, we can use our fitted learner to make a prediction: .. GENERATED FROM PYTHON SOURCE LINES 81-84 .. code-block:: Python test_dataset = skrub.datasets.fetch_employee_salaries(split="test") learner.predict({"employee_data": test_dataset.X, "salary": test_dataset.y}) .. rst-class:: sphx-glr-script-out .. code-block:: none array([119471.41270359, 45057.40117905, 47337.07645869, ..., 106872.37978641, 147682.69636242, 73919.50934215], shape=(1228,)) .. GENERATED FROM PYTHON SOURCE LINES 85-113 We can generate a complete report for the execution of our DataOp by calling: .. code:: python pred.skb.full_report({"employee_data": test_dataset.X, "salary": test_dataset.y}) As the output is usually quite large, it does not display inline in a notebook but is instead opened in a separate browser tab. However here we insert it in the page for convenience. By clicking a node in the graph you can see its result, how long it took, and the scikit-learn estimator that was fitted (if any). .. raw:: html You can also visit the report `here <../_static/employee_salaries_report/index.html>`_. It is also possible to create report about the execution of specific methods of the learner like "fit" and "predict" with :meth:`SkrubLearner.report`. .. GENERATED FROM PYTHON SOURCE LINES 115-123 Cross-validation ---------------- We now make a few refinements on the previous pipeline. DataOps can accept any type of input and perform all processing, so we will extend our pipeline so that it includes the data loading and the creation of our features ``employee_data`` and our target ``salary``. The input will be simply the path to a csv file: .. GENERATED FROM PYTHON SOURCE LINES 123-126 .. code-block:: Python train_dataset.path .. rst-class:: sphx-glr-script-out .. code-block:: none '/home/circleci/skrub_data/employee_salaries/employee_salaries/employee_salaries_train.csv' .. GENERATED FROM PYTHON SOURCE LINES 127-133 Therefore we declare a new variable, to represent the CSV path. We also introduce an important feature of DataOps: interactive preview results. If we pass a value to our variable when creating it, it is used as example data on which skrub runs our pipeline as we define it, so we can see what the result looks like every step of the way. .. GENERATED FROM PYTHON SOURCE LINES 135-138 .. code-block:: Python csv_path = skrub.var("csv_path", train_dataset.path) csv_path .. raw:: html

<Var 'csv_path'>

Show/Hide graph

Result:

'/home/circleci/skrub_data/employee_salaries/employee_salaries/employee_salaries_train.csv'

.. GENERATED FROM PYTHON SOURCE LINES 139-144 Note the added "Result" section in the output, which shows what the current pipeline's output looks like. Similarly to ``.skb.apply`` (which applies an estimator), ``.skb.apply_func`` applies a function: .. GENERATED FROM PYTHON SOURCE LINES 146-151 .. code-block:: Python import pandas as pd full_data = csv_path.skb.apply_func(pd.read_csv) full_data .. raw:: html

<Call 'read_csv'>

Show/Hide graph

Result:

	gender	department	department_name	division	assignment_category	employee_position_title	date_first_hired	year_first_hired	current_annual_salary
0	F	POL	Department of Police	MSB Information Mgmt and Tech Division Records Management Section	Fulltime-Regular	Office Services Coordinator	09/22/1986	1,986	6.92e+04
1	M	POL	Department of Police	ISB Major Crimes Division Fugitive Section	Fulltime-Regular	Master Police Officer	09/12/1988	1,988	9.74e+04
2	F	HHS	Department of Health and Human Services	Adult Protective and Case Management Services	Fulltime-Regular	Social Worker IV	11/19/1989	1,989	1.05e+05

7,998	F	HHS	Department of Health and Human Services	School Health Services	Fulltime-Regular	Community Health Nurse II	07/28/2015	2,015	8.00e+04
7,999	M	CAT	County Attorney's Office	Insurance Defense Litigation	Fulltime-Regular	Assistant County Attorney III	05/19/2014	2,014	1.11e+05

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	gender	StringDtype	False	10 (0.1%)	2 (< 0.1%)
1	department	StringDtype	False	0 (0.0%)	37 (0.5%)
2	department_name	StringDtype	False	0 (0.0%)	37 (0.5%)
3	division	StringDtype	False	0 (0.0%)	681 (8.5%)
4	assignment_category	StringDtype	False	0 (0.0%)	2 (< 0.1%)
5	employee_position_title	StringDtype	False	0 (0.0%)	431 (5.4%)
6	date_first_hired	StringDtype	False	0 (0.0%)	2101 (26.3%)
7	year_first_hired	Int64DType	False	0 (0.0%)	51 (0.6%)	2.00e+03	9.33	1,965	2,005	2,016
8	current_annual_salary	Float64DType	False	0 (0.0%)	3063 (38.3%)	7.34e+04	2.93e+04	9.20e+03	6.92e+04	3.03e+05

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 152-172 Next, from our full dataframe we extract the predictive features and the regression target. The following snippet of code shows 2 important aspects: - Any methods or operators we access on our DataOp ``full_data``, like ``drop`` or the ``[]`` operator below, are recorded in the pipeline and will be applied to the DataOp's result: ``full_data['current_annual_salary']`` is roughly equivalent to ``full_data.skb.apply_func(lambda df: df['curent_annual_salary'])``. This is why all the skrub functionality is behind the ``.skb`` prefix as mentioned earlier: all other attribute access will be replayed directly on the result that the DataOp produces. - Once we have defined the features and targets, we mark them with :meth:`DataOp.skb.mark_as_X()` and :meth:`DataOp.skb.mark_as_y()` respectively. This tells skrub that when performing cross-validation, those are the intermediate results that should be divided into training and testing sets. Therefore, X and y do not need to be constructed and split *outside* the pipeline. Instead, our pipeline can encompass the full processing, and we indicate where the train/test split should happen. .. GENERATED FROM PYTHON SOURCE LINES 172-181 .. code-block:: Python employee_data = full_data.drop( columns="current_annual_salary", errors="ignore" ).skb.mark_as_X() # (errors='ignore' because this column could be absent at the inference stage.) salary = full_data["current_annual_salary"].skb.mark_as_y() salary .. raw:: html

<GetItem 'current_annual_salary'>

Show/Hide graph

Result:

	current_annual_salary
0	6.92e+04
1	9.74e+04
2	1.05e+05

7,998	8.00e+04
7,999	1.11e+05

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	current_annual_salary	Float64DType	False	0 (0.0%)	3063 (38.3%)	7.34e+04	2.93e+04	9.20e+03	6.92e+04	3.03e+05

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 182-184 Finally, we apply the regressor. Note that the X and y nodes, on which train/test split is performed, are colored differently. .. GENERATED FROM PYTHON SOURCE LINES 184-190 .. code-block:: Python pred = employee_data.skb.apply(skrub.TableVectorizer()).skb.apply( HistGradientBoostingRegressor(), y=salary ) pred .. raw:: html

<Apply HistGradientBoostingRegressor>

Show/Hide graph

Result:

	current_annual_salary
0	6.76e+04
1	9.76e+04
2	1.00e+05

7,998	7.54e+04
7,999	1.06e+05

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	current_annual_salary	Float64DType	False	0 (0.0%)	6488 (81.1%)	7.34e+04	2.82e+04	1.40e+04	7.02e+04	2.22e+05

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 191-193 Once we have defined our pipeline, we can tell skrub to perform the cross-validation with :meth:`DataOp.skb.cross_validate`. .. GENERATED FROM PYTHON SOURCE LINES 193-196 .. code-block:: Python pred.skb.cross_validate(scoring="neg_mean_absolute_percentage_error") .. rst-class:: sphx-glr-script-out .. code-block:: none /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) .. raw:: html

	fit_time	score_time	test_score
0	1.75	0.203	-0.0692
1	1.67	0.196	-0.0651
2	1.70	0.194	-0.0662
3	1.78	0.316	-0.0738
4	2.15	0.312	-0.0684

Column	Column name	dtype	Is sorted	Unique values	Mean	Std	Min	Median	Max
0	fit_time	Float64DType	False	5 (100.0%)	1.81	0.194	1.67	1.75	2.15
1	score_time	Float64DType	False	5 (100.0%)	0.244	0.0640	0.194	0.203	0.316
2	test_score	Float64DType	False	5 (100.0%)	-0.0686	0.00336	-0.0738	-0.0684	-0.0651

Column 1	Column 2	Cramér's V	Pearson's Correlation
score_time	test_score	1.00	-0.729
fit_time	test_score	1.00	-0.200
fit_time	score_time	1.00	0.725

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 197-200 Note that the variables used in this pipeline are different than the previous one: we just have ``"csv_path"`` and not ``"employee_data"`` and ``"salary"`` like before. .. GENERATED FROM PYTHON SOURCE LINES 200-204 .. code-block:: Python learner = pred.skb.make_learner(fitted=True) learner.predict({"csv_path": test_dataset.path}) .. rst-class:: sphx-glr-script-out .. code-block:: none array([117656.318758 , 45185.65169354, 46138.80688344, ..., 97099.764518 , 145886.76864681, 70085.23723934], shape=(1228,)) .. GENERATED FROM PYTHON SOURCE LINES 205-211 Tuning arbitrary choices ------------------------ The last feature we present in this first tutorial is hyperparameter tuning. The start of the pipeline is the same as before: .. GENERATED FROM PYTHON SOURCE LINES 213-219 .. code-block:: Python full_data = skrub.var("csv_path", train_dataset.path).skb.apply_func(pd.read_csv) employee_data = full_data.drop( columns="current_annual_salary", errors="ignore" ).skb.mark_as_X() salary = full_data["current_annual_salary"].skb.mark_as_y() .. GENERATED FROM PYTHON SOURCE LINES 220-237 We use functions like :func:`choose_from` or :func:`choose_float` whenever we have a choice for which we want to try several options and keep the one that performs best on the validation data. We simply replace the value by the special "choice" object produced by skrub in our pipeline, and it becomes a tunable hyperparameter of our skrub learner. Here we want to tune: - the choice of encoder applied to high-cardinality categorical columns (:class:`StringEncoder` or :class:`~sklearn.preprocessing.TargetEncoder`) - for the StringEncoder, the number of components - the learning rate of the :class:`~sklearn.ensemble.HistGradientBoostingRegressor`. **Note:** choices are not restricted to estimators or their hyperparameters, we can tune any value used anywhere in a pipeline, or the choice between different pipelines; more details :ref:`here `. .. GENERATED FROM PYTHON SOURCE LINES 237-260 .. code-block:: Python from sklearn.preprocessing import TargetEncoder n_components = skrub.choose_int(10, 80, name="n_components") # choose int in [10, 80[ encoder = skrub.choose_from( # choosing between 2 different estimators { "lse": skrub.StringEncoder(n_components=n_components), # nesting choices "target": TargetEncoder(), }, name="encoder", ) pred = employee_data.skb.apply( skrub.TableVectorizer(high_cardinality=encoder), y=salary ).skb.apply( HistGradientBoostingRegressor( learning_rate=skrub.choose_float(0.01, 0.7, log=True, name="learning_rate") ), y=salary, ) print(pred.skb.describe_param_grid()) .. rst-class:: sphx-glr-script-out .. code-block:: none - learning_rate: choose_float(0.01, 0.7, log=True, name='learning_rate') encoder: 'lse' n_components: choose_int(10, 80, name='n_components') - learning_rate: choose_float(0.01, 0.7, log=True, name='learning_rate') encoder: 'target' .. GENERATED FROM PYTHON SOURCE LINES 261-267 To actually run the search for the best hyperparameters, we use :meth:`DataOp.skb.make_randomized_search` or :meth:`DataOp.skb.make_grid_search`. For the randomized search we can use the powerful Optuna library which provides features like state-of-the-art hyperparameter samplers, live interactive visualization of the search with ``optuna-dashboard``, stopping and resuming searches, etc. .. GENERATED FROM PYTHON SOURCE LINES 267-273 .. code-block:: Python search = pred.skb.make_randomized_search( backend="optuna", fitted=True, n_iter=16, random_state=0 ) search.results_ .. rst-class:: sphx-glr-script-out .. code-block:: none Running optuna search for study skrub_randomized_search_b92b27e5-447d-413c-a79b-8ae96fa549e9 in storage /tmp/tmpx6zxwy68_skrub_optuna_search_storage/optuna_storage /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) /home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros warnings.warn(msg, UserWarning) .. raw:: html

	n_components	encoder	learning_rate	mean_test_score
0	78.0	lse	0.0564	0.911
1	59.0	lse	0.0767	0.908
2	75.0	lse	0.0572	0.904

14		target	0.645	0.868
15		target	0.0183	0.857

Column	Column name	dtype	Is sorted	Null values	Unique values	Mean	Std	Min	Median	Max
0	n_components	Float64DType	False	8 (50.0%)	7 (43.8%)	60.0	23.0	22.0	75.0	79.0
1	encoder	StringDtype	False	0 (0.0%)	2 (12.5%)
2	learning_rate	Float64DType	False	0 (0.0%)	16 (100.0%)	0.196	0.231	0.0183	0.0857	0.658
3	mean_test_score	Float64DType	True	0 (0.0%)	16 (100.0%)	0.889	0.0154	0.857	0.892	0.911

Column 1	Column 2	Cramér's V	Pearson's Correlation
n_components	encoder	1.00
n_components	mean_test_score	0.854	0.330
encoder	learning_rate	0.802
encoder	mean_test_score	0.802
learning_rate	mean_test_score	0.769	-0.397
n_components	learning_rate	0.711	-0.617

Please enable javascript

.. GENERATED FROM PYTHON SOURCE LINES 274-276 .. code-block:: Python search.plot_results() .. raw:: html

.. GENERATED FROM PYTHON SOURCE LINES 277-280 The search can be used with the same interface as the :class:`SkrubLearner` we saw before. Alternatively, we can access its ``best_learner_`` attribute, which is a SkrubLearner. .. GENERATED FROM PYTHON SOURCE LINES 280-282 .. code-block:: Python search.predict({"csv_path": test_dataset.path}) .. rst-class:: sphx-glr-script-out .. code-block:: none array([117953.78973311, 45425.83792057, 50233.10821664, ..., 97320.14629547, 143659.49306584, 73890.48051489], shape=(1228,)) .. rst-class:: sphx-glr-timing **Total running time of the script:** (3 minutes 8.406 seconds) **Estimated memory usage:** 612 MB .. _sphx_glr_download_auto_tutorials_1111_data_ops_quick_tour.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.10.0?urlpath=lab/tree/notebooks/auto_tutorials/1111_data_ops_quick_tour.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../lite/lab/index.html?path=auto_tutorials/1111_data_ops_quick_tour.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 1111_data_ops_quick_tour.ipynb <1111_data_ops_quick_tour.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 1111_data_ops_quick_tour.py <1111_data_ops_quick_tour.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 1111_data_ops_quick_tour.zip <1111_data_ops_quick_tour.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

current_annual_salary

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

current_annual_salary

Please enable javascript

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

current_annual_salary

Please enable javascript

current_annual_salary

Please enable javascript

current_annual_salary

Please enable javascript

fit_time

score_time

test_score

fit_time

score_time

test_score

Please enable javascript

n_components

encoder

learning_rate

mean_test_score

n_components

encoder

learning_rate

mean_test_score

Please enable javascript