Note
Go to the end to download the full example code or to run this example in your browser via JupyterLite or Binder.
Quick overview of DataOps#
Here we give a bird’s eye view of the DataOps workflow on a simple regression task that we saw in an early example: predicting the salaries of US Government employees.
This dataset is so simple that it can be handled without the DataOps, using a
scikit-learn Pipeline, but we will move on to more
challenging datasets in later sections.
Here is the dataset we will work with. The column to predict is
current_annual_salary.
import skrub
train_dataset = skrub.datasets.fetch_employee_salaries(split="train")
skrub.TableReport(train_dataset.employee_salaries)
| gender | department | department_name | division | assignment_category | employee_position_title | date_first_hired | year_first_hired | current_annual_salary | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | F | POL | Department of Police | MSB Information Mgmt and Tech Division Records Management Section | Fulltime-Regular | Office Services Coordinator | 09/22/1986 | 1,986 | 6.92e+04 |
| 1 | M | POL | Department of Police | ISB Major Crimes Division Fugitive Section | Fulltime-Regular | Master Police Officer | 09/12/1988 | 1,988 | 9.74e+04 |
| 2 | F | HHS | Department of Health and Human Services | Adult Protective and Case Management Services | Fulltime-Regular | Social Worker IV | 11/19/1989 | 1,989 | 1.05e+05 |
| 7,998 | F | HHS | Department of Health and Human Services | School Health Services | Fulltime-Regular | Community Health Nurse II | 07/28/2015 | 2,015 | 8.00e+04 |
| 7,999 | M | CAT | County Attorney's Office | Insurance Defense Litigation | Fulltime-Regular | Assistant County Attorney III | 05/19/2014 | 2,014 | 1.11e+05 |
gender
StringDtype- Null values
- 10 (0.1%)
- Unique values
- 2 (< 0.1%)
Most frequent values
department
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
Most frequent values
department_name
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
Most frequent values
division
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
681 (8.5%)
This column has a high cardinality (> 40).
Most frequent values
assignment_category
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 2 (< 0.1%)
Most frequent values
employee_position_title
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
431 (5.4%)
This column has a high cardinality (> 40).
Most frequent values
date_first_hired
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
2,101 (26.3%)
This column has a high cardinality (> 40).
Most frequent values
year_first_hired
Int64DType- Null values
- 0 (0.0%)
- Unique values
-
51 (0.6%)
This column has a high cardinality (> 40).
- Mean ± Std
- 2.00e+03 ± 9.33
- Median ± IQR
- 2,005 ± 14
- Min | Max
- 1,965 | 2,016
current_annual_salary
Float64DType- Null values
- 0 (0.0%)
- Unique values
-
3,063 (38.3%)
This column has a high cardinality (> 40).
- Mean ± Std
- 7.34e+04 ± 2.93e+04
- Median ± IQR
- 6.92e+04 ± 3.94e+04
- Min | Max
- 9.20e+03 | 3.03e+05
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
| Column | Column name | dtype | Is sorted | Null values | Unique values | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | gender | StringDtype | False | 10 (0.1%) | 2 (< 0.1%) | |||||
| 1 | department | StringDtype | False | 0 (0.0%) | 37 (0.5%) | |||||
| 2 | department_name | StringDtype | False | 0 (0.0%) | 37 (0.5%) | |||||
| 3 | division | StringDtype | False | 0 (0.0%) | 681 (8.5%) | |||||
| 4 | assignment_category | StringDtype | False | 0 (0.0%) | 2 (< 0.1%) | |||||
| 5 | employee_position_title | StringDtype | False | 0 (0.0%) | 431 (5.4%) | |||||
| 6 | date_first_hired | StringDtype | False | 0 (0.0%) | 2101 (26.3%) | |||||
| 7 | year_first_hired | Int64DType | False | 0 (0.0%) | 51 (0.6%) | 2.00e+03 | 9.33 | 1,965 | 2,005 | 2,016 |
| 8 | current_annual_salary | Float64DType | False | 0 (0.0%) | 3063 (38.3%) | 7.34e+04 | 2.93e+04 | 9.20e+03 | 6.92e+04 | 3.03e+05 |
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
gender
StringDtype- Null values
- 10 (0.1%)
- Unique values
- 2 (< 0.1%)
Most frequent values
department
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
Most frequent values
department_name
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
Most frequent values
division
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
681 (8.5%)
This column has a high cardinality (> 40).
Most frequent values
assignment_category
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 2 (< 0.1%)
Most frequent values
employee_position_title
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
431 (5.4%)
This column has a high cardinality (> 40).
Most frequent values
date_first_hired
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
2,101 (26.3%)
This column has a high cardinality (> 40).
Most frequent values
year_first_hired
Int64DType- Null values
- 0 (0.0%)
- Unique values
-
51 (0.6%)
This column has a high cardinality (> 40).
- Mean ± Std
- 2.00e+03 ± 9.33
- Median ± IQR
- 2,005 ± 14
- Min | Max
- 1,965 | 2,016
current_annual_salary
Float64DType- Null values
- 0 (0.0%)
- Unique values
-
3,063 (38.3%)
This column has a high cardinality (> 40).
- Mean ± Std
- 7.34e+04 ± 2.93e+04
- Median ± IQR
- 6.92e+04 ± 3.94e+04
- Min | Max
- 9.20e+03 | 3.03e+05
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
| Column 1 | Column 2 | Cramér's V | Pearson's Correlation |
|---|---|---|---|
| department | department_name | 1.00 | |
| assignment_category | current_annual_salary | 0.712 | |
| division | assignment_category | 0.605 | |
| assignment_category | employee_position_title | 0.498 | |
| department | assignment_category | 0.423 | |
| department_name | assignment_category | 0.423 | |
| division | employee_position_title | 0.412 | |
| department_name | employee_position_title | 0.407 | |
| department | employee_position_title | 0.407 | |
| gender | department_name | 0.377 | |
| gender | department | 0.377 | |
| department_name | division | 0.361 | |
| department | division | 0.361 | |
| employee_position_title | date_first_hired | 0.340 | |
| employee_position_title | current_annual_salary | 0.274 | |
| gender | assignment_category | 0.272 | |
| gender | employee_position_title | 0.261 | |
| gender | division | 0.253 | |
| division | current_annual_salary | 0.221 | |
| year_first_hired | current_annual_salary | 0.209 | -0.458 |
| department | current_annual_salary | 0.200 | |
| department_name | current_annual_salary | 0.200 | |
| department_name | date_first_hired | 0.157 | |
| department | date_first_hired | 0.157 | |
| date_first_hired | year_first_hired | 0.154 | |
| employee_position_title | year_first_hired | 0.142 | |
| gender | current_annual_salary | 0.111 | |
| date_first_hired | current_annual_salary | 0.108 | |
| gender | date_first_hired | 0.107 | |
| division | date_first_hired | 0.0950 | |
| department | year_first_hired | 0.0832 | |
| department_name | year_first_hired | 0.0832 | |
| assignment_category | date_first_hired | 0.0814 | |
| division | year_first_hired | 0.0691 | |
| gender | year_first_hired | 0.0608 | |
| assignment_category | year_first_hired | 0.0526 |
Please enable javascript
The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").
A first simple pipeline#
We start by defining our predictive pipeline. We will need to encode the
features with a TableVectorizer then predict with a
HistGradientBoostingRegressor.
Inputs to our pipeline are declared with var():
employee_data = skrub.var("employee_data")
employee_data
Transformation steps are added by calling methods on the intermediate
results. An important one is DataOp.skb.apply(), which applies a
scikit-learn estimator:
from sklearn.ensemble import HistGradientBoostingRegressor
salary = skrub.var("salary")
pred = employee_data.skb.apply(skrub.TableVectorizer()).skb.apply(
HistGradientBoostingRegressor(), y=salary
)
pred
Note that the methods are accessed through the special attribute .skb:
for example .skb.apply. We will explain why shortly.
Once we have added all the steps, we create a learner: an object similar to a
scikit-learn estimator with fit and predict methods.
learner = pred.skb.make_learner()
learner.fit({"employee_data": train_dataset.X, "salary": train_dataset.y})
Regular scikit-learn estimators always take the same fixed inputs: X and y.
Skrub learners can process arbitrary data, so the signature of methods like
fit and predict is different: we pass a dictionary of inputs. The
keys correspond to the names of the variables we used to define our learner,
here "employee_data" and "salary".
Finally, we can use our fitted learner to make a prediction:
test_dataset = skrub.datasets.fetch_employee_salaries(split="test")
learner.predict({"employee_data": test_dataset.X, "salary": test_dataset.y})
array([119471.41270359, 45057.40117905, 47337.07645869, ...,
106872.37978641, 147682.69636242, 73919.50934215], shape=(1228,))
We can generate a complete report for the execution of our DataOp by calling:
pred.skb.full_report({"employee_data": test_dataset.X, "salary": test_dataset.y})
As the output is usually quite large, it does not display inline in a notebook but is instead opened in a separate browser tab. However here we insert it in the page for convenience. By clicking a node in the graph you can see its result, how long it took, and the scikit-learn estimator that was fitted (if any).
You can also visit the report here.
It is also possible to create report about the execution of specific methods
of the learner like “fit” and “predict” with SkrubLearner.report().
Cross-validation#
We now make a few refinements on the previous pipeline. DataOps can accept
any type of input and perform all processing, so we will extend our pipeline
so that it includes the data loading and the creation of our features
employee_data and our target salary. The input will be simply the
path to a csv file:
'/home/circleci/skrub_data/employee_salaries/employee_salaries/employee_salaries_train.csv'
Therefore we declare a new variable, to represent the CSV path.
We also introduce an important feature of DataOps: interactive preview results. If we pass a value to our variable when creating it, it is used as example data on which skrub runs our pipeline as we define it, so we can see what the result looks like every step of the way.
csv_path = skrub.var("csv_path", train_dataset.path)
csv_path
Note the added “Result” section in the output, which shows what the current pipeline’s output looks like.
Similarly to .skb.apply (which applies an estimator), .skb.apply_func
applies a function:
import pandas as pd
full_data = csv_path.skb.apply_func(pd.read_csv)
full_data
Show/Hide graph
| gender | department | department_name | division | assignment_category | employee_position_title | date_first_hired | year_first_hired | current_annual_salary | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | F | POL | Department of Police | MSB Information Mgmt and Tech Division Records Management Section | Fulltime-Regular | Office Services Coordinator | 09/22/1986 | 1,986 | 6.92e+04 |
| 1 | M | POL | Department of Police | ISB Major Crimes Division Fugitive Section | Fulltime-Regular | Master Police Officer | 09/12/1988 | 1,988 | 9.74e+04 |
| 2 | F | HHS | Department of Health and Human Services | Adult Protective and Case Management Services | Fulltime-Regular | Social Worker IV | 11/19/1989 | 1,989 | 1.05e+05 |
| 7,998 | F | HHS | Department of Health and Human Services | School Health Services | Fulltime-Regular | Community Health Nurse II | 07/28/2015 | 2,015 | 8.00e+04 |
| 7,999 | M | CAT | County Attorney's Office | Insurance Defense Litigation | Fulltime-Regular | Assistant County Attorney III | 05/19/2014 | 2,014 | 1.11e+05 |
gender
StringDtype- Null values
- 10 (0.1%)
- Unique values
- 2 (< 0.1%)
department
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
department_name
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 37 (0.5%)
division
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
681 (8.5%)
This column has a high cardinality (> 40).
assignment_category
StringDtype- Null values
- 0 (0.0%)
- Unique values
- 2 (< 0.1%)
employee_position_title
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
431 (5.4%)
This column has a high cardinality (> 40).
date_first_hired
StringDtype- Null values
- 0 (0.0%)
- Unique values
-
2,101 (26.3%)
This column has a high cardinality (> 40).
year_first_hired
Int64DType- Null values
- 0 (0.0%)
- Unique values
-
51 (0.6%)
This column has a high cardinality (> 40).
- Mean ± Std
- 2.00e+03 ± 9.33
- Median ± IQR
- 2,005 ± 14
- Min | Max
- 1,965 | 2,016
current_annual_salary
Float64DType- Null values
- 0 (0.0%)
- Unique values
-
3,063 (38.3%)
This column has a high cardinality (> 40).
- Mean ± Std
- 7.34e+04 ± 2.93e+04
- Median ± IQR
- 6.92e+04 ± 3.94e+04
- Min | Max
- 9.20e+03 | 3.03e+05
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
| Column | Column name | dtype | Is sorted | Null values | Unique values | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | gender | StringDtype | False | 10 (0.1%) | 2 (< 0.1%) | |||||
| 1 | department | StringDtype | False | 0 (0.0%) | 37 (0.5%) | |||||
| 2 | department_name | StringDtype | False | 0 (0.0%) | 37 (0.5%) | |||||
| 3 | division | StringDtype | False | 0 (0.0%) | 681 (8.5%) | |||||
| 4 | assignment_category | StringDtype | False | 0 (0.0%) | 2 (< 0.1%) | |||||
| 5 | employee_position_title | StringDtype | False | 0 (0.0%) | 431 (5.4%) | |||||
| 6 | date_first_hired | StringDtype | False | 0 (0.0%) | 2101 (26.3%) | |||||
| 7 | year_first_hired | Int64DType | False | 0 (0.0%) | 51 (0.6%) | 2.00e+03 | 9.33 | 1,965 | 2,005 | 2,016 |
| 8 | current_annual_salary | Float64DType | False | 0 (0.0%) | 3063 (38.3%) | 7.34e+04 | 2.93e+04 | 9.20e+03 | 6.92e+04 | 3.03e+05 |
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
Please enable javascript
The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").
Next, from our full dataframe we extract the predictive features and the regression target.
The following snippet of code shows 2 important aspects:
Any methods or operators we access on our DataOp
full_data, likedropor the[]operator below, are recorded in the pipeline and will be applied to the DataOp’s result:full_data['current_annual_salary']is roughly equivalent tofull_data.skb.apply_func(lambda df: df['curent_annual_salary']). This is why all the skrub functionality is behind the.skbprefix as mentioned earlier: all other attribute access will be replayed directly on the result that the DataOp produces.Once we have defined the features and targets, we mark them with
DataOp.skb.mark_as_X()andDataOp.skb.mark_as_y()respectively. This tells skrub that when performing cross-validation, those are the intermediate results that should be divided into training and testing sets. Therefore, X and y do not need to be constructed and split outside the pipeline. Instead, our pipeline can encompass the full processing, and we indicate where the train/test split should happen.
employee_data = full_data.drop(
columns="current_annual_salary", errors="ignore"
).skb.mark_as_X()
# (errors='ignore' because this column could be absent at the inference stage.)
salary = full_data["current_annual_salary"].skb.mark_as_y()
salary
Show/Hide graph
| current_annual_salary | |
|---|---|
| 0 | 6.92e+04 |
| 1 | 9.74e+04 |
| 2 | 1.05e+05 |
| 7,998 | 8.00e+04 |
| 7,999 | 1.11e+05 |
current_annual_salary
Float64DType- Null values
- 0 (0.0%)
- Unique values
-
3,063 (38.3%)
This column has a high cardinality (> 40).
- Mean ± Std
- 7.34e+04 ± 2.93e+04
- Median ± IQR
- 6.92e+04 ± 3.94e+04
- Min | Max
- 9.20e+03 | 3.03e+05
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
| Column | Column name | dtype | Is sorted | Null values | Unique values | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | current_annual_salary | Float64DType | False | 0 (0.0%) | 3063 (38.3%) | 7.34e+04 | 2.93e+04 | 9.20e+03 | 6.92e+04 | 3.03e+05 |
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
Please enable javascript
The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").
Finally, we apply the regressor. Note that the X and y nodes, on which train/test split is performed, are colored differently.
pred = employee_data.skb.apply(skrub.TableVectorizer()).skb.apply(
HistGradientBoostingRegressor(), y=salary
)
pred
Show/Hide graph
| current_annual_salary | |
|---|---|
| 0 | 6.76e+04 |
| 1 | 9.76e+04 |
| 2 | 1.00e+05 |
| 7,998 | 7.54e+04 |
| 7,999 | 1.06e+05 |
current_annual_salary
Float64DType- Null values
- 0 (0.0%)
- Unique values
-
6,488 (81.1%)
This column has a high cardinality (> 40).
- Mean ± Std
- 7.34e+04 ± 2.82e+04
- Median ± IQR
- 7.02e+04 ± 3.76e+04
- Min | Max
- 1.40e+04 | 2.22e+05
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
| Column | Column name | dtype | Is sorted | Null values | Unique values | Mean | Std | Min | Median | Max |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | current_annual_salary | Float64DType | False | 0 (0.0%) | 6488 (81.1%) | 7.34e+04 | 2.82e+04 | 1.40e+04 | 7.02e+04 | 2.22e+05 |
No columns match the selected filter: . You can change the column filter in the dropdown menu above.
Please enable javascript
The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").
Once we have defined our pipeline, we can tell skrub to perform the
cross-validation with DataOp.skb.cross_validate().
pred.skb.cross_validate(scoring="neg_mean_absolute_percentage_error")
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
Note that the variables used in this pipeline are different than the previous
one: we just have "csv_path" and not "employee_data" and "salary"
like before.
learner = pred.skb.make_learner(fitted=True)
learner.predict({"csv_path": test_dataset.path})
array([117656.318758 , 45185.65169354, 46138.80688344, ...,
97099.764518 , 145886.76864681, 70085.23723934], shape=(1228,))
Tuning arbitrary choices#
The last feature we present in this first tutorial is hyperparameter tuning.
The start of the pipeline is the same as before:
full_data = skrub.var("csv_path", train_dataset.path).skb.apply_func(pd.read_csv)
employee_data = full_data.drop(
columns="current_annual_salary", errors="ignore"
).skb.mark_as_X()
salary = full_data["current_annual_salary"].skb.mark_as_y()
We use functions like choose_from() or choose_float() whenever we
have a choice for which we want to try several options and keep the one that
performs best on the validation data.
We simply replace the value by the special “choice” object produced by skrub in our pipeline, and it becomes a tunable hyperparameter of our skrub learner. Here we want to tune:
the choice of encoder applied to high-cardinality categorical columns (
StringEncoderorTargetEncoder)for the StringEncoder, the number of components
the learning rate of the
HistGradientBoostingRegressor.
Note: choices are not restricted to estimators or their hyperparameters, we can tune any value used anywhere in a pipeline, or the choice between different pipelines; more details here.
from sklearn.preprocessing import TargetEncoder
n_components = skrub.choose_int(10, 80, name="n_components") # choose int in [10, 80[
encoder = skrub.choose_from( # choosing between 2 different estimators
{
"lse": skrub.StringEncoder(n_components=n_components), # nesting choices
"target": TargetEncoder(),
},
name="encoder",
)
pred = employee_data.skb.apply(
skrub.TableVectorizer(high_cardinality=encoder), y=salary
).skb.apply(
HistGradientBoostingRegressor(
learning_rate=skrub.choose_float(0.01, 0.7, log=True, name="learning_rate")
),
y=salary,
)
print(pred.skb.describe_param_grid())
- learning_rate: choose_float(0.01, 0.7, log=True, name='learning_rate')
encoder: 'lse'
n_components: choose_int(10, 80, name='n_components')
- learning_rate: choose_float(0.01, 0.7, log=True, name='learning_rate')
encoder: 'target'
To actually run the search for the best hyperparameters, we use
DataOp.skb.make_randomized_search() or
DataOp.skb.make_grid_search(). For the randomized search we can use the
powerful Optuna library which provides features like state-of-the-art
hyperparameter samplers, live interactive visualization of the search with
optuna-dashboard, stopping and resuming searches, etc.
search = pred.skb.make_randomized_search(
backend="optuna", fitted=True, n_iter=16, random_state=0
)
search.results_
Running optuna search for study skrub_randomized_search_b92b27e5-447d-413c-a79b-8ae96fa549e9 in storage /tmp/tmpx6zxwy68_skrub_optuna_search_storage/optuna_storage
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
/home/circleci/project/.pixi/envs/doc/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
warnings.warn(msg, UserWarning)
The search can be used with the same interface as the SkrubLearner
we saw before. Alternatively, we can access its best_learner_ attribute,
which is a SkrubLearner.
search.predict({"csv_path": test_dataset.path})
array([117953.78973311, 45425.83792057, 50233.10821664, ...,
97320.14629547, 143659.49306584, 73890.48051489], shape=(1228,))
Total running time of the script: (3 minutes 8.406 seconds)
Estimated memory usage: 612 MB
