.. currentmodule:: skrub .. _user_guide_data_ops_hyperparameter_tuning: Using the skrub ``choose_*`` functions to tune hyperparameters ============================================================== skrub provides a convenient way to declare ranges of possible values, and tune those choices to keep the values that give the best predictions on a validation set. Rather than specifying a grid of hyperparameters separately from the pipeline, we simply insert special skrub objects in place of the value. We define the same set of operations as before: >>> from sklearn.datasets import load_diabetes >>> from sklearn.linear_model import Ridge >>> import skrub >>> diabetes_df = load_diabetes(as_frame=True)["frame"] >>> data = skrub.var("data", diabetes_df) >>> X = data.drop(columns="target", errors="ignore").skb.mark_as_X() >>> y = data["target"].skb.mark_as_y() >>> pred = X.skb.apply(Ridge(), y=y) Now, we can replace the hyperparameter ``alpha`` (which should be a float) with a range created by :func:`skrub.choose_float`. skrub can use it to select the best value for ``alpha``. >>> pred = X.skb.apply( ... Ridge(alpha=skrub.choose_float(1e-6, 10.0, log=True, name="α")), y=y ... ) .. warning:: When we do :meth:`.skb.make_learner() `, the pipeline we obtain does not perform any hyperparameter tuning. The pipeline we obtain by default uses default values for each of the choices. For numeric choices it is the middle of the range (unless an explicit default has been set when creating the choice), and for :func:`choose_from` it is the first option we give it. We can also obtain random choices, or choices suggested by an Optuna :class:`trial `, by passing the ``choose`` parameter. To get a pipeline that runs an internal cross-validation to select the best hyperparameters, we must use :meth:`.skb.make_grid_search() ` or :meth:`.skb.make_randomized_search() `. We can also use `Optuna `_ to choose the best hyperparameters as shown in :ref:`this example `. Here are the different kinds of choices, along with their default outcome when we are not using hyperparameter search: .. _choice-defaults-table: .. list-table:: Default choice outcomes :header-rows: 1 * - Choosing function - Description - Default outcome * - :func:`choose_from([10, 20]) ` - Choose between the listed options (10 and 20). - First outcome in the list: ``10`` * - :func:`choose_from({"a_name": 10, "b_name": 20}) ` - Choose between the listed options (10 and 20). Dictionary keys serve as names for the options. - First outcome in the dictionary: ``10`` * - :func:`optional(10) ` - Choose between the provided value and ``None`` (useful for optional transformations in a pipeline, e.g., ``optional(StandardScaler())``). - The provided ``value``: ``10`` * - :func:`choose_bool() ` - Choose between True and False. - ``True`` * - :func:`choose_float(1.0, 100.0) ` - Sample a floating-point number in a range. - The middle of the range: ``50.5`` * - :func:`choose_int(1, 100) ` - Sample an integer in a range. - The integer closest to the middle of the range: ``50`` * - :func:`choose_float(1.0, 100.0, log=True) ` - Sample a float in a range on a logarithmic scale. - The middle of the range on a log scale: ``10.0`` * - :func:`choose_int(1, 100, log=True) ` - Sample an integer in a range on a logarithmic scale. - The integer closest to the middle of the range on a log scale: ``10`` * - :func:`choose_float(1.0, 100.0, n_steps=4) ` - Sample a float on a grid. - The step closest to the middle of the range: ``34.0`` (steps: ``[1.0, 34.0, 67.0, 100.0]``) * - :func:`choose_int(1, 100, n_steps=4) ` - Sample an integer on a grid. - The step closest to the middle of the range: ``34`` (steps: ``[1, 34, 67, 100]``) * - :func:`choose_float(1.0, 100.0, log=True, n_steps=4) ` - Sample a float on a logarithmically spaced grid. - The step closest to the middle of the range on a log scale: ``4.64`` (steps: ``[1.0, 4.64, 21.54, 100.0]``) * - :func:`choose_int(1, 100, log=True, n_steps=4) ` - Sample an integer on a logarithmically spaced grid. - The step closest to the middle of the range on a log scale: ``5`` (steps: ``[1, 5, 22, 100]``) The default choices for a DataOp, those that get used when calling :meth:`.skb.make_learner() `, can be inspected with :meth:`.skb.describe_defaults() `: >>> pred.skb.describe_defaults() {'α': 0.00316...} We can then find the best hyperparameters. >>> search = pred.skb.make_randomized_search(fitted=True) >>> search.results_ # doctest: +SKIP α mean_test_score 0 0.000480 0.482327 1 0.000287 0.482327 2 0.000014 0.482317 3 0.000012 0.482317 4 0.000006 0.482317 5 0.134157 0.478651 6 0.249613 0.472019 7 0.612327 0.442312 8 2.664713 0.308492 9 3.457901 0.275007 A human-readable description of parameters for a pipeline can be obtained with :meth:`SkrubLearner.describe_params`: >>> search.best_learner_.describe_params() # doctest: +SKIP {'α': 0.000479...} It is also possible to use :meth:`ParamSearch.plot_results` to visualize the results of the search using a parallel coordinates plot. This could also be done with Optuna, either by passing ``backend='optuna'`` to :meth:`DataOp.skb.make_randomized_search`, or by using Optuna directly: >>> import optuna # doctest: +SKIP >>> def objective(trial): # doctest: +SKIP ... learner = pred.skb.make_learner(choose=trial) ... cv_results = skrub.cross_validate(learner, pred.skb.get_data()) ... return cv_results['test_score'].mean() >>> study = optuna.create_study(direction="maximize") # doctest: +SKIP >>> study.optimize(objective, n_trials=10) # doctest: +SKIP >>> best_learner = pred.skb.make_learner(choose=study.best_trial) # doctest: +SKIP >>> best_learner.describe_params() # doctest: +SKIP {'α': 0.0006391165935023005} Rather than fitting a randomized or grid search to find the best combination, it is also possible to obtain an iterator over different parameter combinations to inspect their outputs or to have manual control over the model selection. This can be done with :meth:`.skb.iter_learners_grid() ` or :meth:`.skb.iter_learners_randomized() ` ( which yield the candidate pipelines that are explored by the grid and randomized search respectively), or with the ``choose`` parameter of :meth:`.skb.make_learner() `. A full example of how to use hyperparameter search is available in :ref:`sphx_glr_auto_examples_data_ops_1130_choices.py`, and a full example using Optuna is in :ref:`example_optuna_choices`. | .. _user_guide_data_ops_feature_selection: Feature selection with skrub :class:`SelectCols` and :class:`DropCols` ======================================================================= It is possible to combine :class:`SelectCols` and :class:`DropCols` with :func:`choose_from` to perform feature selection by dropping specific columns and evaluating how this affects the downstream performance. Consider this example. We first define the variable: >>> import pandas as pd >>> import skrub.selectors as s >>> from sklearn.preprocessing import StandardScaler, OneHotEncoder >>> df = pd.DataFrame({"text": ["foo", "bar", "baz"], "number": [1, 2, 3]}) >>> X = skrub.X(df) Then, we use the :ref:`skrub selectors ` to encode each column with a different transformer: >>> X_enc = X.skb.apply(StandardScaler(), cols=s.numeric()).skb.apply( ... OneHotEncoder(sparse_output=False), cols=s.string() ... ) >>> X_enc Result: ――――――― number text_bar text_baz text_foo 0 -1.224745 0.0 0.0 1.0 1 0.000000 1.0 0.0 0.0 2 1.224745 0.0 1.0 0.0 Now we can use :class:`skrub.DropCols` to define two possible selection strategies: first, we drop the column ``number``, then we drop all columns that start with ``text``. We rely again on the skrub selectors for this: >>> from skrub import DropCols >>> drop = DropCols(cols=skrub.choose_from( ... {"number": s.cols("number"), "text": s.glob("text_*")}) ... ) >>> X_enc.skb.apply(drop) Result: ――――――― text_bar text_baz text_foo 0 0.0 0.0 1.0 1 1.0 0.0 0.0 2 0.0 1.0 0.0 We can see the generated parameter grid with :func:`DataOps.skb.describe_param_grid()`. >>> X_enc.skb.apply(drop).skb.describe_param_grid() "- choose_from({'number': …, 'text': …}): ['number', 'text']\n" A more advanced application of this technique is used in `this tutorial on forecasting timeseries `_, along with the feature engineering required to prepare the columns, and the analysis of the results.