User guide

Skrub eases preparing tables for machine learning.

Starting from rich, complex data stored in one or several dataframes, it helps performing the data wrangling necessary to produce a numeric array that is fed to a machine-learning model. This wrangling comprises joining tables (possibly with inexact matches), parsing text into structured data such as dates, extracting numeric features, etc.

For those tasks, skrub does not replace a dataframe library. Instead, it leverages polars or pandas to provide more high-level building blocks that are typically needed in a machine-learning pipeline.

Crucially, the transformations implemented by skrub are stateful: skrub records the transformations that were applied to the training data and replays the same operations when the pipeline is applied to make predictions on unseen data. Implementing data-wrangling steps as transformers that can be fitted is essential to prevent data leakage and ensure generalization.

Skrub highlights:

• eases separating the train and test operations, allowing to tune preprocessing steps to the data and improving the generalization of tabular machine-learning models.

• enables statistical and imperfect assembly, as machine-learning models can typically retrieve signals even in noisy data.

• End-to-end predictive models
  • Create baseline predictive models on heterogeneous dataset
  • Turning a dataframe into a numeric feature matrix
  • Examples
• Encoding: creating feature matrices
  • String entries: categories and open-ended entries
  • Encoding dates
• Skrub Pipeline: fit, tune, and validate arbitrary data wrangling
  • Introduction
  • Skrub expressions
  • Previews
  • Composing expressions
  • Applying machine-learning estimators
  • Deferred evaluation
  • Subsampling to develop pipelines with reduced computation
  • Using only a part of a pipeline
• Tuning and validating Skrub Pipelines
  • Splitting
  • Cross-validation
  • Tuning choices
  • Validating hyperparameter search with nested cross-validation
  • Choices beyond estimator hyperparameters
  • Linking choices
• Assembling: joining multiple tables
  • Fuzzy joining tables
  • Joining external tables for machine learning
  • Column selection inside a pipeline
  • Going further: embeddings for better analytics
• Cleaning
  • Cleaner: sanitizing a dataframe
  • deduplicate(): merging variants of the same entry