Skrub @ CFM

2026-06-10

Roadmap for the presentation

  • What is skrub
  • Contributing to skrub: setting up the environment
  • Contributing to skrub: subjects

What is skrub?

Skrub is a Python library that sits between data stored in dataframes and machine learning with scikit-learn.

Skrub eases preprocessing and machine learning with dataframes.

Skrub compatibility

  • Skrub is mostly written in Python, but it includes some Javascript
  • Skrub is fully compatible with pandas and polars
    • Any feature needs to be supported by both libraries
  • Skrub transformers are fully compatible with scikit-learn
    • Transformers need to satisfy some requirements

skrub.TableReport

from skrub import TableReport
TableReport(employee_salaries)

TableReport Preview

Main features:

  • Obtain high-level statistics about the data
  • Explore the distribution of values and find outliers
  • Discover highly correlated columns
  • Export and share the report as an HTML file

Data cleaning with pandas/polars: setup

import pandas as pd
import numpy as np

data = {
    "Int": [2, 3, 2],  # Multiple unique values
    "Const str": ["x", "x", "x"],  # Single unique value
    "Str": ["foo", "bar", "baz"],  # Multiple unique values
    "All nan": [np.nan, np.nan, np.nan],  # All missing values
    "All empty": ["", "", ""],  # All empty strings
    "Date": ["01 Jan 2023", "02 Jan 2023", "03 Jan 2023"],
}

df_pd = pd.DataFrame(data)
display(df_pd)
Int Const str Str All nan All empty Date
0 2 x foo NaN 01 Jan 2023
1 3 x bar NaN 02 Jan 2023
2 2 x baz NaN 03 Jan 2023

skrub.Cleaner

from skrub import Cleaner
cleaner = Cleaner(drop_if_constant=True, datetime_format='%d %b %Y')
df_cleaned = cleaner.fit_transform(df_pd)
display(df_cleaned)
Int Str Date
0 2 foo 2023-01-01
1 3 bar 2023-01-02
2 2 baz 2023-01-03

Encoding all the features: TableVectorizer

from skrub import TableVectorizer

table_vec = TableVectorizer()
df_encoded = table_vec.fit_transform(df_pd)

The TableVectorizer:

  • Applies the Cleaner to all columns
  • Splits columns by dtype and # of unique values
  • Encodes each column according to its characteristics

Encoding all the features: TableVectorizer

Build a predictive pipeline with tabular_pipeline

import skrub
from sklearn.linear_model import Ridge
model = skrub.tabular_pipeline(Ridge())

Encoding all the features: under the hood

  • DatetimeEncoder: convert datetimes to numerical columns (year, month, day …)
  • SquashingScaler: safely scale numerical features in presence of outliers
  • StringEncoder: quick, robust encoding of categorical and discrete features
  • TextEncoder: encode text and strings using pre-trained language models

Columnwise transformations with ApplyToCols and selectors

from sklearn.preprocessing import StandardScaler
from skrub import ApplyToCols
import skrub.selectors as s
df_id = pd.DataFrame(dict(id=[1000, 2000], A=[-10.0, 10.0], B=[-10.0, 0.0], C=[19, 20], D=["foo", "bar"]))
df_id
exc_scaler = ApplyToCols(StandardScaler(), exclude_cols=s.string() | "id")
exc_scaler.fit_transform(df_id)
id D A B C
0 1000 foo -1.0 -1.0 -1.0
1 2000 bar 1.0 1.0 1.0

A quick intro to DataOps…

Data Ops:

  • Extend the scikit-learn machinery to multi-table operations (join, aggregate…)
  • Can hold onto states, so they can be used to train transformers and models
  • Take care of data leakage
  • Let you tune any operation in a pipeline, including choosing which operations to execute
  • Guarantee that transformations are repeated exactly at prediction time
  • Can be persisted easily

Interested? We can talk about Data Ops during the sprint.

Resources and contacts

Do you want to learn more?

Follow skrub on:

Star skrub on GitHub, or contribute directly:

Contributing to skrub: preparation

Instructions are also available in the Installation page

Setting up the repository

First off, you need to fork the skrub repository

Then, clone the repo on your local machine

git clone https://github.com/<YOUR_USERNAME>/skrub
cd skrub

Add the upstream remote to pull the latest version of skrub:

git remote add upstream https://github.com/skrub-data/skrub.git

You can check that the remote has been added with git remote -v.

Fetch from upstream to update your fork:

git fetch upstream

Opening a PR and contributing upstream

More detail is available in the contributing guide

Start by creating a branch:

# fetch latest updates and start from the current head
git fetch upstream
# create a new branch and switch to it 
git checkout -b my-branch-name

Set the upstream branch to be your remote origin:

git push --set-upstream origin my-branch-name

Make some changes, then:

git add ./the/file-i-changed
git commit -m "my message"
git push

At this point, visit the GitHub PR page and open a PR from there.

About upstream and on avoiding conflicts

upstream/main is the “ground truth” for the repository. It is very important to keep up to date with upstream/main to avoid conflicts. This can be done by routinely merging your branch with upstream/main. If there are conflicts, they must be resolved before merging.

# verify you're on your branch with 
git status
# On branch my-branch-name
# Your branch is up to date with 'origin/my-branch-name'.

# fetch from upstream
git fetch upstream

# merge with upstream
git merge upstream/main

Important

Remember: you’re bringing information from upstream/main into my-branch-name, so you need to be inside my-branch-name and merge with upstream/main.

Setting up the environment

Tip

Use any env tool you’re comfortable with! If you have never used virtual environments before, use venv.

From inside the skrub directory you just cloned:

  • Create the venv (in the current dir):
python -m venv dev-skrub
  • Activate the venv:
source dev-skrub/bin/activate
  • Install skrub and dependencies:
pip install -e ".[dev]"
  • Create the venv (in the current dir):
uv venv dev-skrub
  • Activate the venv:
source activate
  • Install skrub and dev dependencies:
uv pip install -e ".[dev]"
  • Create the conda environment:
conda create -n dev-skrub
  • Activate the enviornment
conda activate dev-skrub
  • Install skrub and dependencies:
pip install -e ".[dev]"
#  Install an environment:
pixi install dev
# activate dev from IDE
# Run a command in a specific environment:
pixi run -e ci-py309-min-deps COMMAND
# Spawn a shell with the given env:
pixi shell -e ci-latest-optional-deps

Note on the pixi.lock file

Important

If you use pixi, it may happen that the pixi.lock file will be updated as you run commands.

Revert changes to this file before adding files and pushing upstream.

Running tests

From inside the root skrub folder, and after activating the environment

pytest --pyargs skrub

Run all tests (you will be prompted to choose an env, you can pick dev):

pixi run test

Run tests in a specific env

pixi run -e dev test

Running tests

Tests are stored in skrub/tests, or in a tests subfolder.

It is possible to run specific tests by providing a path:

# To test `TableVectorizer`
pytest -vsl skrub/tests/test_table_vectorizer.py

-vsl prints out more information compared to the default.

Working on the documentation

Docs are written in RST and use the Sphinx library for rendering, cross-references and everything else.

# From the doc/ folder
# Build the full documentation, including examples
make html

# Build documentation without running examples (faster)
make html-noplot

# Clean previously built documentation
make clean

From the skrub root folder (where pyproject.toml is):

# Build the full documentation, including examples
pixi run build-doc

# Build documentation without running examples (faster)
pixi run build-doc-quick

# Clean previously built documentation
pixi run clean-doc

After rendering the docs, open the doc/_build/html/index.html file with a browser (or open doc/_build/html/index.html).

Working on the documentation on Windows

Warning

On Windows, building the documentation may fail because it requires specific permissions. In that case, let the CI build the documentation and check it from the PR page.

# From the doc/ folder
# Build the full documentation, including examples
make.bat html

# Build documentation without running examples (faster)
make.bat html-noplot

# Clean previously built documentation
make.bat clean

Writing an example

  • Examples are python scripts placed in the examples/ folder.
  • The narrative of the examples should be written in comments and can include RST syntax.
  • Before commmitting, the documentation should be built (at least in quick mode) to check that the example is being rendered correctly.

Full guide

Code formatting and pre-commit

Skrub enforces some strict formatting rules to ensure code quality. This is done through pre-commit hooks. pre-commit must be installed, then it will run before every commit.

  • Make sure pre-commit is installed in the current environment
# install with pip in the env
pip install pre-commit
# install pre-commit using the repository's configuration
pre-commit install

Then, add modified files and run pre-commit on them.

git add YOUR_FILE
pre-commit
# if the file has been formatted, add again
git add YOUR_FILE
# commit the changes
git commit -m "MY COMMIT MESSAGE"
# pre-commit will run again automatically
# push the  commit
git push

If you’re working entirely with pixi:

git add YOUR_FILE
pixi run lint
# if the file has been formatted, add again
git add YOUR_FILE
# commit the changes
git commit -m "MY COMMIT MESSAGE"
# pre-commit will run again automatically
# push the  commit
git push

Code formatting and pre-commit

Note

In some cases, pre-commit will prevent you from committing, but won’t be able to fix the problem on its own. In those cases, you will have to address the issue manually.

Contributing to skrub: open issues

GitHub project

Before you start working on an issue

Important

Write a comment on the issue so we know you’re working on it.

We want to avoid having multiple people working on the same issue in separate PRs.