Choose your columns with selectors

Introduction to Selectors

Complex column selection beyond simple name lists:

import skrub.selectors as s

Selectors are used throughout the skrub package: - SelectCols/DropCols - ApplyToCols - .skb.apply() - TableReport

Selecting by Data Type

Available selectors:

  • .numeric(): Numeric columns (int or float)
  • .integer(): Integer columns only
  • .float(): Floating-point columns
  • .string(): String columns
  • .categorical(): Categorical columns
  • .any_date(): Date or datetime columns
  • .boolean(): Boolean columns

Example: Select String Columns

We can use SelectCols to choose which columns to keep:

from skrub import SelectCols
import pandas as pd

df = pd.DataFrame({
    "age": [25, 30, 35],
    "name": ["Alice", None, "Charlie"],
    "city": ["NYC", "LA", "Chicago"]
})

SelectCols(cols=s.string()).fit_transform(df)
name city
0 Alice NYC
1 None LA
2 Charlie Chicago

Example: select/drop columns

selectors include their own select method, which takes a dataframe and selector:

df = pd.DataFrame({
    "age": [25, 30, 35],
    "name": ["Alice", None, "Charlie"],
    "city": ["NYC", "LA", "Chicago"]
})

selector = s.string()
s.select(df, selector)
name city
0 Alice NYC
1 None LA
2 Charlie Chicago

Selecting by Characteristics

  • .all(): All columns
  • .has_nulls(): Columns with more than a given threshold of nulls (default=0)
  • .cardinality_below(threshold): Few unique values
  • .has_dtype(): Columns that have the provided dtype
df = pd.DataFrame({
    "age": [25, 30, 35],
    "name": ["Alice", None, "Charlie"],
    "city": ["NYC", "LA", "Chicago"]
})
SelectCols(cols=s.has_nulls()).fit_transform(df)
name
0 Alice
1 None
2 Charlie

Example: Selecting by dtype

import polars as pl

df = pl.DataFrame({"A": [[1,2], [1,2,3]], "B": ["A", "B"]})
df.dtypes
[List(Int64), String]

We want to select list columns:

sel = s.has_dtype(pl.List)
SelectCols(cols=sel).fit_transform(df)
shape: (2, 1)
A
list[i64]
[1, 2]
[1, 2, 3]

Selecting by Name

  • .cols("name"): Specific column name(s)
  • .glob("pattern*"): Unix shell-style globbing
  • .regex("pattern"): Regular expressions
df = pd.DataFrame({
    "patient_id": [101, 102, 103],
    "metric_1": [0.3, 0.3, 0.6],
    "metric_2": [3, 5, 10],
})

SelectCols(cols=s.glob("metric_*")).fit_transform(df)
metric_1 metric_2
0 0.3 3
1 0.3 5
2 0.6 10

Combining Selectors

Selectors can be combined: - each selector produces a set of columns - set operations (&, |, ^, -, ~) can be used to combine selectors - column names or lists of columns can be used directly

# Inverse: NOT numeric
~s.numeric()

# OR: datetime columns OR string columns
s.any_date() | s.string()

# AND: string columns without nulls
s.string() & ~s.has_nulls()

# XOR: either string, or name starts with "date_" but not both
s.string() ^ s.glob("date_*")

# Include: string columns OR col "date"
s.string() | "date"

# Exclude: all columns except "datetime-col"
s.all() - "datetime-col"

Extracting Selected Columns

Get the list of selected columns:

df = pd.DataFrame(
    {
        "age": [25, 30, 35],
        "name": ["Alice", None, "Charlie"],
        "city": ["NYC", "LA", "Chicago"],
    }
)

selector = s.has_nulls()
columns_with_nulls = selector.expand(df)

# Use in dataframe operations: capitalize columns with nulls
df[columns_with_nulls].apply(lambda x : x.str.upper())
name
0 ALICE
1 None
2 CHARLIE

Custom Selectors: Filter by Condition

def more_nulls_than(col, threshold=0.5):
    return col.isnull().sum() / len(col) > threshold

df = pd.DataFrame(
    {
        "no-nulls": [1, 2, 3, 4],
        "lotsa-nulls": [None, None, None, 4],
    }
)

selector = s.filter(more_nulls_than, threshold=0.5)
s.select(df, selector)
lotsa-nulls
0 NaN
1 NaN
2 NaN
3 4.0

Custom Selectors: Filter by Name

df = pd.DataFrame(
    {
        "patient_id": [101, 102, 103],
        "metric_1": [0.3, 0.3, 0.6],
        "metric_2": [3, 5, 10],
    }
)

selector = s.filter_names(lambda name: name.startswith("metric_"))
s.select(df, selector)
metric_1 metric_2
0 0.3 3
1 0.3 5
2 0.6 10

Using selectors with the TableReport

The TableReport can use selectors (or simply column names) to focus only on specific columns:

import skrub.selectors as s
from skrub import TableReport
custom_filter = {
    "metrics": s.glob("metric_*")
}

TableReport(df, column_filters=custom_filter)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

What we have seen in this chapters

  • Use selectors for flexible, reusable column selection
  • Combine selectors with logical operators
  • expand() extracts selected column names
  • Create custom selectors for domain-specific logic
  • Perfect with ApplyToCols, SelectCols, DropCols