Choose your columns with selectors

Introduction to Selectors

Complex column selection beyond simple name lists:

import skrub.selectors as s

Selectors are used throughout the skrub package:

• ApplyToCols
• TableReport filters
• SelectCols/DropCols
• .skb.apply()

Example: Select String Columns

We can use SelectCols to choose which columns to keep:

from skrub import SelectCols
import pandas as pd

df = pd.DataFrame({
    "age": [25, 30, 35],
    "name": ["Alice", None, "Charlie"],
    "city": ["NYC", "LA", "Chicago"]
})

SelectCols(cols=s.string()).fit_transform(df)

	name	city
0	Alice	NYC
1	None	LA
2	Charlie	Chicago

Selecting by Data Type

Some of the available selectors:

• .numeric(): Numeric columns (int or float)
• .integer(): Integer columns only
• .float(): Floating-point columns
• .string(): String columns
• .categorical(): Categorical columns
• .any_date(): Date or datetime columns
• .boolean(): Boolean columns

Selecting by Characteristics

• .all(): All columns
• .has_nulls(): Columns with more than a given threshold of nulls (default=0)
• .cardinality_below(threshold): Few unique values
• .has_dtype(): Columns that have the provided dtype

import polars as pl

df = pl.DataFrame({"A": [[1,2], [1,2,3]], "B": ["A", "B"]})
df.dtypes

[List(Int64), String]

sel = s.has_dtype(pl.List)
SelectCols(cols=sel).fit_transform(df)

shape: (2, 1)

A
list[i64]
[1, 2]
[1, 2, 3]

Selecting by Name

• .cols("name"): Specific column name(s)
• .glob("pattern*"): Unix shell-style globbing
• .regex("pattern"): Regular expressions

df = pd.DataFrame({
    "patient_id": [101, 102, 103],
    "metric_1": [0.3, 0.3, 0.6],
    "metric_2": [3, 5, 10],
})

SelectCols(cols=s.glob("metric_*")).fit_transform(df)

	metric_1	metric_2
0	0.3	3
1	0.3	5
2	0.6	10

Combining Selectors

Selectors can be combined:

• each selector produces a set of columns
• set operations (&, |, ^, -, ~) can be used to combine selectors
• column names or lists of columns can be used directly

# Inverse: NOT numeric
~s.numeric()

# OR: datetime columns OR string columns OR the columns in a list
s.any_date() | s.string() | ["date", "datetime-col"]

# AND: string columns without nulls
s.string() & ~s.has_nulls()

# Exclude: all columns except "datetime-col"
s.all() - "datetime-col"

Using selectors with pandas dataframes

Get the list of selected columns:

df = pd.DataFrame(
    {
        "age": [25, 30, 35],
        "name": ["Alice", None, "Charlie"],
    }
)

selector = s.has_nulls()
columns_with_nulls = selector.expand(df)
print(columns_with_nulls)

# Use in dataframe operations: drop columns with nulls
df.drop(columns=columns_with_nulls)

['name']

	age
0	25
1	30
2	35

Custom Selectors: Filter by Condition

Define a function that takes one column as input and returns True/False as output (depending on if the column is selected or not)

def more_nulls_than_half(col):
    return col.isnull().sum() / len(col) > 0.5

df = pd.DataFrame(
    {
        "no-nulls": [1, 2, 3, 4],
        "lotsa-nulls": [None, None, None, 4],
    }
)

selector = s.filter(more_nulls_than_half)
s.select(df, selector)

	lotsa-nulls
0	NaN
1	NaN
2	NaN
3	4.0

Using selectors with the TableReport

The TableReport can use selectors (or simply column names) to focus only on specific columns:

from skrub import TableReport
custom_filter = {"metrics": s.glob("metric_*")}

df = pd.DataFrame({
    "patient_id": [101, 102, 103],
    "metric_1": [0.3, 0.3, 0.6],
    "metric_2": [3, 5, 10],
})

TableReport(df, column_filters=custom_filter)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

What we have seen in this chapters

• Selectors can be used for flexible, reusable column selection
• Selectors can be combined with logical operators as “sets of columns”
• expand() extracts selected column names
• s.filter() can be used to create custom selectors for domain-specific logic
• ApplyToCols and TableReport can use selectors to filter columns