5  Choose your column: selectors

6 Introduction

Very often, column selection is more complex than simply passing a list of column names to a transformer: it may be necessary to select all columns that have a specific data type, or based on some other characteristic (presence of nulls, column cardinality, etc.).

The skrub selectors implement a number of selection strategies that can be combined in various ways to build complex filtering conditions that can then be employed by ApplyToCols, SelectCols, DropCols, .skb.apply() and the TableReport.

6.1 Skrub selectors

Selectors are available from the skrub.selectors namespace:

import skrub.selectors as s

We can use them in conjunction with SelectCols to select, for example, only string columns:

from skrub import SelectCols
import pandas as pd

df = pd.DataFrame({
    "age": [25, 30, 35],
    "name": ["Alice", None, "Charlie"],
    "city": ["NYC", "LA", "Chicago"]
})

SelectCols(cols=s.string()).fit_transform(df)

	name	city
0	Alice	NYC
1	None	LA
2	Charlie	Chicago

Selectors allow to filter columns by data type:

• .float: floating-point columns
• .integer: integer columns
• .any_date: date or datetime columns
• .boolean: boolean columns
• .string: columns with a String data type
• .categorical: columns with a Categorical data type
• .numeric: numeric (either integer or float) columns

from skrub import SelectCols
string_selector = s.string()

SelectCols(cols=string_selector).fit_transform(df)

	name	city
0	Alice	NYC
1	None	LA
2	Charlie	Chicago

Additional conditions include:

• .all: select all columns
• .cardinality_below: select all columns with a number of unique values lower than the given threshold
• .has_nulls: select all columns that include at least one null value
• .has_dtype: select columns that include the provided dtype

SelectCols(cols=s.has_nulls()).fit_transform(df)

	name
0	Alice
1	None
2	Charlie

import polars as pl

df = pl.DataFrame({"A": [[1,2], [1,2,3]], "B": ["A", "B"]})
print(df.dtypes)
sel = s.has_dtype(pl.List)
SelectCols(cols=sel).fit_transform(df)

[List(Int64), String]

shape: (2, 1)

A
list[i64]
[1, 2]
[1, 2, 3]

Various selectors allow to choose columns based on their name:

• .cols: choose the provided column name (or list of names)
  • note that transformers that can accept selectors can also take column names or lists of columns by default
• .glob: use Unix shell style glob to select column names
• .regex: select columns using regular expressions

df = pd.DataFrame({
    "patient_id": [101, 102, 103],
    "metric_1": [0.3, 0.3, 0.6],
    "metric_2": [3, 5, 10],
})

SelectCols(cols=s.glob("metric_*")).fit_transform(df)

	metric_1	metric_2
0	0.3	3
1	0.3	5
2	0.6	10

6.2 Combining selectors

Selectors are designed to be treated like sets of columns, and can be combined by using set operations. Column names or lists of column names are transformed into selectors automatically.

Column selection rules are very powerful and provide a high degree of flexibility:

# Inverse: NOT numeric
~s.numeric()

# OR: datetime columns OR string columns OR the columns in a list
s.any_date() | s.string() | ["date", "datetime-col"]

# AND: string columns without nulls
s.string() & ~s.has_nulls()

# Exclude: all columns except "datetime-col"
s.all() - "datetime-col"

6.3 Extracting selected columns

Selectors can use the expand and expand_index methods to extract the columns that have been selected:

df = pd.DataFrame(
    {
        "age": [25, 30, 35],
        "name": ["Alice", None, "Charlie"],
        "city": ["NYC", "LA", "Chicago"],
    }
)

selector = s.has_nulls()
columns_with_nulls = selector.expand(df)

This can be used, for example, to pass a list of columns to a dataframe library.

df[columns_with_nulls].apply(lambda x : x.str.upper())

6.4 Designing custom filters

Finally, it is possible to define function-based selectors using .filter and .filter_names.

.filter selects columns for which the predicate evaluated by a user-defined function on the given column is True. It is also possible to pass arguments to the function to further tweak the conditions. For example, it is possible to select columns that include a certain amount of nulls by defining a function like the following:

def more_nulls_than(col, threshold=0.5):
    return col.isnull().sum() / len(col) > threshold

df = pd.DataFrame(
    {
        "no-nulls": [1, 2, 3, 4],
        "lotsa-nulls": [None, None, None, 4],
    }
)

selector = s.filter(more_nulls_than, threshold=0.5)
s.select(df, selector)

	lotsa-nulls
0	NaN
1	NaN
2	NaN
3	4.0

.filter_names is similar to .filter in the sense that it takes a function that returns a predicate, but in this case the function is evaluated over the column names.

If we define this example dataframe:

from skrub import selectors as s
import pandas as pd
df = pd.DataFrame(
    {
        "height_mm": [297.0, 420.0],
        "width_mm": [210.0, 297.0],
        "kind": ["A4", "A3"],
        "ID": [4, 3],
    }
)
df

	height_mm	width_mm	kind	ID
0	297.0	210.0	A4	4
1	420.0	297.0	A3	3

We can select all the columns that end with "_mm" as follows:

selector = s.filter_names(lambda name: name.endswith('_mm'))
s.select(df, selector)

	height_mm	width_mm
0	297.0	210.0
1	420.0	297.0

6.5 Using selectors with the TableReport

The TableReport can use selectors (or simply column names) to focus only on specific columns:

import skrub.selectors as s
from skrub import TableReport
custom_filter = {
    "metrics": s.glob("metric_*")
}

df = pd.DataFrame({
    "patient_id": [101, 102, 103],
    "metric_1": [0.3, 0.3, 0.6],
    "metric_2": [3, 5, 10],
})

TableReport(df, column_filters=custom_filter)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

6.6 What we have seen in this chapters

In this chapter we covered the skrub selectors, how they allow to select specific columns either through simple conditions, or by combining different selectors.

Selectors can be combined with other skrub transformers to filter and operate over only specific column.