All the pre-processing in one place: TableVectorizer

The Goal

Convert any mixed-type dataframe into numeric features for ML:

  • Data should be sanitized (clean nulls, correct dtypes)
  • Categorical features need to be encoded into numerical features
  • Datetime features can be transformed into more informative features
  • The transformer should be stateful and robust

The solution: TableVectorizer

The TableVectorizer is a transformer that

  1. Cleans your data to have consistent representation of data types and null values, and
  2. Encodes all columns depending on their data type and characteristics using good defaults.

A toy example

import pandas as pd
from skrub import TableVectorizer

df = pd.DataFrame({
    "age": [25, 30, 35],
    "salary": [50000, 65000, 75000],
    "hire_date": ["2020-01-15", "2021-06-20", "2022-03-10"],
    "department": ["Sales", "Engineering", "HR"]
})

tv = TableVectorizer()
X_numeric = tv.fit_transform(df)
print(X_numeric)
    age   salary  hire_date_year  hire_date_month  hire_date_day  \
0  25.0  50000.0          2020.0              1.0           15.0   
1  30.0  65000.0          2021.0              6.0           20.0   
2  35.0  75000.0          2022.0              3.0           10.0   

   hire_date_total_seconds  department_Engineering  department_HR  \
0             1.579046e+09                     0.0            0.0   
1             1.624147e+09                     1.0            0.0   
2             1.646870e+09                     0.0            1.0   

   department_Sales  
0               1.0  
1               0.0  
2               0.0

Phase 1: Cleaning

The TableVectorizer starts by cleaning:

  • Parse datetime columns (with custom formats)
  • Detect numbers written as strings (“123.45”)
  • Handle missing value markers (“N/A”, “?”)
  • Drop uninformative columns
  • Convert all columns to string if they can’t be parsed as float or datetime
  • Convert to float32 for efficiency

Tip

In practice, a Cleaner followed by conversion to float32.

Phase 2: Column Dispatch

After cleaning, columns are routed to appropriate transformers:

Column Type Cardinality Transformer
Numeric - Passthrough
Datetime - DatetimeEncoder
String/Category ≤ 40 OneHotEncoder
String/Category > 40 StringEncoder 
# Change when categories become "high-cardinality"
tv = TableVectorizer(cardinality_threshold=10)

Customizing Transformers

from skrub import DatetimeEncoder, LLMEncoder

# Add periodic encoding
datetime_enc = DatetimeEncoder(periodic_encoding="circular")
# Use a pre-trained language model
llm_enc = LLMEncoder()

tv = TableVectorizer(
    datetime=datetime_enc,
    high_cardinality=llm_enc
)

Applying to a subset of columns

import pandas as pd
from skrub import ApplyToCols
import skrub.selectors as s

df = pd.DataFrame(
    {
        "metric_1": [10.5, 20.3, 30.1, 40.2],
        "metric_2": [5.1, 15.6, None, 35.8],
        "metric_3": [1.1, 3.3, 2.6, .8],
        "num_id": [101, 102, 103, 104],
        "str_id": ["A101", "A102", "A103", "A104"],
    }
)

We can use ApplyToCols and the skrub selectors:

tv = ApplyToCols(TableVectorizer(), exclude_cols=["num_id", "str_id"])
df_enc = tv.fit_transform(df)
Original
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

Encoded
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

Opening the black box

Plain text: describe_transformations_

tv = TableVectorizer().fit(df)
print(tv.describe_transformations())
Preprocessors
=============
Null values cleaned (1 columns):
    - str_id

Processors by type
==================
PassThrough (numeric - 4 columns):
    - metric_1
    - metric_2
    - metric_3
    - num_id
No datetime columns have been detected.
OneHotEncoder (low_cardinality - 1 columns):
    - str_id
No high_cardinality columns have been detected.

Retrieve the actual transformers, column by column: all_processing_steps_

from pprint import pprint
tv = TableVectorizer().fit(df)
pprint(tv.all_processing_steps_)
{'metric_1': [DropUninformative(),
              ToFloat(),
              PassThrough(),
              {'metric_1': ToFloat()}],
 'metric_2': [DropUninformative(),
              ToFloat(),
              PassThrough(),
              {'metric_2': ToFloat()}],
 'metric_3': [DropUninformative(),
              ToFloat(),
              PassThrough(),
              {'metric_3': ToFloat()}],
 'num_id': [DropUninformative(),
            ToFloat(),
            PassThrough(),
            {'num_id': ToFloat()}],
 'str_id': [CleanNullStrings(),
            DropUninformative(),
            ToStr(),
            OneHotEncoder(drop='if_binary', dtype='float32', handle_unknown='ignore',
              sparse_output=False),
            {'str_id_A101': ToFloat(), 'str_id_A102': ToFloat(), ...}]}

Fine-Grained Control: specific_transformers

Tip

Use this only if you have a specific column that needs to be transformed by a specific custom-made transformer.

specific = [(CustomTransformer(), ["occupation"])]
specific = [("passthrough", ["num_id"])]
tv = TableVectorizer(specific_transformers=specific)

Columns in specific_transformers are forwarded directly to the transformer without any cleaning or parsing.

What we have seen in this chapter

  • One object – the TableVectorizer – handles preprocessing and feature engineering
  • Column types are detected, and columns are encoded accordingly
  • Cleaning and encoding operations can be customized based on the use case
  • Per-column behavior can be set with ApplyToCols
  • The transformations are accessible via describe_transformations() and all_processing_steps_