All the pre-processing in one place: TableVectorizer

The Goal

Convert any mixed-type dataframe into numeric features for ML:

• Data should be sanitized (clean nulls, correct dtypes)
• Categorical features need to be encoded into numerical features
• Datetime features can be transformed into more informative features
• The transformer should be stateful and robust

The solution: TableVectorizer

The TableVectorizer is a transformer that

1. Cleans your data to have consistent representation of data types and null values, and
2. Encodes all columns depending on their data type and characteristics using good defaults.

A toy example

import pandas as pd
from skrub import TableVectorizer

df = pd.DataFrame({
    "age": [25, 30, 35],
    "salary": [50000, 65000, 75000],
    "hire_date": ["2020-01-15", "2021-06-20", "2022-03-10"],
    "department": ["Sales", "Engineering", "HR"]
})

tv = TableVectorizer()
X_numeric = tv.fit_transform(df)
print(X_numeric)

    age   salary  hire_date_year  hire_date_month  hire_date_day  \
0  25.0  50000.0          2020.0              1.0           15.0   
1  30.0  65000.0          2021.0              6.0           20.0   
2  35.0  75000.0          2022.0              3.0           10.0   

   hire_date_total_seconds  department_Engineering  department_HR  \
0             1.579046e+09                     0.0            0.0   
1             1.624147e+09                     1.0            0.0   
2             1.646870e+09                     0.0            1.0   

   department_Sales  
0               1.0  
1               0.0  
2               0.0  

Phase 1: Cleaning

The TableVectorizer starts by cleaning:

• Parse datetime columns (with custom formats)
• Detect numbers written as strings (“123.45”)
• Handle missing value markers (“N/A”, “?”)
• Drop uninformative columns
• Convert all columns to string if they can’t be parsed as float or datetime
• Convert to float32 for efficiency

In practice, a Cleaner followed by conversion to float32.

Phase 2: Column Dispatch

After cleaning, columns are routed to appropriate transformers:

Column Type	Cardinality	Transformer
Numeric	-	Passthrough
Datetime	-	DatetimeEncoder
String/Category	≤ 40	OneHotEncoder
String/Category	> 40	StringEncoder

# Change when categories become "high-cardinality"
tv = TableVectorizer(cardinality_threshold=10)

Customizing Transformers

from skrub import DatetimeEncoder, StringEncoder

# Add periodic encoding
datetime_enc = DatetimeEncoder(periodic_encoding="circular")
# Change the number of output components of the StringEncoder
string_enc = StringEncoder(n_components=10)

tv = TableVectorizer(
    datetime=datetime_enc,
    high_cardinality=string_enc
)

Applying to a subset of columns

import pandas as pd
from skrub import ApplyToCols
import skrub.selectors as s

df = pd.DataFrame(
    {
        "metric_1": [10.5, 20.3, 30.1, 40.2],
        "metric_2": [5.1, 15.6, None, 35.8],
        "metric_3": [1.1, 3.3, 2.6, .8],
        "num_id": [101, 102, 103, 104],
        "str_id": ["A101", "A102", "A103", "A104"],
    }
)
df

	metric_1	metric_2	metric_3	num_id	str_id
0	10.5	5.1	1.1	101	A101
1	20.3	15.6	3.3	102	A102
2	30.1	NaN	2.6	103	A103
3	40.2	35.8	0.8	104	A104

Applying to a subset of columns

We can use ApplyToCols and the skrub selectors:

tv = ApplyToCols(TableVectorizer(), exclude_cols=["num_id", "str_id"])
df_enc = tv.fit_transform(df)

Original
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

Encoded
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

Fine-Grained Control: specific_transformers

Tip

Use this only if you have a specific column that needs to be transformed by a specific custom-made transformer.

specific = [(CustomTransformer(), ["occupation"])]
specific = [("passthrough", ["num_id"])]
tv = TableVectorizer(specific_transformers=specific)

Columns in specific_transformers are forwarded directly to the transformer without any cleaning or parsing.

What we have seen in this chapter

• One object – the TableVectorizer – handles preprocessing and feature engineering
• Column types are detected, and columns are encoded accordingly
• Cleaning and encoding operations can be customized based on the use case
• Per-column behavior can be set with ApplyToCols