6 All the pre-processing in one place: `TableVectorizer`

6.1 Introduction

Machine learning models typically require numeric input features. When working with real-world datasets, we often have a mix of data types: numbers, text, dates, and categorical values.

So far, we’ve seen how skrub helps consolidating those data types by sanitizing and converting when possible. Then, the next step is going from complex and heterogeneous data types like strings and datetimes to a matrix of numeric features ready for machine learning.

This is where the subject of this chapter, the TableVectorizer, comes into play. Instead of manually specifying how to handle each column, the TableVectorizer automatically assigns columns to trasformation strategies, and encodes them accordingly.

6.2 A toy example

import pandas as pd
from skrub import TableVectorizer

df = pd.DataFrame({
    "age": [25, 30, 35],
    "salary": [50000, 65000, 75000],
    "hire_date": ["2020-01-15", "2021-06-20", "2022-03-10"],
    "department": ["Sales", "Engineering", "Sales"]
})

tv = TableVectorizer()
X_numeric = tv.fit_transform(df)

6.3 How does the TableVectorizer work?

The TableVectorizer operates in two phases:

6.3.1 Phase 1: Data cleaning and type detection

First, it runs a Cleaner on the input data to:

Detect and parse datetime columns (possibly, with custom datetime formats)
Detect and parse numerical columns written as strings
Handle missing values represented as strings (e.g., “N/A”)
Remove uninformative columns (those with only nulls, constant values, or all unique values) This ensures that each column has the correct data type before encoding.

Additionally, all numerical features are converted to float32 to reduce the computational cost.

6.3.2 Phase 2: Column dispatch and encoding

After cleaning, the TableVectorizer categorizes columns and dispatches them to the appropriate transformer based on their data type and cardinality. Categorical columns with a cardinality (i.e., number of unique values) larger than 40 are considered “high cardinality”, while all other categorical columns are “low cardinality”.

The TableVectorizer uses the following default transformers for each column type:

Numeric columns: Left untouched (passthrough) - they’re already in the right format
Datetime columns: Transformed by DatetimeEncoder to extract meaningful temporal features
Low-cardinality categorical/string columns: Transformed with OneHotEncoder to create binary indicator variables
High-cardinality categorical/string columns: Transformed with StringEncoder to create dense numeric representations

6.4 Key Parameters

6.4.1 Cardinality threshold

The cardinality threshold that splits columns in “high” and “low” cardinality can be changed by setting the relative parameter:

from skrub import TableVectorizer

tv = TableVectorizer(cardinality_threshold=10)  # Adjust the threshold

6.4.2 Data cleaning parameters

The TableVectorizer forwards several parameters to the internal Cleaner, which behave in the same way:

drop_null_fraction: Fraction of nulls above which a column is dropped (default: 1.0)
drop_if_constant: Drop columns with only one unique value (default: False)
datetime_format: Format string for parsing dates
null_strings: Additional strings to consider as nulls besides the default

tv = TableVectorizer(
    drop_null_fraction=0.9,  # Drop columns that are 90% null
    drop_if_constant=True,
    datetime_format="%Y-%m-%d"
)

6.4.3 Customizing the transformers used by `TableVectorizer`

The TableVectorizer applies whatever transformer is provided to each of the numeric, datetime, high_cardinality, and low_cardinality paramters. To tweak the default parameters of the transformers a new transformer should be provided:

from skrub import TableVectorizer, DatetimeEncoder, StringEncoder
from sklearn.preprocessing import OneHotEncoder

# Create custom transformers
# Add periodic features with the DatetimeEncoder
datetime_enc = DatetimeEncoder(periodic_encoding="circular")
# Change the number of components of the StringEncoder
string_enc = StringEncoder(n_components=10)

# Pass them to TableVectorizer
tv = TableVectorizer(
    datetime=datetime_enc,
    high_cardinality=string_enc,
)

6.4.4 Applying the `TableVectorizer` only to a subset of columns

By default, the TableVectorizer is applied to all the columns in the given dataframe. In some cases, it may be important to keep specific columns “as is”, so that they are not modified by the transformer.

This can be done by wrapping the vectorizer into an ApplyToCols object.

For example, in this case we might want to avoid modifying the two *_id columns.

import pandas as pd
from skrub import ApplyToCols
import skrub.selectors as s

df = pd.DataFrame(
    {
        "metric_1": [10.5, 20.3, 30.1, 40.2],
        "metric_2": [5.1, 15.6, None, 35.8],
        "metric_3": [1.1, 3.3, 2.6, .8],
        "num_id": [101, 102, 103, 104],
        "str_id": ["A101", "A102", "A103", "A104"],
        "description": ["apple", None, "cherry", "date"],
        "name": ["Alice", "Bob", "Charlie", "David"],
    }
)
df

	metric_1	metric_2	metric_3	num_id	str_id	description	name
0	10.5	5.1	1.1	101	A101	apple	Alice
1	20.3	15.6	3.3	102	A102	None	Bob
2	30.1	NaN	2.6	103	A103	cherry	Charlie
3	40.2	35.8	0.8	104	A104	date	David

We can use ApplyToCols and the skrub selectors as follows:

tv = ApplyToCols(TableVectorizer(), exclude_cols=["num_id", "str_id"])
df_enc = tv.fit_transform(df)

print("Original")
print(df[["num_id", "str_id"]])
print("\nEncoded")
print(df_enc[["num_id", "str_id"]])

Original
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

Encoded
   num_id str_id
0     101   A101
1     102   A102
2     103   A103
3     104   A104

The id strings are the same, while all other columns have been encoded as expected.

6.4.5 Using `specific_transformers` for more low-level control

For fine-grained control, we can specify transformers for specific columns using the specific_transformers parameter. This is useful when we want to override the default behavior for particular columns:

from sklearn.preprocessing import OrdinalEncoder
import pandas as pd

df = pd.DataFrame({
    "occupation": ["engineer", "teacher", "doctor"],
    "salary": [100000, 50000, 150000]
})

# Create a custom transformer for the 'occupation' column
specific_transformers = [(OrdinalEncoder(), ["occupation"])]

tv = TableVectorizer(specific_transformers=specific_transformers)
result = tv.fit_transform(df)

Important notes about specific_transformers:

Columns specified here bypass the default categorization logic
The transformer receives the column as-is, without any preprocessing
The transformer must be able to handle the column’s current data type and values
For more complex transformations, consider using ApplyToCols and the selectors API (explained in the previous chapters), or the skrub Data Ops.

6.5 What we have seen in this chapter

The TableVectorizer is a self-contained feature engineering transformer that

Cleans your data to have consistent representation of data types and null values, and
Encodes all columns depending on their data type and characteristics using good defaults.

The idea behind the TableVectorizer is that you should be able to provide any dataframe, and get a good feature matrix based on that dataframe as a result.

The TableVectorizer makes use of most of the objects that have been explained so far, and is an important part of the tabular_pipeline explained in the next chapter.

--- title: "All the pre-processing in one place: `TableVectorizer`" format: html: toc: true revealjs: slide-number: true toc: false code-fold: false code-tools: true --- ## Introduction Machine learning models typically require numeric input features. When working with real-world datasets, we often have a mix of data types: numbers, text, dates, and categorical values. So far, we've seen how skrub helps consolidating those data types by sanitizing and converting when possible. Then, the next step is going from complex and heterogeneous data types like strings and datetimes to a matrix of numeric features ready for machine learning. This is where the subject of this chapter, the `TableVectorizer`, comes into play. Instead of manually specifying how to handle each column, the `TableVectorizer` automatically assigns columns to trasformation strategies, and encodes them accordingly. ## A toy example ```{python} import pandas as pd from skrub import TableVectorizer df = pd.DataFrame({ "age": [25, 30, 35], "salary": [50000, 65000, 75000], "hire_date": ["2020-01-15", "2021-06-20", "2022-03-10"], "department": ["Sales", "Engineering", "Sales"] }) tv = TableVectorizer() X_numeric = tv.fit_transform(df) ``` ## How does the TableVectorizer work? The `TableVectorizer` operates in two phases: ### Phase 1: Data cleaning and type detection First, it runs a `Cleaner` on the input data to: - Detect and parse datetime columns (possibly, with custom datetime formats) - Detect and parse numerical columns written as strings - Handle missing values represented as strings (e.g., "N/A") - Remove uninformative columns (those with only nulls, constant values, or all unique values) This ensures that each column has the correct data type before encoding. Additionally, all numerical features are converted to `float32` to reduce the computational cost. ### Phase 2: Column dispatch and encoding After cleaning, the `TableVectorizer` categorizes columns and dispatches them to the appropriate transformer based on their data type and cardinality. Categorical columns with a cardinality (i.e., number of unique values) larger than 40 are considered "high cardinality", while all other categorical columns are "low cardinality". The `TableVectorizer` uses the following default transformers for each column type: - **Numeric columns**: Left untouched (passthrough) - they're already in the right format - **Datetime columns**: Transformed by `DatetimeEncoder` to extract meaningful temporal features - **Low-cardinality categorical/string columns**: Transformed with `OneHotEncoder` to create binary indicator variables - **High-cardinality categorical/string columns**: Transformed with `StringEncoder` to create dense numeric representations ## Key Parameters ### Cardinality threshold The cardinality threshold that splits columns in "high" and "low" cardinality can be changed by setting the relative parameter: ```{python} from skrub import TableVectorizer tv = TableVectorizer(cardinality_threshold=10) # Adjust the threshold ``` ### Data cleaning parameters The `TableVectorizer` forwards several parameters to the internal `Cleaner`, which behave in the same way: - `drop_null_fraction`: Fraction of nulls above which a column is dropped (default: `1.0`) - `drop_if_constant`: Drop columns with only one unique value (default: `False`) - `datetime_format`: Format string for parsing dates - `null_strings`: Additional strings to consider as nulls besides the default ```{python} tv = TableVectorizer( drop_null_fraction=0.9, # Drop columns that are 90% null drop_if_constant=True, datetime_format="%Y-%m-%d" ) ``` ### Customizing the transformers used by `TableVectorizer` The `TableVectorizer` applies whatever transformer is provided to each of the `numeric`, `datetime`, `high_cardinality`, and `low_cardinality` paramters. To tweak the default parameters of the transformers a new transformer should be provided: ```{python} from skrub import TableVectorizer, DatetimeEncoder, StringEncoder from sklearn.preprocessing import OneHotEncoder # Create custom transformers # Add periodic features with the DatetimeEncoder datetime_enc = DatetimeEncoder(periodic_encoding="circular") # Change the number of components of the StringEncoder string_enc = StringEncoder(n_components=10) # Pass them to TableVectorizer tv = TableVectorizer( datetime=datetime_enc, high_cardinality=string_enc, ) ``` ### Applying the `TableVectorizer` only to a subset of columns By default, the `TableVectorizer` is applied to all the columns in the given dataframe. In some cases, it may be important to keep specific columns "as is", so that they are not modified by the transformer. This can be done by wrapping the vectorizer into an `ApplyToCols` object. For example, in this case we might want to avoid modifying the two `*_id` columns. ```{python} import pandas as pd from skrub import ApplyToCols import skrub.selectors as s df = pd.DataFrame( { "metric_1": [10.5, 20.3, 30.1, 40.2], "metric_2": [5.1, 15.6, None, 35.8], "metric_3": [1.1, 3.3, 2.6, .8], "num_id": [101, 102, 103, 104], "str_id": ["A101", "A102", "A103", "A104"], "description": ["apple", None, "cherry", "date"], "name": ["Alice", "Bob", "Charlie", "David"], } ) df ``` We can use `ApplyToCols` and the skrub selectors as follows: ```{python} tv = ApplyToCols(TableVectorizer(), exclude_cols=["num_id", "str_id"]) df_enc = tv.fit_transform(df) print("Original") print(df[["num_id", "str_id"]]) print("\nEncoded") print(df_enc[["num_id", "str_id"]]) ``` The id strings are the same, while all other columns have been encoded as expected. ### Using `specific_transformers` for more low-level control For fine-grained control, we can specify transformers for specific columns using the `specific_transformers` parameter. This is useful when we want to override the default behavior for particular columns: ```{python} from sklearn.preprocessing import OrdinalEncoder import pandas as pd df = pd.DataFrame({ "occupation": ["engineer", "teacher", "doctor"], "salary": [100000, 50000, 150000] }) # Create a custom transformer for the 'occupation' column specific_transformers = [(OrdinalEncoder(), ["occupation"])] tv = TableVectorizer(specific_transformers=specific_transformers) result = tv.fit_transform(df) ``` Important notes about `specific_transformers`: - Columns specified here bypass the default categorization logic - The transformer receives the column as-is, without any preprocessing - The transformer must be able to handle the column's current data type and values - For more complex transformations, consider using `ApplyToCols` and the selectors API (explained in the previous chapters), or the skrub [Data Ops](https://skrub-data.org/stable/data_ops.html). ## What we have seen in this chapter The `TableVectorizer` is a self-contained feature engineering transformer that 1. Cleans your data to have consistent representation of data types and null values, and 2. Encodes all columns depending on their data type and characteristics using good defaults. The idea behind the `TableVectorizer` is that you should be able to provide any dataframe, and get a good feature matrix based on that dataframe as a result. The `TableVectorizer` makes use of most of the objects that have been explained so far, and is an important part of the `tabular_pipeline` explained in the next chapter.