---
title: "Chapter 1: Introduction"
format:
html:
toc: true
revealjs:
slide-number: true
toc: false
code-fold: false
code-tools: true
execute:
echo: true
---
## A world without skrub {.smaller}
Let's consider a world where skrub does not exist, and all we can do is use
pandas and scikit-learn to prepare data for a machine learning model.
You may already be living in such a world.
### Load and explore the data
```{python}
import pandas as pd
import numpy as np
X = pd.read_csv("../data/employee_salaries/data.csv")
y = pd.read_csv("../data/employee_salaries/target.csv")["current_annual_salary"]
X.head(5)
```
### Explore the target
Let's take a look at the target:
```{python}
y.head(5)
```
This is a **regression** task: we want to predict the value of `current_annual_salary`.
### Strategizing
We can begin by exploring the dataframe with `.describe`, and then think of a
plan for pre-processing our data.
```{python}
X.describe(include="all")
```
### Our plan
We need to:
- Impute some missing values in the `gender` column.
- Encode convert categorical features into numerical features.
- Convert the column `date_first_hired` into numerical features (year, month, day etc.).
- Scale numerical features (we'll be using a linear model).
- Evaluate the performance of the model.
## Feature engineering
### Step 1: Convert date features to numerical {.smaller}
We extract numerical features from the `date_first_hired` column.
```{python}
# Create a copy to work with
X_processed = X.copy()
# Parse the date column
X_processed['date_first_hired'] = pd.to_datetime(X_processed['date_first_hired'])
# Extract numerical features from date
X_processed['hired_month'] = X_processed['date_first_hired'].dt.month
X_processed['hired_year'] = X_processed['date_first_hired'].dt.year
# Drop original date column
X_processed = X_processed.drop('date_first_hired', axis=1)
print("Features after date transformation:")
print("\nShape:", X_processed.shape)
```
### Step 2: Encode categorical features {.smaller}
We encode all categorical features using one-hot encoding.
```{python}
# Find the categorical columns
categorical_cols = X_processed.select_dtypes(include=['object']).columns.tolist()
print("Categorical columns to encode:", categorical_cols)
# Apply one-hot encoding only to categorical columns
X_encoded = pd.get_dummies(X_processed, columns=categorical_cols)
print("\nShape after encoding:", X_encoded.shape)
```
Notice how the number of features has increased massively due to the large number
of unique values in some of the columns.
The large number of features can cause various issues, ranging from **overfitting**,
to performance and memory issues. This is also known as the
[Curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).
Alternative strategies for encoding high-cardinality data (such as text) should
be employed.
### Step 3: Impute missing values {.smaller}
We impute the missing values in the `gender` column to fill any gaps.
In this case, this dataframe does not include many missing values, and
they're concentrated in a single categorical column, so we fill missing
values with the "most frequent" strategy.
```{python}
from sklearn.impute import SimpleImputer
# Impute missing values with most frequent value
imputer = SimpleImputer(strategy='most_frequent')
X_encoded_imputed = pd.DataFrame(
imputer.fit_transform(X_encoded),
columns=X_encoded.columns
)
```
Some models -- like the `Ridge` model we will be using -- cannot handle missing
values, and require imputation in order to work. Other models, such as the
`HistGradientBoostingClassifier`, do not suffer from this limitation.
::: {.callout-important}
Null values are important! The fact that a value is missing is by itself useful
information, and the pattern of missingness is also important.
For this reason, skrub transformers keep missing values as they are to retain the
information content.
:::
Example:
Consider a dataset about **employment**, which among others includes the column
**current position**.
A missing value in this column likely indicates **unemployment**, and imputing it
with a strategy like "most frequent" would introduce wrong information about the
sample and lead the model to wrong conclusions.
### Step 4: Scale numerical features {.smaller}
Linear models are strongly affected by numeric features that haven't been scaled,
so this step should always be done. The `StandardScaler` scales values by removing
the mean and scaling to unit variance.
So, here we scale numerical features for the `RidgeCV` regression model:
```{python}
from sklearn.preprocessing import StandardScaler
# Initialize the scaler
scaler = StandardScaler()
# Fit and transform the data
X_scaled = scaler.fit_transform(X_encoded_imputed)
X_scaled = pd.DataFrame(X_scaled, columns=X_encoded_imputed.columns)
```
::: {.callout-warning}
Outliers (values very far from the mean) make scaling hard to do. **Robust scaling**
is needed.
:::
### Step 5: Train Ridge model with cross-validation {.smaller}
For this example we want to train a `RidgeCV` regression model and evaluate the
model's performance with cross-validation.
```{python}
#| warning: false
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import cross_val_score, cross_validate
import numpy as np
# Initialize Ridge model
ridge = RidgeCV()
# Perform cross-validation (5-fold)
cv_results = cross_validate(ridge, X_scaled, y, cv=5, scoring=["r2", "neg_mean_squared_error"])
# Convert MSE to RMSE
test_rmse = np.sqrt(-cv_results["test_neg_mean_squared_error"])
# Display results
print("Cross-Validation Results:")
print(
f"Mean test R²: {cv_results['test_r2'].mean():.4f} (+/- {cv_results['test_r2'].std():.4f})"
)
print(f"Mean test RMSE: {test_rmse.mean():.4f} (+/- {test_rmse.std():.4f})")
```
## Looking back
### "Just ask an agent to write the code" {.smaller}
That's what I did: I asked an AI agent to turn the "strategizing" section into
code, and the result was not good.
In fact, the strategy I suggested is not laid out in the correct order: the agent
did not spot the problem, followed each item step by step (in the wrong order) and
as a result:
- Imputation was performed before converting categorical features to numeric values,
so it did not impute properly.
- Datetimes were not parsed as datetimes before applying `OneHotEncoder`, so
they were treated as categorical and encoded as such, rather than being converted
to numeric features.
- Finally, `pd.get_dummies` was executed on the full dataframe, rather than looking
for categories only on the training split, and then using those categories on
the test split: this causes **data leakage** by creating an encoding for
categorical features that would appear only in the test set.
The order of our operations is just as important as the operations themselves,
and relying blindly on agents can have bad consequences.
**Disclaimer**: advanced models probably would not fall in the same pitfalls, or
would be able to self-correct to an extent. The main point still stands.
## Small intermission: data leakage and fit/transform
For simplicity, let's try to impute values in a series as our example.
One-hot leakage works in similar ways.
This example dataframe is split into training and test split with a 50/50 split.
```{python}
from sklearn.impute import SimpleImputer
import pandas as pd
df = pd.DataFrame({
"value": [1, 3, None, 5, 10, None, 1000, 5]
})
df_train = df[:5]
df_test = df[5:]
```
A common strategy is to impute numeric missing values by finding the mean of the
features that are still there. We can do this with `stragegy="mean"`.
What happens if you try to find the mean over all data?
```{python}
imputer = SimpleImputer(strategy="mean").set_output(transform="pandas")
df_imputed = imputer.fit_transform(df)
```
The result is that the missing values have been filled with the mean over both
the train and test set. The test set happened to contain an outlier, which skewed
the mean in the training.
To impute properly and avoid leakage, we want to **fit** the imputer on the
training data, and then **transform** the test split using the mean that was learned
in while fitting.
```{python}
print(imputer.fit(df_train).transform(df_test))
```
::: {.callout-important}
This fit-transform paradigm is what scikit-learn (and skrub) uses to avoid data
leakage.
:::
## Waking up from a nightmare {.smaller}
Thankfully, we can `import skrub`:
```{python}
#| warning: false
from skrub import tabular_pipeline
# Perform cross-validation (5-fold)
cv_results = cross_validate(tabular_pipeline("regression"), X, y, cv=5,
scoring=['r2', 'neg_mean_squared_error'],
return_train_score=True)
# Convert MSE to RMSE
train_rmse = np.sqrt(-cv_results['train_neg_mean_squared_error'])
test_rmse = np.sqrt(-cv_results['test_neg_mean_squared_error'])
# Display results
print("Cross-Validation Results:")
print(f"Mean test R²: {cv_results['test_r2'].mean():.4f} (+/- {cv_results['test_r2'].std():.4f})")
print(f"Mean test RMSE: {test_rmse.mean():.4f} (+/- {test_rmse.std():.4f})")
```
### What is happening?
The `tabular_pipeline` is implementing all the steps we have just covered, except
in a more robust and battle-tested fashion. Before encoding, the input table is
sanitized so that features have the proper dtype, null values are marked as such,
and then the appropriate encoder is applied to each column.
{fig-align="center"}
The functioning of the `tabular_pipeline` is explored in more detail in a later
chapter.
## What we saw in this chapter
- We built a predictive pipeline using traditional tools
- We saw some possible shortcomings
- We tested skrub's `tabular_pipeline`
## Roadmap for the course {.smaller}
1. Data exploration with skrub's `TableReport`
2. Data cleaning and sanitization with the `Cleaner`
3. Columnwise operations with `ApplyToCols`
4. Advanced column selectors and how to use them
5. Automatic feature engineering with the `TableVectorizer`
6. A robust baseline for machine learning tasks with the `tabular_pipeline`
7. Feature engineering with the skrub encoders
8. Building dynamic pipelines with the skrub Data Ops (short intro)
