Skrub

Less wrangling, more machine learning

Machine learning and tabular data

ML expects numeric arrays
Real data is more complex:
- Multiple tables
- Dates, categories, text, locations, …

Skrub: bridge the gap between dataframes and scikit-learn.

Skrub helps at several stages of a tabular learning project

What’s in the data? (EDA)
Can we learn anything? (baselines)
How do I represent the data? (feature extraction)
How do I bring it all together? (building a pipeline)

1. What’s in the data?

import skrub
import pandas as pd

employee_salaries = pd.read_csv("https://figshare.com/ndownloader/files/51755822")
employees = employee_salaries.drop("current_annual_salary", axis=1)
salaries = employee_salaries["current_annual_salary"]
employees.iloc[0]

gender                                                                     F
department                                                               POL
department_name                                         Department of Police
division                   MSB Information Mgmt and Tech Division Records...
assignment_category                                         Fulltime-Regular
employee_position_title                          Office Services Coordinator
date_first_hired                                                  09/22/1986
year_first_hired                                                        1986
Name: 0, dtype: object

`TableReport`: interactive display of a dataframe

skrub.TableReport(employees, verbose=0)

	gender	department	department_name	division	assignment_category	employee_position_title	date_first_hired	year_first_hired
0	F	POL	Department of Police	MSB Information Mgmt and Tech Division Records Management Section	Fulltime-Regular	Office Services Coordinator	09/22/1986	1986
1	M	POL	Department of Police	ISB Major Crimes Division Fugitive Section	Fulltime-Regular	Master Police Officer	09/12/1988	1988
2	F	HHS	Department of Health and Human Services	Adult Protective and Case Management Services	Fulltime-Regular	Social Worker IV	11/19/1989	1989
3	M	COR	Correction and Rehabilitation	PRRS Facility and Security	Fulltime-Regular	Resident Supervisor II	05/05/2014	2014
4	M	HCA	Department of Housing and Community Affairs	Affordable Housing Programs	Fulltime-Regular	Planning Specialist III	03/05/2007	2007

9223	F	HHS	Department of Health and Human Services	School Based Health Centers	Fulltime-Regular	Community Health Nurse II	11/03/2015	2015
9224	F	FRS	Fire and Rescue Services	Human Resources Division	Fulltime-Regular	Fire/Rescue Division Chief	11/28/1988	1988
9225	M	HHS	Department of Health and Human Services	Child and Adolescent Mental Health Clinic Services	Parttime-Regular	Medical Doctor IV - Psychiatrist	04/30/2001	2001
9226	M	CCL	County Council	Council Central Staff	Fulltime-Regular	Manager II	09/05/2006	2006
9227	M	DLC	Department of Liquor Control	Licensure, Regulation and Education	Fulltime-Regular	Alcohol/Tobacco Enforcement Specialist II	01/30/2012	2012

Column	Column name	dtype	Null values	Unique values	Mean	Std	Min	Median	Max
0	gender	ObjectDType	17 (0.2%)	2 (< 0.1%)
1	department	ObjectDType	0 (0.0%)	37 (0.4%)
2	department_name	ObjectDType	0 (0.0%)	37 (0.4%)
3	division	ObjectDType	0 (0.0%)	694 (7.5%)
4	assignment_category	ObjectDType	0 (0.0%)	2 (< 0.1%)
5	employee_position_title	ObjectDType	0 (0.0%)	443 (4.8%)
6	date_first_hired	ObjectDType	0 (0.0%)	2264 (24.5%)
7	year_first_hired	Int64DType	0 (0.0%)	51 (0.6%)	2.00e+03	9.33	1,965	2,005	2,016

Column 1	Column 2	Cramér's V
department	department_name	1.00
assignment_category	employee_position_title	0.655
division	assignment_category	0.609
division	employee_position_title	0.512
department_name	employee_position_title	0.403
department	employee_position_title	0.403
department	assignment_category	0.397
department_name	assignment_category	0.397
gender	department	0.363
gender	department_name	0.363
department	division	0.362
department_name	division	0.362
gender	employee_position_title	0.263
gender	division	0.255
gender	assignment_category	0.246
employee_position_title	date_first_hired	0.239
date_first_hired	year_first_hired	0.172
department	date_first_hired	0.146
department_name	date_first_hired	0.146
employee_position_title	year_first_hired	0.122

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We can tell skrub to patch the default display of polars and pandas dataframes.

skrub.patch_display(verbose=0)

2. Can we learn anything?

X, y = employees, salaries

`tabular_learner`: a pre-made robust baseline

learner = skrub.tabular_learner("regressor")
learner

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(high_cardinality=MinHashEncoder(),
                                 low_cardinality=ToCategorical())),
                ('histgradientboostingregressor',
                 HistGradientBoostingRegressor(categorical_features='from_dtype'))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

from sklearn.model_selection import cross_val_score

cross_val_score(learner, X, y, scoring="r2")

array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666])

The tabular_learner adapts to the supervised estimator we choose

from sklearn.linear_model import Ridge

learner = skrub.tabular_learner(Ridge())
learner

Pipeline(steps=[('tablevectorizer', TableVectorizer()),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('standardscaler', StandardScaler()), ('ridge', Ridge())])

cross_val_score(learner, X, y, scoring="r2")

array([0.7332394 , 0.71975525, 0.75090551, 0.74173251, 0.73988883])

3. How do I represent the data?

Skrub helps extract informative features from tabular data.

`TableVectorizer`: apply an appropriate transformer to each column

vectorizer = skrub.TableVectorizer()
transformed = vectorizer.fit_transform(X)

The TableVectorizer identifies several kinds of columns:

categorical, low cardinality
categorical, high cardinality
datetime
numeric
… we may add more

from pprint import pprint

pprint(vectorizer.column_to_kind_)

{'assignment_category': 'low_cardinality',
 'date_first_hired': 'datetime',
 'department': 'low_cardinality',
 'department_name': 'low_cardinality',
 'division': 'high_cardinality',
 'employee_position_title': 'high_cardinality',
 'gender': 'low_cardinality',
 'year_first_hired': 'numeric'}

For each kind, it applies an appropriate transformer

vectorizer.transformers_["department"]  # low-cardinality categorical

OneHotEncoder(drop='if_binary', dtype='float32', handle_unknown='ignore',
              sparse_output=False)

vectorizer.transformers_["employee_position_title"]  # high-cardinality categorical

GapEncoder(n_components=30)

vectorizer.transformers_["date_first_hired"]  # datetime

DatetimeEncoder()

… and those transformers turn the input into numeric features that can be used for ML

transformed[vectorizer.input_to_outputs_["date_first_hired"]]

	date_first_hired_year	date_first_hired_month	date_first_hired_day	date_first_hired_total_seconds
0	1986.0	9.0	22.0	527731200.0
1	1988.0	9.0	12.0	590025600.0
2	1989.0	11.0	19.0	627436800.0
3	2014.0	5.0	5.0	1399248000.0
4	2007.0	3.0	5.0	1173052800.0

9223	2015.0	11.0	3.0	1446508800.0
9224	1988.0	11.0	28.0	596678400.0
9225	2001.0	4.0	30.0	988588800.0
9226	2006.0	9.0	5.0	1157414400.0
9227	2012.0	1.0	30.0	1327881600.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	date_first_hired_year	Float32DType	51 (0.6%)	2.00e+03	9.33	1.96e+03	2.00e+03	2.02e+03
1	date_first_hired_month	Float32DType	12 (0.1%)	6.35	3.48	1.00	7.00	12.0
2	date_first_hired_day	Float32DType	31 (0.3%)	15.3	8.63	1.00	16.0	31.0
3	date_first_hired_total_seconds	Float32DType	2264 (24.5%)	1.08e+09	2.94e+08	-1.34e+08	1.12e+09	1.48e+09

Column 1	Column 2	Cramér's V
date_first_hired_year	date_first_hired_total_seconds	0.930
date_first_hired_month	date_first_hired_day	0.134
date_first_hired_year	date_first_hired_month	0.0849
date_first_hired_year	date_first_hired_day	0.0833
date_first_hired_day	date_first_hired_total_seconds	0.0796
date_first_hired_month	date_first_hired_total_seconds	0.0792

Please enable javascript

For high-cardinality categorical columns the default GapEncoder identifies sparse topics (more later).

transformed[vectorizer.input_to_outputs_["employee_position_title"]]

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	employee_position_title: enforcement, inspector, permitting	Float32DType	443 (4.8%)	1.07	6.91	0.0504	0.0558	64.2
1	employee_position_title: librarian, candidate, telephone	Float32DType	443 (4.8%)	0.703	3.09	0.0507	0.0533	48.9
2	employee_position_title: income, assistance, client	Float32DType	443 (4.8%)	1.19	6.89	0.0500	0.0544	46.9
3	employee_position_title: officer, office, police	Float32DType	443 (4.8%)	2.70	7.19	0.0500	0.0617	23.8
4	employee_position_title: firefighter, rescuer, fire	Float32DType	443 (4.8%)	2.14	7.59	0.0500	0.0600	31.2
5	employee_position_title: manager, engineer, iii	Float32DType	443 (4.8%)	0.847	2.69	0.0500	0.0682	13.8
6	employee_position_title: operator, bus, operations	Float32DType	442 (4.8%)	1.14	3.80	0.0500	0.0540	15.0
7	employee_position_title: administrative, administration, administrator	Float32DType	443 (4.8%)	0.770	4.99	0.0500	0.0550	40.0
8	employee_position_title: coordinator, services, service	Float32DType	443 (4.8%)	1.40	6.09	0.0501	0.0554	37.3
9	employee_position_title: specialist, special, quality	Float32DType	443 (4.8%)	1.42	4.73	0.0500	0.0556	28.2
10	employee_position_title: safety, public, urban	Float32DType	443 (4.8%)	1.09	5.80	0.0500	0.0536	55.9
11	employee_position_title: community, nurse, unit	Float32DType	443 (4.8%)	1.12	5.20	0.0500	0.0537	34.4
12	employee_position_title: information, technology, renovation	Float32DType	443 (4.8%)	1.37	7.35	0.0500	0.0538	56.5
13	employee_position_title: environmental, analyst, enviromental	Float32DType	443 (4.8%)	0.708	4.75	0.0500	0.0556	49.4
14	employee_position_title: school, health, room	Float32DType	443 (4.8%)	1.16	6.61	0.0500	0.0525	43.4
15	employee_position_title: accountant, assistant, attorney	Float32DType	443 (4.8%)	1.01	4.69	0.0500	0.0529	38.5
16	employee_position_title: correctional, correction, corporal	Float32DType	443 (4.8%)	1.63	7.66	0.0500	0.0575	49.0
17	employee_position_title: lieutenant, shift, maintenance	Float32DType	443 (4.8%)	0.660	3.96	0.0500	0.0531	36.9
18	employee_position_title: liquor, clerk, store	Float32DType	443 (4.8%)	0.718	3.91	0.0500	0.0553	28.1
19	employee_position_title: program, programs, projects	Float32DType	443 (4.8%)	0.828	3.88	0.0500	0.0569	30.8
20	employee_position_title: sergeant, cadet, emergency	Float32DType	443 (4.8%)	0.617	3.22	0.0500	0.0550	28.8
21	employee_position_title: captain, chief, mcfrs	Float32DType	443 (4.8%)	0.712	3.91	0.0500	0.0522	35.7
22	employee_position_title: technician, mechanic, supply	Float32DType	443 (4.8%)	1.15	5.27	0.0500	0.0543	51.4
23	employee_position_title: supervisory, therapist, supervisor	Float32DType	443 (4.8%)	0.878	3.82	0.0500	0.0557	28.2
24	employee_position_title: equipment, investigator, investment	Float32DType	443 (4.8%)	0.737	4.01	0.0500	0.0547	39.1
25	employee_position_title: legislative, principal, executive	Float32DType	443 (4.8%)	1.18	6.12	0.0500	0.0524	47.9
26	employee_position_title: recruit, master, registered	Float32DType	443 (4.8%)	1.34	6.60	0.0500	0.0549	42.6
27	employee_position_title: crossing, purchasing, background	Float32DType	443 (4.8%)	0.564	3.13	0.0500	0.0504	36.8
28	employee_position_title: planning, senior, resources	Float32DType	443 (4.8%)	0.824	4.79	0.0500	0.0546	51.9
29	employee_position_title: warehouse, welfare, caseworker	Float32DType	443 (4.8%)	1.12	5.13	0.0500	0.0579	50.6

Column 1	Column 2	Cramér's V
employee_position_title: administrative, administration, administrator	employee_position_title: legislative, principal, executive	0.549
employee_position_title: manager, engineer, iii	employee_position_title: operator, bus, operations	0.453
employee_position_title: income, assistance, client	employee_position_title: specialist, special, quality	0.451
employee_position_title: income, assistance, client	employee_position_title: lieutenant, shift, maintenance	0.447
employee_position_title: environmental, analyst, enviromental	employee_position_title: school, health, room	0.424
employee_position_title: lieutenant, shift, maintenance	employee_position_title: captain, chief, mcfrs	0.406
employee_position_title: safety, public, urban	employee_position_title: sergeant, cadet, emergency	0.393
employee_position_title: lieutenant, shift, maintenance	employee_position_title: equipment, investigator, investment	0.390
employee_position_title: information, technology, renovation	employee_position_title: technician, mechanic, supply	0.383
employee_position_title: administrative, administration, administrator	employee_position_title: safety, public, urban	0.377
employee_position_title: safety, public, urban	employee_position_title: legislative, principal, executive	0.373
employee_position_title: specialist, special, quality	employee_position_title: information, technology, renovation	0.360
employee_position_title: income, assistance, client	employee_position_title: environmental, analyst, enviromental	0.344
employee_position_title: manager, engineer, iii	employee_position_title: program, programs, projects	0.338
employee_position_title: sergeant, cadet, emergency	employee_position_title: supervisory, therapist, supervisor	0.337
employee_position_title: supervisory, therapist, supervisor	employee_position_title: equipment, investigator, investment	0.332
employee_position_title: liquor, clerk, store	employee_position_title: program, programs, projects	0.331
employee_position_title: accountant, assistant, attorney	employee_position_title: sergeant, cadet, emergency	0.329
employee_position_title: safety, public, urban	employee_position_title: community, nurse, unit	0.329
employee_position_title: coordinator, services, service	employee_position_title: technician, mechanic, supply	0.312

Please enable javascript

The transformer used for each column kind can be easily configured.

Preprocessing in the `TableVectorizer`

The TableVectorizer actually performs a lot of preprocessing before applying the final transformers, such as:

ensuring consistent column names
detecting missing values such as "N/A"
dropping empty columns
handling pandas dtypes – float64, nan vs Float64, NA
parsing numbers
parsing dates, ensuring consistent dtype and timezone
converting numbers to float32 for faster computation & less memory downstream
…

pprint(vectorizer.all_processing_steps_["date_first_hired"])

[CleanNullStrings(),
 DropIfTooManyNulls(),
 ToDatetime(),
 DatetimeEncoder(),
 {'date_first_hired_day': ToFloat32(), 'date_first_hired_month': ToFloat32(), ...}]

Extracting good features

Skrub offers several encoders to extract features from different columns. In particular from categorical columns.

`GapEncoder`

Categories are somewhere between text and an enumeration… The GapEncoder is somewhere between a topic model and a one-hot encoder!

import seaborn as sns
from matplotlib import pyplot as plt

gap = skrub.GapEncoder()
pos_title = X["employee_position_title"]
loadings = gap.fit_transform(pos_title).set_index(pos_title.values).head()

loadings.columns = [c.split(": ")[1] for c in loadings.columns]
sns.heatmap(loadings)
_ = plt.setp(plt.gca().get_xticklabels(), rotation=45, ha="right")

`TextEncoder`

Extract embeddings from a text column using any model from the HuggingFace Hub.

import pandas as pd

X = pd.Series(["airport", "flight", "plane", "pineapple", "fruit"])
encoder = skrub.TextEncoder(model_name="all-MiniLM-L6-v2", n_components=None)
embeddings = encoder.fit_transform(X).set_index(X.values)

sns.heatmap(embeddings @ embeddings.T)

`MinHashEncoder`

A fast, stateless way of encoding strings that works especially well with models based on decision trees (gradient boosting, random forest).

4. How do I bring it all together?

Skrub has several transformers that allow peforming typical dataframe operations such as projections, joins and aggregations inside a scikit-learn pipeline.

Performing these operations in the machine-learning pipeline has several advantages:

Choices / hyperparameters can be optimized
Relevant state can be stored to ensure consistent transformations
All transformations are packaged together in an estimator

There are several transformers such as SelectCols, Joiner (fuzzy joining), InterpolationJoiner, AggJoiner, …

A toy example using the AggJoiner:

from skrub import AggJoiner

airports = pd.DataFrame(
    {
        "airport_id": [1, 2],
        "airport_name": ["Charles de Gaulle", "Aeroporto Leonardo da Vinci"],
        "city": ["Paris", "Roma"],
    }
)
airports

	airport_id	airport_name	city
0	1	Charles de Gaulle	Paris
1	2	Aeroporto Leonardo da Vinci	Roma

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)

Column 1	Column 2	Cramér's V
airport_name	city	1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

flights = pd.DataFrame(
    {
        "flight_id": range(1, 7),
        "from_airport": [1, 1, 1, 2, 2, 2],
        "total_passengers": [90, 120, 100, 70, 80, 90],
        "company": ["DL", "AF", "AF", "DL", "DL", "TR"],
    }
)
flights

	flight_id	from_airport	total_passengers	company
0	1	1	90	DL
1	2	1	120	AF
2	3	1	100	AF
3	4	2	70	DL
4	5	2	80	DL
5	6	2	90	TR

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	flight_id	Int64DType	6 (100.0%)	3.50	1.87	1	3	6
1	from_airport	Int64DType	2 (33.3%)	1.50	0.548	1	1	2
2	total_passengers	Int64DType	5 (83.3%)	91.7	17.2	70	90	120
3	company	ObjectDType	3 (50.0%)

Column 1	Column 2	Cramér's V
flight_id	total_passengers	1.00
flight_id	company	1.00
flight_id	from_airport	1.00
total_passengers	company	0.816
from_airport	total_passengers	0.816
from_airport	company	0.745

Please enable javascript

agg_joiner = AggJoiner(
    aux_table=flights,
    main_key="airport_id",
    aux_key="from_airport",
    cols=["total_passengers"],
    operations=["mean", "std"],
)
agg_joiner.fit_transform(airports)

	airport_id	airport_name	city	total_passengers_mean	total_passengers_std
0	1	Charles de Gaulle	Paris	103.33333333333333	15.275252316519467
1	2	Aeroporto Leonardo da Vinci	Roma	80.0	10.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)
3	total_passengers_mean	Float64DType	2 (100.0%)	91.7	16.5	80.0	80.0	103.
4	total_passengers_std	Float64DType	2 (100.0%)	12.6	3.73	10.0	10.0	15.3

Column 1	Column 2	Cramér's V
total_passengers_mean	total_passengers_std	1.00
city	total_passengers_std	1.00
city	total_passengers_mean	1.00
airport_name	total_passengers_std	1.00
airport_name	total_passengers_mean	1.00
airport_name	city	1.00
airport_id	total_passengers_std	1.00
airport_id	total_passengers_mean	1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

More interactive and expressive pipelines

To go further than what can be done with scikit-learn Pipelines and the skrub transformers shown above, we are developing new utilities to easily define and inspect flexible pipelines that can process several dataframes.

A prototype will be shown in a separate notebook.

	employee_position_title: enforcement, inspector, permitting	employee_position_title: librarian, candidate, telephone	employee_position_title: income, assistance, client	employee_position_title: officer, office, police	employee_position_title: firefighter, rescuer, fire	employee_position_title: manager, engineer, iii	employee_position_title: operator, bus, operations	employee_position_title: administrative, administration, administrator	employee_position_title: coordinator, services, service	employee_position_title: specialist, special, quality	employee_position_title: safety, public, urban	employee_position_title: community, nurse, unit	employee_position_title: information, technology, renovation	employee_position_title: environmental, analyst, enviromental	employee_position_title: school, health, room	employee_position_title: accountant, assistant, attorney	employee_position_title: correctional, correction, corporal	employee_position_title: lieutenant, shift, maintenance	employee_position_title: liquor, clerk, store	employee_position_title: program, programs, projects	employee_position_title: sergeant, cadet, emergency	employee_position_title: captain, chief, mcfrs	employee_position_title: technician, mechanic, supply	employee_position_title: supervisory, therapist, supervisor	employee_position_title: equipment, investigator, investment	employee_position_title: legislative, principal, executive	employee_position_title: recruit, master, registered	employee_position_title: crossing, purchasing, background	employee_position_title: planning, senior, resources	employee_position_title: warehouse, welfare, caseworker
	employee_position_title: enforcement, inspector, permitting	employee_position_title: librarian, candidate, telephone	employee_position_title: income, assistance, client	employee_position_title: officer, office, police	employee_position_title: firefighter, rescuer, fire	employee_position_title: manager, engineer, iii	employee_position_title: operator, bus, operations		employee_position_title: coordinator, services, service	employee_position_title: specialist, special, quality	employee_position_title: safety, public, urban	employee_position_title: community, nurse, unit			employee_position_title: school, health, room	employee_position_title: accountant, assistant, attorney	employee_position_title: correctional, correction, corporal	employee_position_title: lieutenant, shift, maintenance	employee_position_title: liquor, clerk, store	employee_position_title: program, programs, projects	employee_position_title: sergeant, cadet, emergency	employee_position_title: captain, chief, mcfrs	employee_position_title: technician, mechanic, supply	employee_position_title: supervisory, therapist, supervisor		employee_position_title: legislative, principal, executive	employee_position_title: recruit, master, registered	employee_position_title: crossing, purchasing, background	employee_position_title: planning, senior, resources	employee_position_title: warehouse, welfare, caseworker
0	0.05583696812391281	0.05122457444667816	0.05442347377538681	0.10461704432964325	0.0547703318297863	0.053633179515600204	0.0702529177069664	0.05421031266450882	37.3344841003418	0.05086061730980873	0.0596134178340435	0.05261371284723282	0.052691005170345306	0.05095909163355827	0.05191970616579056	0.051769595593214035	0.063545361161232	0.05394167825579643	0.05951649323105812	0.05238822475075722	0.07346681505441666	0.056256167590618134	0.05137219280004501	0.0544905960559845	0.0572633221745491	0.05298957973718643	0.05420336127281189	0.05326978489756584	0.06035270541906357	0.05306529626250267
1	0.05176004394888878	0.05721915513277054	0.0809163898229599	21.93833351135254	0.06262846291065216	0.07367730140686035	0.05450122058391571	0.061989977955818176	0.06414863467216492	0.0589413121342659	0.05648364871740341	0.050720177590847015	0.0584520660340786	0.056611575186252594	0.052967995405197144	0.06630362570285797	0.06124555692076683	0.057077161967754364	0.06206129863858223	0.06490938365459442	0.07532013207674026	0.051854681223630905	0.05493974685668945	0.05537518858909607	0.052625011652708054	0.05504601076245308	6.38975715637207	0.05165288597345352	0.06858143955469131	0.05389891564846039
2	0.0545603446662426	0.05353299155831337	0.05502674728631973	0.060591038316488266	0.059118613600730896	0.0751241073012352	0.05370652675628662	0.06461416184902191	0.05217093601822853	0.07353440672159195	0.0515953004360199	0.052403826266527176	0.05793610215187073	0.05863690748810768	0.057864900678396225	0.0516052283346653	0.06391299515962601	0.05008332058787346	0.05776653066277504	0.0586300753057003	0.05177239701151848	0.050209492444992065	0.056745365262031555	0.05361904948949814	0.05557338893413544	0.05646607279777527	0.054961759597063065	0.05019618198275566	0.0595371350646019	20.8485050201416
3	0.12260108441114426	3.6131536960601807	0.05978325754404068	0.08195025473833084	1.6136398315429688	0.5696560144424438	0.07311622053384781	0.06043166667222977	0.1439264863729477	0.06574537605047226	0.06337752938270569	0.05356214940547943	0.06432104110717773	0.12195218354463577	0.050742775201797485	0.07973258942365646	0.0758921205997467	0.1218586415052414	0.08998340368270874	0.06619563698768616	0.12055357545614243	0.15520831942558289	0.05534113943576813	18.319543838500977	5.224661350250244	0.06299922615289688	0.11892539262771606	0.05764332413673401	0.11043545603752136	0.08306732773780823
4	0.07849550992250443	0.05508404225111008	0.08173716068267822	0.07213564217090607	0.06507740169763565	1.7179982662200928	0.05172279477119446	0.1703687608242035	0.05243702977895737	13.797388076782227	0.05287671089172363	0.06050468236207962	0.07158928364515305	0.10916590690612793	0.05766227841377258	0.06153770163655281	0.060601137578487396	0.05558042973279953	0.05503547564148903	0.05713812634348869	0.0573420487344265	0.052376244217157364	0.07469528913497925	0.06678193062543869	0.053371965885162354	0.05736757814884186	0.051163915544748306	0.11711566150188446	15.624563217163086	0.06108585000038147

9223	0.05311708524823189	0.05175912752747536	0.0537041537463665	0.05437226593494415	0.05268342047929764	0.059571556746959686	0.05000019446015358	0.053928442299366	0.051954299211502075	0.054271455854177475	0.051767315715551376	34.4409294128418	0.05184970051050186	0.052476346492767334	0.08262228965759277	0.0519259050488472	0.0539465956389904	0.050835877656936646	0.05216951668262482	0.05147586390376091	0.054165516048669815	0.05120695382356644	0.05323423072695732	0.05566713586449623	0.050660766661167145	0.05225023627281189	0.05034351721405983	0.05023452639579773	0.0526202954351902	0.05425596609711647
9224	0.05065978318452835	0.06747494637966156	0.1045418530702591	0.05184266343712807	0.08036161959171295	0.05001096799969673	0.05000261962413788	0.29809248447418213	0.4271133244037628	0.06373494863510132	0.1418973207473755	0.05056469887495041	0.9262706637382507	0.06233452260494232	0.05883578956127167	0.40221986174583435	0.08885844796895981	0.08327829837799072	0.05219920352101326	0.05000361427664757	0.08156458288431168	32.4133415222168	0.06281514465808868	0.9411808252334595	0.05029449984431267	0.39133164286613464	0.0704251304268837	0.158475860953331	0.08665968477725983	0.08361198008060455
9225	2.9829509258270264	33.53965377807617	0.11803698539733887	0.16333098709583282	0.05985897034406662	0.08463223278522491	0.11329977214336395	2.064655303955078	0.11809763312339783	0.14998266100883484	0.15054181218147278	0.05479539558291435	0.1094961017370224	0.09035434573888779	0.1586700975894928	0.11413414031267166	0.17703355848789215	0.05601043254137039	0.15419574081897736	0.0615767240524292	0.06753755360841751	0.1036735251545906	0.7460574507713318	0.07154794037342072	2.2507364749908447	0.16052520275115967	0.05921342596411705	0.05247887596487999	0.10387630015611649	2.363046407699585
9226	0.05374247580766678	0.0512378066778183	0.051389697939157486	0.06067110598087311	0.05972776934504509	11.890955924987793	0.051837604492902756	0.051339585334062576	0.05167651176452637	0.05189792811870575	0.05062304437160492	0.05178365483880043	0.0513027086853981	0.05740608274936676	0.05122877284884453	0.05179090052843094	0.05559111759066582	0.05338591709733009	0.05302312970161438	0.11964111030101776	0.055003248155117035	0.050003115087747574	0.05355608090758324	0.053275227546691895	0.053124889731407166	0.05000818520784378	0.05520337074995041	0.05005694553256035	0.051653508096933365	0.05786259472370148
9227	14.32251262664795	0.05494850501418114	0.0884532555937767	0.2540455460548401	0.05281653627753258	0.06022859737277031	0.06269559264183044	0.11841216683387756	0.0711812749505043	2.074544668197632	0.056581515818834305	0.06445585191249847	0.1564246565103531	38.363525390625	0.1316186785697937	0.0918332114815712	0.07227492332458496	0.05310409888625145	0.06355835497379303	0.05146981403231621	0.1442510485649109	0.052378080785274506	0.05884139984846115	0.06001986935734749	0.07008115947246552	0.053466178476810455	0.0509156659245491	3.1012489795684814	0.07633450627326965	0.06777606904506683

Machine learning and tabular data

Skrub helps at several stages of a tabular learning project

1. What’s in the data?

TableReport: interactive display of a dataframe

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

Please enable javascript

2. Can we learn anything?

tabular_learner: a pre-made robust baseline

3. How do I represent the data?

TableVectorizer: apply an appropriate transformer to each column

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

Please enable javascript

employee_position_title: enforcement, inspector, permitting

employee_position_title: librarian, candidate, telephone

employee_position_title: income, assistance, client

employee_position_title: officer, office, police

employee_position_title: firefighter, rescuer, fire

employee_position_title: manager, engineer, iii

employee_position_title: operator, bus, operations

employee_position_title: administrative, administration, administrator

employee_position_title: coordinator, services, service

employee_position_title: specialist, special, quality

employee_position_title: safety, public, urban

employee_position_title: community, nurse, unit

employee_position_title: information, technology, renovation

employee_position_title: environmental, analyst, enviromental

employee_position_title: school, health, room

employee_position_title: accountant, assistant, attorney

employee_position_title: correctional, correction, corporal

employee_position_title: lieutenant, shift, maintenance

employee_position_title: liquor, clerk, store

employee_position_title: program, programs, projects

employee_position_title: sergeant, cadet, emergency

employee_position_title: captain, chief, mcfrs

employee_position_title: technician, mechanic, supply

employee_position_title: supervisory, therapist, supervisor

employee_position_title: equipment, investigator, investment

employee_position_title: legislative, principal, executive

employee_position_title: recruit, master, registered

employee_position_title: crossing, purchasing, background

employee_position_title: planning, senior, resources

employee_position_title: warehouse, welfare, caseworker

employee_position_title: enforcement, inspector, permitting

employee_position_title: librarian, candidate, telephone

employee_position_title: income, assistance, client

employee_position_title: officer, office, police

employee_position_title: firefighter, rescuer, fire

employee_position_title: manager, engineer, iii

employee_position_title: operator, bus, operations

employee_position_title: administrative, administration, administrator

employee_position_title: coordinator, services, service

employee_position_title: specialist, special, quality

employee_position_title: safety, public, urban

employee_position_title: community, nurse, unit

employee_position_title: information, technology, renovation

employee_position_title: environmental, analyst, enviromental

employee_position_title: school, health, room

employee_position_title: accountant, assistant, attorney

`TableReport`: interactive display of a dataframe

`tabular_learner`: a pre-made robust baseline

`TableVectorizer`: apply an appropriate transformer to each column

Preprocessing in the `TableVectorizer`

`GapEncoder`

`TextEncoder`

`MinHashEncoder`