Skrub

Less wrangling, more machine learning

Machine learning and tabular data

ML expects numeric arrays
Real data is more complex:
- Multiple tables
- Dates, categories, text, locations, …

Skrub: bridge the gap between dataframes and scikit-learn.

Skrub helps at several stages of a tabular learning project

What’s in the data? (EDA)
Can we learn anything? (baselines)
How do I represent the data? (feature extraction)
How do I bring it all together? (building a pipeline)

1. What’s in the data?

import skrub
import pandas as pd

employee_salaries = pd.read_csv("https://figshare.com/ndownloader/files/51755822")
employees = employee_salaries.drop("current_annual_salary", axis=1)
salaries = employee_salaries["current_annual_salary"]
employees.iloc[0]

gender                                                                     F
department                                                               POL
department_name                                         Department of Police
division                   MSB Information Mgmt and Tech Division Records...
assignment_category                                         Fulltime-Regular
employee_position_title                          Office Services Coordinator
date_first_hired                                                  09/22/1986
year_first_hired                                                        1986
Name: 0, dtype: object

`TableReport`: interactive display of a dataframe

skrub.TableReport(employees, verbose=0)

	gender	department	department_name	division	assignment_category	employee_position_title	date_first_hired	year_first_hired
0	F	POL	Department of Police	MSB Information Mgmt and Tech Division Records Management Section	Fulltime-Regular	Office Services Coordinator	09/22/1986	1986
1	M	POL	Department of Police	ISB Major Crimes Division Fugitive Section	Fulltime-Regular	Master Police Officer	09/12/1988	1988
2	F	HHS	Department of Health and Human Services	Adult Protective and Case Management Services	Fulltime-Regular	Social Worker IV	11/19/1989	1989
3	M	COR	Correction and Rehabilitation	PRRS Facility and Security	Fulltime-Regular	Resident Supervisor II	05/05/2014	2014
4	M	HCA	Department of Housing and Community Affairs	Affordable Housing Programs	Fulltime-Regular	Planning Specialist III	03/05/2007	2007

9223	F	HHS	Department of Health and Human Services	School Based Health Centers	Fulltime-Regular	Community Health Nurse II	11/03/2015	2015
9224	F	FRS	Fire and Rescue Services	Human Resources Division	Fulltime-Regular	Fire/Rescue Division Chief	11/28/1988	1988
9225	M	HHS	Department of Health and Human Services	Child and Adolescent Mental Health Clinic Services	Parttime-Regular	Medical Doctor IV - Psychiatrist	04/30/2001	2001
9226	M	CCL	County Council	Council Central Staff	Fulltime-Regular	Manager II	09/05/2006	2006
9227	M	DLC	Department of Liquor Control	Licensure, Regulation and Education	Fulltime-Regular	Alcohol/Tobacco Enforcement Specialist II	01/30/2012	2012

Column	Column name	dtype	Null values	Unique values	Mean	Std	Min	Median	Max
0	gender	ObjectDType	17 (0.2%)	2 (< 0.1%)
1	department	ObjectDType	0 (0.0%)	37 (0.4%)
2	department_name	ObjectDType	0 (0.0%)	37 (0.4%)
3	division	ObjectDType	0 (0.0%)	694 (7.5%)
4	assignment_category	ObjectDType	0 (0.0%)	2 (< 0.1%)
5	employee_position_title	ObjectDType	0 (0.0%)	443 (4.8%)
6	date_first_hired	ObjectDType	0 (0.0%)	2264 (24.5%)
7	year_first_hired	Int64DType	0 (0.0%)	51 (0.6%)	2.00e+03	9.33	1,965	2,005	2,016

Column 1	Column 2	Cramér's V
department	department_name	1.00
division	assignment_category	0.577
department_name	employee_position_title	0.415
department	employee_position_title	0.415
department	assignment_category	0.398
department_name	assignment_category	0.398
department	division	0.360
department_name	division	0.360
gender	department	0.356
gender	department_name	0.356
division	employee_position_title	0.318
employee_position_title	date_first_hired	0.308
gender	division	0.246
gender	employee_position_title	0.244
gender	assignment_category	0.231
assignment_category	employee_position_title	0.196
employee_position_title	year_first_hired	0.162
department	date_first_hired	0.152
department_name	date_first_hired	0.152
date_first_hired	year_first_hired	0.149

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We can tell skrub to patch the default display of polars and pandas dataframes.

skrub.patch_display(verbose=0)

2. Can we learn anything?

X, y = employees, salaries

`tabular_learner`: a pre-made robust baseline

learner = skrub.tabular_learner("regressor")
learner

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(high_cardinality=MinHashEncoder(),
                                 low_cardinality=ToCategorical())),
                ('histgradientboostingregressor',
                 HistGradientBoostingRegressor())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

from sklearn.model_selection import cross_val_score

cross_val_score(learner, X, y, scoring="r2")

array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666])

The tabular_learner adapts to the supervised estimator we choose

from sklearn.linear_model import Ridge

learner = skrub.tabular_learner(Ridge())
learner

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(datetime=DatetimeEncoder(periodic_encoding='spline'))),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('standardscaler', StandardScaler()), ('ridge', Ridge())])

cross_val_score(learner, X, y, scoring="r2")

array([0.74624262, 0.70865837, 0.7364406 , 0.72896092, 0.75461044])

3. How do I represent the data?

Skrub helps extract informative features from tabular data.

`TableVectorizer`: apply an appropriate transformer to each column

vectorizer = skrub.TableVectorizer()
transformed = vectorizer.fit_transform(X)

The TableVectorizer identifies several kinds of columns:

categorical, low cardinality
categorical, high cardinality
datetime
numeric
… we may add more

from pprint import pprint

pprint(vectorizer.column_to_kind_)

{'assignment_category': 'low_cardinality',
 'date_first_hired': 'datetime',
 'department': 'low_cardinality',
 'department_name': 'low_cardinality',
 'division': 'high_cardinality',
 'employee_position_title': 'high_cardinality',
 'gender': 'low_cardinality',
 'year_first_hired': 'numeric'}

For each kind, it applies an appropriate transformer

vectorizer.transformers_["department"]  # low-cardinality categorical

OneHotEncoder(drop='if_binary', dtype='float32', handle_unknown='ignore',
              sparse_output=False)

vectorizer.transformers_["employee_position_title"]  # high-cardinality categorical

GapEncoder(n_components=30)

vectorizer.transformers_["date_first_hired"]  # datetime

DatetimeEncoder()

… and those transformers turn the input into numeric features that can be used for ML

transformed[vectorizer.input_to_outputs_["date_first_hired"]]

	date_first_hired_year	date_first_hired_month	date_first_hired_day	date_first_hired_total_seconds
0	1986.0	9.0	22.0	527731200.0
1	1988.0	9.0	12.0	590025600.0
2	1989.0	11.0	19.0	627436800.0
3	2014.0	5.0	5.0	1399248000.0
4	2007.0	3.0	5.0	1173052800.0

9223	2015.0	11.0	3.0	1446508800.0
9224	1988.0	11.0	28.0	596678400.0
9225	2001.0	4.0	30.0	988588800.0
9226	2006.0	9.0	5.0	1157414400.0
9227	2012.0	1.0	30.0	1327881600.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	date_first_hired_year	Float32DType	51 (0.6%)	2.00e+03	9.33	1.96e+03	2.00e+03	2.02e+03
1	date_first_hired_month	Float32DType	12 (0.1%)	6.35	3.48	1.00	7.00	12.0
2	date_first_hired_day	Float32DType	31 (0.3%)	15.3	8.63	1.00	16.0	31.0
3	date_first_hired_total_seconds	Float32DType	2264 (24.5%)	1.08e+09	2.94e+08	-1.34e+08	1.12e+09	1.48e+09

Column 1	Column 2	Cramér's V	Pearson's Correlation
date_first_hired_year	date_first_hired_total_seconds	0.909	1.00
date_first_hired_month	date_first_hired_day	0.116	-0.0538
date_first_hired_month	date_first_hired_total_seconds	0.0866	0.0199
date_first_hired_year	date_first_hired_day	0.0854	-0.0432
date_first_hired_day	date_first_hired_total_seconds	0.0818	-0.0424
date_first_hired_year	date_first_hired_month	0.0787	-0.0112

Please enable javascript

For high-cardinality categorical columns the default GapEncoder identifies sparse topics (more later).

transformed[vectorizer.input_to_outputs_["employee_position_title"]]

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	employee_position_title: recreation, correctional, correction	Float32DType	443 (4.8%)	0.869	4.63	0.0500	0.0536	47.6
1	employee_position_title: crossing, engineer, parking	Float32DType	443 (4.8%)	0.474	2.53	0.0500	0.0500	21.8
2	employee_position_title: officer, police, office	Float32DType	443 (4.8%)	2.81	7.21	0.0500	0.0627	28.8
3	employee_position_title: manager, projects, project	Float32DType	443 (4.8%)	0.867	2.98	0.0500	0.0678	27.7
4	employee_position_title: corporal, private, pfc	Float32DType	443 (4.8%)	1.31	7.40	0.0500	0.0587	49.2
5	employee_position_title: operator, bus, operations	Float32DType	440 (4.8%)	1.14	3.80	0.0500	0.0538	15.0
6	employee_position_title: firefighter, rescuer, recruit	Float32DType	439 (4.8%)	2.50	8.55	0.0500	0.0600	43.3
7	employee_position_title: income, assistance, client	Float32DType	442 (4.8%)	1.16	6.61	0.0500	0.0543	45.4
8	employee_position_title: legislative, principal, executive	Float32DType	443 (4.8%)	1.07	5.72	0.0500	0.0522	47.8
9	employee_position_title: school, health, room	Float32DType	443 (4.8%)	1.16	6.62	0.0500	0.0523	43.3
10	employee_position_title: coordinator, services, service	Float32DType	443 (4.8%)	1.78	6.37	0.0500	0.0553	37.3
11	employee_position_title: community, nurse, security	Float32DType	443 (4.8%)	1.06	5.25	0.0500	0.0541	34.4
12	employee_position_title: warehouse, welfare, caseworker	Float32DType	443 (4.8%)	1.17	5.25	0.0500	0.0590	50.4
13	employee_position_title: technician, mechanic, supply	Float32DType	443 (4.8%)	1.10	5.24	0.0500	0.0547	52.9
14	employee_position_title: enforcement, permitting, inspector	Float32DType	443 (4.8%)	1.17	7.01	0.0504	0.0562	64.2
15	employee_position_title: telecommunications, communications, safety	Float32DType	443 (4.8%)	1.18	6.64	0.0501	0.0534	65.5
16	employee_position_title: liquor, clerk, store	Float32DType	443 (4.8%)	0.719	3.91	0.0500	0.0555	28.3
17	employee_position_title: captain, chief, autobody	Float32DType	443 (4.8%)	0.704	3.89	0.0500	0.0522	36.7
18	employee_position_title: representative, administrative, administration	Float32DType	443 (4.8%)	0.898	5.30	0.0500	0.0547	38.6
19	employee_position_title: budget, senior, procurement	Float32DType	443 (4.8%)	0.708	3.98	0.0500	0.0539	52.7
20	employee_position_title: information, technology, technologist	Float32DType	443 (4.8%)	1.15	7.12	0.0500	0.0545	56.5
21	employee_position_title: librarian, psychiatric, psychiatrist	Float32DType	443 (4.8%)	0.604	3.16	0.0503	0.0536	50.4
22	employee_position_title: planning, resources, alcohol	Float32DType	443 (4.8%)	0.781	4.68	0.0501	0.0551	49.8
23	employee_position_title: master, registered, meter	Float32DType	443 (4.8%)	0.984	5.43	0.0500	0.0552	35.7
24	employee_position_title: specialist, special, quality	Float32DType	443 (4.8%)	1.38	4.56	0.0500	0.0559	28.2
25	employee_position_title: equipment, investment, investigator	Float32DType	442 (4.8%)	0.750	4.05	0.0500	0.0553	39.1
26	employee_position_title: program, programs, sr	Float32DType	443 (4.8%)	0.729	3.66	0.0500	0.0591	28.5
27	employee_position_title: lieutenant, shift, maintenance	Float32DType	443 (4.8%)	0.618	3.76	0.0500	0.0531	29.8
28	employee_position_title: therapist, supervisory, sheriff	Float32DType	443 (4.8%)	0.829	3.87	0.0501	0.0560	30.7
29	employee_position_title: accountant, assistant, library	Float32DType	443 (4.8%)	1.13	4.93	0.0500	0.0531	40.0

Column 1	Column 2	Cramér's V	Pearson's Correlation
employee_position_title: income, assistance, client	employee_position_title: specialist, special, quality	0.450	0.310
employee_position_title: operator, bus, operations	employee_position_title: therapist, supervisory, sheriff	0.415	-0.0463
employee_position_title: manager, projects, project	employee_position_title: program, programs, sr	0.383	0.0212
employee_position_title: school, health, room	employee_position_title: planning, resources, alcohol	0.371	0.0151
employee_position_title: recreation, correctional, correction	employee_position_title: specialist, special, quality	0.359	0.143
employee_position_title: officer, police, office	employee_position_title: community, nurse, security	0.355	-0.0532
employee_position_title: liquor, clerk, store	employee_position_title: program, programs, sr	0.353	-0.0103
employee_position_title: information, technology, technologist	employee_position_title: specialist, special, quality	0.348	0.0381
employee_position_title: enforcement, permitting, inspector	employee_position_title: planning, resources, alcohol	0.346	0.0285
employee_position_title: income, assistance, client	employee_position_title: librarian, psychiatric, psychiatrist	0.335	-0.0121
employee_position_title: income, assistance, client	employee_position_title: lieutenant, shift, maintenance	0.333	-0.0219
employee_position_title: officer, police, office	employee_position_title: coordinator, services, service	0.323	-0.0640
employee_position_title: lieutenant, shift, maintenance	employee_position_title: therapist, supervisory, sheriff	0.315	0.00633
employee_position_title: captain, chief, autobody	employee_position_title: lieutenant, shift, maintenance	0.313	-0.0114
employee_position_title: telecommunications, communications, safety	employee_position_title: therapist, supervisory, sheriff	0.297	0.00642
employee_position_title: warehouse, welfare, caseworker	employee_position_title: telecommunications, communications, safety	0.295	-0.00481
employee_position_title: manager, projects, project	employee_position_title: liquor, clerk, store	0.294	0.00832
employee_position_title: manager, projects, project	employee_position_title: operator, bus, operations	0.294	-0.0716
employee_position_title: coordinator, services, service	employee_position_title: technician, mechanic, supply	0.294	-0.0315
employee_position_title: crossing, engineer, parking	employee_position_title: enforcement, permitting, inspector	0.289	-0.0151

Please enable javascript

The transformer used for each column kind can be easily configured.

Preprocessing in the `TableVectorizer`

The TableVectorizer actually performs a lot of preprocessing before applying the final transformers, such as:

ensuring consistent column names
detecting missing values such as "N/A"
dropping empty columns
handling pandas dtypes – float64, nan vs Float64, NA
parsing numbers
parsing dates, ensuring consistent dtype and timezone
converting numbers to float32 for faster computation & less memory downstream
…

pprint(vectorizer.all_processing_steps_["date_first_hired"])

[CleanNullStrings(),
 DropIfTooManyNulls(),
 ToDatetime(),
 DatetimeEncoder(),
 {'date_first_hired_day': ToFloat32(), 'date_first_hired_month': ToFloat32(), ...}]

Extracting good features

Skrub offers several encoders to extract features from different columns. In particular from categorical columns.

`GapEncoder`

Categories are somewhere between text and an enumeration… The GapEncoder is somewhere between a topic model and a one-hot encoder!

import seaborn as sns
from matplotlib import pyplot as plt

gap = skrub.GapEncoder()
pos_title = X["employee_position_title"]
loadings = gap.fit_transform(pos_title).set_index(pos_title.values).head()

loadings.columns = [c.split(": ")[1] for c in loadings.columns]
sns.heatmap(loadings)
_ = plt.setp(plt.gca().get_xticklabels(), rotation=45, ha="right")

`TextEncoder`

Extract embeddings from a text column using any model from the HuggingFace Hub.

import pandas as pd

X = pd.Series(["airport", "flight", "plane", "pineapple", "fruit"])
encoder = skrub.TextEncoder(model_name="all-MiniLM-L6-v2", n_components=None)
embeddings = encoder.fit_transform(X).set_index(X.values)

sns.heatmap(embeddings @ embeddings.T)

`MinHashEncoder`

A fast, stateless way of encoding strings that works especially well with models based on decision trees (gradient boosting, random forest).

4. How do I bring it all together?

Skrub has several transformers that allow peforming typical dataframe operations such as projections, joins and aggregations inside a scikit-learn pipeline.

Performing these operations in the machine-learning pipeline has several advantages:

Choices / hyperparameters can be optimized
Relevant state can be stored to ensure consistent transformations
All transformations are packaged together in an estimator

There are several transformers such as SelectCols, Joiner (fuzzy joining), InterpolationJoiner, AggJoiner, …

A toy example using the AggJoiner:

from skrub import AggJoiner

airports = pd.DataFrame(
    {
        "airport_id": [1, 2],
        "airport_name": ["Charles de Gaulle", "Aeroporto Leonardo da Vinci"],
        "city": ["Paris", "Roma"],
    }
)
airports

	airport_id	airport_name	city
0	1	Charles de Gaulle	Paris
1	2	Aeroporto Leonardo da Vinci	Roma

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)

Column 1	Column 2	Cramér's V
airport_name	city	1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

flights = pd.DataFrame(
    {
        "flight_id": range(1, 7),
        "from_airport": [1, 1, 1, 2, 2, 2],
        "total_passengers": [90, 120, 100, 70, 80, 90],
        "company": ["DL", "AF", "AF", "DL", "DL", "TR"],
    }
)
flights

	flight_id	from_airport	total_passengers	company
0	1	1	90	DL
1	2	1	120	AF
2	3	1	100	AF
3	4	2	70	DL
4	5	2	80	DL
5	6	2	90	TR

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	flight_id	Int64DType	6 (100.0%)	3.50	1.87	1	3	6
1	from_airport	Int64DType	2 (33.3%)	1.50	0.548	1	1	2
2	total_passengers	Int64DType	5 (83.3%)	91.7	17.2	70	90	120
3	company	ObjectDType	3 (50.0%)

Column 1	Column 2	Cramér's V	Pearson's Correlation
flight_id	total_passengers	1.00	-0.466
flight_id	company	1.00
flight_id	from_airport	1.00	0.878
total_passengers	company	0.816
from_airport	total_passengers	0.816	-0.742
from_airport	company	0.745

Please enable javascript

agg_joiner = AggJoiner(
    aux_table=flights,
    main_key="airport_id",
    aux_key="from_airport",
    cols=["total_passengers"],
    operations=["mean", "std"],
)
agg_joiner.fit_transform(airports)

	airport_id	airport_name	city	total_passengers_mean	total_passengers_std
0	1	Charles de Gaulle	Paris	103.33333333333333	15.275252316519467
1	2	Aeroporto Leonardo da Vinci	Roma	80.0	10.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)
3	total_passengers_mean	Float64DType	2 (100.0%)	91.7	16.5	80.0	80.0	103.
4	total_passengers_std	Float64DType	2 (100.0%)	12.6	3.73	10.0	10.0	15.3

Column 1	Column 2	Cramér's V	Pearson's Correlation
total_passengers_mean	total_passengers_std	1.00	1.00
city	total_passengers_std	1.00
city	total_passengers_mean	1.00
airport_name	total_passengers_std	1.00
airport_name	total_passengers_mean	1.00
airport_name	city	1.00
airport_id	total_passengers_std	1.00	-1.00
airport_id	total_passengers_mean	1.00	-1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

More interactive and expressive pipelines

To go further than what can be done with scikit-learn Pipelines and the skrub transformers shown above, we are developing new utilities to easily define and inspect flexible pipelines that can process several dataframes.

A prototype will be shown in a separate notebook.

	employee_position_title: recreation, correctional, correction	employee_position_title: crossing, engineer, parking	employee_position_title: officer, police, office	employee_position_title: manager, projects, project	employee_position_title: corporal, private, pfc	employee_position_title: operator, bus, operations	employee_position_title: firefighter, rescuer, recruit	employee_position_title: income, assistance, client	employee_position_title: legislative, principal, executive	employee_position_title: school, health, room	employee_position_title: coordinator, services, service	employee_position_title: community, nurse, security	employee_position_title: warehouse, welfare, caseworker	employee_position_title: technician, mechanic, supply	employee_position_title: enforcement, permitting, inspector	employee_position_title: telecommunications, communications, safety	employee_position_title: liquor, clerk, store	employee_position_title: captain, chief, autobody	employee_position_title: representative, administrative, administration	employee_position_title: budget, senior, procurement	employee_position_title: information, technology, technologist	employee_position_title: librarian, psychiatric, psychiatrist	employee_position_title: planning, resources, alcohol	employee_position_title: master, registered, meter	employee_position_title: specialist, special, quality	employee_position_title: equipment, investment, investigator	employee_position_title: program, programs, sr	employee_position_title: lieutenant, shift, maintenance	employee_position_title: therapist, supervisory, sheriff	employee_position_title: accountant, assistant, library
		employee_position_title: crossing, engineer, parking	employee_position_title: officer, police, office	employee_position_title: manager, projects, project	employee_position_title: corporal, private, pfc	employee_position_title: operator, bus, operations	employee_position_title: firefighter, rescuer, recruit	employee_position_title: income, assistance, client	employee_position_title: legislative, principal, executive	employee_position_title: school, health, room	employee_position_title: coordinator, services, service	employee_position_title: community, nurse, security	employee_position_title: warehouse, welfare, caseworker	employee_position_title: technician, mechanic, supply	employee_position_title: enforcement, permitting, inspector		employee_position_title: liquor, clerk, store	employee_position_title: captain, chief, autobody		employee_position_title: budget, senior, procurement			employee_position_title: planning, resources, alcohol	employee_position_title: master, registered, meter	employee_position_title: specialist, special, quality		employee_position_title: program, programs, sr	employee_position_title: lieutenant, shift, maintenance	employee_position_title: therapist, supervisory, sheriff	employee_position_title: accountant, assistant, library
0	0.055391132831573486	0.05465637892484665	0.1179402694106102	0.054503608494997025	0.07186415046453476	0.0774315819144249	0.05501769855618477	0.05456649512052536	0.05337824672460556	0.05232129618525505	37.32487106323242	0.05233673006296158	0.05282161757349968	0.051355089992284775	0.056758832186460495	0.05468842759728432	0.06072957441210747	0.05776433274149895	0.05382692441344261	0.052518002688884735	0.053227536380290985	0.051108140498399734	0.05254344642162323	0.055241771042346954	0.05058666691184044	0.05928345024585724	0.05240222066640854	0.05466080084443092	0.05406314134597778	0.05214018002152443
1	0.051088444888591766	0.05002940446138382	21.62775993347168	0.07254762947559357	0.06568593531847	0.054450348019599915	0.059532731771469116	0.0771363377571106	0.05325617641210556	0.05249681696295738	0.06432229280471802	0.050782762467861176	0.053962964564561844	0.05432360619306564	0.051792360842227936	0.053440190851688385	0.05960623919963837	0.05184761807322502	0.05770409107208252	0.05436079576611519	0.05699489638209343	0.06430193036794662	0.06891266256570816	6.754930019378662	0.05635739490389824	0.05266192927956581	0.059721603989601135	0.055518895387649536	0.054512869566679	0.05996069684624672
2	0.05984347313642502	0.050017766654491425	0.05958069488406181	0.0695878192782402	0.063633494079113	0.05378931015729904	0.05848724767565727	0.05486593768000603	0.057124171406030655	0.058061614632606506	0.05226082354784012	0.052549902349710464	20.84235191345215	0.05659961327910423	0.05426747351884842	0.052221883088350296	0.05763429030776024	0.05020422488451004	0.06235050782561302	0.059093985706567764	0.05854439362883568	0.05365617573261261	0.06426448374986649	0.05445589870214462	0.0735359713435173	0.05566619336605072	0.06003148481249809	0.05028039589524269	0.05352978780865669	0.05150934308767319
3	0.14719483256340027	0.06379014998674393	0.08085847645998001	0.17600774765014648	0.07917112112045288	0.07313495129346848	1.4369754791259766	0.060725852847099304	0.10775630176067352	0.050734177231788635	0.14284421503543854	0.05507330223917961	0.08717843145132065	0.05492651090025902	0.11275726556777954	0.05856424942612648	0.09124536067247391	0.15629425644874573	0.06218385323882103	0.14960260689258575	0.0631457269191742	0.05258248373866081	0.07196082174777985	0.1267090141773224	0.0630311593413353	5.372969150543213	0.0746825635433197	0.1299310028553009	22.222854614257812	0.0751139298081398
4	0.055967818945646286	0.07931414991617203	0.06552161276340485	0.11765759438276291	0.05909538269042969	0.05174881964921951	0.06034277379512787	0.08010916411876678	0.05754350870847702	0.05845404788851738	0.05275808647274971	0.061180781573057175	0.06247798725962639	0.0770404040813446	0.07402040809392929	0.05989750102162361	0.05556671693921089	0.05218116566538811	0.11507155746221542	0.07298754900693893	0.07344962656497955	0.05984378606081009	28.486339569091797	0.0508376769721508	2.6592044830322266	0.05511269345879555	0.0601314902305603	0.05542740598320961	0.0680314302444458	0.06268499791622162

9223	0.05308426544070244	0.05000864714384079	0.05383947491645813	0.055719275027513504	0.0540042407810688	0.05000029504299164	0.05229971930384636	0.05390317365527153	0.05207177624106407	0.07939788699150085	0.05267544090747833	34.442848205566406	0.054113686084747314	0.053470827639102936	0.052867498248815536	0.05772652104496956	0.051989246159791946	0.05110325291752815	0.05410536751151085	0.052043307572603226	0.05202293023467064	0.05143876001238823	0.05342757701873779	0.050001487135887146	0.053910478949546814	0.05087826028466225	0.051896072924137115	0.05084214359521866	0.05604195594787598	0.052270036190748215
9224	1.6355334520339966	0.3747715950012207	0.052151769399642944	0.05000999569892883	0.06975682824850082	0.05000387504696846	0.080123670399189	0.1163862869143486	0.2867222726345062	0.05785611644387245	0.3566876947879791	0.050671420991420746	0.08077290654182434	0.058939773589372635	0.05061366409063339	0.0733979120850563	0.052320126444101334	30.70151138305664	0.4078938066959381	0.06395088881254196	0.139220729470253	1.840002417564392	0.07162246108055115	0.07368595153093338	0.06431986391544342	0.05027112737298012	0.050004709511995316	0.09066556394100189	0.22355574369430542	0.2265750914812088
9225	0.07701816409826279	0.05007175728678703	0.09059899300336838	0.08243809640407562	0.09175847470760345	0.14005549252033234	0.06111059710383415	0.07977718859910965	0.1609271913766861	0.08915229141712189	0.07975136488676071	0.052972499281167984	1.8884986639022827	0.1153016909956932	0.15182317793369293	0.07339056581258774	0.13797542452812195	0.07576526701450348	0.30174222588539124	0.08728844672441483	0.09887836873531342	38.988365173339844	0.07841944694519043	0.05655289068818092	0.09030365943908691	3.0345325469970703	0.064872145652771	0.055096469819545746	0.0607161670923233	0.08484359830617905
9226	0.051801905035972595	0.05001617968082428	0.06267242878675461	11.89200496673584	0.05747326835989952	0.051846347749233246	0.0611814484000206	0.051428962498903275	0.05000162869691849	0.051225557923316956	0.05186647176742554	0.05280043184757233	0.060212817043066025	0.053699884563684464	0.05439164116978645	0.050564955919981	0.05351011082530022	0.050001755356788635	0.05203191936016083	0.05382412299513817	0.0518980398774147	0.05244126170873642	0.05161057040095329	0.055562011897563934	0.052779942750930786	0.054448358714580536	0.10995053499937057	0.05239533632993698	0.05447018891572952	0.051886990666389465
9227	0.06175076588988304	0.05018829181790352	0.07864895462989807	0.056595440953969955	0.06452731043100357	0.060972295701503754	0.05263859033584595	0.0770944207906723	0.05269759148359299	0.06457364559173584	0.06282345205545425	0.059019073843955994	0.05953995883464813	0.05866716057062149	16.027219772338867	0.06039658561348915	0.06125618889927864	0.052176471799612045	0.08112183213233948	3.6023056507110596	0.10165814310312271	0.05659399554133415	38.409576416015625	0.05066148191690445	0.21380367875099182	0.1462283879518509	0.05222833901643753	0.05679452046751976	0.06962364166975021	0.09861917048692703

Machine learning and tabular data

Skrub helps at several stages of a tabular learning project

1. What’s in the data?

TableReport: interactive display of a dataframe

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

Please enable javascript

2. Can we learn anything?

tabular_learner: a pre-made robust baseline

3. How do I represent the data?

TableVectorizer: apply an appropriate transformer to each column

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

Please enable javascript

employee_position_title: recreation, correctional, correction

employee_position_title: crossing, engineer, parking

employee_position_title: officer, police, office

employee_position_title: manager, projects, project

employee_position_title: corporal, private, pfc

employee_position_title: operator, bus, operations

employee_position_title: firefighter, rescuer, recruit

employee_position_title: income, assistance, client

employee_position_title: legislative, principal, executive

employee_position_title: school, health, room

employee_position_title: coordinator, services, service

employee_position_title: community, nurse, security

employee_position_title: warehouse, welfare, caseworker

employee_position_title: technician, mechanic, supply

employee_position_title: enforcement, permitting, inspector

employee_position_title: telecommunications, communications, safety

employee_position_title: liquor, clerk, store

employee_position_title: captain, chief, autobody

employee_position_title: representative, administrative, administration

employee_position_title: budget, senior, procurement

employee_position_title: information, technology, technologist

employee_position_title: librarian, psychiatric, psychiatrist

employee_position_title: planning, resources, alcohol

employee_position_title: master, registered, meter

employee_position_title: specialist, special, quality

employee_position_title: equipment, investment, investigator

employee_position_title: program, programs, sr

employee_position_title: lieutenant, shift, maintenance

employee_position_title: therapist, supervisory, sheriff

employee_position_title: accountant, assistant, library

employee_position_title: recreation, correctional, correction

employee_position_title: crossing, engineer, parking

employee_position_title: officer, police, office

employee_position_title: manager, projects, project

employee_position_title: corporal, private, pfc

employee_position_title: operator, bus, operations

employee_position_title: firefighter, rescuer, recruit

employee_position_title: income, assistance, client

employee_position_title: legislative, principal, executive

employee_position_title: school, health, room

employee_position_title: coordinator, services, service

employee_position_title: community, nurse, security

employee_position_title: warehouse, welfare, caseworker

employee_position_title: technician, mechanic, supply

employee_position_title: enforcement, permitting, inspector

employee_position_title: telecommunications, communications, safety

`TableReport`: interactive display of a dataframe

`tabular_learner`: a pre-made robust baseline

`TableVectorizer`: apply an appropriate transformer to each column

Preprocessing in the `TableVectorizer`

`GapEncoder`

`TextEncoder`

`MinHashEncoder`