Skrub

Less wrangling, more machine learning

Machine learning and tabular data

ML expects numeric arrays
Real data is more complex:
- Multiple tables
- Dates, categories, text, locations, …

Skrub: bridge the gap between dataframes and scikit-learn.

Skrub helps at several stages of a tabular learning project

What’s in the data? (EDA)
Can we learn anything? (baselines)
How do I represent the data? (feature extraction)
How do I bring it all together? (building a pipeline)

1. What’s in the data?

import skrub
import pandas as pd

employee_salaries = pd.read_csv("https://figshare.com/ndownloader/files/51755822")
employees = employee_salaries.drop("current_annual_salary", axis=1)
salaries = employee_salaries["current_annual_salary"]
employees.iloc[0]

gender                                                                     F
department                                                               POL
department_name                                         Department of Police
division                   MSB Information Mgmt and Tech Division Records...
assignment_category                                         Fulltime-Regular
employee_position_title                          Office Services Coordinator
date_first_hired                                                  09/22/1986
year_first_hired                                                        1986
Name: 0, dtype: object

`TableReport`: interactive display of a dataframe

skrub.TableReport(employees, verbose=0)

	gender	department	department_name	division	assignment_category	employee_position_title	date_first_hired	year_first_hired
0	F	POL	Department of Police	MSB Information Mgmt and Tech Division Records Management Section	Fulltime-Regular	Office Services Coordinator	09/22/1986	1986
1	M	POL	Department of Police	ISB Major Crimes Division Fugitive Section	Fulltime-Regular	Master Police Officer	09/12/1988	1988
2	F	HHS	Department of Health and Human Services	Adult Protective and Case Management Services	Fulltime-Regular	Social Worker IV	11/19/1989	1989
3	M	COR	Correction and Rehabilitation	PRRS Facility and Security	Fulltime-Regular	Resident Supervisor II	05/05/2014	2014
4	M	HCA	Department of Housing and Community Affairs	Affordable Housing Programs	Fulltime-Regular	Planning Specialist III	03/05/2007	2007

9223	F	HHS	Department of Health and Human Services	School Based Health Centers	Fulltime-Regular	Community Health Nurse II	11/03/2015	2015
9224	F	FRS	Fire and Rescue Services	Human Resources Division	Fulltime-Regular	Fire/Rescue Division Chief	11/28/1988	1988
9225	M	HHS	Department of Health and Human Services	Child and Adolescent Mental Health Clinic Services	Parttime-Regular	Medical Doctor IV - Psychiatrist	04/30/2001	2001
9226	M	CCL	County Council	Council Central Staff	Fulltime-Regular	Manager II	09/05/2006	2006
9227	M	DLC	Department of Liquor Control	Licensure, Regulation and Education	Fulltime-Regular	Alcohol/Tobacco Enforcement Specialist II	01/30/2012	2012

Column	Column name	dtype	Null values	Unique values	Mean	Std	Min	Median	Max
0	gender	ObjectDType	17 (0.2%)	2 (< 0.1%)
1	department	ObjectDType	0 (0.0%)	37 (0.4%)
2	department_name	ObjectDType	0 (0.0%)	37 (0.4%)
3	division	ObjectDType	0 (0.0%)	694 (7.5%)
4	assignment_category	ObjectDType	0 (0.0%)	2 (< 0.1%)
5	employee_position_title	ObjectDType	0 (0.0%)	443 (4.8%)
6	date_first_hired	ObjectDType	0 (0.0%)	2264 (24.5%)
7	year_first_hired	Int64DType	0 (0.0%)	51 (0.6%)	2.00e+03	9.33	1,965	2,005	2,016

Column 1	Column 2	Cramér's V
department	department_name	1.00
division	assignment_category	0.612
assignment_category	employee_position_title	0.500
division	employee_position_title	0.427
department_name	employee_position_title	0.422
department	employee_position_title	0.422
department	assignment_category	0.402
department_name	assignment_category	0.402
gender	department	0.385
gender	department_name	0.385
department	division	0.373
department_name	division	0.373
employee_position_title	date_first_hired	0.313
gender	employee_position_title	0.271
gender	division	0.250
gender	assignment_category	0.250
department	date_first_hired	0.176
department_name	date_first_hired	0.176
division	date_first_hired	0.161
date_first_hired	year_first_hired	0.155

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

We can tell skrub to patch the default display of polars and pandas dataframes.

skrub.patch_display(verbose=0)

2. Can we learn anything?

X, y = employees, salaries

`tabular_learner`: a pre-made robust baseline

learner = skrub.tabular_learner("regressor")
learner

Pipeline(steps=[('tablevectorizer',
                 TableVectorizer(high_cardinality=MinHashEncoder(),
                                 low_cardinality=ToCategorical())),
                ('histgradientboostingregressor',
                 HistGradientBoostingRegressor())])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

from sklearn.model_selection import cross_val_score

cross_val_score(learner, X, y, scoring="r2")

array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666])

The tabular_learner adapts to the supervised estimator we choose

from sklearn.linear_model import Ridge

learner = skrub.tabular_learner(Ridge())
learner

Pipeline(steps=[('tablevectorizer', TableVectorizer()),
                ('simpleimputer', SimpleImputer(add_indicator=True)),
                ('standardscaler', StandardScaler()), ('ridge', Ridge())])

cross_val_score(learner, X, y, scoring="r2")

array([0.73368958, 0.7137886 , 0.74098157, 0.73990674, 0.74342958])

3. How do I represent the data?

Skrub helps extract informative features from tabular data.

`TableVectorizer`: apply an appropriate transformer to each column

vectorizer = skrub.TableVectorizer()
transformed = vectorizer.fit_transform(X)

The TableVectorizer identifies several kinds of columns:

categorical, low cardinality
categorical, high cardinality
datetime
numeric
… we may add more

from pprint import pprint

pprint(vectorizer.column_to_kind_)

{'assignment_category': 'low_cardinality',
 'date_first_hired': 'datetime',
 'department': 'low_cardinality',
 'department_name': 'low_cardinality',
 'division': 'high_cardinality',
 'employee_position_title': 'high_cardinality',
 'gender': 'low_cardinality',
 'year_first_hired': 'numeric'}

For each kind, it applies an appropriate transformer

vectorizer.transformers_["department"]  # low-cardinality categorical

OneHotEncoder(drop='if_binary', dtype='float32', handle_unknown='ignore',
              sparse_output=False)

vectorizer.transformers_["employee_position_title"]  # high-cardinality categorical

GapEncoder(n_components=30)

vectorizer.transformers_["date_first_hired"]  # datetime

DatetimeEncoder()

… and those transformers turn the input into numeric features that can be used for ML

transformed[vectorizer.input_to_outputs_["date_first_hired"]]

	date_first_hired_year	date_first_hired_month	date_first_hired_day	date_first_hired_total_seconds
0	1986.0	9.0	22.0	527731200.0
1	1988.0	9.0	12.0	590025600.0
2	1989.0	11.0	19.0	627436800.0
3	2014.0	5.0	5.0	1399248000.0
4	2007.0	3.0	5.0	1173052800.0

9223	2015.0	11.0	3.0	1446508800.0
9224	1988.0	11.0	28.0	596678400.0
9225	2001.0	4.0	30.0	988588800.0
9226	2006.0	9.0	5.0	1157414400.0
9227	2012.0	1.0	30.0	1327881600.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	date_first_hired_year	Float32DType	51 (0.6%)	2.00e+03	9.33	1.96e+03	2.00e+03	2.02e+03
1	date_first_hired_month	Float32DType	12 (0.1%)	6.35	3.48	1.00	7.00	12.0
2	date_first_hired_day	Float32DType	31 (0.3%)	15.3	8.63	1.00	16.0	31.0
3	date_first_hired_total_seconds	Float32DType	2264 (24.5%)	1.08e+09	2.94e+08	-1.34e+08	1.12e+09	1.48e+09

Column 1	Column 2	Cramér's V
date_first_hired_year	date_first_hired_total_seconds	0.909
date_first_hired_month	date_first_hired_day	0.124
date_first_hired_year	date_first_hired_month	0.0981
date_first_hired_month	date_first_hired_total_seconds	0.0897
date_first_hired_year	date_first_hired_day	0.0832
date_first_hired_day	date_first_hired_total_seconds	0.0832

Please enable javascript

For high-cardinality categorical columns the default GapEncoder identifies sparse topics (more later).

transformed[vectorizer.input_to_outputs_["employee_position_title"]]

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	employee_position_title: accountant, attorney, auditor	Float32DType	443 (4.8%)	1.03	4.76	0.0500	0.0557	29.3
1	employee_position_title: officer, office, traffic	Float32DType	443 (4.8%)	2.63	7.12	0.0500	0.0632	23.8
2	employee_position_title: assistance, income, fiscal	Float32DType	443 (4.8%)	1.11	5.40	0.0500	0.0546	34.8
3	employee_position_title: operator, equipment, bus	Float32DType	443 (4.8%)	1.57	4.80	0.0500	0.0535	31.2
4	employee_position_title: master, registered, meter	Float32DType	443 (4.8%)	0.983	5.44	0.0500	0.0554	35.7
5	employee_position_title: specialist, special, paralegal	Float32DType	443 (4.8%)	1.29	3.99	0.0500	0.0582	27.9
6	employee_position_title: library, associate, locksmith	Float32DType	443 (4.8%)	0.788	3.90	0.0500	0.0533	29.9
7	employee_position_title: enforcement, permitting, inspector	Float32DType	443 (4.8%)	1.26	6.96	0.0501	0.0568	64.3
8	employee_position_title: manager, projects, project	Float32DType	443 (4.8%)	1.27	3.83	0.0500	0.0636	28.9
9	employee_position_title: communications, telecommunications, safety	Float32DType	443 (4.8%)	1.05	6.22	0.0500	0.0537	62.2
10	employee_position_title: corporal, pfc, private	Float32DType	443 (4.8%)	1.24	7.23	0.0500	0.0596	49.2
11	employee_position_title: technician, mechanic, supply	Float32DType	443 (4.8%)	1.11	5.26	0.0500	0.0547	52.9
12	employee_position_title: lieutenant, captain, chief	Float32DType	443 (4.8%)	1.20	5.27	0.0500	0.0528	32.7
13	employee_position_title: supervisor, correctional, correction	Float32DType	443 (4.8%)	0.861	4.45	0.0500	0.0536	46.2
14	employee_position_title: coordinator, transit, coordinating	Float32DType	443 (4.8%)	1.13	5.88	0.0500	0.0562	37.3
15	employee_position_title: school, room, behavioral	Float32DType	443 (4.8%)	1.13	6.58	0.0500	0.0527	43.3
16	employee_position_title: community, security, nurse	Float32DType	443 (4.8%)	0.851	4.85	0.0500	0.0533	34.4
17	employee_position_title: administrative, principal, executive	Float32DType	443 (4.8%)	1.25	6.10	0.0500	0.0528	46.4
18	employee_position_title: information, technology, technologist	Float32DType	443 (4.8%)	1.17	6.98	0.0500	0.0542	56.4
19	employee_position_title: sergeant, police, cadet	Float32DType	443 (4.8%)	0.595	2.71	0.0500	0.0566	19.4
20	employee_position_title: warehouse, craftsworker, welfare	Float32DType	443 (4.8%)	1.17	5.32	0.0500	0.0584	50.7
21	employee_position_title: recreation, renovation, background	Float32DType	443 (4.8%)	0.792	4.24	0.0500	0.0546	43.2
22	employee_position_title: services, service, aide	Float32DType	443 (4.8%)	0.923	4.57	0.0500	0.0567	37.5
23	employee_position_title: firefighter, rescuer, recruit	Float32DType	439 (4.8%)	2.50	8.56	0.0500	0.0594	43.3
24	employee_position_title: program, programs, aquatic	Float32DType	443 (4.8%)	1.02	3.73	0.0500	0.0536	19.4
25	employee_position_title: candidate, sheriff, deputy	Float32DType	443 (4.8%)	0.493	3.05	0.0500	0.0550	32.9
26	employee_position_title: environmental, therapist, enviromental	Float32DType	443 (4.8%)	0.780	4.31	0.0501	0.0547	40.3
27	employee_position_title: crossing, purchasing, engineer	Float32DType	443 (4.8%)	0.578	2.98	0.0501	0.0505	36.2
28	employee_position_title: librarian, psychiatric, psychiatrist	Float32DType	443 (4.8%)	0.499	2.78	0.0505	0.0543	46.6
29	employee_position_title: representative, legislative, customer	Float32DType	443 (4.8%)	0.517	3.89	0.0500	0.0560	48.1

Column 1	Column 2	Cramér's V
employee_position_title: program, programs, aquatic	employee_position_title: environmental, therapist, enviromental	0.448
employee_position_title: specialist, special, paralegal	employee_position_title: program, programs, aquatic	0.444
employee_position_title: officer, office, traffic	employee_position_title: corporal, pfc, private	0.438
employee_position_title: school, room, behavioral	employee_position_title: environmental, therapist, enviromental	0.428
employee_position_title: assistance, income, fiscal	employee_position_title: program, programs, aquatic	0.383
employee_position_title: administrative, principal, executive	employee_position_title: representative, legislative, customer	0.365
employee_position_title: enforcement, permitting, inspector	employee_position_title: services, service, aide	0.363
employee_position_title: assistance, income, fiscal	employee_position_title: specialist, special, paralegal	0.360
employee_position_title: library, associate, locksmith	employee_position_title: administrative, principal, executive	0.348
employee_position_title: sergeant, police, cadet	employee_position_title: services, service, aide	0.347
employee_position_title: specialist, special, paralegal	employee_position_title: administrative, principal, executive	0.342
employee_position_title: manager, projects, project	employee_position_title: program, programs, aquatic	0.338
employee_position_title: manager, projects, project	employee_position_title: environmental, therapist, enviromental	0.333
employee_position_title: warehouse, craftsworker, welfare	employee_position_title: services, service, aide	0.332
employee_position_title: specialist, special, paralegal	employee_position_title: communications, telecommunications, safety	0.322
employee_position_title: supervisor, correctional, correction	employee_position_title: sergeant, police, cadet	0.320
employee_position_title: accountant, attorney, auditor	employee_position_title: officer, office, traffic	0.312
employee_position_title: officer, office, traffic	employee_position_title: community, security, nurse	0.304
employee_position_title: operator, equipment, bus	employee_position_title: coordinator, transit, coordinating	0.292
employee_position_title: operator, equipment, bus	employee_position_title: crossing, purchasing, engineer	0.290

Please enable javascript

The transformer used for each column kind can be easily configured.

Preprocessing in the `TableVectorizer`

The TableVectorizer actually performs a lot of preprocessing before applying the final transformers, such as:

ensuring consistent column names
detecting missing values such as "N/A"
dropping empty columns
handling pandas dtypes – float64, nan vs Float64, NA
parsing numbers
parsing dates, ensuring consistent dtype and timezone
converting numbers to float32 for faster computation & less memory downstream
…

pprint(vectorizer.all_processing_steps_["date_first_hired"])

[CleanNullStrings(),
 DropIfTooManyNulls(),
 ToDatetime(),
 DatetimeEncoder(),
 {'date_first_hired_day': ToFloat32(), 'date_first_hired_month': ToFloat32(), ...}]

Extracting good features

Skrub offers several encoders to extract features from different columns. In particular from categorical columns.

`GapEncoder`

Categories are somewhere between text and an enumeration… The GapEncoder is somewhere between a topic model and a one-hot encoder!

import seaborn as sns
from matplotlib import pyplot as plt

gap = skrub.GapEncoder()
pos_title = X["employee_position_title"]
loadings = gap.fit_transform(pos_title).set_index(pos_title.values).head()

loadings.columns = [c.split(": ")[1] for c in loadings.columns]
sns.heatmap(loadings)
_ = plt.setp(plt.gca().get_xticklabels(), rotation=45, ha="right")

`TextEncoder`

Extract embeddings from a text column using any model from the HuggingFace Hub.

import pandas as pd

X = pd.Series(["airport", "flight", "plane", "pineapple", "fruit"])
encoder = skrub.TextEncoder(model_name="all-MiniLM-L6-v2", n_components=None)
embeddings = encoder.fit_transform(X).set_index(X.values)

sns.heatmap(embeddings @ embeddings.T)

`MinHashEncoder`

A fast, stateless way of encoding strings that works especially well with models based on decision trees (gradient boosting, random forest).

4. How do I bring it all together?

Skrub has several transformers that allow peforming typical dataframe operations such as projections, joins and aggregations inside a scikit-learn pipeline.

Performing these operations in the machine-learning pipeline has several advantages:

Choices / hyperparameters can be optimized
Relevant state can be stored to ensure consistent transformations
All transformations are packaged together in an estimator

There are several transformers such as SelectCols, Joiner (fuzzy joining), InterpolationJoiner, AggJoiner, …

A toy example using the AggJoiner:

from skrub import AggJoiner

airports = pd.DataFrame(
    {
        "airport_id": [1, 2],
        "airport_name": ["Charles de Gaulle", "Aeroporto Leonardo da Vinci"],
        "city": ["Paris", "Roma"],
    }
)
airports

	airport_id	airport_name	city
0	1	Charles de Gaulle	Paris
1	2	Aeroporto Leonardo da Vinci	Roma

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)

Column 1	Column 2	Cramér's V
airport_name	city	1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

flights = pd.DataFrame(
    {
        "flight_id": range(1, 7),
        "from_airport": [1, 1, 1, 2, 2, 2],
        "total_passengers": [90, 120, 100, 70, 80, 90],
        "company": ["DL", "AF", "AF", "DL", "DL", "TR"],
    }
)
flights

	flight_id	from_airport	total_passengers	company
0	1	1	90	DL
1	2	1	120	AF
2	3	1	100	AF
3	4	2	70	DL
4	5	2	80	DL
5	6	2	90	TR

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	flight_id	Int64DType	6 (100.0%)	3.50	1.87	1	3	6
1	from_airport	Int64DType	2 (33.3%)	1.50	0.548	1	1	2
2	total_passengers	Int64DType	5 (83.3%)	91.7	17.2	70	90	120
3	company	ObjectDType	3 (50.0%)

Column 1	Column 2	Cramér's V
flight_id	total_passengers	1.00
flight_id	company	1.00
flight_id	from_airport	1.00
total_passengers	company	0.816
from_airport	total_passengers	0.816
from_airport	company	0.745

Please enable javascript

agg_joiner = AggJoiner(
    aux_table=flights,
    main_key="airport_id",
    aux_key="from_airport",
    cols=["total_passengers"],
    operations=["mean", "std"],
)
agg_joiner.fit_transform(airports)

	airport_id	airport_name	city	total_passengers_mean	total_passengers_std
0	1	Charles de Gaulle	Paris	103.33333333333333	15.275252316519467
1	2	Aeroporto Leonardo da Vinci	Roma	80.0	10.0

Column	Column name	dtype	Unique values	Mean	Std	Min	Median	Max
0	airport_id	Int64DType	2 (100.0%)	1.50	0.707	1	1	2
1	airport_name	ObjectDType	2 (100.0%)
2	city	ObjectDType	2 (100.0%)
3	total_passengers_mean	Float64DType	2 (100.0%)	91.7	16.5	80.0	80.0	103.
4	total_passengers_std	Float64DType	2 (100.0%)	12.6	3.73	10.0	10.0	15.3

Column 1	Column 2	Cramér's V
total_passengers_mean	total_passengers_std	1.00
city	total_passengers_std	1.00
city	total_passengers_mean	1.00
airport_name	total_passengers_std	1.00
airport_name	total_passengers_mean	1.00
airport_name	city	1.00
airport_id	total_passengers_std	1.00
airport_id	total_passengers_mean	1.00
airport_id	city	1.00
airport_id	airport_name	1.00

Please enable javascript

More interactive and expressive pipelines

To go further than what can be done with scikit-learn Pipelines and the skrub transformers shown above, we are developing new utilities to easily define and inspect flexible pipelines that can process several dataframes.

A prototype will be shown in a separate notebook.

	employee_position_title: accountant, attorney, auditor	employee_position_title: officer, office, traffic	employee_position_title: assistance, income, fiscal	employee_position_title: operator, equipment, bus	employee_position_title: master, registered, meter	employee_position_title: specialist, special, paralegal	employee_position_title: library, associate, locksmith	employee_position_title: enforcement, permitting, inspector	employee_position_title: manager, projects, project	employee_position_title: communications, telecommunications, safety	employee_position_title: corporal, pfc, private	employee_position_title: technician, mechanic, supply	employee_position_title: lieutenant, captain, chief	employee_position_title: supervisor, correctional, correction	employee_position_title: coordinator, transit, coordinating	employee_position_title: school, room, behavioral	employee_position_title: community, security, nurse	employee_position_title: administrative, principal, executive	employee_position_title: information, technology, technologist	employee_position_title: sergeant, police, cadet	employee_position_title: warehouse, craftsworker, welfare	employee_position_title: recreation, renovation, background	employee_position_title: services, service, aide	employee_position_title: firefighter, rescuer, recruit	employee_position_title: program, programs, aquatic	employee_position_title: candidate, sheriff, deputy	employee_position_title: environmental, therapist, enviromental	employee_position_title: crossing, purchasing, engineer	employee_position_title: librarian, psychiatric, psychiatrist	employee_position_title: representative, legislative, customer
	employee_position_title: accountant, attorney, auditor	employee_position_title: officer, office, traffic	employee_position_title: assistance, income, fiscal	employee_position_title: operator, equipment, bus	employee_position_title: master, registered, meter	employee_position_title: specialist, special, paralegal	employee_position_title: library, associate, locksmith	employee_position_title: enforcement, permitting, inspector	employee_position_title: manager, projects, project		employee_position_title: corporal, pfc, private	employee_position_title: technician, mechanic, supply	employee_position_title: lieutenant, captain, chief		employee_position_title: coordinator, transit, coordinating	employee_position_title: school, room, behavioral	employee_position_title: community, security, nurse			employee_position_title: sergeant, police, cadet	employee_position_title: warehouse, craftsworker, welfare	employee_position_title: recreation, renovation, background	employee_position_title: services, service, aide	employee_position_title: firefighter, rescuer, recruit	employee_position_title: program, programs, aquatic	employee_position_title: candidate, sheriff, deputy		employee_position_title: crossing, purchasing, engineer
0	0.05737796053290367	0.10020679235458374	0.054879289120435715	0.06472974270582199	0.05543024092912674	0.05179969221353531	0.05051771178841591	0.05480916425585747	0.0540931336581707	0.05432574078440666	0.06392356008291245	0.05166192352771759	0.0546519011259079	0.05735558271408081	37.28716278076172	0.05178763344883919	0.05258779600262642	0.05349650979042053	0.05272148177027702	0.08748094737529755	0.053513579070568085	0.052266206592321396	0.10723209381103516	0.05475832521915436	0.05057666450738907	0.054724689573049545	0.05092926323413849	0.052958350628614426	0.05066924914717674	0.06137316673994064
1	0.058525800704956055	21.634557723999023	0.08611483126878738	0.05368993431329727	6.710700511932373	0.0577864833176136	0.0604577511548996	0.051986437290906906	0.07004567980766296	0.053827088326215744	0.0646471232175827	0.05431704595685005	0.05338016152381897	0.05172603577375412	0.06401882320642471	0.05268576741218567	0.050799231976270676	0.054458729922771454	0.05686930567026138	0.09745483100414276	0.05383677035570145	0.06665493547916412	0.0575433224439621	0.05933864414691925	0.05268023535609245	0.053467437624931335	0.05519871786236763	0.05148470774292946	0.055316898971796036	0.056429553776979446
2	0.05610807612538338	0.06101250648498535	0.0540708564221859	0.05308777838945389	0.054435063153505325	0.08904258906841278	0.057160597294569016	0.05580637976527214	0.06875495612621307	0.05286430940032005	0.06390050053596497	0.05817144736647606	0.05000199005007744	0.061203278601169586	0.05216580256819725	0.05949173495173454	0.05298910290002823	0.057999979704618454	0.05916225165128708	0.05125490948557854	20.821165084838867	0.06469449400901794	0.05179375410079956	0.05878655984997749	0.06163415312767029	0.0522383414208889	0.05993230268359184	0.05010855942964554	0.05575355514883995	0.0552084855735302
3	0.08214592933654785	0.09768906235694885	0.15064707398414612	0.07200082391500473	0.10993237793445587	0.08650316298007965	0.37455230951309204	5.176558971405029	0.17857420444488525	0.05748199671506882	0.07473331689834595	0.058320872485637665	0.12248824536800385	12.04856014251709	0.0704333633184433	0.05118381232023239	0.05516922101378441	0.0577385388314724	0.06632924824953079	0.06246696785092354	0.08537111431360245	0.0660453662276268	10.504438400268555	0.8529676198959351	0.06217220798134804	0.06489139050245285	0.4956369698047638	0.11960155516862869	0.0576261430978775	0.13773949444293976
4	0.05808864161372185	0.0673263892531395	0.07545826584100723	0.05145619437098503	0.050686467438936234	14.647977828979492	0.06028281897306442	0.08501282334327698	0.09377622604370117	0.060385383665561676	0.05923718959093094	0.08172371238470078	0.05388503894209862	0.05419814586639404	0.05317079648375511	0.05870555341243744	0.05705224722623825	0.060048509389162064	0.06790629774332047	0.05979912728071213	0.05997174605727196	13.807762145996094	0.059844180941581726	0.060522470623254776	0.08657856285572052	0.06276968121528625	0.09751822054386139	2.7850849628448486	0.06554565578699112	0.058224260807037354

9223	0.053339883685112	0.055061545222997665	0.05422933027148247	0.05000150203704834	0.050001733005046844	0.05592511221766472	0.05227094516158104	0.05341717228293419	0.05536061152815819	0.06040368229150772	0.05430074781179428	0.0541936494410038	0.05086570233106613	0.052210453897714615	0.051996514201164246	0.0690806433558464	34.42456817626953	0.05225184187293053	0.05223105475306511	0.05363965407013893	0.05391693860292435	0.05216717720031738	0.06462019681930542	0.05300278961658478	0.052476916462183	0.05724189430475235	0.0544067919254303	0.05012940242886543	0.052052970975637436	0.054633185267448425
9224	0.05645890533924103	0.052376605570316315	0.2493654042482376	0.05000150576233864	0.08520420640707016	0.06012314558029175	0.17292481660842896	0.05082643777132034	0.05000133812427521	0.0660768374800682	0.06862062215805054	0.06655819714069366	28.107023239135742	2.4891693592071533	0.22028492391109467	0.06305398046970367	0.050841428339481354	0.46920105814933777	0.20632191002368927	0.05159265175461769	0.08009310811758041	1.8845891952514648	0.17677947878837585	0.09638725221157074	0.05307695269584656	0.05879402905702591	0.062184643000364304	0.20984511077404022	1.9323514699935913	0.2598726153373718
9225	0.26812803745269775	0.15855534374713898	0.11866529285907745	0.2773430347442627	0.05635950341820717	0.11817100644111633	0.08003567159175873	1.785054326057434	0.10391992330551147	0.08188865333795547	0.10430716723203659	0.09805762022733688	0.0554141029715538	0.08335304260253906	0.14405827224254608	0.0771414190530777	0.05322861671447754	0.7100736498832703	0.10037398338317871	0.06549173593521118	0.4715404510498047	0.08393486589193344	0.0639539584517479	0.06745129078626633	0.06279626488685608	0.05352616310119629	0.08638902753591537	0.05093436688184738	40.90443801879883	0.11541547626256943
9226	0.05464360490441322	0.06676299124956131	0.05169437825679779	0.05180729180574417	0.05589008331298828	0.05499241501092911	0.05166955292224884	0.056620948016643524	11.928296089172363	0.0506812259554863	0.05873500183224678	0.05457282438874245	0.050946325063705444	0.05270519107580185	0.05152958258986473	0.0514240525662899	0.05294100567698479	0.05000554397702217	0.052204255014657974	0.054647501558065414	0.06122397631406784	0.050537824630737305	0.05352446436882019	0.0631779134273529	0.052662067115306854	0.054987918585538864	0.053260691463947296	0.05013331025838852	0.05408834293484688	0.05363372713327408
9227	2.903372287750244	0.15880835056304932	0.07997149974107742	0.05999436229467392	0.050562795251607895	5.726363658905029	0.057724479585886	14.5643310546875	0.05323512852191925	0.06507456302642822	0.06967116892337799	0.06066165491938591	0.0538473054766655	0.06296683847904205	0.07032007724046707	0.1120065227150917	0.06353479623794556	0.05368528515100479	0.19311614334583282	0.2137782722711563	0.06486145406961441	0.12897565960884094	0.0650746300816536	0.05188797786831856	0.0895182266831398	0.05276313051581383	28.5845947265625	6.1736555099487305	0.056064244359731674	0.059576958417892456

Machine learning and tabular data

Skrub helps at several stages of a tabular learning project

1. What’s in the data?

TableReport: interactive display of a dataframe

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

gender

department

department_name

division

assignment_category

employee_position_title

date_first_hired

year_first_hired

Please enable javascript

2. Can we learn anything?

tabular_learner: a pre-made robust baseline

3. How do I represent the data?

TableVectorizer: apply an appropriate transformer to each column

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

date_first_hired_year

date_first_hired_month

date_first_hired_day

date_first_hired_total_seconds

Please enable javascript

employee_position_title: accountant, attorney, auditor

employee_position_title: officer, office, traffic

employee_position_title: assistance, income, fiscal

employee_position_title: operator, equipment, bus

employee_position_title: master, registered, meter

employee_position_title: specialist, special, paralegal

employee_position_title: library, associate, locksmith

employee_position_title: enforcement, permitting, inspector

employee_position_title: manager, projects, project

employee_position_title: communications, telecommunications, safety

employee_position_title: corporal, pfc, private

employee_position_title: technician, mechanic, supply

employee_position_title: lieutenant, captain, chief

employee_position_title: supervisor, correctional, correction

employee_position_title: coordinator, transit, coordinating

employee_position_title: school, room, behavioral

employee_position_title: community, security, nurse

employee_position_title: administrative, principal, executive

employee_position_title: information, technology, technologist

employee_position_title: sergeant, police, cadet

employee_position_title: warehouse, craftsworker, welfare

employee_position_title: recreation, renovation, background

employee_position_title: services, service, aide

employee_position_title: firefighter, rescuer, recruit

employee_position_title: program, programs, aquatic

employee_position_title: candidate, sheriff, deputy

employee_position_title: environmental, therapist, enviromental

employee_position_title: crossing, purchasing, engineer

employee_position_title: librarian, psychiatric, psychiatrist

employee_position_title: representative, legislative, customer

employee_position_title: accountant, attorney, auditor

employee_position_title: officer, office, traffic

employee_position_title: assistance, income, fiscal

employee_position_title: operator, equipment, bus

employee_position_title: master, registered, meter

employee_position_title: specialist, special, paralegal

employee_position_title: library, associate, locksmith

employee_position_title: enforcement, permitting, inspector

employee_position_title: manager, projects, project

employee_position_title: communications, telecommunications, safety

employee_position_title: corporal, pfc, private

employee_position_title: technician, mechanic, supply

employee_position_title: lieutenant, captain, chief

employee_position_title: supervisor, correctional, correction

employee_position_title: coordinator, transit, coordinating

employee_position_title: school, room, behavioral

`TableReport`: interactive display of a dataframe

`tabular_learner`: a pre-made robust baseline

`TableVectorizer`: apply an appropriate transformer to each column

Preprocessing in the `TableVectorizer`

`GapEncoder`

`TextEncoder`

`MinHashEncoder`