2  Exploring dataframes with skrub

2.1 Introduction

In this chapter, we will show how we use the skrub TableReport to explore tabular data. We will use the Adult Census dataset as our example table, and perform some exploratory analysis to learn about the characteristics of the data.

2.2 Why do we need to do data exploration?

Before any kind of data processing or usage, we need to know what we are dealing with.

Useful information includes:

• The size of the dataset.
• The data types and names of the columns.
• How values are distributed in each column.
• Whether missing values are present, in what measure and where.
• Which features are discrete/categorical, and how many categories there are.
• Whether columns are strongly correlated with each other. …

2.3 Exploring data with Pandas

First, let’s load the dataset.

import pandas as pd
# Load the Adult Census dataset
data =  pd.read_csv("../data/adult_census/data.csv")
target =  pd.read_csv("../data/adult_census/target.csv")

Let’s first explore the data using Pandas only.

We can get an idea of the content of the table by printing the first few lines, which gives an idea of the datatypes and the columns we are dealing with.

data.head(5)

	age	workclass	fnlwgt	education	education-num	marital-status	occupation	relationship	race	sex	capital-gain	capital-loss	hours-per-week	native-country
0	25	Private	226802	11th	7	Never-married	Machine-op-inspct	Own-child	Black	Male	0	0	40	United-States
1	38	Private	89814	HS-grad	9	Married-civ-spouse	Farming-fishing	Husband	White	Male	0	0	50	United-States
2	28	Local-gov	336951	Assoc-acdm	12	Married-civ-spouse	Protective-serv	Husband	White	Male	0	0	40	United-States
3	44	Private	160323	Some-college	10	Married-civ-spouse	Machine-op-inspct	Husband	Black	Male	7688	0	40	United-States
4	18	NaN	103497	Some-college	10	Never-married	NaN	Own-child	White	Female	0	0	30	United-States

We can use data.info() to find the shape of the dataframe, which dtypes are involved, the number of missing values and the size in memory of the dataframe.

data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 48842 entries, 0 to 48841
Data columns (total 14 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   age             48842 non-null  int64 
 1   workclass       46043 non-null  object
 2   fnlwgt          48842 non-null  int64 
 3   education       48842 non-null  object
 4   education-num   48842 non-null  int64 
 5   marital-status  48842 non-null  object
 6   occupation      46033 non-null  object
 7   relationship    48842 non-null  object
 8   race            48842 non-null  object
 9   sex             48842 non-null  object
 10  capital-gain    48842 non-null  int64 
 11  capital-loss    48842 non-null  int64 
 12  hours-per-week  48842 non-null  int64 
 13  native-country  47985 non-null  object
dtypes: int64(6), object(8)
memory usage: 5.2+ MB

We can also get a richer summary of the data with the .describe() method:

This gives us useful information about all the features in the dataset. Among others, we can find the number of unique values in each column, various statistics for the numerical columns and the number of null values.

data.describe(include="all")

	age	workclass	fnlwgt	education	education-num	marital-status	occupation	relationship	race	sex	capital-gain	capital-loss	hours-per-week	native-country
count	48842.000000	46043	4.884200e+04	48842	48842.000000	48842	46033	48842	48842	48842	48842.000000	48842.000000	48842.000000	47985
unique	NaN	8	NaN	16	NaN	7	14	6	5	2	NaN	NaN	NaN	41
top	NaN	Private	NaN	HS-grad	NaN	Married-civ-spouse	Prof-specialty	Husband	White	Male	NaN	NaN	NaN	United-States
freq	NaN	33906	NaN	15784	NaN	22379	6172	19716	41762	32650	NaN	NaN	NaN	43832
mean	38.643585	NaN	1.896641e+05	NaN	10.078089	NaN	NaN	NaN	NaN	NaN	1079.067626	87.502314	40.422382	NaN
std	13.710510	NaN	1.056040e+05	NaN	2.570973	NaN	NaN	NaN	NaN	NaN	7452.019058	403.004552	12.391444	NaN
min	17.000000	NaN	1.228500e+04	NaN	1.000000	NaN	NaN	NaN	NaN	NaN	0.000000	0.000000	1.000000	NaN
25%	28.000000	NaN	1.175505e+05	NaN	9.000000	NaN	NaN	NaN	NaN	NaN	0.000000	0.000000	40.000000	NaN
50%	37.000000	NaN	1.781445e+05	NaN	10.000000	NaN	NaN	NaN	NaN	NaN	0.000000	0.000000	40.000000	NaN
75%	48.000000	NaN	2.376420e+05	NaN	12.000000	NaN	NaN	NaN	NaN	NaN	0.000000	0.000000	45.000000	NaN
max	90.000000	NaN	1.490400e+06	NaN	16.000000	NaN	NaN	NaN	NaN	NaN	99999.000000	4356.000000	99.000000	NaN

2.4 Exploring data with the skrub TableReport

Now, let’s create a TableReport to explore the dataset.

from skrub import TableReport
TableReport(data, verbose=0)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Tip

If you’re working from a python console rather than a Jupyter notebook (or equivalent), the TableReport must be opened explicitly:

TableReport(data).open()

2.4.1 Default view of the TableReport

The TableReport gives us a comprehensive overview of the dataset. The default view shows all the columns in the dataset, and allows to select and copy the content of the cells shown in the preview.

The TableReport is intended to show a preview of the data, so it does not contain all the rows in the dataset, rather it shows only the first and last few rows by default. Similarly, it stores only the top 10 most frequent values for each column, if column distributions are plotted.

2.4.2 The “Stats” tab

TableReport(data, open_tab="stats")

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

The “Stats” tab provides a variety of descriptive statistics for each column in the dataset. This includes:

• The column name
• The detected data type of the column
• Whether the column is sorted or not
• The number of null values in the column, as well as the percentage
• The number of unique values in the column

For numerical columns, additional statistics are provided:

• Mean
• Standard deviation
• Minimum and maximum values
• Median

Stat columns can also be sorted, for example to quickly identify which columns contain the most nulls, or have the largest cardinality (number of unique values).

2.4.3 The “Distributions” tab

TableReport(data, open_tab="distributions")

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

The “Distributions” tab provides visualizations of the distributions of values in each column. This includes histograms for numerical columns and bar plots for categorical columns.

The “Distributions” tab helps with detecting potential issues in the data, such as:

• Skewed distributions
• Outliers
• Unexpected value frequencies

For example, in this dataset we can see that some columns are heavily skewed, such as “workclass”, “race”, and “native-country”: this is important information to keep track of, because these columns may require special handling during data preprocessing or modeling.

Additionally, the “Distributions” tab allows to select columns manually, so that they can be added to a script and selected for further analysis or modeling.

CautionOutlier detection

The TableReport detects outliers using a simple interquartile test, marking as outliers all values that are beyond the IQR. This is a simple heuristic, and should not be treated as perfect. If your problem requires reliable outlier detection, you should not rely exclusively on what the TableReport shows.

2.4.4 The “Associations” tab

TableReport(data, open_tab="associations")

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

The “Associations” tab provides insights into the relationships between different columns in the dataset. It shows Pearson’s correlation coefficient for numerical columns, as well as Cramér’s V for all columns.

While this is a somewhat rough measure of association, it can help identify potential relationships worth exploring further during the analysis, and highlights highly correlated columns: depending on the modeling technique used, these may need to be handled specially to avoid issues with multicollinearity.

In this example, we can see that “education-num” and “education” have perfect correlation, which means that one of the two columns can be dropped without losing information.

2.4.5 Filtering columns

The TableReport includes various column filters to display only specific columns. Filters can select columns by dtype (for example, to show only numeric columns), or by other characteristics (like the number of unique values, or the presence of missing values).

It is also possible to create custom filters to select columns based on a specific use case:

my_filter = {"only_education": ["education", "education-num"]}

TableReport(data, column_filters=my_filter)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

Skrub selectors can be used for that (more on that later).

2.5 Exploring the target variable

Besides dataframes, the TableReport handles series and mono- and bi-dimensional numpy arrays.

So, let’s take a closer look at the target variable, which indicates whether an individual’s income exceeds $50K per year. We can create a separate TableReport for the target variable to explore its distribution:

TableReport(target)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

2.6 Working with big tables

Plotting and measuring the column correlations are expensive operations and may take a long time for large tables; in these cases it may be beneficial to disable these features when developing and enabling them only when the dataframe has been processed.

By default, both features are disabled when the given dataframe has more than 30 columns, but this behavior can be changed by setting the respective parameters to either True (to always plot/compute associations) or `False (to disable the features entirely):

TableReport(
    data, plot_distributions=False, compute_associations=False, open_tab="distributions"
)

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

It is also possible to change the column threshold from 30 to a different value using the skrub configuration and the table_report_plots_threshold and table_report_associations_threshold parameters:

from skrub import set_config

set_config(table_report_plots_threshold=10)

As with all other skrub configuration parameters, a new default value can be set by using the proper environemntal variables. More detail on the skrub configuration is reported in the User Guide.

2.7 Exporting the TableReport

The TableReport measures a number of statistics that can be used for more than just exploration: for example, they may be provided to other programs for alternative plotting, or shared outside of the starting notebook.

The entire report can be exported as a standalone HTML page that includes all the features:

TableReport(data).write_html("report.html")

Then, the report can be opened using any internet browser, with no need to run a Jupyter notebok or a python interactive console.

Alternatively, the report can be exported in JSON format: this allows to forward it to other programs for programmatic access to the statistics gathered by the report.

json_str = TableReport(data).json()

with open("report.json", "w") as fp:
    fp.write(json_str)

The JSON exported by the report will contain all the distribution plots in SVG format, which may not be necessary for some applications. To avoid exporting the plots, set plot_distributions=False.

Finally, the report can be exported in summarized form as a Markdown-formatted string, which can be useful to share as a message, printed on a command line, or fed to agents ask for insights.

md_str = TableReport(data).markdown()

with open("report.md", "w") as fp:
    fp.write(md_str)

Warning

The TableReport does not do any sanitization of the input data, and prints out column names and most frequent values as part of the output. Do not feed the content of the report to an agent if the dataset is large, or if its content is not trusted.

2.8 Replacing the default dataframe _repr_

It is possible to use the TableReport instead of the default dataframe representation used by Pandas and Polars with patch_display:

from skrub import patch_display, unpatch_display

# replace the default pandas repr 
patch_display()
data.head()

Processing column   1 / 14
Processing column   2 / 14
Processing column   3 / 14
Processing column   4 / 14
Processing column   5 / 14
Processing column   6 / 14
Processing column   7 / 14
Processing column   8 / 14
Processing column   9 / 14
Processing column  10 / 14
Processing column  11 / 14
Processing column  12 / 14
Processing column  13 / 14
Processing column  14 / 14

Please enable javascript

The skrub table reports need javascript to display correctly. If you are displaying a report in a Jupyter notebook and you see this message, you may need to re-execute the cell or to trust the notebook (button on the top right or "File > Trust notebook").

To disable, use unpatch_display:

unpatch_display()
data

	age	workclass	fnlwgt	education	education-num	marital-status	occupation	relationship	race	sex	capital-gain	capital-loss	hours-per-week	native-country
0	25	Private	226802	11th	7	Never-married	Machine-op-inspct	Own-child	Black	Male	0	0	40	United-States
1	38	Private	89814	HS-grad	9	Married-civ-spouse	Farming-fishing	Husband	White	Male	0	0	50	United-States
2	28	Local-gov	336951	Assoc-acdm	12	Married-civ-spouse	Protective-serv	Husband	White	Male	0	0	40	United-States
3	44	Private	160323	Some-college	10	Married-civ-spouse	Machine-op-inspct	Husband	Black	Male	7688	0	40	United-States
4	18	NaN	103497	Some-college	10	Never-married	NaN	Own-child	White	Female	0	0	30	United-States
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
48837	27	Private	257302	Assoc-acdm	12	Married-civ-spouse	Tech-support	Wife	White	Female	0	0	38	United-States
48838	40	Private	154374	HS-grad	9	Married-civ-spouse	Machine-op-inspct	Husband	White	Male	0	0	40	United-States
48839	58	Private	151910	HS-grad	9	Widowed	Adm-clerical	Unmarried	White	Female	0	0	40	United-States
48840	22	Private	201490	HS-grad	9	Never-married	Adm-clerical	Own-child	White	Male	0	0	20	United-States
48841	52	Self-emp-inc	287927	HS-grad	9	Married-civ-spouse	Exec-managerial	Wife	White	Female	15024	0	40	United-States

48842 rows × 14 columns

Warning

After patching, calling methods like df.head() will generate the report only on the relative few lines of the dataframe, thus making stats and distributions unreliable.

2.9 What we have seen in this chapter

In this chapter we have learned how the TableReport can be used to speed up data exploration, allowing us to find possible criticalities in the data.

We covered:

• Creating and configuring a TableReport for fast, interactive data exploration
• Exploring column statistics, value distributions, and associations visually
• Detecting nulls, outliers, and highly correlated columns at a glance
• Filtering columns by type or characteristics using built-in filters
• Saving and sharing reports as HTML, JSON, and Markdown files
• Adjusting TableReport settings for large datasets to optimize performance

In the next chapter, we will find out how to address some of the possible problems using the skrub Cleaner.