Release history#

Release 0.10.0#

New Features#

Transformers:

  • The SessionEncoder is now available. This encoder adds a session_id column, which groups together events that occur within the given session gap. Additionally, it is possible to provide a split_by column or list of columns (e.g., user ID or (user ID, user device)) to compute sessions for each grouping value. #1930 by Riccardo Cappuzzo.

  • The DropSimilar transformer has been added, for removing columns that present high correlation with other columns in a dataframe . #2023 by Eloi Massoulié.

  • ToFloat now allows users to specify decimal and thousand separators to parse numerical columns that use formatting different from the default formatting used in Python, such as 1'234,5. Additionally, negative numbers indicated with parentheses can be converted to the regular numeric format ((432) becomes -432). #1772 by Gabriela Gómez Jiménez.

  • The transformer DurationToFloat has been added. #2069 by Riccardo Cappuzzo.

TableReport:

Data Ops:

Misc:

Changes#

  • SkrubLearner’s SkrubLearner.score() has been enhanced when the DataOp used DataOp.skb.with_scoring(). During scoring, predict(), predict_proba() etc. are cached to avoid recomputation when multiple scorers are used (or one scorer calls them several times). Moreover it is possible to pass return_predictions=True to also retrieve any predictions that have been computed during scoring, in addition to the scores. Finally, in cases where we already have the predictions but want the result of score() without recomputing them, it is possible to provide them in the environment passed to score({..., "_skrub_predictions": {"predict_proba": ...}}). #2195 by Jérôme Dockès.

  • SkrubLearner.find_fitted_estimator() now supports searching for the apply node by ID or callable predicate as alternatives to the node name. #2194 by Jérôme Dockès.

  • The minimum required version of matplotlib has been increased from 3.4.3 to 3.6.1. #2159 by Riccardo Cappuzzo.

  • Gallery examples have been grouped into subject-specific sections. #2102 by Maureen Githaiga.

  • choose_from() now transparently converts outcomes to a list when it is another type of sequence. #2100 by aidbar.

  • An unnecessary warning that was raised when passing a numpy array to the TableVectorizer has been removed. #1908 by Sandrine Henry.

  • The error message displayed in the Associations tab of the TableReport has been improved for reports where only one column is present. #2094 by Alicja Kosak.

  • Added support for numpy arrays in DataOp.skb.concat(). #2096 by Ayesha Siddiqua.

Bugfixes#

  • A bug in how the TableVectorizer and Cleaner treated columns duration columns in pandas and polars has been fixed. Now, both classes convert durations to the total number of seconds (with fractional part). This is done by the new transformer DurationToFloat. #2069 by Riccardo Cappuzzo.

  • An error that could arise when running TableReport on dataframes containing double dollar ($$) signs has been fixed. #2154 by Katerina Michenina, CecilyTS, Eve Rabin.

  • MinHashEncoder with the default hashing="fast" now uses every n-gram size in ngram_range (the upper bound is inclusive, as documented and as already done by hashing="murmur"). Previously the largest size was dropped, so the default ngram_range=(2, 4) ignored 4-grams and a single-size range such as (3, 3) produced the same constant encoding for every string. #2168 by José Maia.

  • An error that happened when running TableReport or column_associations on some dataframes with non-string column names has been fixed in #2179 by Jérôme Dockès.

  • An error that could arise in histograms when running TableReport on data with a very small range (less than 10 representable floating-point numbers between min and max) has been fixed. #2189 by Jérôme Dockès.

Deprecations#

Release 0.9.0#

New Features#

Changes#

Bugfixes#

  • An error that could arise when calling score on a SkrubLearner that contains an inner transformer that has a score method has been fixed. #2052 by Jérôme Dockès.

Deprecations#

  • The parameter numeric_dtype in the Cleaner has been deprecated in favor of cast_to_float in #1910.

  • The parameter drop_if_unique of Cleaner and DropUninformative has been deprecated. #2040 by Riccardo Cappuzzo.

  • The parameters max_plot_columns and max_association_columns of the TableReport have been deprecated in favor of plot_distributions and compute_associations. #1907.

Release 0.8.0#

New Features#

  • The eager_data_ops configuration option has been added. When set to False, no previews are computed and validation is deferred until the DataOp is actually used (e.g. with .skb.eval()) rather than as soon as it is defined. This can make the definition of complex DataOps with many nodes faster (the overhead it removes typically becomes noticeable only in DataOps with 50-100 nodes or more). Moreover, the evaluation of large DataOps has also become faster. #1890 by Jérôme Dockès.

  • The reports produced by DataOp.skb.full_report() and SkrubLearner.report() now also display the values provided in the environment. #1920 by Jérôme Dockès.

  • SkrubLearner, ParamSearch and OptunaParamSearch expose some more attributes for inspection by scikit-learn: __sklearn_tags__, classes_, _estimator_type. #1931 by Jérôme Dockès.

  • It is now possible to pass additional (dynamically computed) arguments to the cross-validation splitter used by DataOp objects for validation, hyperparameter search etc. For example, the groups for a sklearn.model_selection.GroupKFold can be computed as part of the DataOp evaluation and used for splitting. This is achieved by passing the splitter and its arguments to DataOp.skb.mark_as_X(). #1943 by Jérôme Dockès.

  • selectors.has_nulls() now takes a proportion parameter, which allows selecting columns that have a fraction of null values above the given threshold. #1881 by Gabriela Gómez Jiménez.

  • Added a new dataset, fetch_electricity_usage(), which contains electricity usage data for several French cities and corresponding weather data. #2013 by Lisa McBride.

Changes#

  • Increased the minimum version of polars from 0.20 to 1.5.0. #1897 by Riccardo Cappuzzo.

  • ApplyToCols and ApplyToFrame have been merged into a single class, ApplyToCols,that covers the functionality of both the old classes by detecting automatically whether the provided transformer should be applied independently on each column, or on all selected columns as a single dataframe. As a result, ApplyToCols and ApplyToFrame have been removed. #1913, #1919 and #1962 by Riccardo Cappuzzo.

  • The dataset fetcher functions now include a “path” field for each table in the dataset. For example, the dataset “employee_salaries” now has the field employee_salaries_path. Additionally, datasets that include a single table have the field path. These fields contain the paths to the datasets stored in the skrub_data folder. The default skrub_data folder can now be set in the skrub configuration and by setting the SKB_DATA_DIRECTORY environment variable. The environment variable SKRUB_DATA_DIRECTORY is deprecated and will be removed in a future version of skrub. #1852 by Riccardo Cappuzzo. Examples in the gallery have been updated accordingly in #1940 and #1964 by MuditAtrey.

  • SingleColumnTransformer and associated exception RejectColumn (used internally by many skrub estimators) have been added to the public API, in the newly-created skrub.core module. #1851 by Eloi Massoulié.

  • Added the strings "None" and "none" to the list of null string values in Cleaner. Also, exposed the list of null string values that will be set to null by the Cleaner as the parameter null_strings. #1952 and #1954 by Lisa McBride.

  • The configuration parameter “use_table_report” has been removed from the skrub configuration. Use patch_display() instead. #1973 by Riccardo Cappuzzo.

  • Updated how the column_filters parameter of TableReport works. It now accepts a dictionary where the key is the display name for the dropdown menu, and the value is a filter of the columns that will be displayed. Accepts either a list of column indices, a list of column names or an instance of the Selector. #1976 by Lisa McBride.

  • The overplotting of the counts atop the vertical histogram bars in the TableReport has been removed due to formatting issues. #1984 by Lisa McBride.

  • The maximum number of associations that can be displayed in the TableReport has been increased to N=1000, and the associations are now displayed in a scrollable table. #1992 by Lisa McBride.

Bug Fixes#

  • The TableVectorizer now correctly handles the case where one of the provided encoders is a scikit-learn Pipeline that starts with a skrub single-column transformer. #1899 by Jérôme Dockès and #1900 by Jérôme Dockès.

  • Errors raised when a polars LazyFrame is passed where an eager DataFrame is expected are now clearer. #1916 by Jérôme Dockès.

  • DataOp.skb.cross_validate() would raise an error when passed return_indices=True. Now it returns the train and test indices of each fold in the train_indices and test_indices columns of the result dataframe. #1953 by Jérôme Dockès.

  • Polars LazyFrames are no longer collected automatically anywhere in the library; a TypeError is now raised instead. #1941 by Mudit Atrey.

Release 0.7.2#

Changes#

Bugfixes#

Release 0.7.1#

New features#

  • A new dataset, fetch_california_housing(), has been added to the skrub.datasets module. It allows to get a redundancy copy of the scikit-learn fetch_california_housing() function. #1830 by Guillaume Lemaitre.

Bugfixes#

  • DropCols and SelectCols: attributes were renamed to end with an underscore, in order to follow a scikit-learn convention which is used to determine if an estimator is fitted. #1813 by Auguste Baum.

Release 0.7.0#

New features#

Changes#

Bugfixes#

Release 0.6.2#

New features#

Changes#

Bugfixes#

  • Fixed a compatibility bug with Polars 1.32.3 that may cause ToFloat32 to fail when applied to categorical columns. #1570 by Riccardo Cappuzzo.

  • Fixed the display of DataOp objects in google colab cell outputs (no output was displayed). #1590 by Jérôme Dockès.

  • Fixed an error that occurred when using .skb.concat with a pandas dataframe with column names that aren’t strings. #1594 by Riccardo Cappuzzo.

  • Fixed the range from which choose_float() and choose_int() sample values when log=False and n_steps is None. It was between low and low + high, now it is between low and high. #1603 by Jérôme Dockès.

  • DataOp hyperparameter search would raise an error when doing classification and using the scoring parameter, when the dataop contained no variables. Fixed in #1601 by Jérôme Dockès.

  • SkrubLearner used to do a prediction on the train set during fit(), this has been fixed. #1610 by Jérôme Dockès.

  • DataOp would raise errors when containing subclasses of list, tuple or dict that cannot be initialized with an instance of the builtin type (such as classes created by collections.namedtuple), this has been fixed. DataOps now only recurse into the builtin collections to evaluate their items (not into their subclasses). If you need the items evaluated (ie if they contain DataOps or Choices), store them in one of the builtin collections. #1612 by Jérôme Dockès.

  • SkrubLearner.report() with mode="fit" used to display the dataops themselves, rather than their outputs, in the report. This has been fixed in #1623 by Jérôme Dockès.

  • Fixed a bug that happened when get_feature_names_out was called on instances of the DatetimeEncoder. #1622 by Riccardo Cappuzzo.

Release 0.6.1#

Bugfixes#

  • get_feature_names_out now works correctly when used by GapEncoder, DropCols, SelectCols: from within a scikit-learn Pipeline. In addition, DropCols’s get_feature_names_out method now returns the names of the columns that are not dropped, rather than the names of the columns that are dropped. #1543 by Riccardo Cappuzzo.

Release 0.6.0#

Highlights#

  • Major feature! Skrub DataOps are a powerful new way of combining dataframe transformations over multiple tables, and machine learning pipelines. DataOps can be combined to form compled data plans, that can be used to train and tune machine learning models. Then, the DataOps plans can be exported as Learners (skrub.SkrubLearner), standalone objects that can be used on new data. More detail about the DataOps can be found in the User guide and in the examples.

  • The TableReport has been improved with many new features. Series are now supported directly. It is now possible to skip computing column associations and generating plots when the number of columns in the dataframe exceeds a user-defined threshold. Columns with high cardinality and sorted columns are now highlighted in the report.

  • selectors, ApplyToCols and ApplyToFrame are now available, providing utilities for selecting columns to which a transformer should be applied in a flexible way. For more details, see the User guide and the example.

  • The SquashingScaler has been added: it robustly rescales and smoothly clips numeric columns, enabling more robust handling of numeric columns with neural networks. See the example

New features#

  • The skrub DataOps are new mechanism for building machine-learning pipelines that handle multiple tables and easily describing their hyperparameter spaces. Main PR: #1233 by Jérôme Dockès. Additional work from other contributors can be found here: Vincent Maladiere provided very important help by trying the DataOps on many use-cases and datasets, providing feedback and suggesting improvements, improving the examples (including creating all the figures in the examples) and adding jitter to the parallel coordinate plots, Riccardo Cappuzzo experimented with the DataOps, suggested improvements and improved the examples, Gaël Varoquaux , Guillaume Lemaitre, Adrin Jalali, Olivier Grisel and others participated through many discussions in defining the requirements and the public API. See the examples for an introduction.

  • The selectors module provides utilities for selecting columns to which a transformer should be applied in a flexible way. The module was created in #895 by Jérôme Dockès and added to the public API in #1341 by Jérôme Dockès.

  • The DropUninformative transformer is now available. This transformer employs different heuristics to detect columns that are not likely to bring useful information for training a model. The current implementation includes detection of columns that contain only a single value (constant columns), only missing values, or all unique values (such as IDs). #1313 by Riccardo Cappuzzo.

  • get_config(), set_config() and config_context() are now available to configure settings for dataframes display and expressions. patch_display() and unpatch_display() are deprecated and will be removed in the next release of skrub. #1427 by Vincent Maladiere. The global configuration includes the parameter cardinality_threshold that controls the threshold value used to warn user if they have high cardinality columns in their dataset. #1498 by rouk1. Additionally, the parameter float_precision controls the number of significant digits displayed for floating-point values in reports. #1470 by George S.

  • Added the SquashingScaler, a transformer that robustly rescales and smoothly clips numeric columns, enabling more robust handling of numeric columns with neural networks. #1310 by Vincent Maladiere and David Holzmüller.

  • datasets.toy_order() is now available to create a toy dataframe and corresponding targets for examples. #1485 by Antoine Canaguier-Durand.

  • ApplyToCols and ApplyToFrame are now available to apply transformers on a set of columns independently and jointly respectively. #1478 by Vincent Maladiere.

Changes#

Warning

The default high cardinality encoder for both TableVectorizer and tabular_learner() (now tabular_pipeline()) has been changed from GapEncoder to StringEncoder. #1354 by Riccardo Cappuzzo.

Bugfixes#

Documentation#

A large number of improvements to the examples, docstrings, and the documentation website have been made. Contributors include Vincent Maladiere, Riccardo Cappuzzo, Jérôme Dockès, Gael Varoquaux, Gabriela Gómez Jiménez, Sylvain Combettes, Frits Hermans, Vitor Pohlenz, Arturo Amor Quiroz, Marie Sacksick, Emilien Battel, George El Haber, Antoine Canaguier-Durand, and Lionel Kusch.

Release 0.5.4#

Maintenance#

Release 0.5.3#

Changes#

  • The SimpleCleaner has been renamed to Cleaner. Use of the name SimpleCleaner is deprecated and will result in an error in some future release of skrub. #1275 by Riccardo Cappuzzo.

  • A new parameter max_plot_columns has been added to the TableReport and patch_display() to skip column plots when the number of columns exceeds the specified value. #1255 by Priscilla Baah.

Release 0.5.2#

New features#

Changes#

  • The estimator returned by tabular_learner() now uses spline encoding of datetime features when the supervised learner is not a model based on decision trees such as random forests or gradient boosting. #1264 by Guillaume Lemaitre.

  • The “distribution” tab of the TableReport now stacks cards horizontally to avoid adding vertical space. #1259 by Gaël Varoquaux

  • Progress messages when generating a TableReport are now written to stderr instead of stdout. #1236 by Priscilla Baah

  • Optimize the StringEncoder: lower memory footprint and faster execution in some cases. #1248 by Gaël Varoquaux

Bug fixes#

Release 0.5.1#

New features#

Changes#

  • New dataset fetching methods have been added: fetch_videogame_sales(), fetch_bike_sharing(), fetch_flight_delays(), fetch_country_happiness(), and removed fetch_road_safety(). #1218 by Vincent Maladiere

Bug fixes#

Maintenance#

Release 0.4.1#

Changes#

Bug fixes#

  • Generating a TableReport could have an effect on the matplotib configuration which could cause plots not to display inline in jupyter notebooks any more. This has been fixed in skrub in #1172 by Jérôme Dockès and the matplotlib issue can be tracked here.

  • The labels on bar plots in the TableReport for columns of object dtypes that have a repr spanning multiple lines could be unreadable. This has been fixed in #1196 by Jérôme Dockès.

  • Improve the performance of deduplicate() by removing some unnecessary computations. #1193 by Jérôme Dockès.

Maintenance#

Release 0.4.0#

Highlights#

  • The TextEncoder can extract embeddings from a string column with a deep learning language model (possibly downloaded from the HuggingFace Hub).

  • Several improvements to the TableReport such as better support for other scripts than the latin alphabet in the bar plot labels, smaller report sizes, clipping the outliers to better see the details of distributions in histograms. See the full changelog for details.

  • The TableVectorizer can now drop columns that contain a fraction of null values above a user-chosen threshold.

New features#

Major changes#

Minor changes#

  • The column filter selection dropdown in the tablereport is smaller and its label has been removed to save space. #1107 by Jérôme Dockès.

  • The TableReport now uses the font size of its parent element when inserted into another page. This makes it smaller in pages that use a smaller font size than the browser default such as VSCode in some configurations. It also makes it easier to control its size when inserting it in a web page by setting the font size of its parent element. A few other small adjustments have also been made to make it a bit more compact. #1098 by Jérôme Dockès.

  • Display of labels in the plots of the TableReport, especially for other scripts than the latin alphabet, has improved.

    • before, some characters could be missing and replaced by empty boxes.

    • before, when the text is truncated, the ellipsis “…” could appear on the wrong side for right-to-left scripts.

    Moreover, when the text contains line breaks it now appears all on one line. Note this only affects the labels in the plots; the rest of the report did not have these problems. #1097 by Jérôme Dockès and #1138 by Jérôme Dockès.

  • In the TableReport it is now possible, before clicking any of the cells, to reach the dataframe sample table and activate a cell with tab key navigation. #1101 by Jérôme Dockès.

  • The “Column name” column of the “summary statistics” table in the TableReport is now always visible when scrolling the table. #1102 by Jérôme Dockès.

  • Added parameter drop_null_fraction to TableVectorizer to drop columns based on whether they contain a fraction of nulls larger than the given threshold. #1115 and #1149 by Riccardo Cappuzzo.

  • The TableReport now provides more helpful output for columns of dtype TimeDelta / Duration. #1152 by Jérôme Dockès.

  • The TableReport now also reports the number of unique values for numeric columns. #1154 by Jérôme Dockès.

  • The TableReport, when plotting histograms, now detects outliers and clips the range of data shown in the histogram. This allows seeing more detail in the shown distribution. #1157 by Jérôme Dockès.

Bug fixes#

  • The TableReport could raise an exception when one of the columns contained datetimes with time zones and missing values; this has been fixed in #1114 by Jérôme Dockès.

  • In scikit-learn versions older than 1.4 the TableVectorizer could fail on polars dataframes when used with the default parameters. This has been fixed in #1122 by Jérôme Dockès.

  • The TableReport would raise an exception when the input (pandas) dataframe contained several columns with the same name. This has been fixed in #1125 by Jérôme Dockès.

  • The TableReport would raise an exception when a column contained infinite values. This has been fixed in #1150 by Jérôme Dockès and #1151 by Jérôme Dockès.

Release 0.3.1#

Minor changes#

  • For tree-based models, tabular_learner() now adds handle_unknown='use_encoded_value' to the OrdinalEncoder, to avoid errors with new categories in the test set. This is consistent with the setting of OneHotEncoder used by default in the TableVectorizer. #1078 by Gaël Varoquaux

  • The reports created by TableReport, when inserted in an html page (or displayed in a notebook), now use the same font as the surrounding page. #1038 by Jérôme Dockès.

  • The content of the dataframe corresponding to the currently selected table cell in the TableReport can be copied without actually selecting the text (as in a spreadsheet). #1048 by Jérôme Dockès.

  • The selection of content displayed in the TableReport’s copy-paste boxes has been removed. Now they always display the value of the selected item. When copied, the repr of the selected item is copied to the clipboard. #1058 by Jérôme Dockès.

  • A “stats” panel has been added to the TableReport, showing summary statistics for all columns (number of missing values, mean, etc. – similar to pandas.info() ) in a table. It can be sorted by each column. #1056 and #1068 by Jérôme Dockès.

  • The credit fraud dataset is now available with the fetch_credit_fraud function(). #1053 by Vincent Maladiere.

  • Added zero padding for column names in MinHashEncoder to improve column ordering consistency. #1069 by Shreekant Nandiyawar.

  • The selection in the TableReport’s sample table can now be manipulated with the keyboard. #1065 by Jérôme Dockès.

  • The TableReport now displays the pandas (multi-)index, and has a better display & interaction of pandas columns when the columns are a MultiIndex. #1083 by Jérôme Dockès.

  • It is possible to control the number of rows displayed by the TableReport in the “sample” tab panel by specifying n_rows. #1083 by Jérôme Dockès.

  • the TableReport used to raise an exception when the dataframe contained unhashable types such as python lists. This has been fixed in #1087 by Jérôme Dockès.

  • Display’s columns name with the HTML representation of the fitted TableVectorizer. This has been fixed in #1093 by Shreekant Nandiyawar.

  • AggTarget will now work even when y is a Series and not raise any error. This has been fixed in #1094 by Shreekant Nandiyawar.

Release 0.3.0#

Highlights#

  • Polars dataframes are now supported across all skrub estimators.

  • TableReport generates an interactive report for a dataframe. This page regroups some precomputed examples.

Major changes#

  • The InterpolationJoiner now supports polars dataframes. #1016 by Théo Jolivet.

  • The TableReport provides an interactive report on a dataframe’s contents: an overview, summary statistics and plots, statistical associations between columns. It can be displayed in a jupyter notebook, a browser tab or saved as a static HTML page. #984 by Jérôme Dockès.

Minor changes#

  • Joiner and fuzzy_join() used to raise an error when columns with the same name appeared in the main and auxiliary table (after adding the suffix). This is now allowed and a random string is inserted in the duplicate column to ensure all names are unique. #1014 by Jérôme Dockès.

  • AggJoiner and AggTarget could produce outputs whose column names varied across calls to transform in some cases in the presence of duplicate column names, now the output names are always the same. #1013 by Jérôme Dockès.

  • In some cases AggJoiner and AggTarget inserted a column in the output named “index” containing the pandas index of the auxiliary table. This has been corrected. #1020 by Jérôme Dockès.

Release 0.2.0#

Major changes#

  • The Joiner has been adapted to support polars dataframes. #945 by Théo Jolivet.

  • The TableVectorizer now consistently applies the same transformation across different calls to transform. There also have been some breaking changes to its functionality: (i) all transformations are now applied independently to each column, i.e. it does not perform multivariate transformations (ii) in specific_transformers the same column may not be used twice (go through 2 different transformers). #902 by Jérôme Dockès.

  • Some parameters of TableVectorizer have been renamed: high_cardinality_transformer → high_cardinality, low_cardinality_transformer → low_cardinality, datetime_transformer → datetime, numeric_transformer → numeric. #947 by Jérôme Dockès.

  • The GapEncoder and MinHashEncoder are now a single-column transformers: their fit, fit_transform and transform methods accept a single column (a pandas or polars Series). Dataframes and numpy arrays are not accepted. #920 and #923 by Jérôme Dockès.

  • Added the MultiAggJoiner that allows to augment a main table with multiple auxiliary tables. #876 by Théo Jolivet.

  • AggJoiner now only accepts a single table as an input, and some of its parameters were renamed to be consistent with the MultiAggJoiner. It now has a key` parameter that allows to join main and auxiliary tables that share the same column names. #876 by Théo Jolivet.

  • tabular_learner() has been added to easily create a supervised learner that works well on tabular data. #926 by Jérôme Dockès.

Minor changes#

  • GapEncoder and MinHashEncoder used to modify their input in-place, replacing missing values with a string. They no longer do so. Their parameter handle_missing has been removed; now missing values are always treated as the empty string. #930 by Jérôme Dockès.

  • The minimum supported python version is now 3.9 #939 by Jérôme Dockès.

  • Skrub supports numpy 2. #946 by Jérôme Dockès.

  • fetch_ken_embeddings() now add suffix even with the default value for the parameter pca_components. #956 by Guillaume Lemaitre.

  • Joiner now performs some preprocessing (the same as done by the TableVectorizer, eg trying to parse dates, converting pandas object columns with mixed types to a single type) on the joining columns before vectorizing them. #972 by Jérôme Dockès.

skrub release 0.1.1#

This is a bugfix release to adapt to the most recent versions of pandas (2.2) and scikit-learn (1.5). There are no major changes to the functionality of skrub.

skrub release 0.1.0#

Major changes#

Minor changes#

  • Dataset fetcher datasets.fetch_employee_salaries() now has a parameter overload_job_titles to allow overloading the job titles (employee_position_title) with the column underfilled_job_title, which provides some more information about the job title. #581 by Lilian Boulard

  • Fix bugs which was triggered when extract_until was “year”, “month”, “microseconds” or “nanoseconds”, and add the option to set it to None to only extract total_time, the time from epoch. DatetimeEncoder. #743 by Leo Grinsztajn

Before skrub: dirty_cat#

Skrub was born from the dirty_cat package.

Dirty-cat release 0.4.1#

Major changes#

Minor changes#

  • Improvement of date column detection and date format inference in TableVectorizer. The format inference now tries to find a format which works for all non-missing values of the column, and only tries pandas default inference if it fails. #543 by Leo Grinsztajn #587 by Leo Grinsztajn

Dirty-cat Release 0.4.0#

Major changes#

  • SuperVectorizer is renamed as TableVectorizer, a warning is raised when using the old name. #484 by Jovan Stojanovic

  • New experimental feature: joining tables using fuzzy_join() by approximate key matching. Matches are based on string similarities and the nearest neighbors matches are found for each category. #291 by Jovan Stojanovic and Leo Grinsztajn

  • New experimental feature: FeatureAugmenter, a transformer that augments with fuzzy_join() the number of features in a main table by using information from auxiliary tables. #409 by Jovan Stojanovic

  • Unnecessary API has been made private: everything (files, functions, classes) starting with an underscore shouldn’t be imported in your code. #331 by Lilian Boulard

  • The MinHashEncoder now supports a n_jobs parameter to parallelize the hashes computation. #267 by Leo Grinsztajn and Lilian Boulard.

  • New experimental feature: deduplicating misspelled categories using deduplicate() by clustering string distances. This function works best when there are significantly more duplicates than underlying categories. #339 by Moritz Boos.

Minor changes#

  • Add example Wikipedia embeddings to enrich the data. #487 by Jovan Stojanovic

  • datasets.fetching: contains a new function get_ken_embeddings() that can be used to download Wikipedia embeddings and filter them by type.

  • datasets.fetching: contains a new function fetch_world_bank_indicator() that can be used to download indicators from the World Bank Open Data platform. #291 by Jovan Stojanovic

  • Removed example Fitting scalable, non-linear models on data with dirty categories. #386 by Jovan Stojanovic

  • MinHashEncoder’s minhash() method is no longer public. #379 by Jovan Stojanovic

  • Fetching functions now have an additional argument directory, which can be used to specify where to save and load from datasets. #432 by Lilian Boulard

  • Fetching functions now have an additional argument directory, which can be used to specify where to save and load from datasets. #432 and #453 by Lilian Boulard

  • The TableVectorizer’s default OneHotEncoder for low cardinality categorical variables now defaults to handle_unknown="ignore" instead of handle_unknown="error" (for sklearn >= 1.0.0). This means that categories seen only at test time will be encoded by a vector of zeroes instead of raising an error. #473 by Leo Grinsztajn

Bug fixes#

Dirty-cat Release 0.3.0#

Major changes#

  • New encoder: DatetimeEncoder can transform a datetime column into several numeric columns (year, month, day, hour, minute, second, …). It is now the default transformer used in the TableVectorizer for datetime columns. #239 by Leo Grinsztajn

  • The TableVectorizer has seen some major improvements and bug fixes:

    • Fixes the automatic casting logic in transform.

    • To avoid dimensionality explosion when a feature has two unique values, the default encoder (OneHotEncoder) now drops one of the two vectors (see parameter drop="if_binary").

    • fit_transform and transform can now return unencoded features, like the ColumnTransformer’s behavior. Previously, a RuntimeError was raised.

    #300 by Lilian Boulard

  • Backward-incompatible change in the TableVectorizer: To apply remainder to features (with the *_transformer parameters), the value 'remainder' must be passed, instead of None in previous versions. None now indicates that we want to use the default transformer. #303 by Lilian Boulard

  • Support for Python 3.6 and 3.7 has been dropped. Python >= 3.8 is now required. #289 by Lilian Boulard

  • Bumped minimum dependencies:

  • Dropped support for Jaro, Jaro-Winkler and Levenshtein distances.

Notes#

Dirty-cat Release 0.2.2#

Bug fixes#

Dirty-cat Release 0.2.1#

Major changes#

Bug-fixes#

Notes#

Dirty-cat Release 0.2.0#

Also see pre-release 0.2.0a1 below for additional changes.

Major changes#

  • Bump minimum dependencies:

  • datasets.fetching - backward-incompatible changes to the example datasets fetchers:

    • The backend has changed: we now exclusively fetch the datasets from OpenML. End users should not see any difference regarding this.

    • The frontend, however, changed a little: the fetching functions stay the same but their return values were modified in favor of a more Pythonic interface. Refer to the docstrings of functions dirty_cat.datasets.fetch_* for more information.

    • The example notebooks were updated to reflect these changes. #155 by Lilian Boulard

  • Backward incompatible change to MinHashEncoder: The MinHashEncoder now only supports two dimensional inputs of shape (N_samples, 1). #185 by Lilian Boulard and Alexis Cvetkov.

  • Update handle_missing parameters:

    • GapEncoder: the default value “zero_impute” becomes “empty_impute” (see doc).

    • MinHashEncoder: the default value “” becomes “zero_impute” (see doc).

    #210 by Alexis Cvetkov.

  • Add a method “get_feature_names_out” for the GapEncoder and the TableVectorizer, since get_feature_names will be depreciated in scikit-learn 1.2. #216 by Alexis Cvetkov

Notes#

  • Removed hard-coded CSV file dirty_cat/data/FiveThirtyEight_Midwest_Survey.csv.

  • Improvements to the TableVectorizer

    • Missing values are not systematically imputed anymore

    • Type casting and per-column imputation are now learnt during fitting

    • Several bugfixes

    #201 by Lilian Boulard

Dirty-cat Release 0.2.0a1#

Version 0.2.0a1 is a pre-release. To try it, you have to install it manually using:

pip install --pre dirty_cat==0.2.0a1

or from the GitHub repository:

pip install git+https://github.com/dirty-cat/dirty_cat.git

Major changes#

  • Bump minimum dependencies:

    • Python (>= 3.6)

    • NumPy (>= 1.16)

    • SciPy (>= 1.2)

    • scikit-learn (>= 0.20.0)

  • TableVectorizer: Added automatic transform through the TableVectorizer class. It transforms columns automatically based on their type. It provides a replacement for scikit-learn’s ColumnTransformer simpler to use on heterogeneous pandas DataFrame. #167 by Lilian Boulard

  • Backward incompatible change to GapEncoder: The GapEncoder now only supports two-dimensional inputs of shape (n_samples, n_features). Internally, features are encoded by independent GapEncoder models, and are then concatenated into a single matrix. #185 by Lilian Boulard and Alexis Cvetkov.

Bug-fixes#

Dirty-cat Release 0.1.1#

Major changes#

Bug-fixes#

Dirty-cat Release 0.1.0#

Major changes#

  • GapEncoder: Added online Gamma-Poisson factorization through the GapEncoder class. This method discovers latent categories formed via combinations of substrings, and encodes string data as combinations of these categories. To be used if interpretability is important. #153 by Alexis Cvetkov

Bug-fixes#

Dirty-cat Release 0.0.7#

  • MinHashEncoder: Added minhash_encoder.py and fast_hast.py files that implement minhash encoding through the MinHashEncoder class. This method allows for fast and scalable encoding of string categorical variables.

  • datasets.fetch_employee_salaries: change the origin of download for employee_salaries.

    • The function now return a bunch with a dataframe under the field “data”, and not the path to the csv file.

    • The field “description” has been renamed to “DESCR”.

  • SimilarityEncoder: Fixed a bug when using the Jaro-Winkler distance as a similarity metric. Our implementation now accurately reproduces the behaviour of the python-Levenshtein implementation.

  • SimilarityEncoder: Added a handle_missing attribute to allow encoding with missing values.

  • TargetEncoder: Added a handle_missing attribute to allow encoding with missing values.

  • MinHashEncoder: Added a handle_missing attribute to allow encoding with missing values.

Dirty-cat Release 0.0.6#

  • SimilarityEncoder: Accelerate SimilarityEncoder.transform, by:

    • computing the vocabulary count vectors in fit instead of transform

    • computing the similarities in parallel using joblib. This option can be turned on/off via the n_jobs attribute of the SimilarityEncoder.

  • SimilarityEncoder: Fix a bug that was preventing a SimilarityEncoder to be created when categories was a list.

  • SimilarityEncoder: Set the dtype passed to the ngram similarity to float32, which reduces memory consumption during encoding.

Dirty-cat Release 0.0.5#

  • SimilarityEncoder: Change the default ngram range to (2, 4) which performs better empirically.

  • SimilarityEncoder: Added a most_frequent strategy to define prototype categories for large-scale learning.

  • SimilarityEncoder: Added a k-means strategy to define prototype categories for large-scale learning.

  • SimilarityEncoder: Added the possibility to use hashing ngrams for stateless fitting with the ngram similarity.

  • SimilarityEncoder: Performance improvements in the ngram similarity.

  • SimilarityEncoder: Expose a get_feature_names method.