.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/05_deduplication.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code or to run this example in your browser via JupyterLite or Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_05_deduplication.py: .. _examples_deduplication: =================================== Deduplicating misspelled categories =================================== Real world datasets often come with misspellings, for instance in manually inputted categorical variables. Such misspelling break data analysis steps that require exact matching, such as a ``GROUP BY`` operation. Merging multiple variants of the same category is known as *deduplication*. It is implemented in skrub with the |deduplicate| function. Deduplication relies on *unsupervised learning*. It finds structures in the data without providing a-priori known and explicit labels/categories. Specifically, measuring the distance between strings can be used to find clusters of strings that are similar to each other (e.g. differ only by a misspelling) and hence, flag and regroup potentially misspelled category names in an unsupervised manner. .. |deduplicate| replace:: :func:`~skrub.deduplicate` .. |Gap| replace:: :class:`~skrub.GapEncoder` .. |MinHash| replace:: :class:`~skrub.MinHashEncoder` .. GENERATED FROM PYTHON SOURCE LINES 35-49 A typical use case ------------------ Let's take an example: as a data scientist, your job is to analyze the data from a hospital ward. In the data, we notice that in most cases, the doctor prescribes one of three following medications: "Contrivan", "Genericon" or "Zipholan". However, data entry is manual and - either because the doctor's handwriting was hard to decipher, or due to mistakes during input - there are multiple spelling mistakes in the dataset. Let's generate this example dataset: .. GENERATED FROM PYTHON SOURCE LINES 49-63 .. code-block:: Python import pandas as pd import numpy as np from skrub.datasets import make_deduplication_data duplicated_names = make_deduplication_data( examples=["Contrivan", "Genericon", "Zipholan"], # our three medication names entries_per_example=[500, 100, 1500], # their respective number of occurrences prob_mistake_per_letter=0.05, # 5% probability of typo per letter random_state=42, # set seed for reproducibility ) duplicated_names[:5] .. rst-class:: sphx-glr-script-out .. code-block:: none ['Contrivan', 'Cvntrivan', 'Contrivan', 'Coqtrivan', 'Contriian'] .. GENERATED FROM PYTHON SOURCE LINES 64-66 We then extract the unique medication names in the data and visualize how often they appear: .. GENERATED FROM PYTHON SOURCE LINES 66-77 .. code-block:: Python import matplotlib.pyplot as plt unique_examples, counts = np.unique(duplicated_names, return_counts=True) plt.figure(figsize=(10, 15)) plt.barh(unique_examples, counts) plt.ylabel("Medication name") plt.xlabel("Count") plt.show() .. image-sg:: /auto_examples/images/sphx_glr_05_deduplication_001.png :alt: 05 deduplication :srcset: /auto_examples/images/sphx_glr_05_deduplication_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 78-87 We clearly see the structure of the data: the three original medications ("Contrivan", "Genericon" and "Zipholan") are the most common ones, but there are many spelling mistakes or slight variations of the original names. The idea behind |deduplicate| is to use the fact that the string distance of misspelled medications will be closest to their original (most frequent) medication name - and therefore form clusters. .. GENERATED FROM PYTHON SOURCE LINES 89-96 Deduplication: suggest corrections of misspelled names ------------------------------------------------------ The |deduplicate| function uses clustering based on string similarities to group duplicated names. Let's deduplicate our data: .. GENERATED FROM PYTHON SOURCE LINES 96-103 .. code-block:: Python from skrub import deduplicate deduplicated_data = deduplicate(duplicated_names) deduplicated_data[:5] .. rst-class:: sphx-glr-script-out .. code-block:: none Contrivan Contrivan Cvntrivan Contrivan Contrivan Contrivan Coqtrivan Contrivan Contriian Contrivan dtype: object .. GENERATED FROM PYTHON SOURCE LINES 104-112 And that's it! We now have the deduplicated data. .. topic:: Note: The number of clusters will need some adjustment depending on the data. If no fixed number of clusters is given, |deduplicate| tries to set it automatically via the `silhouette score `_. .. GENERATED FROM PYTHON SOURCE LINES 114-115 We can visualize the distribution of categories in the deduplicated data: .. GENERATED FROM PYTHON SOURCE LINES 115-127 .. code-block:: Python deduplicated_unique_examples, deduplicated_counts = np.unique( deduplicated_data, return_counts=True ) deduplicated_series = pd.Series(deduplicated_counts, index=deduplicated_unique_examples) plt.figure(figsize=(10, 5)) plt.barh(deduplicated_unique_examples, deduplicated_counts) plt.xlabel("Count") plt.ylabel("Medication name") plt.show() .. image-sg:: /auto_examples/images/sphx_glr_05_deduplication_002.png :alt: 05 deduplication :srcset: /auto_examples/images/sphx_glr_05_deduplication_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 128-135 Here, the silhouette score finds the ideal number of clusters (3) and groups the spelling mistakes. In practice, the translation/deduplication will often be imperfect and require some tweaks. In this case, we can construct and update a translation table based on the data returned by |deduplicate|. .. GENERATED FROM PYTHON SOURCE LINES 135-144 .. code-block:: Python # create a table that maps original to corrected categories translation_table = pd.Series(deduplicated_data, index=duplicated_names) # remove duplicates in the original data translation_table = translation_table[~translation_table.index.duplicated(keep="first")] translation_table.head() .. rst-class:: sphx-glr-script-out .. code-block:: none Contrivan Contrivan Cvntrivan Contrivan Coqtrivan Contrivan Contriian Contrivan Contaivan Contrivan dtype: object .. GENERATED FROM PYTHON SOURCE LINES 145-149 In this table, we have the category name on the left, and the cluster it was translated to on the right. If we want to adapt the translation table, we can modify it manually. .. GENERATED FROM PYTHON SOURCE LINES 151-158 Visualizing string pair-wise distance between names --------------------------------------------------- Below, we use a heatmap to visualize the pairwise-distance between medication names. A darker color means that two medication names are closer together (i.e. more similar), a lighter color means a larger distance. .. GENERATED FROM PYTHON SOURCE LINES 158-173 .. code-block:: Python from skrub import compute_ngram_distance from scipy.spatial.distance import squareform ngram_distances = compute_ngram_distance(unique_examples) square_distances = squareform(ngram_distances) import seaborn as sns fig, ax = plt.subplots(figsize=(14, 12)) sns.heatmap( square_distances, yticklabels=unique_examples, xticklabels=unique_examples, ax=ax ) plt.show() .. image-sg:: /auto_examples/images/sphx_glr_05_deduplication_003.png :alt: 05 deduplication :srcset: /auto_examples/images/sphx_glr_05_deduplication_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 174-176 We have three clusters appearing - the original medication names and their misspellings that form a cluster around them. .. GENERATED FROM PYTHON SOURCE LINES 178-190 Conclusion ---------- In this example, we have seen how to use the |deduplicate| function to automatically detect and correct misspelled category names. Note that deduplication is especially useful when we either know our ground truth (e.g. the original medication names), or when the similarity across strings does not carry useful information for our machine learning task. Otherwise, we prefer using encoding methods such as |Gap| or |MinHash|. .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 6.062 seconds) .. _sphx_glr_download_auto_examples_05_deduplication.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.1.0?urlpath=lab/tree/notebooks/auto_examples/05_deduplication.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../lite/lab/?path=auto_examples/05_deduplication.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 05_deduplication.ipynb <05_deduplication.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 05_deduplication.py <05_deduplication.py>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_