.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/02_feature_interpretation_with_gapencoder.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_02_feature_interpretation_with_gapencoder.py>`
        to download the full example code. or to run this example in your browser via JupyterLite or Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_02_feature_interpretation_with_gapencoder.py:


.. _example_interpreting_gap_encoder:

==========================================
Feature interpretation with the GapEncoder
==========================================

In this notebook, we will explore the output and inner workings of the
|GapEncoder|, one of the `high cardinality categorical encoders <https://inria.hal.science/hal-02171256v4>`_
provided by skrub.

.. |GapEncoder| replace::
     :class:`~skrub.GapEncoder`

.. |SimilarityEncoder| replace::
     :class:`~skrub.SimilarityEncoder`

.. GENERATED FROM PYTHON SOURCE LINES 20-25

The |GapEncoder| is scalable and interpretable in terms of
finding latent categories, as we will show.

First, let's retrieve the
`employee salaries dataset <https://www.openml.org/d/42125>`_:

.. GENERATED FROM PYTHON SOURCE LINES 25-35

.. code-block:: Python


    from skrub.datasets import fetch_employee_salaries

    dataset = fetch_employee_salaries()

    # Alias X and y
    X, y = dataset.X, dataset.y

    X






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>gender</th>
          <th>department</th>
          <th>department_name</th>
          <th>division</th>
          <th>assignment_category</th>
          <th>employee_position_title</th>
          <th>date_first_hired</th>
          <th>year_first_hired</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>F</td>
          <td>POL</td>
          <td>Department of Police</td>
          <td>MSB Information Mgmt and Tech Division Records...</td>
          <td>Fulltime-Regular</td>
          <td>Office Services Coordinator</td>
          <td>09/22/1986</td>
          <td>1986</td>
        </tr>
        <tr>
          <th>1</th>
          <td>M</td>
          <td>POL</td>
          <td>Department of Police</td>
          <td>ISB Major Crimes Division Fugitive Section</td>
          <td>Fulltime-Regular</td>
          <td>Master Police Officer</td>
          <td>09/12/1988</td>
          <td>1988</td>
        </tr>
        <tr>
          <th>2</th>
          <td>F</td>
          <td>HHS</td>
          <td>Department of Health and Human Services</td>
          <td>Adult Protective and Case Management Services</td>
          <td>Fulltime-Regular</td>
          <td>Social Worker IV</td>
          <td>11/19/1989</td>
          <td>1989</td>
        </tr>
        <tr>
          <th>3</th>
          <td>M</td>
          <td>COR</td>
          <td>Correction and Rehabilitation</td>
          <td>PRRS Facility and Security</td>
          <td>Fulltime-Regular</td>
          <td>Resident Supervisor II</td>
          <td>05/05/2014</td>
          <td>2014</td>
        </tr>
        <tr>
          <th>4</th>
          <td>M</td>
          <td>HCA</td>
          <td>Department of Housing and Community Affairs</td>
          <td>Affordable Housing Programs</td>
          <td>Fulltime-Regular</td>
          <td>Planning Specialist III</td>
          <td>03/05/2007</td>
          <td>2007</td>
        </tr>
        <tr>
          <th>...</th>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
          <td>...</td>
        </tr>
        <tr>
          <th>9223</th>
          <td>F</td>
          <td>HHS</td>
          <td>Department of Health and Human Services</td>
          <td>School Based Health Centers</td>
          <td>Fulltime-Regular</td>
          <td>Community Health Nurse II</td>
          <td>11/03/2015</td>
          <td>2015</td>
        </tr>
        <tr>
          <th>9224</th>
          <td>F</td>
          <td>FRS</td>
          <td>Fire and Rescue Services</td>
          <td>Human Resources Division</td>
          <td>Fulltime-Regular</td>
          <td>Fire/Rescue Division Chief</td>
          <td>11/28/1988</td>
          <td>1988</td>
        </tr>
        <tr>
          <th>9225</th>
          <td>M</td>
          <td>HHS</td>
          <td>Department of Health and Human Services</td>
          <td>Child and Adolescent Mental Health Clinic Serv...</td>
          <td>Parttime-Regular</td>
          <td>Medical Doctor IV - Psychiatrist</td>
          <td>04/30/2001</td>
          <td>2001</td>
        </tr>
        <tr>
          <th>9226</th>
          <td>M</td>
          <td>CCL</td>
          <td>County Council</td>
          <td>Council Central Staff</td>
          <td>Fulltime-Regular</td>
          <td>Manager II</td>
          <td>09/05/2006</td>
          <td>2006</td>
        </tr>
        <tr>
          <th>9227</th>
          <td>M</td>
          <td>DLC</td>
          <td>Department of Liquor Control</td>
          <td>Licensure, Regulation and Education</td>
          <td>Fulltime-Regular</td>
          <td>Alcohol/Tobacco Enforcement Specialist II</td>
          <td>01/30/2012</td>
          <td>2012</td>
        </tr>
      </tbody>
    </table>
    <p>9228 rows × 8 columns</p>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 36-47

Encoding job titles
-------------------

Let's look at the job titles, the column containing dirty data we want to encode:

.. topic:: Note:

  Dirty data, as opposed to clean, are all non-curated categorical
  columns with variations such as typos, abbreviations, duplications,
  alternate naming conventions etc.


.. GENERATED FROM PYTHON SOURCE LINES 47-50

.. code-block:: Python


    X_dirty = X["employee_position_title"]








.. GENERATED FROM PYTHON SOURCE LINES 51-52

Let's have a look at a sample of the job titles:

.. GENERATED FROM PYTHON SOURCE LINES 52-55

.. code-block:: Python


    X_dirty.sort_values().tail(15)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    7753     Work Force Leader II
    1231     Work Force Leader II
    3206     Work Force Leader II
    2602     Work Force Leader II
    6872    Work Force Leader III
    3601    Work Force Leader III
    6922     Work Force Leader IV
    502      Work Force Leader IV
    3469     Work Force Leader IV
    353      Work Force Leader IV
    5838     Work Force Leader IV
    4961     Work Force Leader IV
    2766     Work Force Leader IV
    4556     Work Force Leader IV
    7478     Work Force Leader IV
    Name: employee_position_title, dtype: object



.. GENERATED FROM PYTHON SOURCE LINES 56-59

Then, we create an instance of the |GapEncoder| with 10 components.
This means that the encoder will attempt to extract 10 latent topics
from the input data:

.. GENERATED FROM PYTHON SOURCE LINES 59-64

.. code-block:: Python


    from skrub import GapEncoder

    enc = GapEncoder(n_components=10, random_state=1)








.. GENERATED FROM PYTHON SOURCE LINES 65-67

Finally, we fit the model on the dirty categorical data and transform it
in order to obtain encoded vectors of size 10:

.. GENERATED FROM PYTHON SOURCE LINES 67-71

.. code-block:: Python


    X_enc = enc.fit_transform(X_dirty)
    X_enc.shape





.. rst-class:: sphx-glr-script-out

 .. code-block:: none


    (9228, 10)



.. GENERATED FROM PYTHON SOURCE LINES 72-82

Interpreting encoded vectors
----------------------------

The |GapEncoder| can be understood as a continuous encoding
on a set of latent topics estimated from the data. The latent topics
are built by capturing combinations of substrings that frequently
co-occur, and encoded vectors correspond to their activations.
To interpret these latent topics, we select for each of them a few labels
from the input data with the highest activations.
In the example below we select 3 labels to summarize each topic.

.. GENERATED FROM PYTHON SOURCE LINES 82-87

.. code-block:: Python


    topic_labels = enc.get_feature_names_out(n_labels=3)
    for k, labels in enumerate(topic_labels):
        print(f"Topic n°{k}: {labels}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Topic n°0: employee_position_title: administrative, administration, legislative
    Topic n°1: employee_position_title: equipment, operator, liquor
    Topic n°2: employee_position_title: correctional, correction, officer
    Topic n°3: employee_position_title: firefighter, rescuer, rescue
    Topic n°4: employee_position_title: technology, technician, mechanic
    Topic n°5: employee_position_title: enforcement, crossing, warehouse
    Topic n°6: employee_position_title: manager, worker, program
    Topic n°7: employee_position_title: communications, community, safety
    Topic n°8: employee_position_title: specialist, special, planning
    Topic n°9: employee_position_title: services, supervisor, coordinator




.. GENERATED FROM PYTHON SOURCE LINES 88-93

As expected, topics capture labels that frequently co-occur. For instance,
the labels "firefighter", "rescuer", "rescue" appear together in
"Firefighter/Rescuer III", or "Fire/Rescue Lieutenant".

We can now understand the encoding of different samples.

.. GENERATED FROM PYTHON SOURCE LINES 93-107

.. code-block:: Python


    import matplotlib.pyplot as plt

    encoded_labels = enc.transform(X_dirty[:20])
    plt.figure(figsize=(8, 10))
    plt.imshow(encoded_labels)
    plt.xlabel("Latent topics", size=12)
    plt.xticks(range(0, 10), labels=topic_labels, rotation=50, ha="right")
    plt.ylabel("Data entries", size=12)
    plt.yticks(range(0, 20), labels=X_dirty[:20].to_numpy().flatten())
    plt.colorbar().set_label(label="Topic activations", size=12)
    plt.tight_layout()
    plt.show()




.. image-sg:: /auto_examples/images/sphx_glr_02_feature_interpretation_with_gapencoder_001.png
   :alt: 02 feature interpretation with gapencoder
   :srcset: /auto_examples/images/sphx_glr_02_feature_interpretation_with_gapencoder_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 108-118

As we can see, each dirty category encodes on a small number of topics,
These can thus be reliably used to summarize each topic, which are in
effect latent categories captured from the data.

Conclusion
----------

In this notebook, we have seen how to interpret the output of the
|GapEncoder|, and how it can be used to summarize categorical variables
as a set of latent topics.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 1.704 seconds)


.. _sphx_glr_download_auto_examples_02_feature_interpretation_with_gapencoder.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: binder-badge

      .. image:: images/binder_badge_logo.svg
        :target: https://mybinder.org/v2/gh/skrub-data/skrub/0.2.0?urlpath=lab/tree/notebooks/auto_examples/02_feature_interpretation_with_gapencoder.ipynb
        :alt: Launch binder
        :width: 150 px

    .. container:: lite-badge

      .. image:: images/jupyterlite_badge_logo.svg
        :target: ../lite/lab/?path=auto_examples/02_feature_interpretation_with_gapencoder.ipynb
        :alt: Launch JupyterLite
        :width: 150 px

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: 02_feature_interpretation_with_gapencoder.ipynb <02_feature_interpretation_with_gapencoder.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: 02_feature_interpretation_with_gapencoder.py <02_feature_interpretation_with_gapencoder.py>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_