This tutorial explains how to identify and handle duplicate rows with pyrasgo.

Packages

This tutorial uses:

• pandas
• PyRasgo

Open up a Jupyter Notebook and import the following:

import pandas as pd
import pyrasgo

Connect to Rasgo

If you haven't done so already, head over to https://docs.rasgoml.com/rasgo-docs/onboarding/initial-setup and follow the steps outlined there to create your free account. This account gives you free access to the Rasgo API which will calculate dataframe profiles, generate feature importance score, and produce feature explainability for you analysis. In addition, this account allows you to maintain access to your analysis and share with your colleagues.

rasgo = pyrasgo.login(email='', password='')

Creating the data

We will create a dataframe that contains multiple occurrences of duplication for this example.

df = pd.DataFrame({'A': ['A']*2 + ['A', 'A', 'B', 'A', 'B']*3 + ['A', 'A', 'B'],
                   'B': ['A']*2 + ['A', 'a', 'B', 'A', 'b']*3 + ['A', 'a', 'B'],
                   'C': ['A']*2 + ['A', 'B', 'C']*5 + ['A', 'A', 'B'],
                   'D': ['A']*2 + ['A', 'a', 'B']*5 + ['A', 'A', 'B']
                  })
df

Identify duplicates

Duplicate in all columns

The function evaluate.duplicate_rows will identify duplicates in the data.

dups = rasgo.evaluate.duplicate_rows(df)
dups