Packages

This tutorial uses:


import pandas as pd
import numpy as np

Creating the data

We will create a dataframe that contains multiple time series, one for each group.


np.random.seed(1066)
dates = pd.date_range(start='2010-01-01', end='2010-12-31', freq='D')
df = pd.DataFrame({'date': dates,
                   'group': 'A',
                   'value': np.random.randint(0, 100, size=len(dates))
                  }).append(pd.DataFrame({'date': dates,
                                          'group': 'B',
                                          'value': np.random.randint(0, 100, size=len(dates))
                  })).append(pd.DataFrame({'date': dates,
                                           'group': 'C',
                                           'value': np.random.randint(0, 100, size=len(dates))
                                          })).reset_index(drop=True)
df

Your output should look something like this:


date	group	value
0	2010-01-01	A	57
1	2010-01-02	A	11
2	2010-01-03	A	83
3	2010-01-04	A	83
4	2010-01-05	A	93
...	...	...	...
1090	2010-12-27	C	50
1091	2010-12-28	C	59
1092	2010-12-29	C	85
1093	2010-12-30	C	32
1094	2010-12-31	C	3

Drop some rows randomly to create gaps in the data.


length = df.shape[0]
droplist = np.unique(np.sort(np.random.randint(0, length, size=100))).tolist()
df = df.drop(droplist).reset_index(drop=True)
df

output:


date	group	value
0	2010-01-01	A	57
1	2010-01-02	A	11
2	2010-01-03	A	83
3	2010-01-04	A	83
4	2010-01-05	A	93
...	...	...	...
992	2010-12-27	C	50
993	2010-12-28	C	59
994	2010-12-29	C	85
995	2010-12-30	C	32
996	2010-12-31	C	3

Identify Date Gaps

In a single series

The pandas function diff applied to the date field will generate the time between subsequent datas. Comparing that to 1 day can identify date gaps in the data.


singleseries = df[df.group == 'A']
singleseries['gap'] = singleseries['date'].sort_values().diff() > pd.to_timedelta('1 day')
singleseries[singleseries.gap]

date	group	value	gap
39	2010-02-10	A	97	True
44	2010-02-17	A	93	True
45	2010-02-19	A	88	True
57	2010-03-04	A	92	True
77	2010-03-25	A	44	True
81	2010-03-30	A	94	True
86	2010-04-05	A	7	True
89	2010-04-10	A	65	True
92	2010-04-15	A	85	True
99	2010-04-23	A	7	True
115	2010-05-10	A	46	True
129	2010-05-25	A	50	True
132	2010-05-29	A	71	True
136	2010-06-03	A	42	True
164	2010-07-02	A	26	True
175	2010-07-14	A	50	True
198	2010-08-07	A	6	True
201	2010-08-11	A	84	True
209	2010-08-20	A	14	True
212	2010-08-24	A	60	True
238	2010-09-20	A	39	True
246	2010-09-29	A	34	True
259	2010-10-13	A	27	True
270	2010-10-25	A	19	True
271	2010-10-27	A	3	True
291	2010-11-17	A	1	True
297	2010-11-24	A	39	True
298	2010-11-26	A	28	True
302	2010-12-01	A	3	True

In multiple time series

The same process can be applied to data with multiple time series, we just need to group on the time series identifier (group) before applying the


df['gap'] = df[['group', 'date']].sort_values(by=['group', 'date']).groupby('group').diff() > pd.to_timedelta('1 day')
df[df.gap]

output:


date	group	value	gap
39	2010-02-10	A	97	True
44	2010-02-17	A	93	True
45	2010-02-19	A	88	True
57	2010-03-04	A	92	True
77	2010-03-25	A	44	True
...	...	...	...	...
920	2010-10-11	C	98	True
950	2010-11-11	C	0	True
958	2010-11-20	C	72	True
974	2010-12-08	C	1	True
985	2010-12-20	C	39	True

No-code/low-code data prep and visualization

Request Demo
Try for Free