A Grafana panel showing a month of CPU at a 30 second scrape is not a picture, it is 86,400 rows waiting to become a dataset, and at a 15 second scrape it is 172,800. Once you read the panel as a table instead of a graph, the question stops being what does this line say and becomes what does one row mean, which is the first question a data scientist asks of anything.

Two views of the same numbers, dashboard and dataset

You have stared at telemetry for years, but as dashboards and alert thresholds, not as observations and features. A data scientist looks at the identical bytes and sees a table: each collection is a row, each metric is a column, each label is a category, and the timestamp is what orders it all. Nothing about the data changes, only the frame you put around it. Getting fluent in both frames at once is the actual skill this part builds, and the map below is the artifact to keep, because every later part in this series reshapes telemetry using exactly these correspondences.

Here is the reframe made literal. A single metric from one entity, sampled for 30 days, is a different number of rows depending only on the scrape interval you exported at, and that number is your table before you have modelled a thing. Read the chart as a warning as much as a fact, because the leftmost bar is a size most laptops handle fine for one series and choke on once you multiply by every node.

Last part we made the notebook reproducible, this part we look hard at the raw material and decide what a row is before cleaning or modelling anything. For the running project, export a month of one cluster CPU and memory at a 5 minute interval, which lands near 8,640 rows per metric and stays comfortable in memory. If you want a public stand in to follow along, the Numenta Anomaly Benchmark ships a real AWS CloudWatch export of cluster CPU that reads exactly like your own, and I point to it in the references. For the loading mechanics, the Data Science Series has a full part on getting data into Python, so here I spend the effort on what the numbers mean rather than on read_csv arguments.

One observation per row, tidy telemetry

Tidy data has a precise meaning, from Hadley Wickham: each variable is a column, each observation is a row, each kind of observational unit is its own table. Telemetry rarely arrives tidy. It arrives long, one row per metric per timestamp, or wide with a column per node, or as JSON with labels nested inside. Your first job is to reshape it into one row per timestamp with a clean column per metric, and the first pothole is not the shape at all, it is the timestamp coming back as text.

# tested on Python 3.10.12, pandas 2.3.3, numpy 2.2.6
import pandas as pd

df = pd.read_csv('cpu.csv')          # a Grafana or CloudWatch export
print(df.dtypes)

df.set_index('timestamp').resample('10min').mean()   # try to bucket it

timestamp     object      # the trap, a string, not a time
cpu_pct      float64

TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex,
but got an instance of 'Index'

That object dtype is the same class of bug the last part warned about, a column that looks right and behaves wrong. pandas read the timestamp as an ordinary string, so resample has no time axis to work with and refuses. Parse the dates on load and the same call succeeds, bucketing the 5 minute rows into 10 minute means.

df = pd.read_csv('cpu.csv', parse_dates=['timestamp'])
print(df.dtypes)
print(df.set_index('timestamp').resample('10min').mean())

timestamp    datetime64[ns]
cpu_pct             float64

                     cpu_pct
timestamp
2026-06-01 00:00:00    42.10
2026-06-01 00:10:00    46.55

With a real time index, reshaping long telemetry to tidy wide is one call. Below is the shape every later part assumes, raw export to long frame to wide grid to a labelled table you can model.

flowchart LR
  A[Raw export, long or JSON] --> B[Parse timestamp to datetime]
  B --> C[Pivot to one row per timestamp]
  C --> D[Resample onto a regular grid]
  D --> E[Difference counters into rates]
  E --> F[Tidy dataset, features and label]

# long: one row per metric per timestamp, the common monitoring export
wide = long.pivot(index='timestamp', columns='metric', values='value')
print(wide)

# if the same timestamp appears twice, pivot cannot reshape, aggregate instead
dup.pivot(index='timestamp', columns='metric', values='value')

metric               cpu_pct  mem_pct
timestamp
2026-06-01 00:00:00     22.4     68.0
2026-06-01 00:05:00     61.8     70.2

ValueError: Index contains duplicate entries, cannot reshape
# fix: dup.pivot_table(..., aggfunc='mean') collapses the duplicate scrape

Duplicate timestamps are common when two collectors overlap or a scrape is retried, and pivot draws a hard line there rather than guess. pivot_table with an aggregation collapses the pair, which is what you want as long as you noticed the duplicate first. Reshaping and vectorised pandas get their own treatment in the Data Science Series part on NumPy and pandas, worth a read once your frames outgrow a few thousand rows.

Counters versus gauges, and why the type sets the column

Here is the place the tutorial default is wrong for infrastructure data. Usual advice is to point pandas at your metrics and start modelling, and that works fine for CPU percent or memory used, which are gauges that move up and down and mean what they say. It quietly ruins counters. Prometheus defines a counter as a cumulative value that only increases until a restart resets it to zero, so a counter fed straight in as a feature encodes how long the process has been running, not how busy it is. You must difference it into a rate first, exactly as PromQL does with its rate function under the hood.

import pandas as pd, numpy as np

idx = pd.date_range('2026-06-01', periods=8, freq='5min')
# net_bytes_total is a COUNTER, it climbs until a restart resets it
ctr = pd.Series([100, 250, 500, 900, 1400, 50, 300, 700], index=idx)

delta = ctr.diff()                       # step to step change
print(delta.tolist())

delta[delta < 0] = np.nan                # a restart is unknown, not a real drop
secs = ctr.index.to_series().diff().dt.total_seconds()
print((delta / secs).round(3).tolist())  # bytes per second, a usable feature

[nan, 150.0, 250.0, 400.0, 500.0, -1350.0, 250.0, 400.0]
                                    # the restart shows up as a negative
[nan, 0.5, 0.833, 1.333, 1.667, nan, 0.833, 1.333]

The raw counter climbs from 100 to 1400, then a restart drops it to 50, and the naive difference reports a change of minus 1350 that never physically happened. Clamping negative steps to null and dividing by the seconds between scrapes turns the mess into a clean per second rate, the feature you actually wanted. Applying rate to a counter yields a gauge, which is precisely why the result is now safe to model. Miss this step and every counter in your export becomes a slow ramp that correlates with time itself, and time is the one thing you never want a model leaning on.

Sampling interval and label cardinality as table size

Two dials set how big your dataset gets, and both are decisions you make before exporting, not after. One dial is the sampling interval from the chart earlier, where 15 seconds gives 172,800 rows per series over 30 days and 5 minutes gives 8,640. Cardinality is the other dial, the number of distinct label combinations, and it multiplies. One CPU metric across 50 nodes is 50 series, so at 15 seconds that is 8.6 million rows. Add a second label such as mount point on a disk metric, say 6 mounts per node, and 50 times 6 series at 15 seconds reaches 51.8 million rows for a single metric name.

This is where a naive export detonates, and the fix runs against instinct. An operator wants every node, every mount, every second, because in monitoring more resolution is almost always better. For a first dataset the opposite is true. Pick one metric, one entity, and the coarsest interval that still shows the behaviour you care about, then widen only once a model earns it. The Numenta benchmark makes the same choice for you, aggregating its taxi series into 30 minute buckets and its tweet series into 5 minute counts, because a clean single series teaches more than a raw firehose. Time aware resampling and the backtesting that goes with it is the Data Science Series topic in time series forecasting, and you will lean on it from Part 21 onward.

Missing scrapes, gaps and honest nulls

Real telemetry has holes. A target goes down, a scrape times out, a collector restarts, and the exported CSV simply has no row for those timestamps. If you never line the data up against the interval it should have, those gaps hide, and worse, a naive fill turns them into fiction. Reindex onto the full expected grid first, so every missing scrape becomes an explicit null you can see and count.

full = pd.date_range('2026-06-01 00:00', '2026-06-01 01:00', freq='5min')  # 13 scrapes
reg = frame.reindex(full)                 # line up on the grid that should exist
print('missing scrapes:', reg['cpu_pct'].isna().sum())

# the trap: fillna(0) on a gauge invents idle minutes that never happened
print('fillna(0) mean:', round(reg['cpu_pct'].fillna(0).mean(), 2))
print('skipna   mean:', round(reg['cpu_pct'].mean(), 2))

missing scrapes: 3
fillna(0) mean: 29.85
skipna   mean: 38.8

Three missing scrapes out of thirteen, and filling them with zero pulled the average CPU from 38.8 down to 29.85, a swing of nearly nine points invented from nothing. Zero is a real, believable CPU reading, which is what makes it so dangerous as a fill value, the corrupted number looks perfectly plausible on a dashboard. For a gauge, prefer skipping nulls in the aggregate, or interpolating over short gaps while leaving long outages as nulls a model can learn to ignore. Deciding what a gap means, replay it or exclude it, is the same discipline that keeps tracing and observability honest in language model systems, where a dropped span is missing evidence, never a zero.

Project status and a telemetry to dataset checklist

Where the project stands now: last part gave you a reproducible notebook, this part gave that notebook a real dataset to hold. You can take a raw export, parse its timestamps, pivot it tidy, resample it onto a regular grid, difference the counters, and account for every missing scrape honestly. That is a cleaned, well understood dataset, the raw material this whole series models. Run the short checklist below against your own export before you trust a single number off it.

Export one metric as a dataset this week

If you do one thing after this, export a month of one gauge, CPU percent from one cluster, at a 5 minute interval, and load it with parse_dates so the timestamp is a real datetime. Then reindex it onto the full grid and count the missing scrapes, because that single number tells you how healthy your collection actually is. Label the metric a gauge, resist filling its gaps with zero, and you have the clean, honest dataset the rest of this series builds on. Next part gets data into Python properly, from SQL, APIs and monitoring endpoints, so the export you just did by hand becomes a repeatable pull, bring this dataset with you.

References

• Prometheus documentation, metric types, counter, gauge, histogram and summary
• Prometheus documentation, the data model and labels
• pandas documentation, DataFrame resample
• Hadley Wickham, Tidy Data, Journal of Statistical Software
• Numenta Anomaly Benchmark, the data corpus, including AWS CloudWatch CPU
• Infra to Data Science, the Complete Guide

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts