, ,

Capacity Forecasting From Infrastructure Telemetry (Infra to Data Science Series, Part 22)

A straight line on a seasonal metric tells you when the average crosses your limit, not when the weekly peak does. Forecast with Holt-Winters, read the date off the upper prediction band, and plan against the risk you can afford.

Infra to Data Science Series · Part 22 of 26

A capacity review last quarter opened with one question from the platform owner: when does this cluster run out of headroom. My honest answer was not a date, it was a range, and the distance between the comfortable answer and the honest one was seven weeks. A straight line through the CPU history said the limit was fifty one days away, plenty of runway for next quarter. That same history, forecast with its weekly shape kept in, said the average crossed in twenty one days and the busy Tuesday peak could breach in six.

Key takeaways: A straight line fit to a seasonal metric answers when the average crosses your limit, not when the weekly peak does, and the peak is what pages you. Keep the season inside the model with Holt-Winters, then read the date off the upper prediction band rather than the mean line. On the cluster here a linear trendline promised 51 days of headroom, the seasonal mean forecast said 21, and the upper 95 percent band breached in 6. Plan against the band you can afford to be wrong about, and backtest the model on held out days before you trust any date it prints.
Who this is for: An infrastructure engineer or SRE who has exported a usage metric and needs a defensible answer to when a resource runs out. Terms on first use: capacity forecasting projects a usage metric forward to estimate when it crosses a limit; seasonality is the repeating daily or weekly shape of the metric; Holt-Winters is an exponential smoothing model that carries a level, a trend and a season forward together; a prediction interval is the band a forecast should fall inside with a stated probability; MAPE, mean absolute percentage error, is the average size of a forecast miss as a percent of the actual value.

Linear Trendlines Overpromise on Seasonal Capacity

Where the project stands: last part turned the same exported telemetry into an anomaly stream that needed no labels. This part asks a forward question of that export, when does a cluster run out of headroom, and answers it with a model instead of a ruler laid on a chart. Projecting usage to a limit is the capacity planning reflex every operator already has, and it maps straight onto the cost question that drives cost control and model routing, where knowing when demand crosses a tier is the whole game.

Fitting a straight line and extending it to the limit is where most capacity math begins. It is also where it quietly goes wrong on any metric with a weekly rhythm. A least squares line runs through the middle of the season, so it tracks the average day and sits below the peaks. Extrapolate that middle to 85 percent and it reports when a typical day crosses, which is comfortably later than when the busy day already brushes the ceiling. Below, a daily CPU export of 120 days is fit with numpy and pushed forward until it reaches the threshold. Every line was run against python 3.10, pandas 2.3.3, statsmodels 0.14.6 and numpy 2.2.6.

# capacity.py  tested with python 3.10, pandas 2.3.3, statsmodels 0.14.6, numpy 2.2.6
import numpy as np, pandas as pd

# cpu_daily.csv: one row per day, mean cpu percent for a single cluster, exported from monitoring
s = pd.read_csv('cpu_daily.csv', index_col=0, parse_dates=True)['cpu_pct']
thr, last = 85.0, s.index[-1]
print('last obs', round(float(s.iloc[-1]), 1), 'raw std', round(float(s.std()), 1))

x = np.arange(len(s))
slope, intercept = np.polyfit(x, s.values, 1)          # a straight line through the season
fx = np.arange(len(s), len(s) + 120)
line = slope * fx + intercept
idx = pd.date_range(last + pd.Timedelta(days=1), periods=120, freq='D')
hit = idx[line >= thr]
print('linear slope', round(slope, 3), '%/day  crosses', str(hit[0].date()),
      'in', (hit[0] - last).days, 'days')
last obs 82.7 raw std 8.9
linear slope 0.188 %/day  crosses 2026-09-18 in 51 days

Fifty one days reads like next quarter’s problem. It is wrong in the direction that hurts, because the last real reading was already 82.7 on a busy day while the fitted line for that same day sat near 77. A raw standard deviation of 8.9 points is almost all weekly season, and the line averages straight through it. Keep the season in the model and the answer moves in by a month.

There is a quieter reason to distrust the ruler. A least squares fit minimises error across the whole window, so its projected endpoint is anchored to the seasonal average of the last stretch of history. Export a window that happens to end on a quiet weekend and the slope starts lower and the crossing slides later; end it on a busy Friday and the same data reports a nearer date. A capacity number that swings by weeks depending on which day you pulled the export is not a number to plan on. A seasonal model removes that sensitivity, because it fits the weekly pattern explicitly instead of smearing it into a slope.

Forecasting Daily Utilisation With Holt-Winters

Holt-Winters exponential smoothing carries three things forward at once: a level, a trend, and an additive weekly season of seven days. Because the Data Science Series already works the mechanics and backtesting of forecasting in depth, I keep the theory to a clause here and link the fuller treatment of time series forecasting and backtesting for the parts I skip. The one habit worth importing from that piece is to grade the model on days it never saw before quoting any date. Hold out the last fourteen days, fit on the rest, and measure the miss.

from statsmodels.tsa.holtwinters import ExponentialSmoothing

train, test = s.iloc[:-14], s.iloc[-14:]                 # backtest on unseen days
fit = ExponentialSmoothing(train, trend='add', seasonal='add', seasonal_periods=7).fit()
fc = fit.forecast(14)
mape = float((np.abs(test.values - fc.values) / test.values).mean() * 100)
print('backtest MAPE 14d', round(mape, 2), '%')

full = ExponentialSmoothing(s, trend='add', seasonal='add', seasonal_periods=7).fit()
future = full.forecast(45)
crossed = future[future >= thr]
print('mean crosses', str(crossed.index[0].date()), 'in', (crossed.index[0] - last).days, 'days')
backtest MAPE 14d 1.47 %
mean crosses 2026-08-19 in 21 days

A MAPE of 1.47 percent on held out days says the fit tracks this metric closely, and the seasonal mean crosses 85 percent on 19 August, twenty one days out rather than fifty one. That gap between the two methods is the whole reason to keep the season in the model. One failure sits waiting for anyone who exports real monitoring data, and it does not raise an error you can miss on a bad day.

One argument carries the season and it is easy to set wrong: seasonal_periods is the count of points in one full cycle, seven for daily readings with a weekly rhythm. Leave it at a daily cycle of twenty four out of habit on data that is already one point per day and the model tries to fit a shape that is not there, bending the forecast around noise. Match the period to how your export is sampled, not to the calendar in your head, and confirm the cycle length by plotting one representative week before you trust a single date.

gap = s.drop(s.index[[40, 41, 72]])        # three days missing, a monitoring outage
fut = ExponentialSmoothing(gap, trend='add', seasonal='add', seasonal_periods=7).fit().forecast(5)
print(type(fut.index).__name__, list(fut.index[:3]))
ValueWarning: A date index has been provided, but it has no associated frequency information
ValueWarning: No supported index is available. Prediction results will be given with an integer index
RangeIndex [117, 118, 119]
Production gotcha: Drop three days from a daily series and pandas can no longer read a regular frequency off the index, so statsmodels throws the timestamps away and hands the forecast back on an integer RangeIndex starting at 117. Any code that subtracts dates to compute days to capacity then fails silently or crashes, because 117 is not a date. Monitoring exports have gaps from outages and retention windows constantly. Reindex to a full daily range and interpolate the holes before fitting: s = s.reindex(pd.date_range(s.index.min(), s.index.max(), freq=’D’)).interpolate(), which restores the frequency and puts real dates back on the forecast.

Days to Capacity From a Forecast and a Threshold

The number a capacity review actually wants is not a chart, it is a count of days until a named limit. Wrap the forecast and the crossing test in one function that takes a cleaned series, a horizon and a threshold, and returns the date and the day count or a clear miss. This is the reference artifact of this part; keep it next to the export script and point it at any resource with a ceiling, CPU, memory, disk or connection pool.

def days_to_capacity(series, threshold, horizon=90, period=7):
    series = series.asfreq('D').interpolate()                 # guard the frequency
    model = ExponentialSmoothing(series, trend='add', seasonal='add', seasonal_periods=period).fit()
    fc = model.forecast(horizon)
    over = fc[fc >= threshold]
    if len(over) == 0:
        return {'crosses': None, 'days': None, 'peak': round(float(fc.max()), 1)}
    return {'crosses': str(over.index[0].date()), 'days': (over.index[0] - series.index[-1]).days}

print(days_to_capacity(s, 85.0))
{'crosses': '2026-08-19', 'days': 21}

One choice inside that function decides whether it runs at all: additive season versus multiplicative. Additive assumes the weekly swing is a roughly constant number of points; multiplicative assumes it grows with the level, which fits a metric that fans out as it climbs. Reach for multiplicative on a metric that includes zeros, a cold start cluster or a freshly cut disk, and it stops cold.

ExponentialSmoothing(mem, trend='add', seasonal='mul', seasonal_periods=7).fit()   # mem has zeros
ValueError: endog must be strictly positive when using multiplicative trend or seasonal components.

Pick additive for utilisation percentages, which move in a bounded band and often pass through zero on a new node. Save multiplicative for counts that scale with load, request rate or bytes served, where the weekly peak genuinely grows as the baseline does, and only once you have confirmed the series never touches zero.

One more choice sits outside the model: the threshold itself. The number you forecast toward should be the level where the resource actually starts to hurt, not the round limit on the datasheet. A disk that slows on garbage collection at 80 percent full has an effective ceiling of 80, not 100, and a CPU that starts queueing run tasks at 85 has already run out of useful headroom well below saturation. Set the threshold to where behaviour degrades, leave a buffer for the lead time a purchase actually takes, and the days to capacity number becomes something a change board can sign rather than argue.

Prediction Intervals Turn a Point Guess Into a Range

A single forecast date hides the one thing capacity planning is about, risk. Order hardware to the mean crossing and you are betting the busy week lands exactly on the average, which it never does. The ETSModel interface fits the same additive Holt-Winters model and returns a prediction interval through get_prediction, so every future day comes with a lower and an upper band. Plan against the upper band and you are asking a sharper question: when could the peak week breach, not when does the typical week.

from statsmodels.tsa.exponential_smoothing.ets import ETSModel

m = ETSModel(s, error='add', trend='add', seasonal='add', seasonal_periods=7).fit(disp=False)
sf = m.get_prediction(start=len(s), end=len(s) + 59).summary_frame(alpha=0.05)   # 95 percent band
print(sf[['mean', 'pi_lower', 'pi_upper']].iloc[30].round(1).to_dict())
for name, col in [('mean', 'mean'), ('upper-95', 'pi_upper')]:
    over = sf.index[sf[col].values >= thr]
    print(name, 'crosses', str(over[0].date()), 'in', (over[0] - last).days, 'days')
{'mean': 72.0, 'pi_lower': 68.4, 'pi_upper': 75.6}
mean crosses 2026-08-19 in 21 days
upper-95 crosses 2026-08-04 in 6 days

Three methods, three dates, on one metric: the linear ruler says 51 days, the seasonal mean says 21, and the upper 95 percent band says a peak week could touch 85 percent in 6. None is a lie; they answer different questions. A change request for more capacity should quote the band, because being wrong on the optimistic side of a capacity call is the expensive kind of wrong.

One property of these bands governs how far out to trust them. A prediction interval widens with the horizon, because each forecast day compounds the uncertainty of the days before it, so a band six days out is tight while a band forty days out can be wider than the metric itself. That widening is information, not a defect: the near crossing is a number to act on and the far crossing is a number to recheck. Refit on a weekly cadence as fresh telemetry lands, and let each run tighten the date it reports rather than leaning on one forecast for a whole quarter.

When does the cluster cross 85 percent CPUthree methods, three answers, same 120 day history85 percent limitupper 95 bandmean forecastlinear trendday 6day 21day 51days ahead of the last reading
The linear trend reaches the limit last, the upper band first. Ordering capacity to the mean bets the busy week lands on the average.
MethodCrosses 85 percentDays outWhat it answers
Linear trendline2026-09-1851when the average day crosses
Holt-Winters mean2026-08-1921when the typical week crosses
Holt-Winters upper 952026-08-046when a peak week could breach
War story: I once signed off on a storage plan built from a linear fit that gave a NAS ninety days of runway. That fit ran through a volume whose weekly backup window added twelve percent every Sunday night, which the straight line averaged away. The volume hit 96 percent full on a Sunday nineteen days later and a backup job failed at 2am, and I got the page. Refitting with a weekly season put the honest crossing at eighteen days and the upper band at eleven, both of which would have moved the purchase up by two months. The linear number was not a small error. It was off by a factor of five in the direction that fails at night.

Plan Against the Upper Band, Not the Mean

Reach for a seasonal model whenever the metric has a daily or weekly shape, which covers almost every utilisation series a cluster produces. Fit additive Holt-Winters for bounded percentages, backtest on held out days and quote the MAPE, then report the date off the upper prediction band, not the mean. Steer clear of the two defaults that feel responsible and are not: a linear trendline on a seasonal metric, which reads the average and misses the peak, and a single point date with no interval, which hides the risk the whole exercise exists to measure.

flowchart LR
  E[Exported metric] --> C[Reindex to daily and interpolate]
  C --> B[Backtest on held out days]
  B --> H[Fit additive Holt-Winters]
  H --> P[Forecast with prediction interval]
  P --> T[Days to threshold on upper band]
  T --> O[Capacity order]
Guard the frequency first, prove the model on unseen days, then read the date off the band you plan against.
MetricSeason modelPlan against
CPU or memory percentadditive, weeklyupper 95 band
Request rate or bytesmultiplicative, weeklyupper 95 band
Flat metric, hard ceilingnone neededmean plus buffer

Do this on Monday: export ninety days of one resource from your own monitoring, reindex it to a clean daily frequency, and run days_to_capacity against its real limit. Then fit the interval and compare the mean date with the upper band date; the distance between them is your true lead time. For a public series to rehearse on before touching production, the Backblaze Drive Stats release carries years of daily drive counts you can forecast toward a capacity limit. Next part keeps the same telemetry and turns to the messiest signal a cluster emits, log analysis and clustering at scale.

Infra to Data Science Series · Part 22 of 26
« Previous: Part 21  |  Guide  |  Next: Part 23 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading