Three dashboards in the same company reported active users for March. One said 41,200. One said 28,600. One said 12,900. Nobody had made an error, nobody had a broken query, and all three numbers were correct. The first counted anyone who opened the app, the second counted anyone who opened it on at least two separate days, and the third counted anyone who completed a core action. Three honest definitions of one phrase, and a meeting that spent forty minutes arguing about the data instead of the business.

That meeting is the reason this part exists. In Part 14 you built a dashboard, and in Part 16 you learned to plot in Python. Both of those assume somebody already decided what the numbers mean. Deciding that is the job here. Getting a metric definition right is unglamorous work that saves more hours than any query optimisation you will ever write, and it is a skill hiring managers notice quickly because most junior analysts skip it.

What counts as a metric, and what makes one a KPI

A metric is a number you measure repeatedly, on a defined population, over a defined period. Signups last week, average order value, page load time, tickets closed per agent: all metrics. Analytics tools formalise this. In Google Analytics, for instance, metrics are the quantitative measurements and dimensions are the text attributes you slice them by, so country and device are dimensions while sessions and revenue are metrics. That split is worth carrying in your head, because a metric without a dimension is usually too coarse to act on, and a dimension without a metric measures nothing at all.

A KPI, short for key performance indicator, is a metric that has been promoted. It earns the promotion by being attached to a goal, given a target, given a time frame, and given a person who is accountable for it. Power BI makes this concrete in its KPI visual, which will not render at all until you supply an indicator value, a target, and a trend to measure it against. That is not a quirk of the tool, it is the definition made visible: a number with no target is not a KPI, it is just a number on a screen.

The relationship runs one way. Every KPI is a metric, but the vast majority of metrics should never become KPIs. A healthy company might track four hundred metrics and manage against six KPIs. When I review a dashboard and find fourteen things labelled KPI, I know the label has stopped meaning anything, and I know the team has not yet had the hard conversation about what actually matters this quarter.

Why two people get different numbers for the same metric

Go back to the three dashboards. The disagreement was never about the data, it was about four unwritten choices that each analyst made silently and reasonably. Who counts as a user: every device that opened the app, or every logged in account? What is the window: a calendar month, or a rolling thirty days? What gets excluded: internal staff accounts, bots, test users, refunded orders? And what is the exact arithmetic: distinct accounts, or account days summed and divided?

None of those choices is wrong. What is wrong is leaving them implicit and then calling the result active users, as though the phrase were self explanatory. Here is what those four definitions produced for the same month on the same underlying data.

Same data, same month, four defensible definitions. The widest and narrowest differ by a factor of three.

Anatomy of a metric definition

A definition worth the name fits in a short paragraph and answers five questions without leaving anything for the reader to guess. Start with the name, and make it specific enough that the name itself carries the meaning. Monthly active accounts is a better name than active users, because it already tells you the unit is an account and the window is a month. Vague names invite exactly the confusion the name was supposed to prevent.

Then state the population, meaning who or what is in scope, and be explicit about exclusions such as internal staff, automated traffic and test accounts. State the time window and whether it is a calendar period or a rolling one, because rolling thirty days and calendar month behave very differently at month boundaries. State the calculation in plain words, including whether you are counting distinct things or summing events, and whether refunds, cancellations or duplicates are subtracted. Finally, note the source: which table or which system is the truth, so nobody rebuilds the same number from a different pipe.

Here is a definition that passes. Monthly active accounts is the count of distinct customer account IDs that completed at least one order, saved item or search within a calendar month, in the orders and events tables, excluding accounts flagged as internal or test. Read that once and you could rebuild it, and more to the point, so could a colleague who has never seen your query. That is the whole test.

flowchart LR
  Q[Business question] --> C[Candidate metric]
  C --> D[Write the definition]
  D --> A{Can anyone act on it?}
  A -->|No| C
  A -->|Yes| O[Assign owner and target]
  O --> P[Publish on dashboard]
  P --> R[Review each quarter]
  R --> D

Choosing the few that matter

Once you can define metrics cleanly, the harder question arrives: which ones deserve attention. My filter has three tests, and a candidate KPI has to pass all three. Can somebody change it through a decision they are able to make? Would a move in it change what the team does next week? And can you measure it reliably, on a cadence fast enough to react to? Fail any one and it belongs in the reporting pile, not on the wall.

The actionability test does most of the work. Share of overall market is interesting, but no single team decision moves it inside a quarter, so it makes a poor KPI even though it is a fine metric to report annually. Checkout completion rate, by contrast, responds to work the product team can start on Monday. The table below shows the same set of numbers sorted by which side of the line they fall on.

Two KPIs, two metrics and one vanity number. A team of this size should not be managing more than four or five KPIs at once.

Metric trees, from the top number to the levers

A single top level number tells you that something happened but never why. The fix is to break it down into the parts that multiply or add up to it, which people call a metric tree or a driver tree. Revenue splits into number of customers times average revenue per customer. Customers splits into new customers plus retained customers. New customers splits into visitors times signup rate. Keep going and you eventually reach numbers a team can actually work on.

Building one changes how you investigate. When revenue drops six percent, you do not stare at the revenue line hoping for inspiration; you walk down the tree checking which branch moved. Usually one branch explains most of it, and the tree turns a vague question into a short list of specific ones. This is also how you decide which supporting metrics to put on a dashboard next to a KPI: the useful neighbours are the direct children of that KPI in the tree, not whatever else happened to be easy to query.

flowchart TD
  REV[Revenue] --> CUST[Paying customers]
  REV --> ARPU[Revenue per customer]
  CUST --> NEW[New customers]
  CUST --> RET[Retained customers]
  NEW --> VIS[Visitors]
  NEW --> SIGN[Signup rate]
  RET --> CHURN[Churn rate]
  ARPU --> BASKET[Average order value]
  ARPU --> FREQ[Orders per customer]

Vanity metrics and Goodhart’s law

A vanity metric is a number that reliably goes up, feels good to report, and never changes a decision. Total registered accounts is the classic case, because it is cumulative and therefore cannot fall no matter how badly the product performs. Total downloads, cumulative page views and followers all belong to the same family. The tell is simple: ask what you would do differently if the number were twenty percent higher. If the honest answer is nothing, you are looking at a trophy, not a metric.

The cure is usually to convert the cumulative number into a rate or a ratio over a window. Total accounts becomes monthly active accounts. Total downloads becomes downloads that reached first use within seven days. Cumulative revenue becomes revenue per active customer. Rates can fall, which is precisely what makes them informative, and a number that can deliver bad news is the only kind worth putting on a wall.

The second failure is subtler and arrives later. Goodhart’s law, from a 1975 paper by the economist Charles Goodhart and best known in the phrasing popularised by the anthropologist Marilyn Strathern, holds that when a measure becomes a target it stops being a good measure. Grade a support team on tickets closed per day and they will close tickets faster, sometimes by closing them without solving anything. Grade sales on calls made and you will get calls. The behaviour is rational, and the metric quietly detaches from the outcome it was supposed to represent.

Write the definition before you build the dashboard

If you take one working habit from this part, take this one: write the metric definition in plain language, get the person who requested it to agree to the wording, and only then open your query editor. It feels slower for about ten minutes and it is faster for the rest of the project. Nearly every dashboard rebuild I have watched was caused by a definition that was assumed rather than agreed, and no amount of careful SQL rescues a number that means two things to two people.

Alongside that, keep the count small. Four or five KPIs with owners and targets beat twenty numbers with neither, because attention is the scarce resource and a wall of gauges spends it on nothing. Add the guardrail metric whenever you add a target, and revisit the whole set once a quarter, retiring anything that has not changed a decision since the last review. Metrics should be pruned like a garden, not accumulated like a collection.

You now know how to define a number precisely, tell a KPI from a metric, break a top line into its drivers, and spot the two ways metrics go wrong. Part 18 takes the next step and asks how you prove that a change actually moved one of these numbers, through A/B tests and experiments explained without the statistics jargon. Keep the Data Analyst guide open as your map, and revisit Part 11 if the word average starts feeling slippery again.

This week, pick the single number your team quotes most often and try to write its definition from memory in five sentences. If you cannot, you have found your first entry for the metric dictionary, and probably the source of an argument that has been running longer than anyone realises.

References

• Dimensions and metrics introduction, Google Analytics Help
• Key Performance Indicator (KPI) visuals, Microsoft Learn
• Goodhart’s law, background and origin

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts