To try the hands-on steps you will need Docker, to run Prometheus and Grafana locally. Everything here is free.
To try the hands-on steps you will need Docker, to run Prometheus and Grafana locally. Everything here is free.
Your app is running in production right now. How do you know it is actually healthy?
Not by guessing, and not by waiting for an angry customer. You know because the system is measuring itself and telling you. Monitoring is collecting those measurements. Observability is being able to ask why when a number looks wrong. Together they are the difference between finding a problem in a dashboard at 4pm and hearing about it on social media at midnight.
Who this is for: the college passout or career switcher who has built things that run but has never had to watch them stay up, and wonders how teams catch problems before users do.
Everything in this series so far has been about building and shipping. This part is about the other half of the DevOps loop from Part 3, the half beginners forget: watching what you shipped. A good place to start is knowing what to watch. Google is site reliability teams boiled it down to four numbers, the four golden signals, and they are the checklist to start any monitoring with.
✓ Latency: how long requests take to answer
✓ Traffic: how many requests are coming in
✓ Errors: how many requests are failing
✓ Saturation: how full your resources are, like CPU and memory
Watch those four and you catch the large majority of real problems. Everything else in monitoring is detail on top of these.
Monitoring tells you what, observability tells you why
The two words get used interchangeably, but the distinction is useful. Monitoring is watching known measurements and alerting when they cross a line: error rate above five percent, tell me. It answers what is wrong. Observability is broader. It is designing your system so you can explore its behavior and answer questions you did not think to ask in advance. It answers why is it wrong. Monitoring is the smoke alarm. Observability is being able to walk through the house and find which room is burning.
In practice, observability rests on three kinds of data: metrics, the numbers over time you will meet in this part; logs, the detailed event records that are the subject of Part 13; and traces, which follow one request across many services. You do not need all three perfectly on day one. Start with metrics and logs, and you can answer most questions a small system will throw at you.
| Signal | What it tells you | Kind of data |
|---|---|---|
| Metrics | Trends and totals over time | Numbers |
| Logs | What happened, line by line | Text events |
| Traces | One request across services | Timed spans |
Metrics are numbers over time
A metric is a measurement recorded again and again with a timestamp, so you can see how it moves. Request count, error count, response time, memory used: each becomes a line on a graph. The power of a metric is not the single value, it is the shape over time. A latency of 200 milliseconds means little on its own. A latency that was flat at 120 all week and suddenly spiked to 600 this afternoon tells you exactly when something went wrong.
Apps expose metrics as plain text that a monitoring system reads. Here is what raw request metrics look like when you fetch them directly.
$ curl -s localhost:9090/metrics | grep http_requests
http_requests_total{method="GET",status="200"} 48213
http_requests_total{method="GET",status="500"} 37Two counters: successful requests and failed ones. Divide the failures by the total over time and you have your error rate, one of the four golden signals, straight from the app.
The stack: Prometheus and Grafana
The most common open source setup pairs two tools. Prometheus collects and stores metrics. It works by pulling: on a schedule, it reaches out to each app and scrapes the metrics page you just saw, then keeps the history. Grafana sits on top and turns that history into dashboards, the graphs a team watches. Prometheus is the memory, Grafana is the eyes.
From metric to alert to a human
A dashboard nobody is looking at catches nothing at 3am. That is what alerts are for. You write a rule that says when this metric crosses this line for this long, notify someone. Here is a real Prometheus alert that fires when more than five percent of requests fail for ten minutes straight.
groups:
- name: web
rules:
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status="500"}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "Error rate above 5 percent for 10 minutes"The for: 10m matters more than it looks. Without it, a one second blip would page someone at 3am for nothing. Requiring the condition to hold for ten minutes filters out noise and only wakes a human for a real, sustained problem. That closes the monitoring loop: the app emits a metric, Prometheus stores it, a rule watches it, and a person is told only when it truly matters.
Dashboards: what to put on the screen
A dashboard is where the numbers become a picture a team can read at a glance, and the mistake beginners make is cramming forty graphs onto one. A good dashboard answers one question fast: is this service healthy right now. Start it with the four golden signals and little else, arranged so a glance tells you the story. Latency and error rate at the top, traffic and saturation below, all on the same timeline so you can line up a spike in one against a change in another.
The reason to share a timeline across panels is correlation. When errors jump at 2:14 and you can see traffic also jumped at 2:14, you have a lead in seconds. When latency climbs while CPU sits flat, you have ruled out one cause without touching a thing. A dashboard built this way turns a vague something is wrong into a specific here is where and when, which is most of the work in any incident. Extra detail graphs are fine, but keep them on a second screen, below the four signals that tell you whether to care at all.
SLOs turn signals into a promise
Once you can measure a signal, you can set a target for it, and that is where service level objectives come in. The vocabulary sounds formal but the idea is plain. You pick a signal to measure, decide the target you promise to hit, and accept that the gap below perfect is a budget you are allowed to spend. Chasing one hundred percent is a trap, because the last fraction of reliability costs more than it is worth and no system is ever perfect.
| Term | What it means | Example |
|---|---|---|
| SLI | A signal you measure | Percent of requests under 300ms |
| SLO | The target for that signal | 99.9 percent under 300ms |
| Error budget | How much you may miss | The 0.1 percent left over |
The error budget is the clever part. If you are meeting your objective with budget to spare, you can ship faster and take more risk. If you have burned through it with outages, you slow down and shore things up before adding more. It turns an argument about whether to ship into a number both developers and operators can look at together. You will not set objectives in your first week, but knowing the words, and that the goal is a sensible target rather than perfection, marks you as someone who thinks about reliability the way mature teams do.
Alert on symptoms, not on everything
The most common monitoring mistake is alerting on too much. Teams wire up an alarm for every metric, and soon a channel is buzzing all day with warnings nobody reads. This is called alert fatigue, and it is worse than no alerts, because when a real one finally fires, it is buried in noise and ignored. I would rather a team have five alerts they trust than fifty they mute.
The fix is to alert on symptoms your users actually feel, not on every internal cause. Do not page someone because CPU hit eighty percent, because high CPU might be perfectly fine. Page them because requests are failing or the site is slow, the things a user would notice. If the symptom is healthy, the system is doing its job no matter what the internal numbers say. Cause metrics are for investigating once an alert fires, not for waking people up. Build alerts around the four golden signals, keep them few and trustworthy, and people will actually respond when one goes off.
What are the four golden signals, and why alert on them? Answer: latency, traffic, errors, and saturation. They matter because they capture what users experience, so watching them catches the problems that actually affect people. Add that you alert on symptoms like errors and latency rather than on every internal cause like CPU, to avoid alert fatigue, and you have shown judgment, not just recall. That judgment is what separates someone who has run a system from someone who has only read about one.
Your first on-call shifts are not about designing dashboards. They are about reading them. When an alert fires, being able to open Grafana, see that latency spiked at a certain time, line it up with a deploy, and say this started right after the 2pm release is often the whole diagnosis. Nobody expects a junior to have built the monitoring. They expect you to read it calmly and connect a spike to a cause. That single skill makes you useful the first time something breaks on your watch.
Free, about thirty minutes. Run Prometheus and Grafana locally with Docker from Part 8, using their official images. Point Prometheus at its own metrics endpoint so it scrapes itself, then open Grafana and add Prometheus as a data source. Build one graph of a metric like
up, which is 1 when a target is reachable. How to check you did it right: the graph shows a live line, and if you stop the Prometheus container the value drops, which is monitoring catching a failure in front of you.Health checks are the simplest monitoring
Before any fancy stack, there is the humblest signal of all: a health check. Most apps expose a small endpoint, often called /health or /healthz, that returns a 200 when the app is fine and an error when it is not. It is a single yes or no answer to the question are you alive. This tiny thing does a surprising amount of work across the tools you already met in this series.
The load balancer from Part 6 pings it to decide whether to send a server traffic. Kubernetes from Part 9 uses it as a liveness and readiness probe, restarting a pod whose health check fails and holding traffic back until a new pod reports ready. Your monitoring pings it to know a service is up at all. One trivial endpoint, reused by three systems, is a good reminder that monitoring is not always complex. Sometimes it is one honest yes or no, checked often, that keeps a whole system self-correcting without a human in the loop.
Monitoring questions beginners ask
What is the difference between metrics and logs?
A metric is a number over time, like error count, cheap to store and perfect for trends and alerts. A log is a detailed text record of a single event, richer but heavier. You use metrics to notice something is wrong and logs to find out exactly what. Part 13 covers logs in depth.
What does pull based mean for Prometheus?
Instead of apps sending metrics out, Prometheus reaches in and scrapes each app on a schedule. This keeps apps simple, they just expose a metrics page, and lets Prometheus detect when a target stops responding, which is itself a useful signal that something is down.
What is a good number of alerts?
Few and trustworthy. If every alert demands action and none can be safely ignored, you have it right. If people mute the channel, you have too many. Quality of alerts beats quantity every time, because an ignored alert protects nothing.
Do I need to learn Prometheus specifically?
Learn the ideas first: metrics, the golden signals, symptom based alerting. Prometheus and Grafana are the most common tools to learn them on, and cloud providers offer managed versions. The concepts carry to any monitoring stack, which is the pattern across this whole series.
You can now watch a running system and know when it is unwell. Next up is Part 13 on logging and alerting, where you go from noticing a problem to reading the exact events that explain it.
New here? Start with Part 3 on the DevOps loop, whose feedback half this covers, and Part 6 on how the web works. The Cloud for Beginners series pairs well for where monitoring runs.
References
Prometheus documentation
Grafana documentation
Google SRE Book: Monitoring
About The Author
DrJha