Metrics tell you something is wrong. Logs tell you what. A log is a timestamped record of an event, and when you write them as structured data and ship them all to one searchable place, you can answer almost any what happened question in seconds instead of SSHing into forty machines. Alerting then watches those logs and metrics so a person hears about the real problems and only the real ones.
For the college passout or career switcher who has printed things to a console but never had to find one error among millions of lines spread across many servers.
A payment fails in production. On a single small app you open one log file and read it. Now imagine the checkout runs as forty containers spread across a dozen machines, and the failed request touched five of them. Without a plan, you are SSHing into box after box, running the log commands from Part 4, and trying to stitch a story together by hand while the clock runs. Centralized, structured logging is how teams turn that nightmare into a single search.
It starts with what a single log line looks like. Here is one written as structured data, which means as fields a machine can filter, not just a sentence.
{"time":"2026-07-02T21:14:12Z","level":"ERROR",
"service":"checkout","request_id":"a1b2c3",
"msg":"payment failed","err":"gateway timeout"}Each piece is a named field: when it happened, how serious it is, which service, which request, a message, and the error. Because it is structured, you can later ask for every log where level is ERROR and service is checkout, and get an exact answer.
Part 12 taught you to watch metrics and get alerted when a number goes wrong. This part is what you do next: open the logs and read the actual events behind the number. Metrics point at the fire, logs show you which wire sparked.
Structured beats a wall of text
Traditional logs are lines of plain text a human reads one at a time. That is fine for one app on one machine. It falls apart at scale, because searching millions of freeform lines for the ones that matter is slow and unreliable. Structured logs solve this by writing each event as named fields, usually JSON. Now a computer can filter by field: show me errors, from this service, for this user, in this ten minute window. The difference between the two is the difference between reading a book to find a fact and querying a database.
| Question | Plain text logs | Structured logs |
|---|---|---|
| Find all errors | Grep and hope | Filter by a field |
| Filter by user or request | Hard, brittle | Exact and fast |
| Machine readable | Not really | Yes |
| Good for humans | Yes, in small amounts | Yes, with a viewer |
Log levels turn down the noise
Not every log deserves equal attention, so each carries a level marking how serious it is. Levels let you record rich detail but only surface what matters. In development you might show everything down to DEBUG. In production you usually keep INFO and above, and alert only on ERROR. The level is the volume knob for your logs.
| Level | Use it for |
|---|---|
DEBUG | Fine detail while developing |
INFO | Normal events worth recording |
WARN | Something odd but not broken yet |
ERROR | A request or job actually failed |
FATAL | The app cannot continue |
Centralized logging: one place to search
The core move at scale is aggregation. Every app and container ships its logs to one central system that stores and indexes them, so you search across your whole fleet from a single screen. Common stacks are the Elastic based tools and Grafana Loki, plus managed cloud services. The plumbing varies, the idea does not: collect everywhere, store centrally, search once.
Finding one request across many logs
Here is the trick that makes structured logs shine. Give every incoming request a unique id and attach it to every log line that request produces, in every service it touches. That is the request_id in the log at the top. Now a single search for that id pulls the entire journey of one request across your whole system, in order. Without it, you are guessing which lines belong together. With it, you replay exactly what happened.
$ cat app.log | jq 'select(.request_id=="a1b2c3")'
{ "service": "api", "msg": "received order" }
{ "service": "checkout", "msg": "charging card" }
{ "service": "checkout", "level": "ERROR", "msg": "payment failed" }The tool jq filters JSON on the command line, and here it pulls one request out of a noisy file. The story is right there: the order came in, checkout tried to charge the card, and the charge failed. Three lines, one id, whole picture.
Alerting: metrics wake you, logs explain
Alerting sits across both parts. The cleanest division is to alert mostly on metrics, which are cheap to watch and good at spotting trends, and use logs to investigate once an alert fires. You can also alert on logs directly, for example when errors of a certain kind spike, but the same discipline from Part 12 applies: alert on meaningful patterns, not on every stray ERROR. One error in a million requests is not an emergency. A sudden burst of the same error is.
The path a log takes
It helps to see the full journey of a single log line, because each stage is a place a team makes a choice. The app writes the line, usually to its standard output rather than a file, which fits the disposable containers from Part 8. A small agent running alongside collects that output and ships it to the central store. The store indexes it so search is fast. You query it through a dashboard. And after a set time, the line expires and is deleted to control cost. Five stages, and every one is a decision: what to write, how to ship, how long to keep.
The reason writing to standard output matters is that it hands the hard parts to the platform. Your app does not open files, rotate them when they fill the disk, or worry about shipping. It just prints, and the surrounding system, whether Docker or Kubernetes or a cloud agent, does the rest. This is the same lesson as the disposable container: keep the app simple and let the platform handle plumbing. An app that writes its own log files inside a container is fighting the model, and it is usually the first thing to break when that container is replaced.
When you join a team, find out three things about logging on your first day: where the logs go, how to search them, and what the request id field is called. With those three answers you can debug almost anything the system does. Without them, you will waste your first incident just hunting for where the logs even live. It is a two minute question to a teammate that saves you an hour under pressure later, so ask it before you need it.
Log less, and never log secrets
Beginners are told to log everything, and it is bad advice. Logs cost money to store and index, and a flood of noise makes the useful lines harder to find, not easier. Log with intent: enough to reconstruct what happened, not a running commentary of every function call. High volume systems often sample, keeping a fraction of routine logs and all of the errors. Less, chosen well, beats more.
The rule I will not soften: never log secrets or personal data. Passwords, API keys, card numbers, and personal details must never land in a log, because logs get shipped, stored, and read widely, and a secret in a log is a secret leaked. This is a real and common breach, not a theoretical one. Scrub sensitive fields before they are ever written. A log is a record many people and systems can see, so treat it as public inside your company and act accordingly.
How would you trace a single failing request across several services? Answer: attach a unique request id to the request at the edge and include it in every log line it produces in every service, then search your central log store for that id to see the whole path in order. Mention structured logging so the search is exact, and that this is why correlation ids exist. That answer shows you understand debugging a distributed system, which is exactly the pain logging is meant to solve.
When something breaks, the person who can find the right logs fast is the one who resolves it. Knowing how to open your log tool, filter to ERROR for the affected service in the right time window, and pull one request by its id turns a two hour hunt into a five minute one. Nobody expects you to have designed the logging pipeline. They expect you to search it well under pressure, and that skill makes you the person others turn to when the alert fires.
Free, about twenty minutes. Run any container from Part 8 and use
docker logs to see its output. If the app emits JSON logs, pipe them through jq to filter by a field, for example only the ERROR lines. If not, use grep from Part 4 to pull matching lines. How to check you did it right: you can show only the error lines out of a full, noisy log, and if you have a request id field, you can isolate a single request. You just did centralized search in miniature.Turning logs back into numbers
Logs and metrics are not separate worlds. A common and useful move is to count log lines to produce a metric, which links this part straight back to Part 12. Count the ERROR lines per minute and you have an error rate you can graph and alert on, built entirely from logs. Watch the rate of a specific message like payment failed and a slow leak shows up as a rising line long before anyone opens the raw logs.
This is where the two halves of observability meet. The metric, cheap and fast, tells you the payment failure rate jumped from 2 an hour to 200 an hour at 2:14. The logs, rich and detailed, tell you every one of those failures said gateway timeout, so the problem is the payment provider, not your code. Neither alone gives the full picture. The number tells you how bad and when, the logs tell you what and why, and a good engineer moves between them without thinking about it. That fluid motion, notice in a metric, confirm in the logs, is the core loop of running software in production, and it is the habit this part and the last one together are meant to build.
Logging questions beginners ask
Logs or metrics, which should I set up first?
Both are basic, but if forced to pick, logs come first for a small app, because they tell you what actually happened. Metrics become essential as soon as you care about trends and alerting. In practice you want both quickly, with metrics for noticing and logs for explaining.
How long should I keep logs?
Long enough to investigate incidents and meet any legal rules, but not forever, because storage adds up. A common pattern is keep detailed logs for a few weeks and summaries for longer. Retention is a cost and privacy decision as much as a technical one.
What is the difference between logging and tracing?
A log is one event. A trace follows one request across services and times each step, which is great for finding where time is spent. Correlation ids in logs give you much of the same power, and full tracing tools add the timing view on top. Start with structured logs and ids.
Do containers change how I log?
Yes, in a helpful way. Containers are told to write logs to their standard output, and the platform collects them, so you do not manage log files inside the container. This fits the disposable nature of containers from Part 8, and it is why docker logs and Kubernetes log commands just work.
You can now read the story a running system tells about itself. Next up is Part 14 on cloud basics, the ground all of this has quietly been running on.
New here? Start with Part 12 on monitoring, which this pairs with, and Part 4 on the command line for grep and jq. The Python for Beginners series pairs well too.
References
Grafana Loki documentation
Elastic: What is the ELK stack?
The Twelve-Factor App: Logs
About The Author
DrJha