The site is down. Someone is standing at your desk. Your pulse is up, and every instinct says do something fast: restart the service, reboot the box, change three settings at once and hope. That instinct is why outages last longer than they should. Rebooting often makes the symptom vanish for an hour and takes the evidence with it, so the same failure returns that night with nobody any wiser.
The admins who fix things quickly are not smarter or faster typists. They follow a method. They turn a vague report into a precise question, check the cheap things first, and change one thing at a time so they always know what did what. This part is that method, plus the handful of commands that answer most questions in the first two minutes.
Early in your career? The discipline here matters more than any single command; the method is what stops you flailing. Been on call for years? You know this, but it is worth having written down for the day the pressure makes you skip a step. It pulls together the logs and process skills from earlier parts into one workflow.
Do not change anything yet. First turn the report into a precise problem: what is actually broken, for whom, and since when. Reproduce it if you can. Check the cheap, common causes: is the disk full, is the service running, is the network reachable. Read the logs before you touch a setting. Then form one hypothesis and change one thing, so you can always undo it and always know what helped. Guessing changes many things and learns nothing; the method changes one and learns every time.
Turn the report into a real problem
The site is down is not a problem statement, it is a feeling. Sharpen it before you act. Is the whole site down or one page? For everyone or one user? Since when, and what changed around then? A deploy, a full disk, a certificate that expired at midnight. Half of all fixes fall out of this step alone, because the honest answer to what changed points straight at the cause. The most useful question in this job is what changed just before it broke.
Then reproduce it. A bug you can trigger on demand is a bug you can watch, instrument, and confirm fixed. A bug you only hear about secondhand is a rumor you are chasing. If you cannot reproduce it, that itself is a clue: the trigger is something you have not accounted for yet.
The first two minutes: four commands
Before any theory, run the cheap checks. Most incidents are one of a small set of boring causes, and four commands catch the majority of them. Is the service running, what do its logs say, is the disk full, and is memory exhausted.
$ systemctl status nginx # is it even running, and why did it stop
$ journalctl -u nginx -n 50 --no-pager # the last 50 log lines
$ df -h # is any filesystem full
$ free -h # is memory or swap exhausted| Question | Command | Red flag |
|---|---|---|
| Is the service up | systemctl status | failed or inactive |
| What went wrong | journalctl -u name | an error near the stop time |
| Is the disk full | df -h and df -i | 100% used or inodes gone |
| Out of memory | free -h, dmesg | no free memory, OOM killer messages |
| Is the network ok | ping, ss -tlnp | no route, or nothing listening |
A full disk deserves special mention because it breaks things in confusing ways. When a filesystem fills, programs cannot write logs, sessions, or temp files, so a database refuses connections and a web app throws errors that look nothing like a disk problem. Whenever the symptoms are weird and scattered, run df -h early. It rules out or catches a whole family of strange failures in one line.
Read the signal, not just the number
The number that fools the most people is the load average. A load of 8 does not mean the CPU is 8 times too busy. Load counts processes that are running plus processes stuck waiting on the disk or network, in an uninterruptible state. So a machine can show a frightening load while the CPU sits nearly idle, because everything is blocked waiting on a slow disk or a hung network mount. Read load next to CPU usage, never alone.
You see this split in top. Watch the CPU line: a high us or sy number means real work, while a high wa number means the processor is idle, waiting on input and output. The two point at completely different fixes, one at code or capacity, the other at a slow or failing disk. Reading the wrong one sends you hours in the wrong direction.
Keep the five fields on the top CPU line straight, because each one points at a different kind of problem:
| Field on the top CPU line | What it counts | A high value points at |
|---|---|---|
us | Time in user code | An application doing real work or a busy loop |
sy | Time in the kernel | Heavy system calls, drivers, or networking |
wa | Waiting on input and output | A slow or failing disk, not the CPU |
id | Idle time | Spare capacity, the CPU is not the limit |
st | Steal time | A noisy neighbor on a shared or cloud host |
The pair that catches people is wa against us. A machine pinned at high wa looks overloaded but has an idle processor, so throwing more CPU at it changes nothing. The fix is the storage, and the top line tells you that in one glance once you know which field to read.
Rebooting is the troubleshooting equivalent of turning it off and on again, and it has a real cost: it destroys the state that would have told you the cause. The full memory, the stuck process, the open file that was never released, all gone. The symptom clears, everyone relaxes, and the bug comes back at 3am with no evidence left to study.
Reboot when you have gathered what you need or when nothing else will restore service, not as the first move. Capture the logs, note the memory and load, save the output, and then restart if you must. A reboot with the evidence saved is fine. A reboot that erases the only clues is how a one hour incident becomes a recurring one.
Change one thing, and write it down
Once the basics are ruled out, form a single hypothesis and test it with a single change. This is the rule that separates fixing from fiddling. If you change five settings and the problem clears, you have no idea which one mattered, whether the other four introduced new problems, or how to explain it later. Change one thing, test, and if it does not help, put it back before trying the next. The habit feels slow and is far faster than the alternative, because you are never lost.
Keep a running note as you go: the time, what you observed, what you changed, what happened. On a long incident this note is the difference between a clean handoff to the next person and starting from zero. It is also the raw material for the writeup afterward, so the same failure teaches the whole team once instead of surprising each person in turn.
Cut the problem in half
When a system has many moving parts, do not poke at them in a random order. Split the path in half and test the middle. A web request travels a chain: the client, then DNS, then the network and firewall, then the web server, then the application, then the database. Each hop is a place it can break. Test one point in the middle and you learn which half the fault lives in, then repeat inside that half. This is binary search applied to a running system, and it turns a big vague area into a small certain one in a few steps.
The single most useful cut is inside versus outside the machine. Ask the server to talk to itself, then ask the outside world to reach it. If the local test works and the remote one does not, the application is healthy and the gap is in the network or firewall in front of it. One test just eliminated half the stack.
$ curl -I http://localhost # does the app answer on the box itself
HTTP/1.1 200 OK
$ curl -I http://203.0.113.10 # does it answer from outside
curl: (7) Failed to connect after 3000 ms
# app is fine locally, so the fault is the network or firewallThat single pair of commands rules out the whole application and database layer and points you at the firewall rules from the hardening part. Without the split, you might have spent an hour reading application logs that were never going to show anything, because the request was not reaching the application at all.
On a test VM, break something on purpose and practice finding it without guessing. Stop a service, then work the method: status, logs, basics.
sudo systemctl stop nginx # cause the fault
curl -I localhost # observe the symptom
systemctl status nginx # define: it is stopped
journalctl -u nginx -n 20 --no-pagerNotice how the symptom (a refused connection) and the cause (a stopped service) are two different facts, and the method walks you from one to the other. Then start it again and confirm the fix.
A user reports the website is down. Walk me through what you do.
Do not jump to a cause. Start by defining the problem: down for everyone or one user, whole site or one page, since when, what changed. Then the cheap checks in order, service status, its logs, disk, memory, network. Say you would read the logs before changing anything and change one thing at a time. The interviewer is testing for a calm, ordered process, not whether you can name a clever command. Ending with I would note what I did and write it up afterward is what a senior answer sounds like.
Questions from real outages
Where do I look first, the logs or the metrics? Both, quickly. A glance at service status and its recent logs tells you what failed, while disk, memory, and load tell you the condition of the machine. Start with whichever your symptom points at, but do not skip the other.
The logs show nothing. Now what? Widen the window and the sources. Check the system journal as well as the app log, look at the time just before the symptom, and raise the log level if the app supports it. Silence in one log is not silence everywhere.
Is it really always DNS? Often enough that it is a running joke. A service that seems slow or half broken is frequently waiting on name resolution. Test with an address instead of a hostname; if that works and the name does not, you have found it.
When should I escalate? When you have exhausted the cheap checks and one or two solid hypotheses without progress, or when the fix risks data. Escalate with your notes, not just a plea for help. The evidence you gathered is what lets the next person start where you left off.
What if the fault is intermittent and I cannot reproduce it? These are the hardest, because the method needs something to observe. Add logging or a small monitor and wait for it to happen while you are watching, capturing the load, memory, and connections at the moment it occurs. Then correlate the times it struck against deploys, cron jobs, backups, and traffic peaks. The trigger is almost always hiding in that timeline, and the pattern is the clue that a single snapshot cannot give you.
How do I stop the same thing recurring? Write a short blameless note: what broke, the root cause, how you confirmed it, and what would prevent it. Fixing the incident restores service; the note is what stops the next one.
A method turns panic into a sequence of small, answerable questions, and that is the skill that makes an admin trusted with the important systems. You now have the whole toolkit from this series. The next part puts it somewhere safe to practice: your own Linux home lab, where you can break things on purpose and fix them without fear. Work the series in order from the Complete Guide.
References
About The Author
DrJha