A server fell over at 03:00. You were asleep, the watchdog rebooted it, and by morning it was serving traffic again as if nothing happened. Now you want to know what killed it. You log in, reach for the logs from before the reboot, and the machine tells you they were never there.
$ journalctl -b -1
Specifying boot ID or boot offset has no effect, no persistent
journal was found.That one line is the whole lesson of this part in miniature. The systemd journal was recording everything, but it was writing to memory, and memory does not survive a reboot. The tool you drive it with is journalctl, and it can slice logs by service, by boot, by priority and by time. First you learn to read the journal. Then you make sure it is still there when you need it most.
Here is the same tool working the way it should, annotated so nothing is a mystery.
$ journalctl -u ssh -b # ssh unit, this boot only
$ journalctl -u ssh -b -p err # only error priority and worse
$ journalctl -f # follow new lines live, like tail -f
$ journalctl --since '10 min ago' # time window, plain English works
$ journalctl -k -b # kernel messages, this boot (dmesg)
$ journalctl -u nginx -e # jump to the end, newest linesEvery one of those is a filter. -u is by unit, -b is by boot, -p is by priority, --since is by time, -k is kernel only. Stack them and you go from ten thousand lines to the five that matter.
On modern distros the systemd journal collects log messages from the kernel, services and everything in between into one binary store, and
journalctl reads it. Learn four filters, -u, -b, -p and --since, and you can find any event fast. Then set the journal to persistent so those events outlive a reboot, and cap its size so it never fills the disk.If you came from Windows Event Viewer or a hosting panel where logs were a scrolling web page, this is the command-line version of the same job, only faster once your fingers know it. And if you already run a fleet, the part that pays for itself is the priority and boot filtering, because it turns a blind grep through a giant file into a precise query.
Where Linux logs actually live
There are two log worlds on a typical Linux box, and knowing which one you are in saves a lot of confusion. The old world is plain text files under /var/log, written by a syslog daemon such as rsyslog. The new world is the systemd journal, a binary store that journald writes and only journalctl reads. On Debian and Ubuntu both worlds run at once: journald collects everything, and rsyslog also writes the familiar /var/log/syslog. On RHEL 9, Rocky and AlmaLinux the journal is the primary store and rsyslog is present but leans on the journal underneath.
Why a binary store at all, when text files worked for decades? Because the journal keeps structured fields on every entry, so you can filter by the exact unit, the priority, the process ID or the boot, without a regular expression guessing at line shapes. A few plain text files still live on under /var/log and are worth knowing.
| Path | What it holds | Family |
|---|---|---|
/var/log/syslog | General system messages | Debian, Ubuntu |
/var/log/messages | General system messages | RHEL family |
/var/log/auth.log | Logins, sudo, sshd | Debian, Ubuntu |
/var/log/secure | Logins, sudo, sshd | RHEL family |
/var/log/journal/ | Persistent binary journal | Both, when enabled |
Here is where I disagree with a lot of older tutorials. Reaching for grep sshd /var/log/auth.log still works, but it is the slow path now. journalctl -u ssh -p warning -b asks a sharper question and gets a cleaner answer, because you are filtering on real fields instead of hoping a word appears on the right line. Learn the journal first. Fall back to the text files when a tool only writes there.
The journalctl flags you will use every day
Running journalctl with no arguments dumps the whole journal into a pager, oldest first, which is rarely what you want. The value is in the filters. These are the ones that carry the day.
| Flag | What it does | Example |
|---|---|---|
-u | Filter to one unit | journalctl -u nginx |
-b | This boot; -b -1 the previous one | journalctl -b |
-p | Priority and worse | journalctl -p err |
-f | Follow live, like tail -f | journalctl -f |
--since, --until | Time window | journalctl --since '09:00' |
-k | Kernel messages only | journalctl -k |
-e | Jump to newest entries | journalctl -u nginx -e |
-r | Reverse, newest first | journalctl -r |
They combine. The single query I type more than any other, when a service just died, is this:
$ journalctl -u myapp -b -p err --no-pager
Jul 03 03:07:12 web01 myapp[1834]: FATAL: could not connect to
database: connection refused
Jul 03 03:07:12 web01 systemd[1]: myapp.service: Main process
exited, code=exited, status=1/FAILURE
Jul 03 03:07:12 web01 systemd[1]: myapp.service: Failed with result
'exit-code'.That is the crash from the opening scenario, and it took one line to find. The application could not reach its database, exited with status 1, and systemd marked the unit failed. No scrolling, no grep. The --no-pager flag prints straight to the terminal instead of opening a pager, which is what you want when you are copying output into a ticket or a script.
Priority levels, and why -p err is your friend
Every journal entry carries a priority, a number from 0 to 7 borrowed straight from the old syslog standard. Lower numbers are more severe. When you pass -p err you are saying show me priority 3 and everything worse, so errors, criticals, alerts and emergencies, and drop the routine chatter. On a busy box that filter alone cuts the noise by an order of magnitude.
| Number | Name | Means |
|---|---|---|
| 0 | emerg | System is unusable |
| 1 | alert | Act immediately |
| 2 | crit | Critical condition |
| 3 | err | Something failed |
| 4 | warning | Might become a problem |
| 5 | notice | Normal but notable |
| 6 | info | Routine information |
| 7 | debug | Verbose detail |
You can name the level or use the number, so -p 3 and -p err are the same. A pair I lean on when triaging a fresh incident:
$ journalctl -p err -b # everything that errored this boot
$ journalctl -p warning..err -b # just warnings through errorsThe range form with two dots is the one most people never learn, and it is the sharpest of the lot. It hides the emergencies you probably already know about and the info noise you do not care about, leaving the middle band where the early signs of trouble usually hide.
The default that loses your logs
This is the part that bites real people, so it gets the most room. Whether your logs survive a reboot comes down to one setting, Storage, in /etc/systemd/journald.conf, and its behaviour is not obvious. The value that ships on most systems is auto, and auto means the journal is persistent only if the directory /var/log/journal already exists. If it does not, the journal writes to /run/log/journal, which is memory, and memory is wiped on reboot.
Why the default catches Ubuntu users out
On many minimal and cloud Ubuntu images the /var/log/journal directory does not exist, so Storage=auto quietly resolves to volatile, and your logs live only until the next reboot. RHEL 9 ships Storage=persistent in its config, which creates the directory for you, but even there a bare install may not have it populated until the setting takes effect. The honest rule is to not trust the default. Check it, and make it persistent on purpose.
Making it persistent is two commands. Set Storage=persistent, then tell journald to pick it up.
$ sudo mkdir -p /var/log/journal
$ sudo sed -i 's/^#Storage=auto/Storage=persistent/'
/etc/systemd/journald.conf
$ sudo systemctl restart systemd-journald
$ journalctl --list-boots # now shows more than one line
-1 8f3c... Thu 2026-07-02 22:10:04 UTC ... Fri 2026-07-03 03:00:11 UTC
0 a17d... Fri 2026-07-03 03:00:44 UTC ... Fri 2026-07-03 09:31:52 UTCThe moment journalctl --list-boots shows a boot with a -1 next to it, you have history that survives a restart, and journalctl -b -1 will hand you the logs from before the last reboot instead of that flat refusal at the top of this article. My verdict is plain: on any machine you will ever need to investigate after a crash, which is all of them, set Storage=persistent the day you build it. The volatile default trades away exactly the logs you reach for at the worst moment.
A junior admin swears a service logged an error before the box rebooted, but
journalctl -b -1 comes up empty and they start doubting themselves. Nothing is broken. The journal was volatile, the reboot wiped it, and there was never anything to find. The tell is a single line at the top of any boot query: no persistent journal was found. See that once and the first fix is always the same, create /var/log/journal and restart journald, before you chase a ghost.Keeping the journal off your disk
Once the journal is persistent, the obvious worry is that it grows without bound. It does not. journald caps itself, and the default cap is SystemMaxUse set to 10 percent of the filesystem the journal lives on, with that calculated value capped at 4 GB. It also keeps 15 percent of the disk free by default through SystemKeepFree. So on a 100 GB root, the journal will use at most about 4 GB, not 10, because the 4 GB ceiling wins. Check what it is using now with one command.
$ journalctl --disk-usage
Archived and active journals take up 1.2G in the file system.When you want to reclaim space now, or trim history before archiving a box, vacuum does it by size or by age.
$ sudo journalctl --vacuum-size=500M # keep at most 500 MB
$ sudo journalctl --vacuum-time=2weeks # drop anything older than 2 weeks
Vacuuming done, freed 712.3M of archived journals.To make a smaller cap permanent, set SystemMaxUse=500M in journald.conf and restart journald, the same edit-and-restart pattern as the storage change. On a small VPS with a 20 GB disk, I set an explicit cap rather than trust the percentage, because 10 percent of a tiny disk is still enough log to matter when the disk fills.
"A service failed to start after the last reboot and you need to know why. Walk me through it." The answer they want, in order:
systemctl status name for the summary, then journalctl -u name -b -1 -p err to read that unit’s errors from the previous boot. The follow-up trap is whether -b -1 even works. If the journal was volatile, it does not, and naming that failure mode out loud is what separates a real operator from someone reciting flags.Make your journal persistent and prove it. Run
sudo mkdir -p /var/log/journal then sudo systemctl restart systemd-journald. Reboot the machine, log back in, and run the exact verify command:journalctl --list-boots | wc -lIf that number is 2 or more, your logs now cross reboots. Before the change it would have printed
1, because only the current boot existed.Questions I actually get
Do I still need rsyslog if I have the journal? Not for most single servers. The journal covers you. Keep rsyslog when something depends on the text files, or when you forward logs to a central collector that speaks syslog. Otherwise the journal is enough on its own.
What does the -x flag actually add? journalctl -xe attaches catalog text with suggested explanations to some messages. It helps on unfamiliar errors and becomes noise once you know the system. Useful when you are stuck, skippable when you are not.
Why does journalctl open a pager and how do I stop it? It pipes through less so long output is scrollable. Add --no-pager for a straight dump, which is what you want in scripts, or set -e to jump to the end and keep the pager.
Can a normal user read the journal? A user sees their own messages. To read the full system journal, either use sudo or add the account to the systemd-journal group, which grants read access without full root. Groups came up back in Part 5.
How do I watch two services at once? Repeat the flag: journalctl -u nginx -u php-fpm -f follows both live in one stream, which is how you catch a web server and its backend failing in the same instant.
Logs stop being a wall of text once you filter them. You ask for one unit, one boot, one priority, one time window, and the journal answers. Set it persistent on day one, cap it so it never fills the disk, and the next 03:00 incident leaves you a trail to follow instead of an empty prompt. Part 11 moves from reading the system to the disks underneath it, partitions, filesystems and mounting. Following the whole path? The Linux for Beginners complete guide lists every part in order, and the same investigate-after-the-fact instinct shows up in the VMware for Beginners and Cloud for Beginners guides.
References
• journalctl(1) manual page
• journald.conf, Storage and SystemMaxUse
• systemd-journald service
• Red Hat: enable persistent journal logging
About The Author
DrJha