A support ticket lands: the site threw errors around 3am, find out what happened. You have a log file with two million lines and a few minutes. Opening it in an editor is hopeless. This is the work an admin does more than any other, turning a mountain of text into a single clear answer, and Linux ships three tools built exactly for it. Learn them and a log stops being a wall of noise and becomes a question you can ask.

The three are grep to find lines, sed to change them, and awk to pull them apart by column. Around those sit a few small helpers. Together they handle almost every text job you will meet before you ever need a real programming language.

Reach for grep to find lines that match a pattern, sed to substitute or delete text in a stream, and awk to treat each line as columns and work with fields. Feed them with pipes and finish with the small helpers, cut, sort, uniq, tr, and wc. Most real questions about a log or a config file are answered by three or four of these joined in a single line, no script required.

grep prints lines that match a pattern and drops the rest. It is the fastest way to pull the needle from the haystack. A few flags cover almost everything: -i ignores case, -r searches a whole directory, -n shows line numbers, -c counts matches instead of printing them, and -v inverts the match to show lines that do not contain the pattern.

Two habits pay off early. Use grep -v '^#' to strip comments and read the real settings in a config file, and use -E when you want alternation, so grep -E 'error|fatal|panic' catches any of the three. The caret in '^#' is a regular expression anchor for the start of a line, and learning a handful of these symbols multiplies what grep can do.

Where grep sees whole lines, awk sees each line as a set of fields split on whitespace, numbered $1, $2, and so on, with $NF meaning the last field and $0 the whole line. That makes awk the tool for anything arranged in columns, which is most logs and many config files. Change the separator with -F when the file uses something other than spaces.

Counting fields from one is the whole idea. Once you can name a column by its number, you can print it, filter on it, or add it up, and $NF lets you grab the last field even when the number of columns changes from line to line.

The move that makes people love awk is running totals. It can keep a variable across every line and print the result at the end, so summing a column is one short program. Here is the total bytes served from an access log, where the size is the tenth field:

Read that first line as three parts: for every line add the tenth field to sum, and after the last line print the total in megabytes. That pattern, an action per line and a summary at END, is the heart of awk and covers a huge share of real reporting.

sed changes text as it flows past. Its most used form is substitution, s/old/new/, with a trailing g to replace every match on a line rather than just the first. By default sed prints to the screen and leaves the file alone, which is the safe way to preview a change before committing it.

That last line is the one to handle with care. The -i flag rewrites the file in place, and adding a suffix like .bak makes sed save the original first. Always run the command without -i once to see the result on screen, then add -i when you are sure. An in place edit with a bad pattern rewrites the file instantly, and without the backup there is nothing to fall back on.

A script with sed -i that works on your Linux box fails on a Mac or a BSD server with a confusing error. The reason is that the two versions of sed disagree about -i. GNU sed, on Linux, treats -i.bak as in place with a .bak backup, and bare -i as in place with none. BSD sed, on macOS, requires an explicit argument, so bare -i eats the next word as the backup suffix and mangles the command.

For anything that must run on both, always give an explicit suffix and mind the spacing, or avoid -i in portable scripts and write to a new file then move it. On a single Linux fleet this never comes up; the day you move a script to a Mac, it will.

A handful of tiny commands round out the toolkit, and they show up at the end of most pipelines. cut pulls fixed fields or characters, sort orders lines, uniq -c collapses and counts adjacent duplicates, tr swaps or deletes characters, and wc -l counts lines. None does much alone, but chained they are how you turn a stream into a tidy answer.

One caution that trips everyone once: uniq only collapses duplicates that sit next to each other, so it almost always follows sort. Run uniq on unsorted input and it misses duplicates scattered through the file, which looks like a bug but is doing exactly what it was told.

When a small text job appears, many people open a Python file out of habit. For counting, filtering, and summing columns, that is slower to write and slower to run than a one line pipeline of grep and awk that is already on every machine. My rule is to reach for the shell tools first, because for the common cases they are faster to think in and need nothing installed.

The honest other half of that rule is knowing when to stop. Once an awk program grows past a few lines, sprouts nested conditionals, or needs real data structures, it has outgrown the job and become hard to read. That is the moment to switch to Python or another proper language. grep and awk are superb at the ninety percent of text work that is simple; forcing them through the last ten percent produces a one liner nobody, including you next month, can maintain.

Answer a real question with one pipeline. Find the ten addresses that appear most in your auth log, which is how you spot who is hammering a server:

grep 'Failed password' /var/log/auth.log 
  | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head

Build it up one stage at a time and watch each pipe narrow the result. The $(NF-3) grabs the address field counting back from the end, which is steadier than a fixed number when the message wording shifts.

From an access log, how would you total the bytes served, and separately count requests per status code?

For the total, awk '{sum += $10} END {print sum}', because awk can carry a running sum and print it at the end. For the counts, pull the status field and pipe through sort | uniq -c | sort -rn. The interviewer is checking that you know awk owns column math and running totals, while sort plus uniq -c is the standard way to tally, and that you remember uniq needs sorted input. Reaching for a spreadsheet or a script here is the weaker answer.

When do I need grep -E or grep -P? Plain grep uses basic regular expressions, where + and | are literal. Use -E for extended expressions so those work as operators, and the rarer -P for Perl style patterns when you need features like lookahead. For most work, -E is enough.

Why did my grep find nothing when the text is clearly there? Usually case. grep is case sensitive by default, so Error does not match error. Add -i. The other common cause is a regex character like . or * being taken as a pattern rather than a literal.

Is awk a whole programming language? Yes, a small and capable one, with variables, conditionals, and loops. For an admin, the useful ninety percent is printing fields, filtering on a condition, and summing with END. You can go deeper, but you rarely need to.

How do I replace a string across many files at once? Combine find and sed, for example find . -name '*.conf' -exec sed -i.bak 's/old/new/g' {} +. Preview first by running the sed without -i on one file, and keep the backup suffix so a mistake is recoverable.

What is the difference between cut and awk for columns? cut is simpler and faster for fixed delimiters, like a colon separated file. awk is better when fields are separated by variable whitespace, when you need math, or when you want to filter on a field before printing.

With grep, sed, and awk in hand, no log or config file is a wall of text anymore; it is a set of questions you can answer in one line. Next we protect the data all this runs on, with proper copying and backups using tar and rsync, including the trailing slash that quietly changes what rsync does. Work the series in order from the Complete Guide.

GNU grep manual
GNU sed manual
GNU awk user guide