Linux/Basic commands/grep

The famous tool grep is a command-line utility for searching plain-text data sets for lines that match a regular expression.

Pre-requisites: Basic familiarity with directories and paths in the shell, and (if working with PCRE) the perl-doc package.

grep fits in as the simplest component of the grep / AWK / Perl triumvirate from which you can choose the optimal tool for the job. grep can search in either a stream of text via a pipe or in one or more files. In its most basic form, it prints the lines which match the pattern(s) given as run time arguments, in this case a simple string is sought in a single file:

grep 'search' access.log

Preparatory Exercise: If it's not already installed, add the perl-doc package to your system or arrange for your system administrator to install it for you.

Exercise: Check the reference manual for grep and read the two sections titled "Description" and "Synopsis" there: man grep

The Fundamentals of grep (45 minutes)

Exercise: Fetch a plain text book into a file to be named book.txt using wget -O book.txt https://www.gutenberg.org/cache/epub/2701/pg2701.txt. Use grep to find the chapters, grep 'CHAPTER' book.txt and use the terminal window's scroll bar to view the whole output. What else did grep find besides chapter headings?

The text found can be anywhere on a line when anchors are not applied. Using a caret ^ will designate that grep should look for the text at the start of the line. Likewise, a dollar sign $ will force grep to look for the text at the end of a line.

Exercise: Use grep with an anchor to find the actual chapters, grep '^CHAPTER' book.txt Compare the result to the previous pattern. What is the difference?

Exercise: Use grep while treating uppercase and lowercase letters alike via the -i option. grep -i '^CHAPTER' book.txt Compare the result to the first pattern. What is the difference?

Searches can be negated with the -v option. So, if we want all the lines except those with the word "search", it could look like this:

grep -v 'search' access.log

Furthermore, if we're looking for the exact word, "search", and not variations like "searches", "searched", and "researched" and so on, we can apply the -w option. The following will find lines with the string "search" but only when it is a whole word.

grep -w 'search' access.log

Exercise: Try the pattern 'inkling', grep 'inkling' book.txt then try it with the -w option to find only whole words grep -w 'inkling' book.txt What is the difference between the two searches?

And, of course, multiple options can often be used together.

grep -v -w 'search' access.log

Below, the number of lines with the whole word "search" is reported:

grep -c 'xmlUrl' blogroll.opml

Exercise: How many times is the narrator mentioned by name, or his good friend Queequeg by his correct name? grep -c 'Ishmael' book.txt and grep -c 'Queequeg' book.txt

Next, it is possible to track a live file as it updates using tail which grep then receives as input via a pipe. Thus as each new line is added to the file, it is sent to grep for processing. A web server log is a typical example of a file which is continually growing:

tail -f access.log | grep 'search'

In either case above, reading directly or through a pipe, any lines containing the fixed string "search" will be printed out.

Exercise: In one terminal window, start a loop to add the current time in seconds to a text file, while sleep 1; do date > /tmp/times.txt; done and in a second terminal window, try following that file with tail like this tail -f /tmp/times.txt and you can press ctrl-c when ready. Then try with grep with a pipe, tail -f /tmp/times.txt | grep '0 ' and mind the trailing space in the pattern. What is it finding? Press ctrl-c when ready and try another pattern. Press ctrl-c in both windows when done to stop everything.

For the full set of options and an authoritative answer as to what they do, see the reference manual on your system using man and scroll down to the section labeled, "OPTIONS".

man grep

Although there is a lot which can be done with just fixed strings, grep's search is not limited only to exact strings, it can search for patterns of strings by adding special characters to the search. ., *, ?, ^, and $ are among the special characters which provide extra capabilities through what are known as regular expression pattern matching, often shorted as "regex".

Exercise: Read what the reference manual for grep has to say about its options. What do -i, -E, -w, and -P do? What are the long format options for each of them?

Create several files with two lines in each of them:

echo -e "one\ntwo\n" > test-1.txt
echo -e "three\nfour\n" > test-2.txt
echo -e "five\nsix\n" > test-3.txt
echo -e "seven\neight\n" > test-4.txt

Try the -l option to suppress the normal output and show only the names of files with matching content. grep -l "one" test-?.txt Which file had the string "one" in it?

Exercise: Which files have the letter "i" in them?

Exercise: How does the output from the -h option differ from that of the -l option?

Basic and Extended Regular Expressions in grep (45 minutes)

There are several styles of regex. Those styles are Basic, Extended, and Perl-compatible. Modern grep covers all three. By default, Basic Regular Expression pattern matching is used. However, the -E and the -P options provide Extended Regular Expression and Perl-compatible Regular Expression pattern matching, respectively. The latter, the Perl-compatible Regular Expression pattern matching is by the most powerful, but for full power, actual Perl is required so that is covered in a separate lesson.

Basic Regular Expressions

Basic Regular Expressions (BRE) are defined as part of the POSIX standard. There are various "metacharacters" which allow flexibility. There are also full sets of characters, called "character classes", which can stand in place of a range of charcters. First, here is an overview of BRE metacharacters.

A . matches any single character. So if you wanted to find all five-letter words in a file, you could look for ..... like this:

Here a dot stands in for any character, whether letter, digit, or punctuation.

grep -w '.....' book.txt

Exercise: Try the above pattern with and without the -o option. What is the difference in the presentation of the results?

Another way to do that would be with quantifies using curly brackets, { and }. Unlike in Extended Regular Expressions, in Basic Regular Expressions the curly brackets will need to be escaped with a backslash. Quantifiers specify a minimum and/or maximum number of times the preceding pattern should be repeated. So another way to write the above pattern would be with a quantifier:

grep -o -w '.\{5\}' book.txt

That will do the same thing, find five-letter strings. It also finds all five-digit numbers like 12345 or 32027 and so on. One can use the metacharacters [ and ]to find a set or range of letters or digits. Below, the first example finds a line with a capitalized five-letter word, and the second a lowercase five-letter word. The third example finds lines with a capitalized five-letter word, and the fourth finds those containing four-digit number.

grep -o -w '[A-Z]\{5\}' book.txt

grep -o -w '[a-z]\{5\}' book.txt

grep -o -w '[A-Z][a-z]\{4\}' book.txt

grep -o -w '[0-9]\]{4\}' book.txt

However, many languages which use the Latin alphabet have more than just the letters A-Z, so one can use named classes of characters. The [:alpha:] and [:alnum:] classes do not differentiate between upper and lower cases. More on such classes in the sections on Perl-Compatible Regular Expressions.

grep -o -w '[[:alpha:]]\{5\}' book.txt

grep -o -w '[[:alnum:]]\{5\}' book.txt

grep -o -w '[[:digit:]]\{4\}' book.txt

See man 7 regex about BRE. Scroll down to the paragraph on character classes for more details.

Logical OR and AND

With BRE, a logical OR can be produced by adding additional patterns.

Exercise: Compare the output of both of these commands, grep 'Queequeg' book.txt and grep -e 'Queequeg' -e 'Quohog' book.txt What is the difference?

With BRE, a logical AND can only be produces by piping several uses of grep together in sequence.

Exercise: Try the next two and compare the results, grep -w -e 'Ahab' -e 'whale' book.txt and grep -w -e 'Ahab' book.txt | grep -e 'whale' What is the difference?

Extended Regular Expressions

Extended Regular Expressions (ERE) add a bit to the basic functionality. Most notable is the ability to use the alternation pattern | to look for more than one pattern at a time.

Exercise: Try the following three BRE searches. grep 'Queequeg' book.txt and grep 'Quohog' book.txt and grep 'Queequeg|Quohog' book.txt Why does the last one not find anything? Now try the last search as an ERE, grep -E 'Queequeg|Quohog' book.txt what is the difference?

That's basically a logical OR operator, but one which always returns the longer match.

Exercise: Explain the output of the following pattern.

echo "one two three four" | grep -o -E 'o|one'

Quantifiers are a little different in ERE in that they do not need to be escaped.

grep -o -E -w '[[:alpha:]]{1,5}' book.txt

grep -o -E -w '[[:alnum:]]{3,5}' book.txt

grep -o -E -w '[[:alpha:]]{5,}' book.txt

Exercise: What does the first line above do? What else does the second line find that the first line does not? What is the overlap between what is found by the first line and what is found by the third line?

For extra credit, try the following refinements.

grep -o -E -w '[[:alpha:]]{1,5}' book.txt | sort

grep -o -E -w '[[:alpha:]]{1,5}' book.txt | sort | uniq

grep -o -E -w '[[:alpha:]]{1,5}' book.txt | sort | uniq -c | less -X

grep -o -E -w '[[:alpha:]]{1,5}' book.txt | sort | tr 'A-Z' 'a-z' | uniq -c | less -X

What did each of those pipes do? Why are contractions like "ain’t", "don’t", "can’t", and so on not processed correctly?

Perl-Compatible Regular Expressions in grep (45 minutes)

A large subset of Perl's pattern matching abilities has been exported to a great many other languages and tools under the name "Perl-Compatible Regular Expressions". It is of Perl's main strengths, and perhaps the most visible one. grep is one of the tools which can utilize PCRE. These powerful capabilities are invoked with the -P option.

In Perl, there are characters with special meanings. The the backslash \, the pipe |, the left parenthesis (, the left square bracket [, the left curly bracket {, the caret ^, the dollar sign $, the asterisk *, the plus sign +, the question mark ?, and the period . which provide special functionality.

Exercise: See the reference manual for Perl regular expression under the section "Metacharacters". man perlre What do each of the above listed characters officially do?

Find lines with roman numerals, or strings which happen to appear very much like a roman numeral:

grep -P -w '[CDILVX]+\.' book.txt

Exercise: What are the false positives found by the above formula? Why are they found in addition to roman numerals?

Character Classes and other Special Escapes

POSIX character classes as seen above in BRE can still be used within Perl-compatible regular expressions, e.g. :alpha:, :space:, and :upper: However, Perl has shorter expressions for those sets of characters.

Exercise: See the reference manual for Perl regular expression under the section "Character Classes and other Special Escapes". man perlre What do \w, \d, \s, and \S find?

We can rewrite some of the expressions above to use some Perl sequences instead of POSIX notation. The \w stands for any letter or number. The quantifiers in the curly brackets work the same as in the BRE. However, the plus means one or more of the preceding letters.

grep -o -P -w '\w{2,5}' book.txt

grep -o -P -w '\w+' book.txt

Exercise: What is the difference between what the two patterns above find?

PCRE allows capture groups, patterns which can be noted and kept for later reuse. These are marked by a pair of parentheses. Then the capture groups are referred to by the order which they appear. The first capture group is \1, a second capture group would be \2, the third \3, and so on. The pattern below will find all the hyphenated double words in the book using one capture group:

grep -o -P -w '(\w+)-\1' book.txt

Exercise: Compose a pattern which will find the words with at least one pair of double letters.

Advanced PCRE

There are also several noteworthy zero-width assertions. These are markers which indicate a boundaries of a string, but which don't actually absorb any letters or other characters. Specifically those are \b, \A, and \Z. They're more useful in actual Perl scripts because they are mostly made redundant by the -w' option in grep. However, look around assertions can be used to exclude part of the pattern from the output.

Exercise: Find the section, "Lookaround assertions". What is the difference between a look ahead assertion and a look behind assertion?

With PCRE, one can make quite advanced patterns. Here is a sample. We find words which are preceded by the word "introduced" in the first pattern below. And, in the second pattern we find the words which are followed by the word "why". In both cases, only the preceding or following word is displayed in the results, not the zero-width part of the pattern which.

grep -o -P -w '(?<=introduced )\w+' book.txt | sort | uniq -c

grep -o -P -w '\w+(?= why)' book.txt | sort | uniq -c

The specific notation for the lookaround assertions is not as important to remember as the fact that look ahead and look behind assertions exist and how to look up the notation when needed.

Review

In this lesson, we covered Basic Regular Expressions (BRE), Extended Regular Expressions (ERE), and Perl-Compartible Regular Expressions (PCRE). It does not matter much which style is used, as long as the choice result in the job getting done.

For in-depth material on Basic and Extended regular expressions, see Bruce Barnett's Grymoire

For in-depth material on Perl's pattern matching capabilities, see the reference manual man perlre on your system and see also chapter 5 of Programming Perl by Tom Christiansen, brian d foy, and Larry Wall.

Exercise: What do each of the following grep options do? -c, -E, -h, -i, -l, -o, -P, and -w

See also

• AWK
• Perl