Linux/Basic commands/awk

AWK is a language/utility designed and used mainly for text processing.

Prerequisites: a system with an interactive shell and one of the AWK interpreters.

AWK is a venerated, simple pattern scanning and text processing language for writing short, interpreted scripts. It fits in as the middle part of the grep / AWK / Perl triumvirate as far as power and flexibility go. It was originally written in the 1970s by Alfred V Aho, Peter J Weinberger and Brian W Kernigan. There are several variants of AWK known as mawk, gawk, and nawk which extend the basic language in various ways.

Exercise: Check the reference manual on your system to see which version is present there.

man awk

Exercise: Read the section 1. Program structure in the reference manual for the basic concept of AWK.

man awk

It's short, isn't it?

AWK Basics

The basic structure of an AWK script is each line is shorthand for if-then statements. More specifically, it reads input line by line and a pattern is looked for. If the pattern is found then an action is executed.

awk '/pattern/' file.txt

If the action is omitted, then the default is to print the whole line. If the pattern is omitted, then the default is to apply the action to each and every line.

Note that the above is the short hand equivalent for the following three variations. All four do the same thing, but are written either more concisely or more tersely. The variable $0 stands for the current input line in its entirety. The tilde ~ means check that specific variable for the pattern given. And the slashes / mark the boundaries of the pattern to be used:

awk '/pattern/ { print }' file.txt
awk '/pattern/ { print $0 }' file.txt
awk '$0 ~ /pattern/ { print $0 }' file.txt

Exercise: Using any of the above styles, write an AWK one-liner which finds lines with a pattern of your choice in a text file of your choice.

Specific columns can be examined or printed by using the relevant special variable, $1, $2, ..., $NF specifically. So to print the first column of each line of a white space delimited text file, the following would work.

awk '{ print $1 }' file.txt

The delimiter is generally known as the field separator and the columns referred to as fields.

CSV

Caution, comma-separated value (CSV) files are a deceptively complex beast. The specification is defined fully in RFC 4180 and it is beyond simply looking for white space or using a comma as the field separator. Instead, back in 2022, UNIX legend Brian Kernighan, the "K" in AWK itself, added CSV parsing according to RFC 480 to basic AWK.

awk '{ print $2 }' --csv spreadsheet-table.csv

The above reads a file and parses it according to RFC 4180 and prints the second column of each line.

Exercise: Download the Air Quality for the City of New York, NY open data set. Print all the place names. There are found in column eight. Bonus for compacting the list by piping the output through sort and uniq and the pager less.

Note, not all distros have updated their copies of AWK to support parsing CSV files.

AWK One-Liners

Here is a one-liner which uses a variable to count the number of non-empty lines by checking the built-in variable $0 to see if it is empty or not. If it is not empty, the variable count is incremented by one. After there is no more input, the result is printed.

awk '$0 { count++ } END { print count " non-empty lines" }' file.txt

Exercise: Make a short file containing dozen or so lines of text, some of which contain the string "foobar". Write a one-liner which checks a file for how many lines contain the string "foobar".

Here is a one-liner which finds permanent redirections, status 301 and 308, in an Apache2 log file. If one uses white space as a delimiter, which is the default for AWK, then the ninth column will hold the HTTP Response Code:

awk '$9=="301" || $9=="308"' access.log

If one is looking for all responses, that is any response code from 300 to 399, then one way of several is to use a pattern, say /^3[0-9][0-9]$/. Or it could be treated like a number:

awk '$9 >= 300 && $9 <= 399' access.log

While white space is the default field separator, it can be set to any character using the -F option.

Exercise: Set the delimiter to a colon : and print out the first column whenever the third column is greater than or equal to 1000. Use the file, /etc/passwd, it is world-readable and not secret.

In some variants the field separator can even be set to a pattern.

Exercise: Check whether your version of AWK can use a pattern as a field separator or not:

man awk

Standalone AWK

The -f in the shebang is essential for a standalone script.

#!/usr/bin/awk -f

{ print $1 }

Exercise: Create a text file with dozen or so lines, with at least one space per line. Create a text file using nano or the editor of your choice and fill it with the script above: nano first-script.awk Then use chmod to make it executable: chmod u+rwx first-script.awk and then use the script to print the first field of each line ./first-script.awk file.txt

AWK also has associative arrays, they can be defined on-the-fly. The following uses each whole line as a key in an associative array, thus allowing de-duplication of a file.

#!/usr/bin/awk -f

! a[$0]++ { 
        print
        next
}

! $0 {
        print
}

Exercise: Write the above to a file, second-script.awk, and make the file executable. Then create a file of a dozen or so lines with a few duplicate lines. Run the above deduplication script over the file. Try removing the ! a[$0]++ clause. What was the effect? Try removing the ! $0 clause. What effect did that have?

See also

• grep
• Perl