5 Class 6. Unix sorting, redirection, filtering with head/tail

Command line practice: sorting, redirecting, head/tail

Content modified from The Software Carpentries

Now that we know a few basic commands, we can finally look at the shell’s most powerful feature: the ease with which it lets us combine existing programs in new ways. We’ll start with a directory called molecules that contains six files describing some simple organic molecules.

What are these files? Where did they come from?

The .pdb extension indicates that these files are in Protein Data Bank format, a simple text format that specifies the type and position of each atom in the molecule.

Let’s go into that directory with cd and run the command wc *.pdb. wc is the “word count” command: it counts the number of lines, words, and characters in files (from left to right, in that order).

The * in *.pdb matches zero or more characters, so the shell turns *.pdb into a list of all .pdb files in the current directory:

If we run wc -l instead of just wc, the output shows only the number of lines per file:

Remember that we can also use -w to get only the number of words, or -c to get only the number of characters.

Which of these files is shortest? It’s an easy question to answer when there are only six files, but what if there were 6000? Our first step toward a solution is to run the command:

The greater than symbol, >, tells the shell to redirect the command’s output to a file instead of printing it to the screen. (This is why there is no screen output: everything that wc would have printed has gone into the file lengths.txt instead.) The shell will create the file if it doesn’t exist. If the file exists, it will be silently overwritten, which may lead to data loss and thus requires some caution. ls lengths.txt confirms that the file exists:

We can now send the content of lengths.txt to the screen using cat lengths.txt. cat stands for “concatenate”: it prints the contents of files one after another. There’s only one file in this case, so cat just shows us what it contains:

What Does sort -n do?

Create a text file called numbers and enter the following numbers.

If we run sort on a file containing the following lines by typing

We will also use the -n flag to specify that the sort is numerical instead of alphabetical. This does not change the file; instead, it sends the sorted result to the screen:

We can put the sorted list of lines in another temporary file called sorted-lengths.txt by putting > sorted-lengths.txt after the command, just as we used > lengths.txt to put the output of wc into lengths.txt. Once we’ve done that, we can run another command called head to get the first few lines in sorted-lengths.txt:

It’s a very bad idea to try redirecting the output of a command that operates on a file to the same file. For example:

Doing something like this may give you incorrect results and/or delete the contents of lengths.txt.

Tail follows same the format as the head command. You use –n to specify number of lines to show. For example:

This command would show you the last line of the sorted-lengths.txt file.

What Does >> Mean?

We have seen the use of >, but there is a similar operator >> which works slightly differently. By using the echo command to print strings, test the commands below to reveal the difference between the two operators:

Hint: Try executing each command twice in a row and then examining the output files.

Using the >> operator will append data at the end of the file, while using the > will overwrite the contents of the file if already existing.

Command review from this tutorial:

• cat displays the contents of its inputs.
• head displays the first 10 lines of its input. You can adjust the # of lines to display with the -n option.
• tail displays the last 10 lines of its input. You can adjust the # of lines to display with the -n option.
• sort sorts its inputs.
• wc counts lines, words, and characters in its inputs.
• command > file redirects a command’s output to a file (overwriting any existing content).