Introduction
The redirection capabilities built into Linux provide you with a robust set of tools to optimize many workflows. The “Unix philosophy” of software development was to make tools that each do one thing well, and this philosophy has been carried forward to modern command-line tools, which are individually powerful, and exponentially more so when combined. Whether you’re writing complex software or just working on the command line, knowing how to manipulate the different I/O streams in your environment will greatly increase your productivity.
Prerequisites
To follow along with this guide, you will need to have access to a Linux server. If you need information about connecting to your server for the first time, you can follow our guide on connecting to a Linux server using SSH.
Streams
Input and output in the Linux environment is distributed across three streams. These streams are:
standard input (stdin)
standard output (stdout)
standard error (stderr)
The streams are also numbered:
stdin (0)
stdout (1)
stderr (2)
During standard interactions between the user and the terminal, standard input comes from the user’s keyboard. Standard output and standard error are displayed on the user’s terminal as text. Collectively, the three streams are referred to as the standard streams.
Standard Input
The standard input stream typically carries data from a user to a program. Programs that expect standard input usually receive input from a device, such as a keyboard. Later in this tutorial, you will see examples of using one program’s output as Standard Input to another.
Standard
Standard output is the output that is generated by a program. When the standard output stream is not redirected, it will output text directly to the terminal. Try to output some arbitrary text, using echo:
echo Sent to the terminal
Sent to the terminal
When used without any additional options, the echo command outputs any argument that is passed to it on the command line.
Run echo without any arguments:
echo
It will return an empty line. Some programs do not do anything without provided arguments.
Standard Error
Standard error contains errors generated by a program that has failed in some way. Like standard output, the default destination for this stream is the terminal display.
Let’s see a basic example of standard error using the ls command. ls lists a directory’s contents.
When run without an argument, ls lists the contents within the current directory. If ls is run with a directory as an argument, it will list the contents of the provided directory.
ls %
Since % is not an existing directory, this will send the following text to standard error:
ls: cannot access %: No such file or directory
A program does not have to crash or finish running in order to generate Standard Error, and whether some output is sent to either Standard or Standard Error is down to the behavior of the program. They are not technically different from one another in any way — just that one output stream is supposed to be reserved for error messages, and some tools will assume that Standard Error being empty means that a program ran successfully. Some programs will even output minor errors to Standard Error without crashing or failing to also produce the intended output. It is only used as a convention to separate intended output from unintended output.
Stream Redirection
Linux includes redirection commands for each stream. These can be used to write standard output or standard error to a file. If you write to a file that does not exist, a new file with that name will be created prior to writing.
Commands with a single bracket overwrite the destination’s existing contents.
Overwrite
– standard output
< – standard input
2> – standard error
Commands with a double bracket do not overwrite the destination’s existing contents.
Append
– standard output
« – standard input
2» – standard error
Pipes
Pipes are used to redirect a stream from one program to another. When a program’s standard output is sent to another through a pipe, the first program’s output will be used as input to the second, rather than being printed to the terminal. Only the data returned by the second program will be displayed.
The Linux pipe is represented by a vertical bar: |
Here is an example of a command using a pipe:
ls | less
This takes the output of ls, which displays the contents of your current directory, and pipes it to the less program. less displays the data sent to it one line at a time.
ls normally displays directory contents across multiple rows. When you run it through less, each entry is placed on a new line.
Though the functionality of the pipe may appear to be similar to that of > and >>, the distinction is that pipes redirect data from one command to another, while > and » are used to redirect exclusively to files.
Filters
Filters are are a class of programs that are commonly used with output piped from another program. Many of them are also useful on their own, but they illustrate piping behavior especially well.
find – returns files with filenames that match the argument passed to find.
grep – returns text that matches the string pattern passed to grep.
tee – redirects standard input to both standard output and one or more files.
tr – finds-and-replaces one string with another.
wc – counts characters, lines, and words.
Examples
Now that you have been introduced to redirection, piping, and basic filters, let’s look at some common redirection patterns and examples.
The command > file pattern redirects the standard output of a command to a file.
ls ~ > root_dir_contents.txt
The command above passes the contents of your home directory (~) as standard output, and writes the output to a file named root_dir_contents.txt. It will delete any prior contents in the file, as it is a single-bracket command.
The command > /dev/null pattern redirects standard output to nowhere. /dev/null is a special file that is used to trash any data that is redirected to it. It is used to discard standard output that is not needed, and that might otherwise interfere with the functionality of a command or a script. Any output that is sent to /dev/null is discarded.
ls > /dev/null
This command discards the standard output stream returned from the command ls by passing it to /dev/null.
This command 2> file pattern redirects the standard error stream of a command to a file, overwriting existing contents.
mkdir '' 2> mkdir_log.txt
This redirects the error raised by the invalid directory name '', and writes it to log.txt. Note that the error is still sent to the terminal and displayed as text.
The command >> file pattern redirects the standard output of a command to a file without overwriting the file’s existing contents.
echo Written to a new file > data.txt
echo Appended content to an existing file >> data.txt
This pair of commands first redirects the text inputted by the user through echo to a new file. It then appends the text received by the second echo command to the existing file, without overwriting its contents.
The command 2>> file pattern above redirects the standard error stream of a command to a file without overwriting the file’s existing contents. This pattern is useful for creating error logs for a program or service, as the log file will not have its previous content wiped each time the file is written to.
find '' 2> stderr_log.txt
wc '' 2>> stderr_log.txt
The above command redirects the error message caused by an invalid find argument to a file named stderr_log.txt. It then appends the error message caused by an invalid wc argument to the same file.
The command | command pattern redirects the standard output from the first command to the standard input of the second command.
find /var lib | grep deb
This command searches through /var and its subfolders for filenames and extensions that match the string deb, and returns the file paths for the files, with the matching portion in each path highlighted in red.
The command | tee file pattern (which includes the tee command) redirects the standard output of the command to a file and overwrites its contents. Then, it displays the redirected output in the terminal. It creates a new file if the file does not already exist.
In the context of this pattern, tee is typically used to view a program’s output while simultaneously saving it to a file.
wc /etc/magic | tee magic_count.txt
This pipes the counts for characters, lines, and words in the /etc/magic file (used by the Linux shell to determine file types) to the tee command, which then splits wc’s output in two directions, and sends it to the terminal display and the magic_count.txt file. For the tee command, imagine the letter T. The bottom part of the letter is the initial data, and the top part is the data being split in two different directions (standard output and the terminal).
Multiple pipes can be used to redirect output across multiple commands and/or filters.
Conclusion
Learning how to use the redirection capabilities built into the Linux command line is a crucial skill. Now that you have seen the basics of how redirections and pipes work, you’ll be able to begin your journey into the world of shell scripting, which makes frequent use of the programs and patterns highlighted in this guide.
Searching for specific commands, or for something that you would like to do in the command line (e.g. “delete all files in a directory that begin with an uppercase letter”) can also prove helpful when you need to accomplish a specific task using the command line.
