A special case in scripting: How can I have a shell function that passed data through untouched and without interruption, while at the same time modifying a copy of the data?

The Problem

The problem occurs sometimes in shellscripting, you would like to branch text a stream from stdin and continue processing it in different ways.

The tee utility provides this functionality within limits. tee writes a stream to multiple files and at the same time passes it through to stdout. For example:

echo foobar |tee file1 |tr 'oa' 'ei'

will write 'feebir' to the output, and set up file1, containing the string 'foobar'. You can then process the files further.

But what if I don't like using temporary files?

Of course, tempfiles are messy. You run the danger of writing sensitive data to nonvolatile storage, and in any case you have to clean them up afterwards. You are also wasting time because you cannot process a continuous stream as it flies in.

Without digging to deep into shell scripting, there is one other form of the problem, which you can handle easily: if your intended data modification can be done by sed, and should be written to a file.

Most text utilities can be covered by a function in sed, for example the tr example given above can be written as

sed 'y;oa;ei;'

But sed can not only write data to stdout, but also to files - both at the same time. For example the tee command from above can be replaced by

sed -n 'p;wfile1'

So consider the case where you want to read an input and pass it through while writing modifications to a file. Instead of using a tempfile like this:

echo foobar | tee tempfile | tr 'oa' 'ei' > outputfile
cat tempfile | other_program

you can write:

echo foobar |sed -n 'p;y;oa;ei;;woutputfile' |other_program

Be careful, if you want to write to multiple output files. The write commands have to be separated by newlines within the sed command:

echo foobar |sed 'wfile1
                  wfile2
                  wfile3'

This is awkward to enter into an interactive shell. At the cost of a mostly negligible overhead you can also use multiple invocations of sed:

echo foobar |sed 'wfile1' |sed 'wfile2' |sed 'wfile3'

which in this case would be equivalent to the simpler

echo foobar |tee 'file1' 'file2' 'file3'