shunt v1.7.3

Introduction:

shunt is a general purpose command-line tool that is used to shunt data around closed pipes to restarted processes. By creative use of the scripts that are restarted or by recursively calling shunt, you should be able to do some interesting things, but I wrote it primarily to allow me to stream multi-volume data backups directly to CD or DVD without the need for any temporary files on disk.

exactly is a trivial program that is used to exactly control the number of bytes consumers in the pipeline consume. This is especially useful when dealing with programs that use the buffered C/C++ standard I/O library or when you want an unbuffered program to be less greedy. You need to use this program with mkisofs (but not with the included flyisofs that serves the same purpose).

flyisofs creates an ISO 9660 file system on the fly from a stream of data on standard input. The data is stored sequentially in files the size of your choosing.

shunt was originally written for burning multi-set CDROM backups using mkiosfs and cdrecord. You can get both by downloading the latest cdrtools package. For burning DVDs, mkisofs (currently) has limitations on how much data it can stream while creating an ISO 9660 file system on the fly. Because of this, you need to replace mkisofs with flyisofs (which is part of the shunt source package that you can download below). Also, depending on your setup, you may have a forked or old version of cdrecord that cannot burn DVDs. If so, you probably will want to use growisofs instead.

Reasoning:

Backup utilities that are capable of handling multi-volume archives have to directly control the opening, reading or writing, and closing of the underlying device. Apparently, this is not supported by the current generation of device drivers for CDs and DVDs.

To my knowledge, cdrecord and growisofs write to only a single CD or DVD before exiting. The CD-Writing HOWTO only has this to say: "You have to take special care within the backup utility if the backup spans multiple CD-Rs." The HOWTO does not elaborate further.

This is where shunt, exactly, and flyisofs prove themselves. By using a combination of these programs, you can convince your backup utility that it is streaming to one large file while simultaneously convincing flyisofs and cdrecord (or growisofs) that they are dealing with sets of data each of which is small enough to fit on the media. When you want to restore from your multi-volume set, shunt handles the chore by itself. In both cases, no temporary files are used. This will save you time and tedium and also allow you to backup systems that are low on hard drive space.

Warning #1:

You will be frustrated if you do not have a reliable CD or DVD burner because the nature of these backups require all disks to burn without error; otherwise, you will have to start over from the beginning. As a work around, if you have enough hard drive space for a single CD or DVD, you can alter the examples below to have the ISO image created by flyisofs redirected to a temporary file. When shunt tells you that flyisofs has exited, it pauses the backup program. You can take this opportunity to manually burn the ISO image and then verify that the CD or DVD is correct. If the verification fails, just keep manually reburning the ISO image until it succeeds. You can then have shunt restart the flyisofs portion of the pipeline in order to generate the next ISO image.

Warning #2:

shunt uses standard output to prompt the user and standard input to read the response. The necessary consequence of this is that the pipeline set up by shunt is disconnected from standard input and standard output. This means if you are performing a backup across a network using rsh or ssh, the rsh or ssh command should be embedded as one of shunt's command-line parameters.

Usage:

Usage: shunt <flags> <child> [+ <flags> <child>] . . .

    Flags:

         + : add another child
        -c : automatically close the pipe when child exits.  This should
             start a chain reaction that eventually causes all processes
             that are part of the pipeline to close.
        -h : help
        -r : automatically restart child

The basic idea is that you want to use something like the following to stream data to cdrecord: 

archive_program | cdrecord

The above command does not work because cdrecord exits after burning the first CD or DVD. This causes archive_program to receive a premature EOF which in turn cause archive_program to exit before the archive can be fully written. To fix this, you use shunt to suppress the EOF when cdrecord exits. shunt will then restart cdrecord and put it back in place at the end of the pipeline. What follows is a simplification, but it is more informative than a full example: 

shunt -c archive_program + cdrecord

The trick to getting the flags right is in realizing that shunt allows you to control many child processes (no hard-coded limit) and that the -c and -r flags can be applied to each process individually. So, you are allowed to use the -c and the -r flags more than once per command line. shunt keeps track of which process the -c and -r flags belong to by separating each child with a '+' flag (analogous to how shells separate processes using the '|' symbol).

The most important flag is the -c flag. Pipelines are always set up like dominoes. When one process (usually the last or first in the pipeline) runs out of things to do, it exits which causes the process next to it to exit and so on. The -c flag lets shunt know which process (or processes) you want to endow with this attribute. If you do not endow the right process with the automatic-close attribute you could end up with the pipeline not shutting down when there is nothing left to do, or the pipeline could shutdown unexpectedly in the middle of things. If you get the -c flag right, everything should go smoothly. If you look at the examples below, you can see how this works in practice.

If you set the -r flag of a child process, shunt will automatically restart that child process without prompting you. When burning CDs this is not what you want because you need things to slow down while you replace the medium after each burn, but it is useful for other things.

Environment Variables:

shunt puts SHUNT_STARTS and SHUNT_ALL_STARTS into the environment of the child process. SHUNT_STARTS tells the child process how many times it has been started. SHUNT_ALL_STARTS tells the process how many times all the processes have been started. You can use these values to generate unique names for the files created by mkisofs, for instance.

libcalc:

Though unlikely when burning CDs, it is conceivable in other situations that the values for SHUNT_STARTS and SHUNT_ALL_STARTS could overflow. To eliminate this possibility, the shunt code can be optionally built with support for arbitrary precision calculations using libcalc. The only problem with using libcalc is that it adds substantially to the size of the resulting executable. So, if you want to build a version of shunt to include on a disaster recovery image, the default configuration which does not depend upon libcalc will get you a binary weighing in at only 10K.

The Linux Situation:

If you need to run shunt and flyisofs on linux, make sure you are using linux-2.6.8 or higher. Previous kernels work for burning CDs, but they do not work for DVDs because the inode numbering scheme overflows at 4GB. Starting with linux-2.6.8, the kernel includes a patch I wrote that does not overflow until 128GB. This new scheme should be adequate for two or three more generations of media.

Examples:

Warning #3:

As mentioned in Warning #2 above, the pipeline set up by shunt is disconnected from standard input. Thus, if you encrypt your backups and you are having trouble getting your encryption program to read your password, the likely problem is that your encryption program is reading its passwords from standard input (which has been disconnected) rather than directly from the terminal. You should be able to work around this using something similar to the following: 

shunt "sleep 4 ; { printf 'Password: ' ; read passwd < `tty` ; echo \"\$passwd\" ; } | cat"

Source:

You should be able to compile and run shunt, exactly, and flyisofs on (at least) FreeBSD, Linux, NetBSD, OpenBSD, and Solaris.

ChangeLog:

The major changes are documented in the ChangeLog.