8. Advanced Shell Topics

1. More I/O
2. Synchronicity
3. Arithmetic
4. Eval Statements
5. Exec Statements
6. Setting Shell Flags
7. Signals and Traps

8.1 More I/O

The shell permits the combination of the basic I/O constructs with a file descriptor. A file descriptor is nothing more than a non-negative digit that points at a file.  File descriptors are also known as file handles.  They are very useful for interprocess and network programming.  The reader should not be terribly concerned if this concept is confusing.  From the shell's perspective, only stdin, stdout, and stderr are truly important.  Anything else probably requires a more powerful language such as C, which can make system calls to create and gain access to other file descriptors.

The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively.  Any of these may be redirected to a file or to each other.  In other words, it is quite possible to send the output from stdin and stderr to the same file.  This is quite useful when a user would rather check a script's results after it has completed processing.  The methods for doing this are shown in Table 8.1-1.

$ cd /bogus >>out
/bogus: does not exist
$ cat out

The user starts by trying to change directories to a non-existant directory.  In the process, the output from the command is redirected to the file out.  Unfortunately, the user neglects that fact that the output generated by cd is an error and is output on stderr.  So when the user displays the contents of out, it contains nothing.  This time, the user tries redirecting stderr to the same file in which stdout is redirected.

$ cd /bogus >>out 2>&1
$ cat out
/bogus: does not exist

This time, out contains the error message.  The key is the addition of 2>&1.  This is a very important concept, and to understand it, the line should be read, "Change directory to /bogus, redirect stdout to the file out, and then redirect stderr to the same file that stdout uses."  The last part is exactly what 2>&1 means.  Moreover, most scripters find that the 2>&1 construct is the most widely if not the only of these constructs used.  As stated previously, it is extremely useful for saving the output to a file for reviewing after the script completes.  For example, a backup script that runs each night might do this and then email the output to the system administrator.

As a quick note, the same results could have been generated simply by preceding the output redirection with stderr's file descriptor.

$ cd /bogus 2>>out
$ cat out
/bogus: does not exist

As the last example of advanced redirection, the results of printing the working directory are suppressed by closing stdout.

$ pwd >&-
$

Now using these within a script can become a bit tiresome.  A programmer needs to precede each command that could produce output or errors with appropriate redirection.  Actually, to get around this, the user just executes the script at the command line with the same redirection.  But if the scripter chooses to do the work for the user, then the shell provides a method for redirecting all of a script's input and output.  A form of the exec command allows this: exec construct file, where construct is a redirection and file is the input or output to use.  For example,

#!/bin/sh
exec >>${0}.out

redirects the output for the script into the file given by ${0}.out.  ${0}.out is the new file named the same as the script but with .out appended.

Next, redirection can be used with loops.  To do so, the programmer adds the redirection construct to the end of the loop declaration.  In fact, loops can use redirection, piping, and even backgrounding by the same method, though redirection is easily the most useful.  Below is a short script demonstrating the technique by printing the disk usage of each user on a system.

$ USERS=`cat /etc/passwd | cut -f1 -d:`
$ for user in ${USERS}
> do
> du -sk /home/${user}
> done >>diskusage 2>/dev/null
$ cat diskusage
2748    /home/ftp
38931   /home/jac
109     /home/mlambi
26151   /home/rsayle
5       /home/bobt
8185    /home/mann_al
13759   /home/bheintz

The script starts by getting a list of users from the /etc/passwd file.  It stores the list in the variable USERS.  Then it parses through the list with a for loop.  Each iteration of the loop performs a disk usage summary on the current user's home directory.  The results are in turn saved in the file diskusage by redirecting standard output into it.  Standard error is sent to the great bit bucket in the sky because some accounts listed in /etc/passwd do not actually have home directories.  This would cause du to complain that /home/${user} for the current user does not exist.  The results are then displayed by catting diskusage.

Standard input works well with loops.  As it turns out, redirecting stdin with a loop is the way one can read a file line by line in order to process its contents.  Combining this with a little output redirection creates a slick way to generate files from a template and a list of data to fill the template.

$ cat mkraddb
#!/bin/sh
#
# mkraddb: generate the RADIUS configuration of a set of users
#
#

# Global Vars
PROG=`basename ${0}`
USAGE="${PROG}: usage: ${PROG} users_file"
DATE=`date +%m%d%y`
RADDBUSERS="raddb.users.${DATE}"

##############################################
# buildRaddbUsers
##############################################

buildRaddbUsers () {

cat >>${RADDBUSERS} <<EOF
${name}-out     Password = "ascend",
        User-Service = Dialout-Framed-User
        User-Name = ${name},
        Framed-Protocol = MPP,
        Framed-Address = ${ipaddr},
        Framed-Netmask = 255.255.255.255,
        Ascend-Metric = 2,
        Framed-Routing = None,
        Ascend-Route-IP = Route-IP-Yes,
        Ascend-Idle-Limit = 30,
        Ascend-Send-Auth = Send-Auth-CHAP,
        Ascend-Send-Passwd = "password",
        Ascend-Receive-Secret = "password"

EOF

}
 

##############################################
# Main
##############################################

if [ $# -ne 1 ]; then
  echo ${USAGE}
  exit 1
fi

# Create the configs
while read name ipaddr
do
  buildRaddbUsers
done <${1}

# Clean up
exit 0

So here is a script called mkraddb.  Its sole purpose is to read a file and generate the RADIUS profiles for the users listed in the file.  Focusing on the redirection within the script, the program first uses redirection with the while loop under the main program.  The loop reads from the file passed to the script as its argument.  The redirection of the first positional parameter into the while loop does this.  The read command following the while serves as its test.  It takes each line from the file and reads the line's contents into the temporary variables name and ipaddr.  As long as there are lines in the file to read, read returns true and the loop executes.  When there are no more entires left, the loop terminates and, consequently, so does the script.

To process the data, the loop calls the function buildRaddbUsers.  Turning back to this function shows the second usage of redirection within the script.  Input redirection causes the function to create a new file as given by the value of RADDBUSERS.  The output redirection of cat instructs the script to write into this file, the text between the end of the file markers, EOF.  The shell expands the variables name and ipaddr as it writes the text to create the user's RADIUS profile.  Here is how it works.

$ ls
mkraddb  users
$ cat users
bob 192.168.0.1
joe 172.16.20.2
mary 10.0.25.3

The user first gets a directory listing that shows only the script, mkraddb, and the users file.  Then the operator cats users and notes that the script should create RADIUS entries for bob, joe, and mary.  Below, the operator executes mkraddb passing it users.  A quick listing shows that mkraddb created raddb.users.082698, and inspecting the contents of the new file shows the profiles for these users.

$ mkraddb users
$ ls
mkraddb           raddb.users.082698  users
$ cat raddb.users.082698
bob-out Password = "ascend",
        User-Service = Dialout-Framed-User
        User-Name = bob,
        Framed-Protocol = MPP,
        Framed-Address = 192.168.0.1,
        Framed-Netmask = 255.255.255.255,
        Ascend-Metric = 2,
        Framed-Routing = None,
        Ascend-Route-IP = Route-IP-Yes,
        Ascend-Idle-Limit = 30,
        Ascend-Send-Auth = Send-Auth-CHAP,
        Ascend-Send-Passwd = "password",
        Ascend-Receive-Secret = "password"

joe-out Password = "ascend",
        User-Service = Dialout-Framed-User
        User-Name = joe,
        Framed-Protocol = MPP,
        Framed-Address = 172.16.20.2,
        Framed-Netmask = 255.255.255.255,
        Ascend-Metric = 2,
        Framed-Routing = None,
        Ascend-Route-IP = Route-IP-Yes,
        Ascend-Idle-Limit = 30,
        Ascend-Send-Auth = Send-Auth-CHAP,
        Ascend-Send-Passwd = "password",
        Ascend-Receive-Secret = "password"

mary-out        Password = "ascend",
        User-Service = Dialout-Framed-User
        User-Name = mary,
        Framed-Protocol = MPP,
        Framed-Address = 10.0.25.3,
        Framed-Netmask = 255.255.255.255,
        Ascend-Metric = 2,
        Framed-Routing = None,
        Ascend-Route-IP = Route-IP-Yes,
        Ascend-Idle-Limit = 30,
        Ascend-Send-Auth = Send-Auth-CHAP,
        Ascend-Send-Passwd = "password",
        Ascend-Receive-Secret = "password"

8.2 Synchronicity

$ find / -name samunix.tar -print 2>/dev/null &
[1] 10950
$ cd proj/book
$ mkdir advanced
$ cd advanced
/home/rsayle/proj/ucb/samunix.tar
$ touch example.sh
[1]+  Exit 1                  find / -name samunix.tar -print 2>/dev/null 
$

In this case, the operator decides to search for a missing tar file.  The user runs the find command and instructs the program to start the search from the root directory.  Any errors produced by find, namely directories that find is unable to search due to illegal priviliges for the user, are redirected to /dev/null.  This way, the user can continue to do work without being interrupted.  The user backgrounds the command as well.  The shell returns a message indicating that the command is running with process identifier 10950.  If the user chooses, a kill signal can be sent to the command to end it prematurely.  While the find runs, the operator continues with work.  The operator changes the working directory, creates a new subdirectory, and then changes working directories once more into the newly created subdirectory.  After this last action, the find command returns a successful hit.  It found the tar file in /home/rsayle/proj/ucb.  The user continues by creating a new shell script called example.sh.  The backgrounded job finishes its processing just after this, and it returns a message stating that it is complete.

Sometimes it may be necessary to synchronize a script with an asynchronous process, ie. a backgrounded job.   The shell provides the wait built-in command for such synchronization. Wait takes an optional process number as its argument.  It pauses the shell's execution until the specified process had terminated.  If only one process has been sent to the background, then wait can be issued without an argument.  It will automatically wait for the job to complete.  If more than one job is running, it becomes necessary to store their process identifiers in variables and pass the appropriate job number to wait.

#!/bin/sh
#
# stopOV: stop HP OpenView
#

PROG=`basename ${0}`

# check for the root user
if [ "${USER}" != "root" ]; then
  echo "${PROG}: only the root user can halt OpenView."
  exit 1
fi

# stop OpenView using its utility
ovstop >>ovstop.$$ 2>&1 &
echo "${PROG}: attempting to stop all OpenView processes..."
wait

# check for runaway OpenView processes
STILLRUNNING=`grep "RUNNING" ovstop.$$`
if [ "${STILLRUNNING}" != "" ]; then
  for ovprocess in ${STILLRUNNING}
  do
    ovpid=`echo ${ovprocess} | cut -f1 -d" "`
    echo "${PROG}: killing OpenView process ${ovpid}"
    kill ${ovpid} &
    LASTOVPID=$!
  done
fi

# wait for the last one to die
wait ${LASTOVPID}

# notify administrator if there are still run away processes
for ovproc in ${STILLRUNNING}
do
  runaway=`ps -ef | grep ${ovproc}`
  if [ "${runaway}" != "" ]; then
    echo "${PROG}: could not stop OpenView process: ${runaway}"
  fi
done

# Clean up
exit 0

The script aboves uses both forms of the wait command.  StopOV is a shell script that halts the HP OpenView^TM network management platform.  When the root user runs it, the script first tries using the stop utility that comes with the software.  It runs ovstop sending its output to the file ovstop.$$.  If all goes well, then this file will be empty.  The ovstop, as issued by the script also redirects stderr to the same file.  If any OpenView processes fail to terminate, it lists them in ovstop.$$ beginning with their process identifier.  The script sends the command to the background, prints a friendly message indicating that it is working, and then waits.  Here the script uses wait without any arguments since ovstop is the only job.

Upon completion of ovstop, the script continues by checking ovstop.$$ for any errors.  If it finds any processes still running, as stored in the STILLRUNNING variable, then it tries to halt them with the kill command.  To do this, the script employs a for loop.  It parses through the contents of STILLRUNNING and picks out the errant processes.  It sends a kill signal to these processes and backgrounds each attempt.  Through each iteration, it stores the attempt in LASTOVPID.  The final iteration of the loop resets LASTOVPID to the last kill issued.  The script then waits for the last kill to complete.  This of course assumes that all other kills issued complete prior to the last one finishing.

This may be an errant assumption, but as it turns out, it is sufficient for this program because when the last kill terminates, the scripts continues by checking the process table for any run away processes.  StopOV shows this in the last for loop.  If it finds any, it notifies the OpenView administrator so that the manager can take further action.

8.3 Arithmetic

In any event, there are moments when a script calls for basic mathematics.  A good example is when a programmer needs to increment a counter variable for a loop.  The expr command suits such a task well.

As an example of how to use expr, the script below demonstrates the calculation of the Pythagorean Theorem.

$ cat csquared
#!/bin/sh
#
# csquared: almost calculate the hypotenuse of a rt triangle
#

PROG=`basename ${0}`
USAGE="${PROG}: usage: ${PROG} base height"

if [ $# -ne 2 ]; then
  echo ${USAGE}
        exit 1
fi

ASQUARED=`expr ${1} \* ${1}`
BSQUARED=`expr ${2} \* ${2}`
expr ${ASQUARED} + ${BSQUARED}

$ csquared 3 4
25

Because expr cannot handle complex math such as exponentials, the script breaks the calculation into three steps.  It evaluates the squares of each side of the right triangle and stores the results in the temporary variables ASQUARED and BSQUARED.  During the evaluation of each expr, the programmer must escape the multiplication symbol so that the shell does not expand it as a wildcard.  The final calculation, which is the sum of the squares, is handled directly by expr.  It returns the result to the user's screen as demonstrated by the invokation of csquared.

UNIX has a much more precise and powerful calculator called bc.  It can evaluate real valued numbers, arrays, and exponentials, just to name a few.  Bc utilizes an interactive interface that reads from stdin and writes to stdout, but it also supports reading from and writing to files.  This fact makes it useful for shell scripts.

Now csquared can be rewritten with bc making it a much more compat program.

$ cat csquared
#!/bin/sh
#
# csquared: almost calculate the hypotenuse of a rt triangle
#

PROG=`basename ${0}`
USAGE="${PROG}: usage: ${PROG} base height"

if [ $# -ne 2 ]; then
  echo ${USAGE}
        exit 1
fi

bc <<EOF
${1}^2 + ${2}^2
quit
EOF

$ csquared 3 4
25

This version of csquared does not need intermediate variables since bc can evaluate complex expressions.  To pass the calculation to bc, the script employs input redirection, and to pass it the values for the expression, the shell uses the positional parameters.  Further, the calculation is handled in one line as the theorem itself is expressed, a² + b².  Bc prints the results on the user's terminal, but it could also easily be redirected to a file if needed.

8.4 Eval Statements

$ spool=/var/spool
$ ptr='$spool'
$ echo; echo $ptr; echo

$spool

$ eval echo $ptr
/var/spool

The example above demonstrates the double scan by creating a pointer variable.  First, the operator creates the variable spool and stores the directory /var/spool into it.  Next, the user declares a pointer, ptr, and stores in this variable $spool.  The single quotes prevent the shell from evaluating $spool; hence, ptr does not contain spool's value.  To show the value of spool, ptr's contents are echoed to the screen.  Additional echos help to differentiate the value from other commands.  Then, to get back the value ptr points to, the user employs eval.

When the shell invokes eval, it scans the arguments that follow and does a variable substitution on ptr.  This leaves the shell with echo $spool.  After this, the shell processes this command.  It performs another scan and expands $spool into its contents, /var/spool.  With the second scan complete, the shell prints /var/spool to the terminal.

8.5 Exec Statements

execute_custom_script () {
  if [ "${CUSTOMSCRIPT}" = "" ]; then
    exit
  else
    if [ -x ${CUSTOMSCRIPT} ]; then
      echo "Executing ${CUSTOMSCRIPT}"
      exec ${CUSTOMSCRIPT}
    fi
  fi
}

8.6 Setting Shell Flags

$ set goodbye cruel world
$ while [ $# -gt 0]
> do
> echo ${1}
> shift
> done
goodbye
cruel
world

8.7 Signals and Traps

In this form, trap instructs the shell to catch the signals listed by their numbers in the list of signals.  The shell handles the signals by executing commands, a list of valid UNIX commands.  To demonstrate, handletraps catches SIGINT and SIGTERM.

$ cat handletraps
#!/bin/sh
OUTPUT=out.$$
trap "if [ -f ${OUTPUT} ]; then rm ${OUTPUT}; exit; fi" 2 15
touch ${OUTPUT}
while true
do
  sleep 5
done
$ handletraps &
[1] 836
$ ls
handletraps out.836
$ kill -15 836
$ ls
handletraps

The program above starts by defining a variable whose value is out.$$.  It then sets up the signal handling with a trap.  The trap catches signals 2 and 15.  When these signals are sent to handletraps, the program checks for the existence of out.$$.  If it is present, it removes the file and then terminates.  The script continues by touching the file and then sits in an endless loop.

The user executes handletraps sending it to the background.  The shell shows it is running as process number 836.  A quick directory listing shows that the directory now contains the script plus the touched file, out.836.  The operator then sends a SIGTERM to handletraps with the kill command.  Another directory listing shows that the script caught the signal and ran the if block which removed out.836.

In its other forms, trap can be issued with no arguments.  This second form of trap just displays the list of currently handled signals.  In its third form, signals can be reset.  To do this, the programmer omits the set of commands from trap but provides a list of signals.  This resets the signals listed to their defaul behavior.  In its fourth form, trap can be used to ignore a set of signals.  This is done by giving an empty command list, denoted by "", to trap.  The last form of trap uses a colon as the command list.  This causes the parent shell to ignore a signal but its subshells terminate if they receive any signals listed in signals.

A couple properties of the trap command are important to remember.  Subshells inherit the trap behavior of their parents.  Also, the shell scans command once when it encounters the trap and then again when it processes a signal; consequently, it may be necessary to use single quotes to prevent expansion until the signal is caught.