Technical article 104919

How can I monitor disk filesystem space and the syslog automatically?

OpenServer 6.0.0OpenServer 5.0.0OpenServer 5.0.2OpenServer 5.0.4OpenServer 5.0.5OpenServer 5.0.6OpenServer 5.0.7UnixWare 7.0.0UnixWare 7.0.1UnixWare 7.1.0UnixWare 7.1.1UnixWare 7.1.2UnixWare 7.1.3UnixWare 7.1.4Open UNIX 8

Written · Last revised

Problem

I want to monitor certain filesystems on my host. If the space left on a particular filesystem is lower than a threshold, the System Administrator should be notified.

I also want to monitor my syslog for any errors, warnings or
failures which may have occured overnight.  An example, might be HP's
Extended Feature Supplement, EFS, writes information to the 'syslog'
but unless you have configured the Insight Management services on
OpenServer6 and UnixWare7, you would not be notified of these events.

Whilst this is not essential, the TA assumes that the server has been configured to send email to a remote smarthost either with your MMDF or Sendmail configuration.

Solution

The following shell script will use "df -v" to check the server's filesystems is less than 15%.

Save this script in a System Admin area such as /usr/bin or make your own, such as /local/sysadm and create "DiskSpaceMonitor" and modify root's crontab according to the description in the shell script's header.

This assumes that cron has $PATH access to these utilities within the script.

----------------------- Shell script starts here ---------------------------
:
#!/bin/sh
#
# DiskSpaceMonitor : Ideally used with cron to notify the System
#                    Administrator that a finite resource (here :
#                    file system space) is running out.
#                    The following entry in root's crontab file
#                    will run it every 15 Minutes Monday through
#                    Friday between 8am and 5.45pm
#
#                    0,15,30,45  8-17 * * 1-5 /local/sysadm/DiskSpaceMonitor >
#                    /local/sysadm/DiskSpaceMonitor.log 2>&1
#
#                    Based on the article "Have your system page
#                    you automatically" of Ernest H. Rice III in
#                    the Unix Review, Edition March 1995
#
#                    In this example, a mail to "root" is sent, if the
#                    filesystem "/" has less than 15%, along with a message to
#                    the screen, console and syslog.
#
/bin/df -v | tr -d '%' | awk '{if ($6 >= 85) printf ("%s%%\n",$0)}' \
| /bin/grep -v "Mount" > /tmp/check.$$
if [ -s /tmp/check.$$ ]; then
        MSG="`uname -n`: WARNING: DISK SPACE THRESOLD EXCEEDED ON:" export MSG
        /bin/logger "$MSG `cat /tmp/check.$$`"
        echo "$MSG `cat /tmp/check.$$`"
        echo "$MSG `cat /tmp/check.$$`" > /dev/console
        /usr/bin/mail -s "$MSG" root sysadm@mydomain.com < /tmp/check.$$
fi
rm /tmp/check.$$
----------------------- Shell script ends here ---------------------------
NOTES:  In addition the following tools can be used to monitor disk space
        usage:
        What is using the most disk space?
        ---------------------------------
dfspace
df -kv                          - The -k option displays size in 1KB blocks
df -i                           - The -i option displays "INODE" usage
du / | sort -nr | head          - Display the largest 10 directories
du -ks /<...>                   - Display -k size in 1k blocks / -s total
quot /dev/u1                    - How much space is being used by each user
        Thresholds:
        ----------
df -v | tail +2 | tr -d '%' | awk '{if ($6 >= 85) printf ("%s%%\n",$0)}'
It has been seen that SCO OpenServer filesystem performance will begin
to degrade, ie. slow down, when filesystem the free space drops below
15%.
To Find Large Files:
-------------------
ls -l|sort +4 -nr
find / -size +n -print
      for example where "n" is 6155 to find files greater than 6MB.
find / -user <username> -print
Log Files that can be truncated:
-------------------------------
/etc/wtmp                       Accounts Record File
/usr/adm/messages
/usr/adm/syslog
/usr/spool/lp/admins/lp/logs/*
   eg. /usr/spool/lp/logs/requests

After truncating the file with the command "> <filename>" then the server should be re-booted.

        Other Log Files to look at:
        --------------------
/usr/spool/mmdf/lock/home               Look for mail notes, DeadLetters
/usr/mmdf/log                           chan.log, msg.log
/usr/adm/                               Trim syslog, messages
/usr/adm/acct                           fiscal/nite/sum directories
/usr/adm/sa                             sa* SAR files
/usr/tmp
/lost+found
/usr/spool/mail                         Mail files can build up.  For example,
                                        if you have a cron job writing sending
                                        email to root and the permissions are
                                        not 1777 then Metamail can fail and
                                        build up errors in the root mail file;
                                        "Postmaster notify" errors,
eg:

N 20 Mail Delivery Subs ... Postmaster notify: see ...

Invoking metamail...
UX:metamail: ERROR: You can not run
MetaMail as root unless you use -r or
set MM _RUNASROOT
UX:metamail: TO FIX: Usage:  metamail
Checking the Root Filesystem
----------------------------
When you run out of disk space in the root filesystem sometimes the
only option is to re-boot the server into Single-User mode and remove
files as defined above.  If the server has had to run an "fsck"
automatically on the root filesystem, in Single User Mode run:
/tcb/bin/authck -av                     Check the tcb database

then re-boot the server again to ensure the server boots up with a clean filesystem, ie. one that does not require "fsck" to be run automatically.

If running manually with the -ofull option then it is recommended to check the character device rather than the block device, eg /dev/rroot rather than /dev/root, see:

From UnixWare fsck(1M) man page:

> special represents a block or character special device (for example, > /dev/rdsk/*, where the value of * is machine dependent). It is > preferable that a character special device be used. fsck will not > work on a block device if it is mounted.

http://uw714doc.xinuos.com/en/man/html.1M/fsck.1M.html
NOTE:   It is recommended to have the "root" disk on a separate physical disk
        with application and data being, at least, in a separate filesystem.
        Where possible it is preferable to use a RAID controller with the RAID
        controller mirrored using RAID1 technology.

The disk space requirements for the installation of OpenServer are available from:

http://osr5doc.ca.sco.com:457/RNOTES/RN_diskspace.html

It is suggested though that, at least, 1GB of disk space would be recommended for normal production.

NOTE:   If you are not able to boot into Single-User Mode then alternative
        emergency measures of mounting the root hard disk are available by
        contacting your SCO Support provider.
NOTE:   Checking the syslog can be performed in a similar way to the
        DiskSpaceMonitor script above, for example:
----------------------- Shell script starts here ---------------------------
:
#!/bin/sh
#
# CheckSysLog : Ideally used with cron to notify the System
#               Administrator that an issue has been detected in the
#               server's syslog file.
#
#               The following entry in root's crontab file
#               will run it every evening.
#
#               59 23 * * * /local/sysadm/CheckSysLog >
#               /local/sysadm/CheckSysLog.log 2>&1
#
# What's Today's Date
MONTH=`date '+%b'` export MONTH
DAY=`date '+%d'` export DAY
case "$DAY" in
"01")
	DAY=" 1"
	;;
"02")
	DAY=" 2"
	;;
"03")
	DAY=" 3"
	;;
"04")
	DAY=" 4"
	;;
"05")
	DAY=" 5"
	;;
"06")
	DAY=" 6"
	;;
"07")
	DAY=" 7"
	;;
"08")
	DAY=" 8"
	;;
"09")
	DAY=" 9"
	;;
esac
# Is there a syslog.old to check against?
if [ ! -f /usr/adm/sylog.old ]; then
	touch /usr/adm/syslog.old
fi
diff /usr/adm/syslog /usr/adm/syslog.old > /tmp/tailed_syslog
# Check for Today's Issues.
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i error > /tmp/todays.log
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i notice >> /tmp/todays.log
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i warning | grep -v arp >> /tmp/todays.log
# Anythin to report?
if [ -s /tmp/todays.log ]; then
	echo "Emailing ... "
	cat /tmp/todays.log
/usr/bin/mail -s "`uname -n`: CheckSysLog Report" root sysadm@mydomain.com < /tmp/todays.log
fi
# Now Prepare for Tomorrow.
cp /usr/adm/syslog /usr/adm/syslog.old
----------------------- Shell script ends here ---------------------------

This could also be expanded to include application logs or layered products such as the Office Mail Server's Insight Logs.

If you simply want to keep a regular eye on a server though, just add this to your root cron:

59 23 * * * /local/sysadm/status.sh > /local/sysadm/status.log 2>&1
and have /local/sysadm/status.sh as:
----------------------- Shell script starts here -------------------------
:
# Daily Status Report
#
OUTFILE=/tmp/status.txt export OUTFILE
uname -a > $OUTFILE
echo " " >> $OUTFILE
who -b >> $OUTFILE
echo " " >> $OUTFILE
memsize >> $OUTFILE
echo " " >> $OUTFILE
dfspace >> $OUTFILE
sar >> $OUTFILE
echo " " >> $OUTFILE
FILE=/tmp/proc.out
echo "Top on the Process Queue ... " > ${FILE}
# top?
TOP=/usr/local/bin
if [ -x ${TOP}/top ]; then
        echo " " >> ${FILE}
        echo "Top:" >> ${FILE}
        echo "--- " >> ${FILE}
        ${TOP}/top -S -q -o cpu 10 >> ${FILE}
fi
echo " " >> ${FILE}
echo "CPU Time:" >> ${FILE}
echo "-------- " >> ${FILE}
ps -elf | head -1 >> ${FILE}
ps -elf | grep -v "00:00:00" | sort -k14 -r | grep -v WCHAN | head -10 >> ${FILE}
echo " " >> ${FILE}
echo "Memory Usage:" >> ${FILE}
echo "------------ " >> ${FILE}
ps -elf | head -1 >> ${FILE}
ps -elf | sort -k 11 -n -r | grep -v WCHAN | head -10 >> ${FILE}
cat ${FILE} >> $OUTFILE
echo " " >> $OUTFILE
tail /usr/adm/syslog >> $OUTFILE
mail -s "`uname -n`: Status Report" sysadm@mydomain.com < /tmp/status.txt
----------------------- Shell script ends here ---------------------------

See also

df(C)
dfspace(C)
quot(ADM)
wtmp(F)

From the Xinuos technical article database, record 104919. Reproduced with permission. The original is at www.sco.com/ta/104919, which is reachable but blocked from search engines.

Next step

Need this fixed rather than explained?

Tell us the release, what the system runs and what it is doing. We will reply by email with what we would do next.

  • You deal with the people who do the work
  • No obligation, and no upgrade required before we will help
  • We tell you plainly if something is not worth doing

Send an enquiry

We reply by email. If a server is down, say so in the first line. You can also write to help@scoexpert.com.