Problem
I want to monitor certain filesystems on my host. If the space left on a particular filesystem is lower than a threshold, the System Administrator should be notified.
I also want to monitor my syslog for any errors, warnings or
failures which may have occured overnight. An example, might be HP's
Extended Feature Supplement, EFS, writes information to the 'syslog'
but unless you have configured the Insight Management services on
OpenServer6 and UnixWare7, you would not be notified of these events.Whilst this is not essential, the TA assumes that the server has been configured to send email to a remote smarthost either with your MMDF or Sendmail configuration.
Solution
The following shell script will use "df -v" to check the server's filesystems is less than 15%.
Save this script in a System Admin area such as /usr/bin or make your own, such as /local/sysadm and create "DiskSpaceMonitor" and modify root's crontab according to the description in the shell script's header.
This assumes that cron has $PATH access to these utilities within the script.
----------------------- Shell script starts here ---------------------------
:
#!/bin/sh
#
# DiskSpaceMonitor : Ideally used with cron to notify the System
# Administrator that a finite resource (here :
# file system space) is running out.
# The following entry in root's crontab file
# will run it every 15 Minutes Monday through
# Friday between 8am and 5.45pm
#
# 0,15,30,45 8-17 * * 1-5 /local/sysadm/DiskSpaceMonitor >
# /local/sysadm/DiskSpaceMonitor.log 2>&1
#
# Based on the article "Have your system page
# you automatically" of Ernest H. Rice III in
# the Unix Review, Edition March 1995
#
# In this example, a mail to "root" is sent, if the
# filesystem "/" has less than 15%, along with a message to
# the screen, console and syslog.
#
/bin/df -v | tr -d '%' | awk '{if ($6 >= 85) printf ("%s%%\n",$0)}' \
| /bin/grep -v "Mount" > /tmp/check.$$
if [ -s /tmp/check.$$ ]; then
MSG="`uname -n`: WARNING: DISK SPACE THRESOLD EXCEEDED ON:" export MSG
/bin/logger "$MSG `cat /tmp/check.$$`"
echo "$MSG `cat /tmp/check.$$`"
echo "$MSG `cat /tmp/check.$$`" > /dev/console
/usr/bin/mail -s "$MSG" root sysadm@mydomain.com < /tmp/check.$$
fi
rm /tmp/check.$$
----------------------- Shell script ends here ---------------------------NOTES: In addition the following tools can be used to monitor disk space
usage: What is using the most disk space?
---------------------------------
dfspace
df -kv - The -k option displays size in 1KB blocks
df -i - The -i option displays "INODE" usage
du / | sort -nr | head - Display the largest 10 directories
du -ks /<...> - Display -k size in 1k blocks / -s total
quot /dev/u1 - How much space is being used by each user Thresholds:
----------
df -v | tail +2 | tr -d '%' | awk '{if ($6 >= 85) printf ("%s%%\n",$0)}'It has been seen that SCO OpenServer filesystem performance will begin
to degrade, ie. slow down, when filesystem the free space drops below
15%.To Find Large Files:
-------------------
ls -l|sort +4 -nr
find / -size +n -print
for example where "n" is 6155 to find files greater than 6MB.
find / -user <username> -printLog Files that can be truncated:
-------------------------------
/etc/wtmp Accounts Record File
/usr/adm/messages
/usr/adm/syslog
/usr/spool/lp/admins/lp/logs/*
eg. /usr/spool/lp/logs/requestsAfter truncating the file with the command "> <filename>" then the server should be re-booted.
Other Log Files to look at:
--------------------
/usr/spool/mmdf/lock/home Look for mail notes, DeadLetters
/usr/mmdf/log chan.log, msg.log
/usr/adm/ Trim syslog, messages
/usr/adm/acct fiscal/nite/sum directories
/usr/adm/sa sa* SAR files
/usr/tmp
/lost+found
/usr/spool/mail Mail files can build up. For example,
if you have a cron job writing sending
email to root and the permissions are
not 1777 then Metamail can fail and
build up errors in the root mail file;
"Postmaster notify" errors,eg:N 20 Mail Delivery Subs ... Postmaster notify: see ...
Invoking metamail...UX:metamail: ERROR: You can not run
MetaMail as root unless you use -r or
set MM _RUNASROOTUX:metamail: TO FIX: Usage: metamailChecking the Root Filesystem
----------------------------
When you run out of disk space in the root filesystem sometimes the
only option is to re-boot the server into Single-User mode and remove
files as defined above. If the server has had to run an "fsck"
automatically on the root filesystem, in Single User Mode run:/tcb/bin/authck -av Check the tcb databasethen re-boot the server again to ensure the server boots up with a clean filesystem, ie. one that does not require "fsck" to be run automatically.
If running manually with the -ofull option then it is recommended to check the character device rather than the block device, eg /dev/rroot rather than /dev/root, see:
From UnixWare fsck(1M) man page:> special represents a block or character special device (for example, > /dev/rdsk/*, where the value of * is machine dependent). It is > preferable that a character special device be used. fsck will not > work on a block device if it is mounted.
http://uw714doc.xinuos.com/en/man/html.1M/fsck.1M.htmlNOTE: It is recommended to have the "root" disk on a separate physical disk
with application and data being, at least, in a separate filesystem.
Where possible it is preferable to use a RAID controller with the RAID
controller mirrored using RAID1 technology.The disk space requirements for the installation of OpenServer are available from:
http://osr5doc.ca.sco.com:457/RNOTES/RN_diskspace.htmlIt is suggested though that, at least, 1GB of disk space would be recommended for normal production.
NOTE: If you are not able to boot into Single-User Mode then alternative
emergency measures of mounting the root hard disk are available by
contacting your SCO Support provider.NOTE: Checking the syslog can be performed in a similar way to the
DiskSpaceMonitor script above, for example:----------------------- Shell script starts here ---------------------------
:
#!/bin/sh
#
# CheckSysLog : Ideally used with cron to notify the System
# Administrator that an issue has been detected in the
# server's syslog file.
#
# The following entry in root's crontab file
# will run it every evening.
#
# 59 23 * * * /local/sysadm/CheckSysLog >
# /local/sysadm/CheckSysLog.log 2>&1
## What's Today's Date
MONTH=`date '+%b'` export MONTH
DAY=`date '+%d'` export DAY
case "$DAY" in
"01")
DAY=" 1"
;;
"02")
DAY=" 2"
;;
"03")
DAY=" 3"
;;
"04")
DAY=" 4"
;;
"05")
DAY=" 5"
;;
"06")
DAY=" 6"
;;
"07")
DAY=" 7"
;;
"08")
DAY=" 8"
;;
"09")
DAY=" 9"
;;
esac# Is there a syslog.old to check against?
if [ ! -f /usr/adm/sylog.old ]; then
touch /usr/adm/syslog.old
fidiff /usr/adm/syslog /usr/adm/syslog.old > /tmp/tailed_syslog# Check for Today's Issues.
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i error > /tmp/todays.log
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i notice >> /tmp/todays.log
grep "$MONTH $DAY" /tmp/tailed_syslog | grep -i warning | grep -v arp >> /tmp/todays.log# Anythin to report?
if [ -s /tmp/todays.log ]; then
echo "Emailing ... "
cat /tmp/todays.log
/usr/bin/mail -s "`uname -n`: CheckSysLog Report" root sysadm@mydomain.com < /tmp/todays.log
fi# Now Prepare for Tomorrow.
cp /usr/adm/syslog /usr/adm/syslog.old----------------------- Shell script ends here ---------------------------This could also be expanded to include application logs or layered products such as the Office Mail Server's Insight Logs.
If you simply want to keep a regular eye on a server though, just add this to your root cron:
59 23 * * * /local/sysadm/status.sh > /local/sysadm/status.log 2>&1and have /local/sysadm/status.sh as:----------------------- Shell script starts here -------------------------:
# Daily Status Report
#
OUTFILE=/tmp/status.txt export OUTFILE
uname -a > $OUTFILE
echo " " >> $OUTFILE
who -b >> $OUTFILE
echo " " >> $OUTFILE
memsize >> $OUTFILE
echo " " >> $OUTFILE
dfspace >> $OUTFILE
sar >> $OUTFILE
echo " " >> $OUTFILE
FILE=/tmp/proc.out
echo "Top on the Process Queue ... " > ${FILE}# top?
TOP=/usr/local/bin
if [ -x ${TOP}/top ]; then
echo " " >> ${FILE}
echo "Top:" >> ${FILE}
echo "--- " >> ${FILE}
${TOP}/top -S -q -o cpu 10 >> ${FILE}
fiecho " " >> ${FILE}
echo "CPU Time:" >> ${FILE}
echo "-------- " >> ${FILE}
ps -elf | head -1 >> ${FILE}
ps -elf | grep -v "00:00:00" | sort -k14 -r | grep -v WCHAN | head -10 >> ${FILE}echo " " >> ${FILE}
echo "Memory Usage:" >> ${FILE}
echo "------------ " >> ${FILE}
ps -elf | head -1 >> ${FILE}
ps -elf | sort -k 11 -n -r | grep -v WCHAN | head -10 >> ${FILE}
cat ${FILE} >> $OUTFILE
echo " " >> $OUTFILE
tail /usr/adm/syslog >> $OUTFILE
mail -s "`uname -n`: Status Report" sysadm@mydomain.com < /tmp/status.txt----------------------- Shell script ends here --------------------------- See also
df(C)
dfspace(C)
quot(ADM)
wtmp(F)