Problem
My system's behavior is slow. How do I quickly determine the most
likely cause of the bottleneck?For OpenServer5/6, ensure "sar" is enabled with:
# /usr/lib/sa/sar_enable -yUnixware automatically runs sar by default. Solution
The following is a two-step guide to quickly finding the bottleneck on a system with slow performance.
Note: Running two sar commands as a "before and after" test, (once before
the system slowdown and once during the system slowdown), will make
it easier to see differences in sar data.Characters after the '#' sign indicate commands to be run.
To review existing "sar" data from a previous day or file then:
# cd /var/adm/sa# sar -A -f sa<day>STEP 1
------
Check the general system activity:# sar 1 5
09:35:13 %usr %sys %wio %idle (-u)
09:35:14 17 0 0 83
09:35:15 5 0 0 95
09:35:16 5 0 0 95
09:35:17 5 0 0 95
09:35:18 5 1 0 94Average 7 0 0 92This command should always be run first to get a general idea of the primary location of the bottleneck.
- If %usr is high, then the system is waiting for user commands to finish (i.e. sort, data gathering/processing programs).
CAUSES: Non-interactive programs running unnecessarily at peak hours, slow CPU, not enough CPUs, unnecessary programs running, inefficient third party progams running, daemons processing data, bad nice values.
- If %sys is high, then the system is waiting for kernel driver calls to complete (i.e. hardware issues, spurious interrupts, third party drivers).
CAUSES: Inefficient third party drivers, bad hardware causing spurious interrupts, slow CPU, not enough CPUs.
- If BOTH %usr and %sys are high, then the system is waiting for all types of system calls, whether they are user generated or kernel generated.
CAUSES: Slow CPU, not enough CPUs.- If %wio is high, the system is waiting for the disk subsystem to retrieve data.
CAUSES: Not enough disk cache (NBUF/NHBUF), slow hard drive system, not enough memory, memory leak from process, process grabbing too much memory.
Check the basic disk performance speed with, a more advanced test would be to use more random writes to the disk, say with a creating and unpacking a large "cpio" archive or to use a benchmarking tool.
For example, this will generate a 1GB file:
# timex dd if=/dev/zero of=big bs=1024k count=1000A more detailed test file is shown below.
STEP 2
------
- IF %usr IS HIGH:1) Check for processes consuming too much CPU time:
# ps -el | more
F S UID PID PPID C PRI NI ADDR SZ TTY TIME CMD
71 S 0 0 0 0 95 20 fb117000 0 ? 00:00:01 sched
20 S 0 1 0 0 66 20 fb117158 148 ? 00:00:00 init
.
.
.
20 S 0 347 1 0 76 24 fb119db0 312 ? 00:00:00 snmpd
20 S 17 349 1 1 66 20 fb119f08 156 ? 01:05:53 deliver
20 S 0 413 410 0 75 20 fb11a060 128 ? 00:00:00 lockd(WCHAN removed to fit in screen)Check the C and TIME values. If the TIME value (minutes:seconds:100ths of
a second) is unusually high and C is positive for a specific process, then
that process could be taxing the system. In the above example, the deliver
daemon is processing leftover admin mail. The mail can be removed.NOTE: WCHAN (l)
The address of an event for which the process is sleeping or in
SXBRK state; if blank, the process is running. For an
individual LWP if -L is specified.You can also use "who" to check on the number of idle processes clocking up CPU time:
# w
4:41pm up 5:04, 3 users, load average: 0.00 0.00 0.00
User tty login@ idle JCPU PCPU what
root tty01 11:51am 21:38 bash
root tty03 4:31pm 9 -sh
root ttyp1 1:55pm w# who -u | sort -k 6 -r
root tty03 Jul 6 16:31 0:09 1774
root ttyp1 Jul 6 13:55 0:01 3902
root tty01 Jul 6 11:51 . 17732) Check for system call activity:# sar -c 1 5
SCO_SV tuvok 3.2v5.0.5 i80386 06/21/200109:55:08 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s (-c)
09:55:09 1216 67 12 0.99 0.99 178441 3988
09:55:10 147 31 6 0.00 0.00 168723 8421
09:55:11 74 27 4 0.00 0.00 163644 3342
09:55:12 245 37 6 0.00 0.00 171821 8928
09:55:13 151 29 4 0.00 0.00 163770 3468Average 367 38 6 0.20 0.20 169280 5629Check before and after for system calls, forks, execs., etc. If system
calls are high, it indicates one or more of the following:- Programs are suddenly being used more actively. - More programs in general are being run on the system.
Use ps to check to see if this is necessary. If forks/execs, or reads/writes are specifically high, check for programs that may be calling specific calls.
- IF %sys IS HIGH:If you have a muliprocessing system, run the following command to see if any device is sending thousands of interrupts to slow down the CPU.
1) For OpenServer5:#sar -j 1 5For UnixWare7/Open UNIX 8/OSR6:#sar -P ALL 1 5Check programs that could be accessing the tape drive, third party smart boards, or other non-disk drivers.
- IF BOTH USR AND SYS ARE HIGH:
1) Check the run queue, for swapping:
# sar -q 1 5SCO_SV lunasco 3.2v5.0.4 Pentium 06/21/200110:46:29 runq-sz %runocc swpq-sz %swpocc (-q)
10:46:30 3.0 100
10:46:31
10:46:32 1.0 100
10:46:33 1.0 100
10:46:34 1.0 100Average 1.5 100Normally the Average time should be less than 3 on a taxed system. If the
average time is constantly higher than that, the processes are not being
serviced quick enough; the CPU could be to blame. Either increase the CPU
speed or add CPUs, if possible.- IF %wio IS HIGH:1) Check for greedy processes:# ps -el | more
F S UID PID PPID C PRI NI ADDR SZ TTY TIME CMD
71 S 0 0 0 0 95 20 fb117000 0 ? 00:00:01 sched
20 S 0 1 0 0 66 20 fb117158 148 ? 00:00:00 init
71 S 0 2 0 0 95 20 fb1172b0 0 ? 00:00:00 vhand
71 S 0 3 0 0 95 20 fb117408 0 ? 00:00:16 bdflush
71 S 0 4 0 0 95 20 fb117560 0 ? 00:00:00 kmdaemon
71 S 0 5 1 0 95 20 fb1176b8 0 ? 00:00:18 htepi_daemon
.
.
.
20 S 0 252 1 0 76 20 fb118830 152 ? 00:00:00 cron
20 S 0 354 1 0 76 24 fb118988 233504 ? 00:00:03 report
20 S 0 496 1 0 76 24 fb118ae0 200 ? 00:00:00 calserverCheck the SZ value to see if a process is either grabbing too much memory or
not freeing it up when needed. In the above example, the "report" program is
grabbing a lot of memory.2) Check the amount of useable memory:
# sar -r 1 5SCO_SV tuvok 3.2v5.0.5 i80386 06/21/200110:34:13 freemem freeswp availrmem availsmem (-r)
10:34:14 8262 389120 28765 56421
10:34:15 8262 389120 28765 56421
10:34:16 8262 389120 28765 56421
10:34:17 8262 389120 28765 56421
10:34:18 8262 389120 28765 56421Average 8262 389120 28765 56421If freemem (listed in 4K pages) is below 500 and freeswap is dynamic, the system is paging in data because it can't fit what it needs in memory. If this is happening all the time, increase RAM.
Pleae note: In OpenServer sar -r reports the amount of swap space on disk.
In UnixWare7/Open UNIX 8, it reports the swap space in virtual memory
(RAM plus swap).3) Check the disk i/o caching usage:
# sar -b 1 5SCO_SV tuvok 3.2v5.0.5 i80386 06/21/200110:37:17 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s (-b)
10:37:18 0 0 0 0 0 0 0 0
10:37:19 0 0 0 0 0 0 0 0
10:37:20 0 60 100 0 1 100 0 0
10:37:21 0 -1 100 0 0 0 0 0
10:37:22 0 56 100 0 1 100 0 0Average 0 24 100 0 0 100 0 0If %rcache is continuously < 85 and/or %wcache is < 80, then the system is having to go to the hard drive to load the disk cache. Increase the disk cache by increasing NBUF by 50 percent and adjusting NHBUF appropriately.
For UnixWare7/Open UNIX 8.0.0, there are the additional kernel tunables:
"FDFLUSHR", this is the interval in seconds to check the need to write the
buffer cache and file pages to disk. The default is 1."NAUTOUP", this is the humber of seconds between filesystems updates. The
default is 60 seconds. Increasing NAUTOUP can improve performance, but also
increase the risk of data loss should a system crash occur.4) Check the actual disk i/o with:sar -dA %busy figure averaging > 50% can indicate a disk bottleneck.
The "avserv" column shows service tume after a request has arrived at the disk.
The "await" column shows the average wait time for an I/O request to be serviced.
The "avque" column shows the average length of the wait queue for an I/O request.
# sar -d
UnixWare omega 5 7.1.3 i386 02/09/0500:00:00 device MB %busy avque r+w/s blks/s avwait avserv
10:40:01 c0b0t0d0s1 12867 90 23.7 8 83 2556.6 112.8
10:40:01 c0b0t0d0s10 50 0 1.0 0 0 0.0 790.0
10:40:01 c0b0t0d0 17359 90 23.7 8 83 2556.4 112.8
10:40:01 c0b0t1d0s12 65458 1 1.6 2 21 4.1 6.9
10:40:01 c0b0t1d0 69459 1 1.6 2 21 4.1 6.9
10:40:01 c0b0t2d0s1 138918 2 3.2 4 86 13.3 6.1
10:40:01 c0b0t2d0 138919 2 3.2 4 86 13.3 6.1Here with %busy and no process consuming memory there is a massive amount of
IO occuring on the root disk. If this is a RAID set, is the logical volume
recovering from a disk failure in the set. For example, for the HP CISS SMART
ARRAY contoller, visit /usr/bin/compaq/bin/diags/ciss_menu and look at the
status of each logical volume.If the disk performance appears to be acceptable, but you have unacceptable levels of CPU time devoted to waiting for I/O, then you may have a memory bottleneck.
5) Check the use of file access system routines for disk fragmentation issues with:sar -aReport use of file access system routines (per second):
iget/s
number of S5, SFS, VXFS, BFS, and UFS files located by
inode entry
namei/s
number of filesystem path searches
dirbk/s
number of S5 directory block reads issued
%dnlc
hit rate of directory name lookup cacheIf -R is specified then %dnlc is replaced by dnlc-hits and dnlc-miss, the counts of cache hits and misses.
slab6(7.1.4)# sar -aUnixWare slab6 5 7.1.4 i386 07/16/0900:00:00 iget/s namei/s dirbk/s %dnlc
01:00:00 8 32 8 91
02:00:00 8 32 8 91
03:00:00 8 32 8 91
04:00:00 8 32 8 91
05:00:00 8 32 8 91
06:00:00 14 52 20 89
Average 9 35 10 90See TA 112921Please Note: Make sure you do not increase NBUF by so much that you run out of regular general purpose memory (check with sar -r).
If neither memory nor the disk cache is a problem, check the disk i/o system as you may need a RAID system or a faster host adapter system.
For OpenServer5 see:http://osr507doc.xinuos.com/en/PERFORM/tuning_IO_rsc.htmlFor OpenServer6/UnixWare7 see:'rtpm' and check the IO Stats, rdblk and wrblk to determine read and write bottlenecks.
Best displayed on the console or a graphical interface which shows high usage in RED.
Here's an example where there are a large number of TCP/IP failures and high CPU (%sys) usage:
1) netstat -s in the sysinfo shows some errors, mainly icmp. Within rtpm if you use the arrow keys to highlight "TCP/IP:" and hit enter you will see details on exactly what types or error are being encountered. For example, are you are getting ICMP destination unreachable errors and this can be confirmed by hitting enter with "ICMP:" highlighted?
2) The vflts value indicates the number of times a second the CPU failed to perform an address translation. This is an indication of how busy the CPU or CPU group is.
3) The CALLS/s value is an indication of the number of system calls a second being made by the applications running on the server. The higher this value the busier the server is. You can highligh CALLS/s: and hit enter to see the types of call being made. For example, you might be seeing a lot of read or write calls given that your users are using Samba and a Database.
NOTE: In general, ensure that the server has the latest patches, host bus adaptors and network drivers are installed available from:
http://www.sco.com/support/download.htmlNOTE: There are other tools available to assist in identifying the processes and analysing the output of sar.
These are:Check the Message Queues using:# sar -m# ipcs -qopTA105414, "How do I find out the open files and stack trace used by a hanging process? "
Check the Streams Buffers using:# netstat -mTA116684, "OpenServer 5, How to Debug STREAMS failures."
List of Open Files utility, lsof, available from:
http://www.sco.com/skunkwareFiles in /varhttp://uw714doc.xinuos.com/en/SM_concepts/_Files_in_var.htmlFor OpenServer5/OpenServer6 and UnixWare7:"top" available from Skunkware at http://www.sco.com/skunkwareTCP/IP Monitoring:"inconfig"See /etc/inet/inet.dfl or /etc/default/inet for OpenServer5# inconfig arpprintfs 10
# inconfig igmpprintfs 10
# inconfig ipprintfs 10
# inconfig icmpprintfs 10
# inconfig tcpprintfs 10
# inconfig udpprintfs 10For OpenServer5 only you can also add:
# inconfig mbclprintfs 10
# inconfig nbprintfs 10This can increase the debug level for these protocols and should generate additional debug messages in your /var/adm/syslog.
They can be turned off with the 0 parameter.
The verbosity increases with the value:0=none, 1=minimal, 10=most verboseThe debug messages will have error codes and a reverse hex representation of the IP address where the bad packet originated.
For example you may see:NOTICE:icmp_error(F08296E4,3,3)In this case the IP associated with F08296E4 is translated by breaking it into hex pairs and reversing the 4 octets.
F0 = 240 82 = 130 96 = 150 E4 = 228
So this is from the IP 228.150.130.240..
In the file /usr/include/netinet/ip_icmp.h we can translate the 3, 3
part of the message to mean:#define ICMP_UNREACH 3 /* dest unreachable, codes: */
#define ICMP_UNREACH_PORT 3 /* bad port */Meaning that the packet was destined to a non-existent port or to a port which did not have any services.
To get further information about these packets might require using a sniffer to see what was actually in the packet and where it was destined.
Monitoring and tuning the system:OpenServer5:
-----------
- http://osr507doc.xinuos.com/en/PERFORM/autotune_desc.html- http://osr507doc.xinuos.com/en/PERFORM/kernel_configure.html- SkunkWare Products:"hog" package available from Skunkware, above, these include "memhog", "iohog" and "cpuhog".
SarCheck available for OSR5 from:ftp://ftp2.sco.com/pub/skunkware/osr5/sysadmin/SarCheck/"u386mon" for OSR5 available from Skunkware from:
http://www.sco.com/skunkware/faq.html#u386UnixWare7/OpenServer6:
---------------------
- http://uw714doc.xinuos.com/en/SM_perform/CONTENTS.html- http://uw714doc.xinuos.com/en/FEATS/new_features_710_performance.html- Real Time Performance Monitor for LWP Processes:
rtpm(1M) for UnixWare7/Open UNIX 8.0.0/OpenServer6- http://uw714doc.xinuos.com/en/SM_perform/_Real-Time_Performance_Monitor.html- http://uw714doc.xinuos.com/en/man/html.1M/rtpm.1M.html- The System Monitor:http://uw714doc.xinuos.com/en/SM_perform/_System_Monitor.html- Summary of system administration tasks:http://uw714doc.xinuos.com/en/HANDBOOK/saT.tasksummary.htmlAIM Benchmarking:This can be obained from:http://sourceforge.net/projects/aimbenchDownload the source and compile using the "make" file supplied. A "cc" compiler from the Development System will be required. Compile with the "-lsocket -lnsl" linker options.
For OpenServer5, edit disk1.c to comment out the following definition:
To:/* #define _M_XOUT so O_SYNC can be used on
Acer/Altos */sysinfo:The latest sysinfo see: http://www.sco.com/support/sysinfo.htmlOnce installed, use: /usr/lib/sysinfo.d/bin/stune_verify to check the values in your "stune" file do not exceed the maximum or are below the default recommended values.
RAID Levels and their impact: (for %wio):
RAID LVL Striping Mirroring Error Correction
RAID0 Block Level No n/a
RAID1 n/a Yes n/a
RAID2 Bit Level No Hammering ECC
RAID3 Byte Level No Dedicated Parity
RAID4 Block Level No Dedicated Parity
RAID5 Block Level No Distributed Parity
RAID6 Block Level No Dual Distributed Parity
RAID7 Cached No Dedicated Parity
RAID0+1 Block Level Yes n/a
RAID0+3 Block & Byte Level No Dedicated Parity
RAID0+5 Block Level No Distributed Parity
RAID1+5 Block Level No Distributed Parity
RAID1+0 Block Level Yes n/a
RAID3+0 Block & Byte Level No Dedicated Parity
RAID5+0 Block Level No Distributed Parity
RAID5+1 Block Level No Distributed ParityRAID LVL Read Speed Write Speed Fault Tolerance Market Acept'nc Cost RAID0 5 5 1 5 Lowest RAID1 3 3 5 5 High RAID2 1 1 3 1 Highest RAID3 3 1 3 5 Medium RAID4 5 1 3 1 Medium RAID5 5 1 3 5 Low RAID6 5 1 5 1 High RAID7 5 5 3 3 Highest RAID0+1 5 5 5 5 High RAID0+3 5 1 3 1 High RAID0+5 5 3 3 1 High RAID1+5 5 3 5 1 Highest RAID1+0 5 5 5 5 High RAID3+0 5 1 3 1 High RAID5+0 5 3 3 1 High RAID5+1 5 3 5 1 Highest
Marks out of 5.RAID LVL Application RAID0 Non Critical Data requiring high speed and low cost RAID1 High Fault Tolerance RAID2 Rarely Used RAID3 Large file applications requiring high speed and redundancy RAID4 A compromise between RAID3 and RAID5 RAID5 General purpose, relational database & ERP applications RAID6 Similar to RAID5 with additional fault tolerance RAID7 Specialised high end applications RAID0+1 High Performance and Reliability RAID0+3 Higher Performance than RAID3 for large file applications RAID0+5 Increased Capacity over RAID5 RAID1+5 Very High Fault Tolerance RAID1+0 High Performance and Reliability RAID3+0 Higher Performance than RAID3 for large file applications RAID5+0 Increased Capacity over RAID5 RAID5+1 Very High Fault Tolerance
NOTES: IT IS RECOMMENDED TO ANALYSE THE sa DATA FILES ON THE SAME OPERATING SYSTEM AS THE ORIGINAL.
eg. Do not take a "sa" sar data file from OSR6 and analyse it on a UW714 server because the results will not be correct.
Also, when taking the data for analysis between different timezones then this can also result in missing data around the midnight mark where the timezones differ.
It is recommended to simply use:# cd /var/adm/sa
# sar -A -f sa<day> > /tmp/sa<day>.txtand examine the /tmp/sa<day>.txt file.TROUBLESHOOTING:
---------------
Performance Tuning:- Identify bottlenecksar, Rtpm (OSR6/UW7), (prfstat, prof, lprof)
CPU performancesar -u
00:00:00 %usr %sys %wio %idle %intr
00:00:01 30 10 10 46 4high usr, investigate with truss, prof high sys, intr, investigate with prfstat high wio, storage throughput
Storage Performance:- Hardware configurationDevice topology:don't connect slow devices and fast devices on the same bus e.g. put your slow tape drive on a separate controller
Cabling:ensure your cables are up to specificationsHardware RAID:performance RAID 0 vs integrity RAID 1 RAID 5
Filesystem tuning:fsadm, block size, increase logsize (@ mkfs only)
mount options; tmplogMemory:Avoid swapping DEDICATED_MEMORY, use if using shared memory mkdev dedicated Dedicated memory reserves physical Saves kernel virtual Reduces paging, uses large mappings (PSE) SEGKMEM_PSE_BYTES Add more memory!
Tuning for largefile support:HDATLIM, SDATLIM, HVMMLIM, SVMMLIM, HFSZLIM, SFSZLIM set to 0x7fffffff
(unlimited)
/etc/conf/bin/idbuild -B && init 6
fsadm /mountpoint or raw device
fsadm -o largefiles /
OSR6 defaults to largefiles, UW7 does notBuilding large file aware applications:-D_FILE_OFFSET_BITS=64NOTE: To assist in investigating a process using 'truss' try using truss# truss -o /tmp/binary.truss.$$ -aef -wall -rall -vall <command>Alternatively, try a wrapper script?, eg:# mv binary binary.origThen use a script like this saved as the original binary name.
: truss -o /tmp/binary.truss.$$ -aef -wall -rall -vall binary.orig $*
NOTE: To gather statistics on a 'sar' you can run it manually with:# sar -A -o <output file>and analyse with, for example:# sar -u -f <output file>A record of output files is stored in /usr/adm/sa
A script to automatically gather performance statistics can be found here:
./sar_check 10----- cut here -----#!/bin/sh# 1 minute
period=${1:-"60"}# For example 17 hours worth
iterations=${2:-"`expr 86400 / $period`"}echo "sar: $iterations iterations at $period second intervals"
/usr/sbin/sar -A -o /local/sar/sa`date +%d` $period $iterations > /dev/null----- cut here -----This assumes /local/sar exists.It can also be run in conjunction with the Server Certification Tests to give performance information, from:
http://www.sco.com/developers/hdk/testsuites/NOTE: Other simple tasks to save on resource would be to run on all SCO
Unix's:# scologin disableand for OpenServer5 and OpenServer6:Turn off all of the unneccessary tty's running on the console, if you like, to save on memory:
echo "Disabling tty05...tty12 ..."
for i in 05 06 07 08 09 10 11 12
do
echo $i
disable tty${i}
doneLastly, you may wish to disable the un-used Calendar Server (not OMS related) with:
# cd /etc/rc2.d
# ./P95calserver stop
# mv P95calserver p95calserverNOTE: Troubleshooting Test file (test.sh):----- Start to cut here -----echo "Turn Kernel Profiling on ... "
prfstat on
echo "Empty Statistic Files ... "
rm -f prfsnap.out
rm -f sadc.out
echo "Take a Kernel Snapshot ... "
prfsnap prfsnap.out
echo "Collect 5 mins of Sar info ... "
/usr/lib/sa/sadc 5 60 sadc.out &
dfspace
echo "Generate a count=2000 (2GB) file or count=10000 (10GB) file if Large files are supported ... "
timex dd if=/dev/zero of=testfile bs=1048576 count=2000
dfspace
echo "Remove Test File ... "
rm testfile
dfspace
echo "Take a Kernel Snapshot ... "
prfsnap prfsnap.out
echo "Turn Kernel Profiling off ... "
prfstat off
echo "Example Kernel Profile Analysis ... "
prfpr -PALL prfsnap.out 0
echo "Example Sar Analysis ... "
sar -f sadc.out----- Finish Cut Here ----- See also
sar(ADM), vmstat(C), idtune(ADM)Differences in Kernel Tuning between OpenServer5 and OpenServer6/ UnixWare7 see:
http://osr600doc.xinuos.com/en/SM_perform/osr507kerntuns.htmlTA117424, How can I analyze "on-the-spot" performance data with SarCheck?"
TA114622, "How do I install and run SarCheck?"http://www.aplawrence.com/Unixart/slow.html