Problem
I have identified a disk buffering bottleneck using sar. How do I increase disk buffering for SCO OpenServer 5?
Cause
sar -b reports buffer cache activity:# sar -b 1 5SCO_SV tiger5 3.2v5.0.6 Pentium 03/08/200111:26:50 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s (-b)
11:26:51 145 145 0 0 0 0 0 0
11:26:52 134 134 0 0 0 0 0 0
11:26:53 130 522 75 121 196 38 0 0
11:26:54 96 96 0 0 0 0 0 0
11:26:55 106 106 0 0 0 0 0 0Average 122 200 39 24 39 38 0 0The columns show the numbers of actual disk read/write
operations (cache misses), and the number of buffer cache
read/write operations (cache hits). Each operation is a
1 Kbyte transfer. The columns also show read/write hit
ratios, and the number of raw I/O requests.The most useful statistic is the read hit ratio (%rcache column),
which shows you the effectiveness of the buffer cache. On a typical
system you should expect a hit ratio of 85% OR HIGHER. If this
degrades significantly, try to increase the size of the buffer
cache.You would probably need to increase buffer cache if %wcache is
less than circa 65%, even if %rcache is 100%, as long as the
system is not paging or swapping. You can see this with the
"sar -p" command.Alternatively, you could get a faster drive to increase the
throughput onto the disk or reorganize your application so
that several spindles can be utilized. See sar(ADM).The disk load caused by the actual disk read/write operations can be investigated with "sar -d":
# sar -d 1 5SCO_SV tiger5 3.2v5.0.6 Pentium 03/08/200111:27:26 device %busy avque r+w/s blks/s avwait avserv (-d)
11:27:27 Sdsk-0 100.00 1.43 100.00 1797.03 8.91 20.79
11:27:28 Sdsk-0 100.00 1.38 113.86 2208.91 6.70 17.83
11:27:29 Sdsk-0 100.00 1.44 112.87 2168.32 7.46 16.93
11:27:30 Sdsk-0 100.00 1.34 109.90 2417.82 7.39 21.71
11:27:31 Sdsk-0 100.00 1.52 121.57 2741.18 10.56 20.32Average Sdsk-0 100.00 1.42 111.66 2267.59 8.23 19.49To simulate this load, NBUF value was set deliberately low, to
"200". Then the following commands were run to demonstrate
some heavy test load on the server:dd if=/dev/zero of=/tmp/testfile bs=1024 count=500000 &
dd if=/dev/zero of=/tmp/testfile1 bs=1024 count=500000 &This generated 2 x 500MB files in /tmp.
Solution
Prior to SCO OpenServer 5.0.6, the automatic buffer cache setting would set aside 10% of memory to disk buffers and cap at about 6.3MB (6652 1k buffers).
For example, SCO OpenServer 5.0.5 would allocate the following I/O buffers for 64MB of RAM:
mem: total = 65148k, kernel = 11188k, user = 53960k swapdev = 1/41, swplo = 0, nswap = 300000, swapmem = 150000k rootdev = 1/42, pipedev = 1/42, dumpdev = 1/41 kernel: Hz = 100, i/o bufs = 6300k
The i/o bufs are 10% of memory.
For OpenServer 5.0.6 this cap has been removed and will scale up to a new limit of 450MB.
However, in order to achieve better results with your buffer cache,
you may wish to increase the amount of memory used for the buffer
cache. In this example, 25% of memory is allocated:mem: total = 65148k, kernel = 23240k, user = 41908k swapdev = 1/41, swplo = 0, nswap = 300000, swapmem = 150000k Autoboot from rootdev = 1/42, pipedev = 1/42, dumpdev = 1/41 kernel: Hz = 100, i/o bufs = 16384k (high bufs = 11472k)
The i/o bufs are now 25% of memory.
These disk buffering caches can be increased manually.
Changing Kernel parameters to anything above this may improve the performance of the disk buffering, but may direct valuable memory from an important resource to disk buffering. Any changes to kernel parameters should be undertaken with caution.
To change these values, use the following command:
/etc/conf/cf.d/configureSelect 1 - Buffer Management Enter the figure for NBUF, when prompted Enter the figure for NHBUF, when prompted Enter RETURN to accept the current values for the remaining parameters OR you can simply enter "q" to skip the remaining parameters.
If NHBUF is not a power of 2 then this will be highlighted when you attempt to "q" from the main menu and answer "y" to update the system configuration files with your changes.
To adjust the value of NBUF so that it takes up 25% of memory, just take .25 times the amount of memory on your system, expressed in MB, and multiply by 1024 to get the value of NBUF, as:
RAM in MB x (1024*1024) bytes/1MB x (1 disk buffer/1024 bytes) x 25%
Example: NBUF <25% of memory> eg, 64MB = ((64 x 1024) x 0.25) = 16384NBUF has a companion parameter, NHBUF. This is best left at 0,
telling the kernel to choose an appropriate value based on NBUF.If you do change NHBUF remember that it must be an exact power of 2.
The algorithms for the various versions of SCO OpenServer, detailing what value NHBUF should be set to, are:
For OpenServer 5.0.0, 5.0.2, 5.0.4, and 5.0.5:
NHBUF <HALF NBUF (to the next power of 2)
for Single Processor Servers>
<Next exact power of 2 larger than NBUF
for Multi Processor Servers>For OpenServer 5.0.6/5.0.7:The autotune algorithm for NHBUF changed in OpenServer 5.0.6/5.0.7 so that Single and MultiProcessor servers use the same values:
NHBUF <Next exact power of 2 larger than NBUF>Typical settings from the field are:MEM NBUF ONE CPU NHBUF MULTI-CPU NHBUF
64MB 16384 8192 n/a - leave at auto values
128MB 32768 16384 - as above
256MB 65536 32768 - as above
512MB 131072 65536 - as above
1GB 262144 131072 - as above
2GB 450000 262144 - as above
3GB 450000 262144 - as above
4GB 450000 262144 - as aboveNow relink the kernel with this command:
/etc/conf/cf.d/link_unixReboot your server for the changes to take effect.
NOTE: 25% is just an example. The actual needs of individual systems
vary. Also, it is not possible to assign 25% of the largest
systems' RAM to buffers since the upper limit on NBUF is 450000,
while OpenServer 5.0.4 (and later) support up to 4GB of RAM.By default, "sar" is not enabled on SCO OpenServer. To enable it, use this command:
/usr/lib/sa/sar_enable -y See also
ta103492, "Tuning large systems for performance."ta106129, "Suggestions for improving UNIX disk performance."
ta107228, "Large document for tuning SCO UNIX System and SCO Open Desktop kernels."
ta110628, "Large document for tuning SCO UNIX System and SCO Open Desktop kernels - cont'd."
The Online Performance Guide at:http://your_server:457/PERFORM/CONTENTS.htmlThis web site provides UNIX performance tuning tools:
http://www.sarcheck.com