Technical article 112921

I want to improve the performance of a vxfs filesystem.

OpenServer 6.0.0UnixWare 7.1.1UnixWare 7.1.2UnixWare 7.1.3UnixWare 7.1.4Open UNIX 8

Written · Last revised

Problem

I want to improve the performance of a vxfs filesystem.

Solution

Coverage of improving performance is covered at:

http://uw714doc.xinuos.com/en/HANDBOOK/fsD.vxfsopt.html
and in more detail at:
http://osr600doc.xinuos.com/en/ODM_FSadmin/fssag-8.html

More information on these options are available from the command:

man mount_vxfs

General speed testing, other than installing and testing your application, would be:

dd if=/dev/zero of=/<filesystem>/tmp/testfile bs=1024k count=5000
(creates a 5GB file)

This will monitor the timings of any differences set; use "sar -d" to monitor disk activity of each logical disk.

Mounting:
--------
Below are some examples of how to add options to the mounts:
For UW7:

- Adding an entry to the mount options in /etc/vfstab, for example:

From:
/dev/filesys   /dev/rfilesys   /   vxfs   1    no
-      SYS_RANGE_MAX
To:
/dev/filesys   /dev/rfilesys   /   vxfs   1    no
delaylog=nolog,mincache=closesync      SYS_RANGE_MAX
For OSR6:
Use the mntopts= line in /etc/vfstab (ie. /etc/default/filesys)
Defragmentation:
---------------
Once in production, you may wish to look at the disk fragmentation
options available with the "fsadm" command, eg.:
Extents Fragmentation:
fsadm -p 1 -E -e -s -v /<filesystem>
Directory Fragmentation:
fsadm -p 1 -D -d -s -v /<filesystem>
For more information see:
http://uw714doc.xinuos.com/en/man/html.1M/fsadm_vxfs.1M.html

You can run this on any 'vxfs' filesystem, including root '/', in production but may take considerable time so it is recommended to run out of hours, if possible.

Should you see the error:

"vxfs: fsadm: could not get file system reorg back operation not applicable"

Then this particular 'fsadm' is not enabled in the Maintenance and you must have an appropriate Veritas license running on your server.

For UW7, you must have an 'Enterprise' or 'Data Centre' license in place OR have a Veritas license added to your 'Base', 'Business' or 'Departmental' license.

As a general rule, if your application is CPU/Memory bound then use a Data Centre or Enterprise license but if your application is IO bound then use a lesser license but appropriate to the memory you have.

See: TA117262, "How do I enable the Advanced VxFS Feature Set within UnixWare 7 or Open UNIX 8?"

and:

For OpenServer6, you will need to have a Veritas license added to your Enterprise server.

TA127233, "On SCO OpenServer 6 how can I access the advanced Veritas Filesystem (VxVM) features? "

However, use just the -E and -D options to determine whether the file system actually needs defragmentation and afterwards repeat the -E and -D options to see whether the defragmentation has actually had any affect, ie. it is recommended to run the command twice.

Take care not to interrupt the process once it has begun as this may
result in a corrupted filesystem.  It is recommended to ensure the
filesystem is backed up prior to running these commands.

In general, the cleanest way to handle defragmentation is to back the data up, remove the filesystem, re-add it and put the data back.

This is the only option for operating systems like SCO OpenServer5.

vxtunefs:
--------
If you want to improve the performance of writing files then you can
use the vxtunefs command.
# vxtunefs -p /filesystem
See:  http://docsrv.sco.com/cgi-bin/man/man?vxtunefs+1M

Again, the licensing requirements are required, as above, otherwise you will see:

vxfs vxtunefs: Cannot get parameters for /<filesystem>

However, this will enable you to view and change some of the parameters but not set them all:

For example:
# vxtunefs -s -o max_direct_iosz=1048586 /
vxfs vxtunefs: Parameters successfully set for /
or:
# vxtunefs -s -o read_nstream=2 /
vxfs vxtunefs: Parameters successfully set for /
but:
# vxtunefs -s -oqio_cache_enable=1 /
vxfs vxtunefs: failed to set parameters for /

Please note that after a reboot these values will be reset to their original values.

Even if you optimally tune your vxfs filesystem, you might still
encounter performance problem when restoring from tape a large
number of small files.  This overhead is only due to the vxfs
design.  For instance, while using cpio to restore around 300,000
files with size of 4-14k, it has been reported that a drop in
performance was experienced from a 1-2M transfer rate (tape
specification) to a 500K transfer rate.
discovered_direct_iosz IO (Active by default with Enterprise License)
By default:
# timex dd if=/dev/zero of=/data1/testfile bs=1048576 count=1000
1000+0 records in
1000+0 records out
real     1:04.75
user        0.00
sys         1.15
# vxtunefs -o discovered_direct_iosz=1048576 /data1
vxfs vxtunefs: Parameters successfully set for /data1
# vxtunefs -p /data1
Filesystem i/o parameters for /data1
read_pref_io = 65536
read_nstream = 1
read_unit_io = 65536
write_pref_io = 65536
write_nstream = 1
write_unit_io = 65536
pref_strength = 10
buf_breakup_size = 131072
discovered_direct_iosz = 1048576  <-- New value
max_direct_iosz = 1048576
default_indir_size = 8192
qio_cache_enable = 0
max_diskq = 1048576
initial_extent_size = 4
max_seqio_extent_size = 2048
# timex dd if=/dev/zero of=testfile bs=1048576 count=1000
1000+0 records in
1000+0 records out
real       37.75
user        0.00
sys         0.06
See: http://uw714doc.xinuos.com/en/man/html.1M/vxtunefs.1M.html for more
information.
RAID Configuration Options:
--------------------------
In order to achieve the best performance, it's helpful to look at
the type of disk configurations available:
No Raid - Disks independent, no resilience

Raid 0 - Fast read and write speeds. 1 disk min, max depends on controller. No Fault Tolerance. Use: Non-critical data requiring high speed and low cost.

Raid 1 - Mirrored disks.
         Medium speed on read and writes.
         2 disks required min.
         Fault Tolerance.
         Hot Spare available.
         Use: High Fault Tolerance

Raid 5 - Resilience but slow as parity spread across all the disks. Fast Read, very slow writes 3 disk min, max depends on controller. Fault Tolerance. Hot spare available. Use: General purpose data, database and applications.

Raid 5+1 - An entire Raid 5 configuration mirrored. 6 disks min, max depends on controller. Fault Tolerant. Excellent resilience. Rarely used. Use: Very high fault tolerance.

DISK1 + DISK2 + DISK3  DISK4 + DISK5 + DISK6
-Raid 5 Strip+Parity-  -Raid 5 Strip+Parity-
-------------- Raid 1 Mirroring ------------

Raid 0+1 - Raid 0 configuration but each disk is mirrored. 2 disks min, max depends on controller. High performance and reliability Hot Spare available. Costly, because if you wanted a 100GB Logical Drive made from 10GB disks you would need 10 disks for the 100GB, and then a further 10 disks for the mirror.

eg.:
DISK1+DISK2+DISK3  DISK4+DISK5+DISK6
-Raid 0 Striping   -Raid 0 Striping
----------- Raid 1 Mirroring -------

Raid 1+0 - As above, but Raid 1 mirrored then striped.

eg:
DISK1+DISK2+DISK3  DISK4+DISK5+DISK6
-Raid 1 Mirroring  -Raid 1 Mirroring
----------- Raid 0 Striping --------

Bus Speed - There are three components: the controller card, PCI bus inside the server, and the driver for the controller. All three need to be 64 bit to get true 64 bit, otherwise there will be a 32 bit bottleneck.

Controller - If the controller has memory with a battery backup
             then "Write Back" can be used to use the full benefit
             of the on-board cache of the controller. The other
             option is the "Write Thru" option which is slower
             but more reliable.

Please note: If the database used is transaction based, then data can be written to the disk from the cache during a boot up if the power had gone down earlier. If the database is not fully opened then the transaction logs can get confused.

You may want to have separate controllers for separate filesystem activities.

You can also gain performance by ensuring that if the controller is multi-channel, each logical drive uses physical disks on the same channel.

It is recommended to have a separate controller for
tape(s) and CDROM(s), rather than use the same controller
as the disks.  It is also recommended to not have too
many SCSI devices on the same bus.
Sizes  -  You can set the Stripe Size on the controller and the
        Block Size of the filesystem to be the same to optimize
        performance for large file processing with large
        databases, such as Oracle, especially if these databases
        are on a separate controller. This would mean increasing
        the blocksize of the filesystem, when it is created, to
        the same size as the Stripe Size.
The larger the block size is, then each file created will
consume that amount in the inode table.  For example, a
block size of 8k will consume 8k in the inode table for
each file created.  A typical example of this would be an
Oracle database with a 8k block size setting within Oracle's
configuration.  You would make Oracle and the filesystem's
block size consistent.

For some servers you may want to lower the Stripe Size if lots of small reads and writes are taking place so the controller is not waiting for its cache to be filled each time. However, the more read/writes there are, the more times an interrupt will be created for the action.

The norm though is to use the default RAID configuration setting for the RAID Stripe Size and the default filesystem block size for UnixWare 7, which is 1024.

Disks  -  In general, the more disks, the better performance
        as more spindles/heads are being used. The faster
        the disk revolution, the better.  You should also
        check the cache provided by the disk from the
        disk's manufacturer.
Filesystems - Assign individual filesystems to separate tasks to
        reduce activity to the inode tables.  In general,
        accept the number of inodes recommended by UnixWare
        7 when adding the filesystem, even when you are expecting
        there to be a large number of small files written there.
        In this case, you would normally expect to double the number
        of recommended inodes.

However, the "diskadd" command will actually give you an inode value of unlimited.

To check these settings, run this command on the filesystem (slice) in question:

mkfs -m /dev/rdsk/cXbXtXdXsX
Please note: During the Initial System Load (ISL), UnixWare2
             & UnixWare 7.0.x will provide an Advanced option
             during the creation of the filesystems, in the
             Customise Filesystems option.  By default, these
             "vxfs" filesystems will have a 64K inode limit but
             this can be changed to "Unlimited" in this
             Advanced option.

In the case of SCO Unix, the best compromise for resilience and performance is to use Raid 1 for the root filesystem on one Logical Disk.

For each filesystem required, have a Logical Drive consisting of
the necessary Raid 0+1 disks required for the size.  If the number
of disks is a factor, which is the norm, then have a reduced number
of Logical Drives splitting the filesystems between them.

For any server, this value may need to be altered to find the best optimization, if the server is doing a combination of database and i/o work.

Also, before the server goes into production, remove disk(s) from the RAID configurations to ensure that the RAID controller detects that a disk has failed and that when it is re-inserted the RAID is successfully rebuilt.

See also

ta110169, "UnixWare 7, How to enable large files support on UnixWare 7."

For the latest supported Host Bus Adapter (HBA) drivers:

https://ftp.scoexpert.com/unixware7/drivers/storage

For the latest patch revisions for your server including the latest fixes for the HBAs:

https://ftp.scoexpert.com/unixware7

Check the hardware manufacturer documentation of the controller(s) and disk(s) for more detailed information on configuration and performance.

http://docsrv.sco.com:80/search97en_US.ISO8859-1/ODM_FSadmin/fssag-8.html

From the Xinuos technical article database, record 112921. Reproduced with permission. The original is at www.sco.com/ta/112921, which is reachable but blocked from search engines.

Next step

Need this fixed rather than explained?

Tell us the release, what the system runs and what it is doing. We will reply by email with what we would do next.

  • You deal with the people who do the work
  • No obligation, and no upgrade required before we will help
  • We tell you plainly if something is not worth doing

Send an enquiry

We reply by email. If a server is down, say so in the first line. You can also write to help@scoexpert.com.