Technical article 105935

How do I create customized system dump images on demand?

OpenServer 5.0.4OpenServer 5.0.5OpenServer 5.0.6OpenServer 5.0.7

Written · Last revised

Problem

How can I create system dump images of a live SCO OpenServer
Release 5.0.4 and 5.0.5 system?  How can I reduce a large
panic dump image, omitting portions of memory that are not
useful in a typical panic analysis?

Solution

By default, when the server boots, /etc/bcheckrc is run and this will
run /etc/dumpsave.  If there is a dump in memory then it will be
saved from the default dump device which is /dev/swap.  However, if
the server auto boots then /etc/dumpsave will not prompt you to save
the dump.  Therefore, after an automatic reboot you can run
"dumpsave" to check to see if a dump is available to be saved.
If not then,
The sysdump(ADM) utility, new to SCO OpenServer Release 5.0.4 and
5.0.5, can create system dump images on demand.  sysdump is
described below; the information is organized under these
headings:
I.   What tasks sysdump can perform
II.  Examples of sysdump and corresponding crash(ADM) commands
III. Notes on usage of sysdump and crash
IV.  A sysdump(ADM) man page
I. What tasks sysdump(ADM) can perform
sysdump(ADM) can perform the following tasks:

a) Create a system dump image on the running system, without disrupting the normal operation of the system, and save this image to a file, automatically incorporating within the file the kernel whose symbol table is necessary for examining the image subsequently.

b) Copy a panic dump image to a file while reducing it by excluding large portions of the panic dump which are usually not needed for subsequent analysis with the crash(ADM) utility; such a reduced image is not compatible with crash(ADM) but is usually compact enough to transfer across a WAN, even if the original amount of memory on the system is very large (hundreds of MB).

c) Restore a reduced dump image (as created in b) above) to a file that can be read by crash(ADM).

d) Create a file, based on a relevant kernel, that contains only a table of symbols relevant to that kernel, and can be read by crash(ADM) for analysis on the corresponding memory image.

II. Examples of sysdump and corresponding crash(ADM) commands

a) Copy the current live system memory image to the file "livedump":

/etc/sysdump -i /dev/mem -n /unix -o livedump

unix is automatically included in "livedump", which has not been reduced in any way, and is compatible with the crash utility. The appropriate crash command would be:

/etc/crash -n livedump -d livedump

b) Copy the current live system memory image, excluding user memory pages (-u), free memory pages (-f), multiphysical tape buffers (-m -- not relevant for most analyses), and process page tables (-t), to the file "minidump":

/etc/sysdump -i /dev/mem -n /unix -fumto minidump
In order to examine the dump image with crash, minidump needs to
be "inflated".  See example d) below.

See the sysdump(ADM) man page, included below, for more information on the various sysdump command line options.

c) Copy a panic dump residing on the swap device to a file "minidump", excluding user memory pages (-u), free memory pages (-f), multiphysical tape buffers (-m), and buffer cache pages (-b), to the file "minidump":

/etc/sysdump -i /dev/swap -n /unix -fumbo minidump

This would be a typical way to create a "deflated" system dump image of a large memory system that has experienced a panic (for which it has been concluded) that has not been caused by a problem with the filesystem or block I/O. The -b option can be left out if a filesystem-related panic is suspected, or if no possible causes can be ruled out.

In order to examine the dump image with crash, minidump needs
to be "inflated".   See example d) below.

See the sysdump(ADM) man page, included below, for more information on the various sysdump command line options.

d) Inflate the dump image "minidump", as obtained in examples b) or c) above, into a full-size dump image "maxidump" that can be read by crash(ADM):

/etc/sysdump -i minidump -O maxidump

Note the use of the -O (versus -o in previous examples) to inflate the input dump image.

The crash utility may now be used to analyze the image:

/etc/crash -n maxidump -d maxidump

See the discussion below on "Usage with Crash" for more details.

e) Create a Unix symbols file "unix.syms" from the kernel /unix:

/etc/unixsyms -n /unix -o unix.syms
Please Note: /etc/unixsyms should be a hard link to /etc/sysdump, but
this link was inadvertently omitted in OpenServer 5.0.4.  Remedy
this by creating the link:
ln /etc/sysdump /etc/unixsyms
III. Notes on Usage of sysdump and Crash(ADM)

In the examples b) and c) above, the "deflated" dump images named "minidump" cannot be read by crash(ADM) as a dump file. However, the restored, or "inflated" dump image named "maxidump" (example d) above), created by specifying the deflated image as input (-i) and specifying maxidump as output with the -O option, is compatible with the crash utility. The appropriate command to access the image with crash is:

/etc/crash -n maxidump -d maxidump
which was indicated in example d).  Also, in example a), the
dump file "livedump" is not reduced, and can therefore be
used with crash directly, as indicated in that example.
Please Note: maxidump also contains the symbol table for the kernel
that was originally packaged in minidump, and is also created in
a format that can be read by crash as a namelist.  Therefore,
maxidump is specified as the namelist that corresponds to the
dump file--which is maxidump as well.  Both minidump and maxidump
contain this same set of symbols and are both in a format
readable by crash.  Therefore, we could have also specified:
/etc/crash -n minidump -d maxidump
Please Note: dump files can also be compressed with the compress(C)
command to further reduce their size.  When re-inflating such
dump files, there is no need to uncompress them before specifying
them as input with the -i option to sysdump.  As stated in the
manual page below, sysdump can detect when a file is compressed
and automatically uncompress and re-inflate the dump file.
IV. A sysdump(ADM) Man Page

Below is the manual page for sysdump, which will be included in the next release of SCO OpenServer 5.

sysdump(ADM)
************
____________________________________________________________________________
sysdump -- copy, reduce or inspect a system memory image dump
unixsyms -- make unix symbols file, or report symbol values
Syntax
======
/etc/sysdump -i dumpin [ -n unix ] [ -o|O dumpout ] [-a|fumbthp]
/etc/unixsyms [ -n unix ] [ -o symsfile ] [ Symbols | 0xValues ]
Description
===========

sysdump -- copy, reduce or inspect a system memory image dump.

Whenever possible, sysdump displays a kernel stack backtrace for the input dump, together with a report of what parts have been omitted (these displays are to stderr, since stdout may be used for output). Whenever possible, sysdump uses lseek(S) to skip over blocks of zeroes: so an output regular file has "sparse" blocks which use no disk space. Unless dump-specific options are given, sysdump can be used to copy a file which is not a system dump: to take advantage of this sparse writing.

-i dumpin Specifies the location of the input dump to be copied, reduced or inspected e.g. "-i /dev/swap". The input dump may already be reduced, in either "-o" or "-O" format. "-i /dev/mem" takes the memory image from the running system. "-i -" takes the image from standard input; but is not needed for piping from /usr/bin/compress, since sysdump can detect a compressed file (provided the default algorithm was used, not -H) and uncompress it.

-n unix       Specifies the kernel or symbols file to be used for
              deciphering the input dump, and for attaching symbols
              to the output dump.  If unspecified, matching symbols are
              taken from the dump itself, or from the dumping kernel
              (if that is still available).
-o dumpout    Specifies the location of the output dump if any,
              to be created in deflated format: zeroed pages are
              not substituted for any pages omitted.  This format
              is suitable for long-term storage on tape or disk,
              but cannot be understood by crash(ADM) - unless the
              running kernel has the "dump" driver configured in.
              "o" and "O" options are mutually exclusive.  "-o -"
              gives the dump to standard output e.g. for piping
              to /usr/bin/compress.
-O dumpout    Specifies the location of the output dump if any,
              to be created in inflated format: zeroed pages are
              substituted for any pages omitted.  This format is
              always understood by crash(ADM) - unless such a dump
              would exceed 2Gb in size; but is unsuitable for storage
              except as a regular file, since only then can sparse
              writing avoid wasting space on the zeroed pages.
              "O" and "o" options are mutually exclusive.  "-O -"
              gives the dump to standard output e.g. for piping
              to /usr/bin/compress.
-a            The "asking" option presents a different format of
              report on what parts have been or could be omitted
              from the dump to reduce its size.  If an output file
              has been specified, the "asking" option then prompts
              the user whether to omit each remaining part - of the
              kinds described below.  The "a" option (prompting) and
              "fumbthp" options (no prompting) are mutually exclusive.
-f            Omits unused pages ("freemem") from the output dump.
              These are seldom useful in analyzing a panic dump, and
              a dump can often be made much smaller by omitting them.
-u            Omits user data pages ("usermem") from the output dump.
              These are seldom useful in analyzing a panic dump, and
              a dump can often be made much smaller by omitting them.
-m            Omits multiphysical and tape buffers ("mpbuffs") from the
              output dump.  These are seldom useful in analyzing a panic
              dump, and are often modified by the act of dumping itself:
              a dump can be made somewhat smaller by omitting them.
-b            Omits data blocks of the I/O buffer cache ("buffers")
              from the output dump.  Some may be useful in analyzing a
              panic dump, e.g. indirect or inode blocks when analyzing
              a filesystem panic.  But they often use a lot of space,
              and may be irrelevant to the panic in question.
-t            Omits process page tables ("ptables") from the output
              dump.  Some may be useful in analyzing a panic dump, e.g.
              when analyzing a k_trap type E panic.  But they often use a
              lot of space, and may be irrelevant to the panic in question.
              ptables cannot be omitted unless usermem is also omitted.
-h            Omits buffer cache headers ("bufhdrs") from the output dump.
              These are often useful in analyzing a panic dump.  So this
              option is not recommended, unless the bufhdrs occupy a very
              large amount of space, or prior analysis has proved them to
              be irrelevant to the panic.  bufhdrs cannot be omitted
              unless buffers are also omitted.
-p            Omits page frame data ("pfdata") from the output dump.
              This is often useful in analyzing a panic dump.  So this
              option is not recommended, unless pfdata occupies a very
              large amount of space, or prior analysis has proved it to
              be irrelevant to the panic.  pfdata cannot be omitted
              unless freemem is also omitted.

unixsyms -- make unix symbols file, or report symbol values.

unixsyms creates or updates a COFF-format unix symbols file to match a
specified kernel.  This file may later be attached to a panic dump, to
facilitate later analysis.
-n unix       Specifies the kernel whose symbol table is to be used.
              If unspecified, the running kernel is used if that can be
              found.
-o symsfile   Specifies output symbols file to be created.  The file
              Format according to file(C) is "iAPX 386 COFF demand-paged
              executable not stripped", but contains no executable text
              or data.
Symbols       Symbol names and/or hexadecimal addresses may also be
0xValues      specified on the command line: unixsyms then displays
              the hexadecimal address and symbol+offset for each.
/etc/unixsyms is linked to /etc/sysdump.
NOTE:     The symbolic link for /etc/unixsyms was inadvertently omitted
          in SCO OpenServer Release 5.0.4.  To remedy this, become root
          and run:
ln /etc/sysdump /etc/unixsyms

See also

bcheckrc(ADM), bootstring(HW), compress(C), crash(ADM),
dumpsave(ADM), ldsysdump(ADM), sassign(F)

From the Xinuos technical article database, record 105935. Reproduced with permission. The original is at www.sco.com/ta/105935, which is reachable but blocked from search engines.

Next step

Need this fixed rather than explained?

Tell us the release, what the system runs and what it is doing. We will reply by email with what we would do next.

  • You deal with the people who do the work
  • No obligation, and no upgrade required before we will help
  • We tell you plainly if something is not worth doing

Send an enquiry

We reply by email. If a server is down, say so in the first line. You can also write to help@scoexpert.com.