Problem
I have an SCO OpenServer system that is experiencing one of the two following problems:
1. The system panics but does not save a valid dump, either due to insufficient swap/dump space or some other source corruption.
2. The system hangs and does not provide any information on the cause of the hang.
Please Note: If the system is producing a valid dump, you won't need to follow the instructions below; instead see TA105619, "Panic/Crash Analysis if a dump is available".
Cause
Although there can be several reasons why a dump is not saved, often it is due to insufficient swap space. Increasing swap for the purpose of obtaining a valid system dump, however depends on additional disk space availability and usually, re-dividing the root disk partition into differently-sized "divvy(ADM)" divisions.
***********************IMPORTANT********************************The information that follows refers to creating more space to hold a system dump.
You _MUST_ have a FULL SYSTEM BACKUP (using either the backup utility provided with SCO OpenServer or a third party backup program with appropriate emergency boot/root floppies) before attempting the following.
To create emergency boot/root floppies, see the man page "mkdev(ADM)" (the "fd" section of the man page refers to creating the emergency floppies). For information on creating a backup using the Server backup utility, see the online "Help" documentation, accessed through the GUI, "System Administration Guide, Chapter 3, Backing up filesystems" or the SCO System Administration Guide Book.
On most systems, swap and dump are the same device, but this is not
necessarily the case. To determine if swap and dump are the same
device for your system, do the following:# cd /etc/conf/cf.d <Enter>View the "sassign" file.Below is an example of the sassign file where the swap and dump devices are the same:
swap hd 41
dump hd 41The value "41" may be different on your system, however it is the
default value. If both swap and dump have the same number, they
are the same device. If the numbers are different, they are
different devices.The appropriate action to take to obtain a larger dump device depends on what situation applies to you:
1. If the swap device is the dump device and you want to increase
its size, you must use the divvy (ADM) utility to alter the
division layout of the drive on which the swap area is usually
located. Since this is usually the root drive and the swap area
is usually located physically on the drive before the root
(and perhaps other) filesystem, the procedure involves
overwriting root, and involves the same steps as an emergency
restore procedure from a set of up-to-date backup tapes.See TA104767, "Recover OpenServer 5 root filesystem after crash if backups, boot/root exist" in order to perform this procedure; when running divvy(ADM) to create the disk layout (under section II of the aforementioned Technical Article) you can modify the divisions in order to accommodate a larger swap area.
2. If you have an additional drive on which to allocate a separate
dump device, you must initialize that drive and alter the default
bootstring to recognize the new dump device. See TA105920,
"In OpenServer 5.0.x, I don't have enough swap space to complete
a panic dump" for the detailed procedure.***************************************************************When a system hangs, there may be no indication of the reason for
the hang and getting information from the system at this point may
be extremely difficult or impossible. However, in some cases, it
may be possible to use the SCO Kernel Debugger, scodb, to
troubleshoot the problem. The information below describes how to
configure and use scodb.As always, for system hangs or panics, refer to the following logfiles for information that may be relevant to the problems a system is experiencing:
# /usr/adm/messages
# /usr/adm/syslog Solution
The SCO Kernel Debugger, scodb, may be an extremely valuable tool for analyzing panics and hangs on your system. The primary features of scodb, when used in these situations, are:
1. When in scodb debug mode, kernel execution is suspended,
allowing the system administrator to observe snapshots of
important system information, such as the system stack
trace of the currently active process. You can manually
enter debug mode at any time by typing "<ctrl> x" on the
console keyboard on a 5.0.4 or 5.0.5 system, or by typing
"<cntl-alt> d" on a 5.0.6 system, after configuring scodb
on the system.2. After scodb is configured on the system, debug mode is automatically entered when a panic occurs on the system,
3. When used in combination with a serial console, scodb enables the System Administrator to troubleshoot software-related system hangs.
When OpenServer 5.0.x is installed, scodb is also installed, but it is not active until it is linked into the kernel.
To link scodb into the kernel:Change directories to:# /etc/conf/sdevice.dEdit the following file with vi or your favorite text editor:
scodbOnce the scodb file is open it should look like this:
scodb N 1 0 0 0 0 0 0 0To allow scodb to be linked into the kernel, change the "N" to a "Y" so that it looks like this:
scodb Y 1 0 0 0 0 0 0 0Once the file has been changed and saved you can relink the kernel by typing:
# /etc/conf/cf.d/link_unix -yPlease Note: It is necessary to relink the kernel twice, as the kernel symbol table has been adjusted during the first relink.
You may get the following message after this first relink:
db_symtable in unix would only fit 200000 out of 216984 bytes
Corrected table sizes in /var/opt/K/SCO/link/1.1.1Eb/etc/conf/pack.d/scodb/tune.h ready for next re-link
This is normal; after relinking the second time you will not receive the error.
After the kernel is relinked, restart the system.
*********************** IMPORTANT ****************************The X Desktop should not be the active console multiscreen
while running scodb. If the X Desktop is active when scodb is
started by a panic, or if started manually, you will not be
able to get to a character tty session to run scodb. (scodb
will not run in an X Window.)Thus, it may be good practice to not run the X Desktop at all
while scodb is configured in the kernel. For instructions
on how to disable scologin, which controls Desktop startup, see
Technical Article TA109072, "Can the console be the default
screen after bootup instead of scologin?"**************************************************************Manually Starting the SCO Kernel Debugger (scodb):
To manually enter scodb you will need to be at a character tty. The key sequence to enter scodb on a 5.0.4 or 5.0.5 system is
# <ctrl> xand on a 5.0.6 system,# <cntl-alt> dPlease Note: If you have an application that uses the "<ctrl> x" sequence to run a command on a 5.0.4 or 5.0.5 system, or the "<cntl-alt> d" sequence to run a command on a 5.0.6 system, scodb will start and the application's "<ctrl> x" or "<cntl-alt> d", in effect, will no longer be active while scodb is configured into the kernel.
Reconfiguring the scodb Command Key Sequence:If you need to configure the key sequence of "<ctrl> x" on
OpenServer 5.0.4 or 5.0.5 to another key sequence see Chapter 1 of
the SCODB User's Guide under "Running SCODB" and "Modifying tuneable
parameters" and see the examples that follow. On an OpenServer
5.0.6 system, see the comments at the bottom of the file/etc/conf/pack.d/scodb/tune.hin order to modify the key sequence for entering scodb.
To see the currently configured parameters for scodb, type:
# cd /etc/conf/pack.d/scodb <Enter># bin/modtunes -f tune.h <Enter>The key sequence for entering scodb is next to the "DBKEY" value. The value corresponds to the ASCII table decimal value. For example, "<ctrl> x" is represented by "24" on the ASCII table.
To change the key sequence for entering scodb, type:
# cd /etc/conf/pack.d/scodb <Enter># bin/modtunes -f tune.h DBKEY=26This example will set the key sequence to be "<ctrl> z" (an ASCII value of "26"), instead of "<ctrl> x", to enter scodb manually.
After making the above change the kernel will have to be
relinked for the change to take effect. To relink the kernel,
type:# /etc/conf/cf.d/link_unix -y <Enter>Once the kernel is relinked the system will have to be shut down and restarted.
==============================================================System Panics:When a system panics, scodb should start up automatically and you will see the following prompt:
debug0:1>where "0" is the entry level of the debugger and the "1" is the command number in the history.
Once in scodb there are several commands you can use.
The command that should show where the system panicked is stack (that is, at the debug prompt you would type in "stack" and <Enter> as in the example below):
debug0:1> stack <Enter>The "stack" command does not list the processes that were running; rather, it shows the kernel stack of the active process in a readable format.
For multiprocessor (SMP) computers the next command would be:
debug0:2> eps() <Enter>The above command will give information for each processor on the system.
Please Note: You should write down the information from the "stack"
and eps() commands and have that available for your SCO Technical
Support provider.To exit out of scodb, type:quit <Enter>In some cases you may need to press <Enter> a second time.
Once scodb is exited, the system will attempt to save a system dump image to the dump device. Whether or not this dump finishes will depend on the factors listed in the "Cause" section of this article. Once the dump completes or quits, you will see the "Safe to Power Off" message and the system will need to be restarted.
====================================================================System Hangs:System hangs can be more problematic. Depending on the type
of system hang scodb may not be useful since the keyboard will
be "locked up" and won't respond. If the keyboard is responding,
you can enter the following to manually start scodb:# <ctrl> xPress the <Ctrl> and <x> keys at the same time or the key sequence that you have changed to.
If the keyboard responds this will start scodb and you will see the debug prompt as above.
Using the stack command, gather the information that is displayed, as in the section entitled "System Panics" above.
For system hangs, a dump will not be run when you exit scodb.
If you have enough space to dump, you can do so by entering the following at the scodb debug prompt:
debug0:1> sysdump() <Enter>Please Note: The debug prompt may have different numeric values than in the example above.
Whether or not a dump will complete will depend on several factors such as having enough room to dump to, the state of the system and whether or not it can dump, and so on.
If a system dump is generated, see TA105935, "How do I create customized system dump images on demand?"
To extract the dump from swap, you could use the /etc/sysdump command:
# /etc/sysdump -i /dev/swap -n /unix -fumbo my.minidumpAgain, to exit out of scodb type in "quit" <Enter> <Enter>.
If the keyboard does not respond, there is still a chance that the
hang can be analyzed the next time it occurs by configuring a serial
console to take the place of the default console device. This is
valuable in those cases where the system is hung inside the
operating system, and can be interrupted by a device driver running
at a sufficiently high priority. Since the serial driver runs at the
highest priority level (7) on the system, these types of hangs can
be analyzed by typing <Ctrl><x> (or the remapped key sequence) at
the serial console keyboard when the hang occurs.A serial console is used as if it were the "normal" console. Once
again, use the stack command at the debug prompt to gather
information as in the steps above.For information on how to set up a serial terminal, see TA109287, "How do I set up a serial console on SCO OpenServer 5?"
===================================================================Removing the SCO Kernel Debugger (scodb):To remove or deconfigure the SCO Kernel Debugger, scodb, do the following:
Change directories to:# /etc/conf/sdevice.dEdit the following file with vi or your favorite text editor:
# scodbOnce the scodb file is open it should look like this:
# scodb Y 1 0 0 0 0 0 0 0To remove scodb from the kernel change the "Y" to an "N" so it looks like this:
# scodb N 1 0 0 0 0 0 0 0Once the file has been changed and saved you can relink the kernel by typing:
# /etc/conf/cf.d/link_unix -yAfter the kernel relinks, shut down and restart the system and scodb will no longer be functional on the system.
************************* IMPORTANT ****************************The SCO kernel debugger, scodb, is a powerful tool and should be
used only when necessary. The kernel debugger should not be used
on systems (particularly production systems) unless there is no
other method of finding out information about what is causing
system panics/hangs. If possible scodb should be installed on a
test system so that the user can become familiar with it before
attempting to configure it on a system that contains important,
critical or unrecoverable data.****************************************************************One possible problem of running scodb on a production system is
the use of the "ctrl x" option to start scodb. If an application
that is on the system also uses "<ctrl> x" to save a file, or some
other operation, the use of "<ctrl> x" will start scodb, freezing
the kernel; the system may need to be restarted if exiting out of
scodb restores connectivity or other functionality. See the section
for configuring the "<ctrl> x" option under the "System Hangs"
portion of this technical article for reconfiguring this key
sequence for scodb.This technical article is not meant to be a tutorial for scodb nor is it a comprehensive overview of scodb; rather, it is intended for use on a system that is panicking or hanging and not providing either a dump or information regarding the hang.
If the system is producing valid dumps and the crash utility can be run against the dumps scodb does not have to be used.
Running scodb should have a minimal effect on system performance.
See also
man pages scodb(ADM), crash(ADM), divvy(ADM), mkdev(ADM)SCO System Administration Guide book or the Online Help for the System Administration Guide.
Online Help for scodbTo access the online help, double-click on the "Help" icon
on the SCO OpenServer GUI Desktop, select "Navigate" then
"Search". When the new window opens, click on "Entire
Library" and enter "scodb" in the "Search for" field and
click on the "Search" button.Online Help for configuring scodb: Chapter 1 of the SCODB User's Guide.
See:http://osr507doc.xinuos.com/en/SCODB/CONTENTS.htmlhttp://osr507doc.xinuos.com/en/man/html.ADM/scodb.ADMTA105935, "How do I create customized system dump images on demand?"
TA105411, "Filesystem Repair Utilities for SCO OpenServer 5.0.0, 5.0.2 and 5.0.4."
TA105619, "Panic/Crash Analysis if a dump is available."
TA105840, "Techniques to help identify the failing function of a kernel panic."
TA105920, "In OpenServer 5.0.x, I don't have enough swap space to complete a panic dump."
TA109072, "Can the console be the default screen after bootup instead of scologin?"
TA104767, "Recover OpenServer 5 root filesystem after crash if backups, boot/root exist"
TA109287, "How do I set up a serial console on OpenServer 5?"