Problem
My process is hanging; how do I find out what the process is doing?
Solution
The following is a walk-through the crash utility to determine the function stack trace and the open files used by a specific process. We will include an example of examining the "scoterm" process. The format will be a generic command, followed by the specific command used to examine the "scoterm" process:
# command (generic)
# command (specific example)(1) Determine the process id (pid) of the process:
# ps -ef | grep <process name>
# ps -ef | grep scotermThis will print a line that contains the process id (pid) of your process.
(2) Run the crash utility. You will come to a ">" prompt:
# crash
dumpfile = /dev/mem, namelist = /unix, outfile = /tmp/writefile
>You can run "proc" to get the full process table:
> proc
PROC TABLE SIZE = 315
SLOT ST PID PPID PGRP UID PRI CPU EVENT NAME FLAGS
0 s 0 0 0 0 0 0 f021d9e4 sched load sys nwak
1 s 1 0 0 0 39 0 e0000000 init load
2 s 2 0 0 0 0 0 f00bb998 vhand load sys nwak
nxec
3 s 3 0 0 0 20 0 f00b28b0 bdflush load sys nwak
nxec
4 s 465 1 465 0 30 0 f00ff404 sh load
5 s 466 1 466 0 30 0 f00ff55c sh load
6 s 245 1 0 0 28 0 f016dcf4 strerr load
7 s 119 1 0 0 26 0 f00e0694 logger load
8 s 208 1 208 0 26 0 f01d3e9a cron load nxec
9 s 237 1 237 0 26 0 f013247c cpd load nxec
10 s 219 1 219 0 26 0 f01d467c lpsched load nxec ntrc
11 s 239 1 239 0 39 0 e0000000 slink load nxec
...
52 s 4888 4887 468 0 28 0 f0170f94 rlogin load nxec
53 s 2708 469 469 0 26 0 f013247c telnet load
54 s 4789 854 854 0 26 0 f013247c scoterm load
55 s 4791 4789 4791 0 28 0 f00fc930 rlogin load pollNotice how the first field shows the process slot (pslot) and the third field shows the process id (pid).
(3) Determine the process slot (pslot) of the process by doing a pattern search of the entry with the appropriate process id (pid):
> proc ! grep <pid> > proc ! grep 4789
54 s 4789 854 854 0 26 0 f013247c scoterm loadIn this line, the pslot value is 54.
(4) List the stack trace of the process:
> trace <pslot>
> trace 54
STACK TRACE FOR PROCESS 54:
STKADDR FRAMEPTR FUNCTION POSSIBLE ARGUMENTS
e0000d18 e0000d3c swtch (f013247c,1a,24,f00f22b0)
e0000d44 e0000ddc select (e0000e34,f00f1ecc,e0000e34,7ffff544)
e0000de4 e0000df4 cxenix (0,20,7,2)
e0000dfc e0000e28 systrap (e0000e34)
e0000e34 sys_call from 00029424
ax: 2428 cx: 20 dx: 5 bx: 7 fl: 202 ds: 1f fs: 0
sp:e0000e64 bp:7ffff558 si: 0 di: 20 err: 2428 es: 1f gs: 0This lists the function stack for the scoterm process. At the moment, the "swtch" function is on the top of the stack, indicating the scoterm process is probably waiting for input from the user.
(5) Determine the files being used by the process. You may want to use the "! more" suffix to scroll the information one screen at a time:
> u -f <pslot> ! more
> u -f 54 ! more
PER PROCESS USER AREA FOR PROCESS 54
USER ID's: uid: 0, gid: 1, real uid: 0, real gid: 1
supplementary gids: 1
PROCESS TIMES: user: 571, sys: 1471, child user: 1, child sys: 12
PROCESS MISC:
command: scoterm, psargs: /usr/bin/X11/scoterm -sb -rw -si -ut -icon
+menu -n srv100c -T srv100a -e rlogi
proc slot: 54, cntrl tty: maj(??) min(??)
start: Thu Jan 9 08:44:50 1997
mem: 152d, type: exec
proc/text lock: none
inode of current directory: 40
OPEN FILES AND POFILE FLAGS:
[ 0]: F#167 [ 1]: F#159 w [ 2]: F#159
[ 3]: F#143 r [ 4]: F#194 r [ 5]: F#181 c r w
[ 6]: F#151 [ 7]: F#15 r w [ 8]: F#189
[11]: F#173
FILE I/O:
u_base: 7fffed64, file offset: 550696, bytes: 0,
segment: sys, cmask: 0022, ulimit: 2097151
file mode(s): read write ndelay
SIGNAL DISPOSITION:
sig# signal oldmask sigmask
1: ignore -
2: ignore -
...
27: ignore -
28: default -
29: default -
ux_uid: 0, ux_gid: 0, ux_mode:
comp: e0000cc3, nextcp: e0000cc7
bsize: 1024, pgproc: 0, qsav: 446544, error: 0
ap: e0001148, u_r: 0, pbsize: 24
pboff: 0, pbdev: 1,40 , rablock: 0, errcnt: 6
dirp: 8, dent.d_ino: 0 dent.d_name: pmp1, pdir: 159
ttyip: - , tsize: 2b, dsize: d9, ssize: 6
arg[0]: 8, arg[1]: 7ffff63c, arg[2]: 7ffff5f8
arg[3]: 7ffff608, arg[4]: 0, arg[5]: 0
ar0: e0000e34, ttyp: 0, ticks: 1b89560
pr_base: 0, pr_size: 0, pr_off: 0, pr_scale: 0
ior: 117, iow: 4, iosw: e4, ioch: a5ab4
sysabort: 0, systrap: 0
entrymask: 00000000 00000000 00000000 00000000
exitmask: 00000000 00000000 00000000 00000000
EXDATA:
ip: 574, tsize: 29dfc, dsize: 15f90, bsize: 5dd0, lsize: 50
magic#: 413, toffset: 260, doffset: 2a05c, loffset: 3ffec
txtorg: 260, datorg: 40005c, entloc: 814, nshlibs: 3
execsz: 186The information to look at is the list of files that are opened by this process ("OPEN FILES AND POFILE FLAGS"). The numbers after "F#" indicate the file slot (fslot) for each open file:
OPEN FILES AND POFILE FLAGS:
[ 0]: F#167 [ 1]: F#159 w [ 2]: F#159
[ 3]: F#143 r [ 4]: F#194 r [ 5]: F#181 c r w
[ 6]: F#151 [ 7]: F#15 r w [ 8]: F#189
[11]: F#173(6) Determine the inode slot (islot) for a chosen file (for example, "F#167"):
> f <fslot>
> f 167
FILE TABLE SIZE = 938
SLOT RCNT I/FL OFFSET FLAGS
167 4 I904 0 readThe inode slot (islot) is 904.(7) Determine the inode number (inumb) represented by the inode slot (islot):
> i -f <islot>
> i -f 904
INODE TABLE SIZE = 938
SLOT MAJ/MIN FS INUMB RCNT LINK UID GID SIZE MODE MNT M/ST RCVD FLAGS
904 1,40 1 154 18 1 0 3 0 c---666 0 - -
FORW BACK AFOR ABCK
- 838 875 777
RMAJ/MIN FSTYPP FSPTR FILOCKS VCODE WCNT FS
4,2 f00dd138 f01cb4c8 0 23 0 S51K LASTR MAP/DOFF FLAGS
ffffffff free
NEXT: -The inode number is 154. We can now determine the actual open file name using the ncheck(ADM) program.
(8) Determine the open file name. First, quit out of the crash(ADM) utility:
> quitNext, run the ncheck command to determine the file name:
# ncheck -i <inumb> /dev/root
# ncheck -i 154 /dev/root
/dev/root: (NOTE: This search may take a while.)/dev/nullOne of the open files that the "scoterm" program is using is "/dev/null". If your program were hanging, you could find out which files were open and use that information to determine if it was failing to open any other necessary files.
NOTE: In SCO OpenServer Release 5.0.0 and 5.0.2, you can bypass step 3 by prefixing a process id with a "#" symbol. This will automatically replace the parameter with the appropriate process slot. For example, you could skip step 3 and go directly to steps 4 and 5 with the following commands:
(4) > trace #<pid>
> trace #4789(5) > u -f #<pid> | more
> u -f #4789 | moreNOTE: TLS076a includes a utility called "lsof" that will automatically list the files that are opened by a process, if given the process id ("lsof -p <pid>"). TLS076a is available from SCO's ftp site (ftp.sco.com).
NOTE: "top" is also a utility available from http://www.sco.com/skunkware which will place the process using the most memory to the top of the queue.
NOTE: "ipcs" will show you the amount of shared memory in use.
See also
crash(ADM), ncheck(ADM), ps(C), grep(C), ipcs(ADM)