Technical article 105414

How do I find out the open files and stack trace used by a hanging process?

OpenServer 5.0.0OpenServer 5.0.2OpenServer 5.0.4OpenServer 5.0.5OpenServer 5.0.6OpenServer 5.0.7

Written · Last revised

Problem

My process is hanging; how do I find out what the process is doing?

Solution

The following is a walk-through the crash utility to determine the function stack trace and the open files used by a specific process. We will include an example of examining the "scoterm" process. The format will be a generic command, followed by the specific command used to examine the "scoterm" process:

# command (generic)
# command (specific example)

(1) Determine the process id (pid) of the process:

# ps -ef | grep <process name>
# ps -ef | grep scoterm

This will print a line that contains the process id (pid) of your process.

(2) Run the crash utility. You will come to a ">" prompt:

# crash
dumpfile = /dev/mem, namelist = /unix, outfile = /tmp/writefile
>

You can run "proc" to get the full process table:

> proc
PROC TABLE SIZE = 315
SLOT ST PID   PPID  PGRP   UID PRI CPU   EVENT     NAME        FLAGS
   0 s     0     0     0     0   0   0 f021d9e4 sched          load sys nwak
   1 s     1     0     0     0  39   0 e0000000 init           load
   2 s     2     0     0     0   0   0 f00bb998 vhand          load sys nwak
nxec
   3 s     3     0     0     0  20   0 f00b28b0 bdflush        load sys nwak
nxec
   4 s   465     1   465     0  30   0 f00ff404 sh             load
   5 s   466     1   466     0  30   0 f00ff55c sh             load
   6 s   245     1     0     0  28   0 f016dcf4 strerr         load
   7 s   119     1     0     0  26   0 f00e0694 logger         load
   8 s   208     1   208     0  26   0 f01d3e9a cron           load nxec
   9 s   237     1   237     0  26   0 f013247c cpd            load nxec
  10 s   219     1   219     0  26   0 f01d467c lpsched        load nxec ntrc
  11 s   239     1   239     0  39   0 e0000000 slink          load nxec
...
  52 s  4888  4887   468     0  28   0 f0170f94 rlogin         load nxec
  53 s  2708   469   469     0  26   0 f013247c telnet         load
  54 s  4789   854   854     0  26   0 f013247c scoterm        load
  55 s  4791  4789  4791     0  28   0 f00fc930 rlogin         load poll

Notice how the first field shows the process slot (pslot) and the third field shows the process id (pid).

(3) Determine the process slot (pslot) of the process by doing a pattern search of the entry with the appropriate process id (pid):

> proc ! grep <pid> > proc ! grep 4789

54 s  4789   854   854     0  26   0 f013247c scoterm        load

In this line, the pslot value is 54.

(4) List the stack trace of the process:

> trace <pslot>
> trace 54
STACK TRACE FOR PROCESS 54:
STKADDR   FRAMEPTR  FUNCTION  POSSIBLE ARGUMENTS
e0000d18  e0000d3c  swtch    (f013247c,1a,24,f00f22b0)
e0000d44  e0000ddc  select   (e0000e34,f00f1ecc,e0000e34,7ffff544)
e0000de4  e0000df4  cxenix   (0,20,7,2)
e0000dfc  e0000e28  systrap  (e0000e34)
          e0000e34  sys_call from 00029424
  ax:    2428 cx:      20 dx:       5 bx:       7 fl:      202 ds:  1f fs:   0
  sp:e0000e64 bp:7ffff558 si:       0 di:      20 err:    2428 es:  1f gs:   0

This lists the function stack for the scoterm process. At the moment, the "swtch" function is on the top of the stack, indicating the scoterm process is probably waiting for input from the user.

(5) Determine the files being used by the process. You may want to use the "! more" suffix to scroll the information one screen at a time:

> u -f <pslot> ! more
> u -f 54 ! more
PER PROCESS USER AREA FOR PROCESS 54
USER ID's:      uid: 0, gid: 1, real uid: 0, real gid: 1
        supplementary gids: 1
PROCESS TIMES:  user: 571, sys: 1471, child user: 1, child sys: 12
PROCESS MISC:
        command: scoterm, psargs: /usr/bin/X11/scoterm -sb -rw -si -ut -icon
+menu -n srv100c -T srv100a -e rlogi
        proc slot: 54, cntrl tty: maj(??) min(??)
        start: Thu Jan  9 08:44:50 1997
        mem: 152d, type: exec
        proc/text lock: none
        inode of current directory: 40
OPEN FILES AND POFILE FLAGS:
            [ 0]: F#167        [ 1]: F#159     w  [ 2]: F#159
            [ 3]: F#143   r    [ 4]: F#194   r    [ 5]: F#181 c r w
            [ 6]: F#151        [ 7]: F#15    r w  [ 8]: F#189
            [11]: F#173
FILE I/O:
        u_base: 7fffed64, file offset: 550696, bytes: 0,
        segment: sys, cmask: 0022, ulimit: 2097151
        file mode(s): read write ndelay
SIGNAL DISPOSITION:
           sig#  signal   oldmask sigmask
              1: ignore      -
              2: ignore      -
   ...
            27: ignore      -
            28: default     -
            29: default     -
           ux_uid: 0, ux_gid: 0, ux_mode:
           comp: e0000cc3, nextcp: e0000cc7
           bsize: 1024, pgproc: 0, qsav: 446544, error: 0
           ap: e0001148, u_r: 0, pbsize: 24
           pboff: 0, pbdev:   1,40 , rablock: 0, errcnt: 6
           dirp: 8, dent.d_ino: 0 dent.d_name: pmp1, pdir: 159
           ttyip:  - , tsize: 2b, dsize: d9, ssize: 6
           arg[0]:        8, arg[1]: 7ffff63c, arg[2]: 7ffff5f8
           arg[3]: 7ffff608, arg[4]:        0, arg[5]:        0
           ar0: e0000e34, ttyp: 0, ticks: 1b89560
           pr_base: 0, pr_size: 0, pr_off: 0, pr_scale: 0
           ior: 117, iow: 4, iosw: e4, ioch: a5ab4
           sysabort: 0, systrap: 0
           entrymask: 00000000 00000000 00000000 00000000
           exitmask:  00000000 00000000 00000000 00000000
EXDATA:
        ip:  574, tsize: 29dfc, dsize: 15f90, bsize: 5dd0, lsize: 50
        magic#: 413, toffset: 260, doffset: 2a05c, loffset: 3ffec
        txtorg: 260, datorg: 40005c, entloc: 814, nshlibs: 3
        execsz: 186

The information to look at is the list of files that are opened by this process ("OPEN FILES AND POFILE FLAGS"). The numbers after "F#" indicate the file slot (fslot) for each open file:

OPEN FILES AND POFILE FLAGS:
            [ 0]: F#167        [ 1]: F#159     w  [ 2]: F#159
            [ 3]: F#143   r    [ 4]: F#194   r    [ 5]: F#181 c r w
            [ 6]: F#151        [ 7]: F#15    r w  [ 8]: F#189
            [11]: F#173

(6) Determine the inode slot (islot) for a chosen file (for example, "F#167"):

> f <fslot>
> f 167
FILE TABLE SIZE = 938
SLOT  RCNT  I/FL   OFFSET  FLAGS
 167     4  I904         0 read
The inode slot (islot) is 904.

(7) Determine the inode number (inumb) represented by the inode slot (islot):

> i -f <islot>
> i -f 904
INODE TABLE SIZE = 938
SLOT MAJ/MIN FS INUMB RCNT LINK  UID   GID    SIZE     MODE  MNT M/ST RCVD FLAGS
 904   1,40   1   154   18    1     0     3        0 c---666   0   -    -
       FORW BACK AFOR ABCK
         -   838  875  777
       RMAJ/MIN  FSTYPP   FSPTR    FILOCKS    VCODE     WCNT  FS
         4,2   f00dd138 f01cb4c8        0       23        0 S51K
  LASTR   MAP/DOFF  FLAGS
ffffffff            free
NEXT:  -

The inode number is 154. We can now determine the actual open file name using the ncheck(ADM) program.

(8) Determine the open file name. First, quit out of the crash(ADM) utility:

> quit

Next, run the ncheck command to determine the file name:

# ncheck -i <inumb> /dev/root
# ncheck -i 154 /dev/root
/dev/root:                  (NOTE: This search may take a while.)
/dev/null

One of the open files that the "scoterm" program is using is "/dev/null". If your program were hanging, you could find out which files were open and use that information to determine if it was failing to open any other necessary files.

NOTE: In SCO OpenServer Release 5.0.0 and 5.0.2, you can bypass step 3 by prefixing a process id with a "#" symbol. This will automatically replace the parameter with the appropriate process slot. For example, you could skip step 3 and go directly to steps 4 and 5 with the following commands:

(4) > trace #<pid>
    > trace #4789
(5) > u -f #<pid> | more
    > u -f #4789 | more

NOTE: TLS076a includes a utility called "lsof" that will automatically list the files that are opened by a process, if given the process id ("lsof -p <pid>"). TLS076a is available from SCO's ftp site (ftp.sco.com).

NOTE: "top" is also a utility available from http://www.sco.com/skunkware which will place the process using the most memory to the top of the queue.

NOTE: "ipcs" will show you the amount of shared memory in use.

See also

crash(ADM), ncheck(ADM), ps(C), grep(C), ipcs(ADM)

From the Xinuos technical article database, record 105414. Reproduced with permission. The original is at www.sco.com/ta/105414, which is reachable but blocked from search engines.

Next step

Need this fixed rather than explained?

Tell us the release, what the system runs and what it is doing. We will reply by email with what we would do next.

  • You deal with the people who do the work
  • No obligation, and no upgrade required before we will help
  • We tell you plainly if something is not worth doing

Send an enquiry

We reply by email. If a server is down, say so in the first line. You can also write to help@scoexpert.com.