Problem
I have a program that is consuming all the memory resource on my machine. My machine is slowing down because of this and seems to hang for a while.
Cause
One possible cause you should check is stopio(S). The stopio(S) call is used by default on Traditional, Improved and High security systems to ensure that a tty is not held open by another process once it has been reallocated. If stopio is called on your controlling tty and your program attempts to write(S) a message to the tty, the write(S) will fail with EBADF and a SIGHUP signal is sent to your process. Your process will then call the signal handler, attempt to write(S) a message to the tty, and the result is a nested signal handler loop. The result of this loop is a process stack that will grow until all the memory on your system (availsmem) is exhausted. When all memory has been exhausted your process will be killed and it will dump core. The memory used by the process will not be released until the dump of the core file is complete; this can take a substantial amount of time. During this time, no other process on the system will be able to fork or allocate memory.
stopio(S) is called by telnetd(ADMN), rlogind(ADMN) and by init(ADM) when an I/O session on a tty is terminated.
Any program that traps SIGHUP and attempts to print a message to the screen as part of the signal handling routine may start to consume memory if the controlling tty on which the program is running is stopio'd.
Solution
This has been resolved in SCO OpenServer 5.0.5.
As a workaround, for OpenServer 5.0.4, you can use the configure(ADM) utility to disable the stopio(S). This can be done by setting SECSTOPIO to 0 in section 8, "Security" of the configure(ADM) menu, and relinking and rebooting your system.
The preferred solution, for OpenServer 5.0.4, would be to rewrite the application to prevent the SIGHUP tight loop described above.