Recovery - Avaya S8700 Maintenance Manual

Recovery

Recovery
Watchdog and Applications
The Watchdog monitors the sanity of the various applications it initially created. It
does this using two mechanisms. The Watchdog receives:
A SIGCHLD signal if any process it created dies
Periodic heartbeat messages from those processes
If heartbeat messages go away for a certain time (as specified in the Watchdog's
configuration file), the application is killed. When an application terminates (either
unintentionally died or intentionally killed), Watchdog runs an application-specific
"recovery" script that:
May try to kill every process in the application
Checks for and corrects any resource problems
Tries to recreate the application
The Watchdog tries to recreate the application a specified number of times. If
unsuccessful after that number of tries within the specified retry interval, the
Watchdog runs the application's "total failure" script.
For MultiVantage, the recovery script kills every MultiVantage process. Its total
failure script kills off the MultiVantage processes and causes a Linux reboot.
Watchdog and Linux
The Watchdog also monitors several Linux services/daemons. Since the Linux
"init" process originally started these processes, Watchdog can't use the
SIGCHLD signal to monitor these processes. Instead, Watchdog uses a thread to
periodically check the validity of the process identifier for each monitored
processes. If invalid, the Watchdog calls a Linux script to stop and then restart the
particular service. The Linux services monitored by Watchdog are:
atd – at daemon (runs programs at specific times)
crond – cron daemon (runs programs periodically)
dbgserv – provides debugging services
httpd – Apache hypertext transfer protocol server
inetd – Internet server daemon (provides telnet/rlogin/etc. connectivity)
klogd – Linux kernel log daemon (manages logging from Linux
kernel/drivers)
prune – monitors and cleans up partitions
555-233-143 Issue 1 May 2002 3-5