postfix-users October 2010 archive
Main Archive Page > Month Archives  > postfix-users archives
postfix-users: Re: intermittent hang on "postfix stop"

Re: intermittent hang on "postfix stop"; doesn't return "terminating on signal"

From: Wietse Venema <wietse_at_nospam>
Date: Fri Oct 15 2010 - 14:10:20 GMT
To: Postfix users <postfix-users@postfix.org>

Scott Brown:
> Usually, when the update-postfix.pl script runs, it tells Postfix to shut down
> and we get a logged message that says "postfix/postfix-script: stopping the
> Postfix mail system". Right after that, postfix responds with something like
> "postfix/master[11211]: terminating on signal 15"
>
> However, sometimes (once every day or so), the script runs and we get the first
> message "postfix/postfix-script: stopping the Postfix mail system", but then
> postfix does not respond to it and keeps running for a while, until it sees the
> virtual.domain2.com.db was updated, at which point it logs
> "postfix/trivial-rewrite[16529]: table
> hash:/etc/postfix/usermanaged/virtual.domain2.com(0,lock|fold_fix) has changed
> -- restarting", and then after that, it appears to be hung.

You mean, the master did not terminate trivial-rewrite and
then itself, as it is normally does.

When you say "postfix stop", the postfix-script file sends a SIGTERM
signal to the process ID in $queue_directory/pid/master.pid.

There are a number of possible explanations:

1) The master.pid file is bad - it contains a different PID than
that of the running master daemon, and therefore "postfix stop"
does not send. In that case, Postfix keeps handling mail, but that
is not what you observe, because trivial-rewrite is still running.

2) The master.pid file is good, and the master is stuck
after it receives the SIGTERM signal.

The code in the master's sigterm handler is trivially simple. When
I omit the comments (which contain more text that the code itself):

     1 static void master_sigdeath(int sig)
     2 {
     3 const char *myname = "master_sigdeath";
     4 struct sigaction action;
     5 pid_t pid = getpid();
     6
     7 killme_after(5);
     8
     9 sigemptyset(&action.sa_mask);
    10 action.sa_flags = 0;
    11 action.sa_handler = SIG_IGN;
    12 if (sigaction(SIGTERM, &action, (struct sigaction *) 0) < 0)
    13 msg_fatal("%s: sigaction: %m", myname);
    14 if (kill(-pid, SIGTERM) < 0)
    15 msg_fatal("%s: kill process group: %m", myname);
    16 msg_info("terminating on signal %d", sig);
    17 ...

(The killme_after() function is a workaround for Linux that
sets up a time bomb to terminate the master in 5 seconds).

Line 14 terminates all master's child processes including
trivial-rewrite. Apparently this line did not not execute, so I
wonder where the execution has stopped.

Next time, can you do:

        gdb -p pid-of-master-daemon
        ...garbage...
        (gdb) bt

And report the result.

I would also like to mention that "postfix stop" is a very disruptive
way to update a table. Instead, consider doing a "safe update" of
the Berkeley DB map as described in:

http://www.postfix.org/DATABASE_README.html#safe_db

This updates the file without overwriting it.

        Wietse