From: Wietse Venema Date: Sat, 9 Dec 2006 05:00:00 +0000 (-0500) Subject: postfix-2.4-20061209 X-Git-Tag: v2.4.0-RC1~31 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=32c25f2151e2df8c54d2e7d3230717021ab04556;p=thirdparty%2Fpostfix.git postfix-2.4-20061209 --- diff --git a/postfix/HISTORY b/postfix/HISTORY index bc6e7d1ad..931279f9a 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -10760,7 +10760,7 @@ Apologies for any names omitted. Files: global/mail_params.[hc]. --------- - + 20050415-20050615 As of 20050525, DSN support does not involve new queue file @@ -12814,6 +12814,33 @@ Apologies for any names omitted. response on a cached SMTP/LMTP connection. Report by Brian Kantor. Fix by Victor Duchovni. File: smtp/smtp_reuse.c. +20061106 + + The following is implemented using snapshot 20061019 as the + code base. + + Feature: new retry delivery agent, to avoid the synchronous + defer service client in the queue manager. This code is + co-located with the error(8) server. File: error/error.c. + + Performance: the queue manager could spend too much time + in the synchronous defer service client, causing the watchdog + timer to go off. Where possible, the queue manager now + bounces or defers recipients asynchronously, by routing + them to the error or the retry delivery agent. Code by + Wietse and Patrik Rak. Files: global/recipient_list.c, + *qmgr/qmgr_error.c, *qmgr/qmgr_defer.c, *qmgr/qmgr_entry.c, + *qmgr/qmgr_deliver.c, *qmgr/qmgr_message.c. + + Performance: refined recipient and job grouping, and more + agressive early refill of in-memory recipients to prevent + a worst-case scenario where the queue manager became starved + until after the last batch of slow in-memory recipients of + jumbo multi-recipient mail. Code by Patrik Rak. Files: + global/mail_conf_time.c, qmgr/qmgr_message.c, qmgr/qmgr.c, + qmgr/qmgr.h, qmgr/qmgr_entry.c, qmgr/qmgr_job.c, + qmgr/qmgr_message.c, qmgr/qmgr_transport.c. + 20061113 Bugfix: the Postfix install/upgrade procedure broke with @@ -12837,21 +12864,15 @@ Apologies for any names omitted. 20061123 - Workaround: more agressive early refill of in-memory - recipients to prevent a worst-case scenario where the queue - manager became starved until after the last batch of slow - in-memory recipients of jumbo multi-recipient mail. Files: - qmgr/qmgr_job.c. - Safety: don't read more than 5000 recipients at a time, to avoid spending too much time away from interrupts. File: qmgr/qmgr_message.c. - + 20061201 - Workaround: don't complain in the trivial-rewrite, verify, - proxymap or connection cache client when the server exits - after the client sends its request. We still complain, + Workaround: don't complain with "Error 0" in the trivial-rewrite, + verify, proxymap or connection cache client when the server + exits after the client sends its request. We still complain, however, when the problem persists. Files: global/rewrite_clnt.c, global/resolve_clnt.c, global/verify_clnt.c, global/scache_clnt.c, global/dict_proxy.c. @@ -12878,8 +12899,93 @@ Apologies for any names omitted. Postfix daemons now properly detect write timeout errors on internal connections. Files: util/vbuf.h. + Workaround: some broken SMTP servers reply and hang up in + the middle of DATA. The Postfix SMTP client now stops sending + and tries to receive the server response. This can help to + avoid repeated delivery attempts. Initial implementation + by Wietse, later work by Victor Duchovni. Files: + smtp/smtp_proto.c, smtpstone/smtp-sink.c, util/vstream.c, + plus trivial mods for code thatr calls vstream_fpurge(). + +20061204 + + Compatibility: The Postfix installation/upgrade procedure + no longer sets "unknown_local_recipient_code = 450" in + main.cf. This was a safety net for upgrades from Postfix + 1.x. Four years later is no longer needed. File: + conf/post-install. + + Cleanup: removed vstream_fclose() error warning in the code + that disconnects from a delivery agent. There is no need + to report errors here because they would already be reported + earlier. Files: *qmgr/qmgr_deliver.c. + + Robustness: "kill me after N seconds" feature to ensure + that a daemon process does not get stuck while preparing + for exit after signal arrival. File: util/killme_after.[hc], + util/watchdog.c, master/master_sig.c. + +20061206 + + Robustness: low-cost re-entrancy guard that allows daemons + to safely call msg_fatal() etc. from a signal handler, + without risking memory corruption, or deadlock on Redhat + Linux. This works provided that the signal handler terminates + the process. In that special case we need not guarantee + after-the-fact consistency of the thread that was interrupted. + File: util/msg_output.c. + + Robustness: replace exit() calls by _exit(). File: util/msg.c. + +20061207 + + Workaround: on systems with usable futimes() or equivalent + (Solaris, *BSD, MacOS, but not Linux), always explicitly + set the queue file last modification time stamps while + creating a queue file. With this, Postfix can avoid logging + warnings when the file system clock is ahead of the local + clock. Clock skew can be a problem, because Postfix does + not deliver mail until the local clock catches up with the + queue file's last modification time stamp. File: + global/mail_stream.c. + + Workaround: on systems without usable futimes() or equivalent, + log a warning when the file system clock is more than 100 + seconds behind the local clock. This does not cause mail + delivery problems, but it just looks silly in message + headers. File: global/mail_stream.c. + + On systems without usable futimes() (Linux, and ancient + versions of Solaris, SunOS and *BSD) Postfix will keep using + the slower utime() system call to update queue file time + stamps when the file system clock is off with respect to + the local system clock. + + Compatibility with Postfix < 2.3: undo the change to bounce + instead of defer after pipe-to-command delivery fails with + a signal. File: global/pipe_command.c. + +20061208 + + Workaround: apparently, some mail software removes or hides + "" in the Postfix bounce text, because it + processes the text as if it were HTML. This confuses users. + The bounce template has been updated to remove the < and + >. File: bounce/bounce_templates.c. + + Cleanup: when smtp_generic_maps is turned on, don't parse + MIME structures in the message body. Victor Duchovni. File: + smtp/smtp_proto.c. + Wish list: + Update MILTER_README with Martinec info. + + Make postcat header/body aware so people can grep headers. + + Make postmap header/body aware so people can test multi-line + header checks. + Investigate if clients of single-instance servers such as tlsmgr, verify, can close sockets earlier. @@ -13097,12 +13203,6 @@ Wish list: Low: smtp-source may block when sending large test messages. - Med: make qmgr recipient bounce/defer activity asynchronous - or add a multi-recipient operation that reduces overhead. - One possibility is to pass delivery requests to a retry(8) - delivery agent which is error(8) in disguise, and which - calls defer_append() instead of bounce_append(). - Med: find a way to log the sender address when MAIL FROM is rejected due to lack of disk space. diff --git a/postfix/README_FILES/BACKSCATTER_README b/postfix/README_FILES/BACKSCATTER_README index fcde09c05..e5dc8f9f3 100644 --- a/postfix/README_FILES/BACKSCATTER_README +++ b/postfix/README_FILES/BACKSCATTER_README @@ -21,7 +21,16 @@ WWhhaatt iiss bbaacckkssccaatttteerr mmaaiill?? When a spammer or worm sends mail with forged sender addresses, innocent sites are flooded with undeliverable mail notifications. This is called backscatter -mail, and if your system is flooded then you will find out soon enough. +mail. With Postfix, you know that you're a backscatter victim when your logfile +goes on and on like this: + + Dec 4 04:30:09 hostname postfix/smtpd[58549]: NOQUEUE: reject: + RCPT from xxxxxxx[x.x.x.x]: 550 5.1.1 : + Recipient address rejected: User unknown; from=<> + to= proto=ESMTP helo= + +What you see are lots of "user unknown" errors with "from=<>". These are error +reports from MAILER-DAEMONs elsewhere on the Internet. HHooww ddoo II bblloocckk bbaacckkssccaatttteerr mmaaiill ttoo rraannddoomm rreecciippiieenntt aaddddrreesssseess?? @@ -37,6 +46,9 @@ waste time. # Not needed with Postfix 2.1 and later. smtpd_error_sleep_time = 0 + # Not needed with Postfix 2.4 and later. + unknown_local_recipient_reject_code = 550 + HHooww ddoo II bblloocckk bbaacckkssccaatttteerr mmaaiill ttoo rreeaall rreecciippiieenntt aaddddrreesssseess?? When backscatter mail passes the "unknown recipient" barrier, there still is no diff --git a/postfix/RELEASE_NOTES b/postfix/RELEASE_NOTES index 8cc6422af..98e5f9ad9 100644 --- a/postfix/RELEASE_NOTES +++ b/postfix/RELEASE_NOTES @@ -17,12 +17,58 @@ Incompatibility with Postfix 2.2 and earlier If you upgrade from Postfix 2.2 or earlier, read RELEASE_NOTES-2.3 before proceeding. +Incompatible changes with Postfix snapshot 20061209 +=================================================== + +After upgrading Postfix you MUST execute "postfix reload", otherwise +the queue manager may log a warnings with: + + warning: connect to transport retry: Connection refused + +The upgrade procedure adds a new "retry" service to the master.cf +file. If you make the mistake of copying old Postfix configuration +files over the new files, the queue manager may log warnings with: + + warning: connect to transport retry: Connection refused + +To fix your master.cf file, use "postfix upgrade-configuration" +followed by "postfix reload". + +Small changes were made to the default bounce message templates, +to prevent HTML-aware software from hiding or removing "" +and producing really misleading text. + +Major changes with Postfix snapshot 20061209 +============================================ + +Better interoperability with non-conforming SMTP servers that reply +and disconnect before Postfix has sent the end-of-data. + +Improved worst-case queue manager performance when deferring or +bouncing lots of mail. Instead of talking to the bounce or defer +service directly, this work is now done in the background by the +error or retry service. The retry service is new. + +Improved worst-case queue manager performance when delivering +multi-recipient mail. The queue manager now more agressively reloads +recipients from the queue file as soon as slots become available, +instead of waiting for slow deliveries to complete. + Incompatible changes with Postfix snapshot 20061006 =================================================== The format of SMTP server TLS session cache lookup keys has changed. The lookup key now includes the master.cf service name. +Major changes with Postfix snapshot 20061006 +============================================ + +Individual CISCO PIX bug workarounds are now on/off configurable. +This introduces new parameters: smtp_pix_workarounds (default: +disable_esmtp, delay_dotcrlf) and smtp_pix_workaround_maps (workarounds +indexed by server IP address). The default settings are backwards +compatible. + Incompatible changes with Postfix snapshot 20060806 =================================================== diff --git a/postfix/conf/master.cf b/postfix/conf/master.cf index c37982a8b..77daf91c6 100644 --- a/postfix/conf/master.cf +++ b/postfix/conf/master.cf @@ -35,6 +35,7 @@ relay unix - - n - - smtp # -o smtp_helo_timeout=5 -o smtp_connect_timeout=5 showq unix n - n - - showq error unix - - n - - error +retry unix - - n - - error discard unix - - n - - discard local unix - n n - - local virtual unix - n n - - virtual diff --git a/postfix/conf/post-install b/postfix/conf/post-install index 54b3b93bc..755bb53ee 100644 --- a/postfix/conf/post-install +++ b/postfix/conf/post-install @@ -588,16 +588,16 @@ EOF # would be accepted by a previous Postfix version. # This safety net is also documented in LOCAL_RECIPIENT_README. - unknown_local=unknown_local_recipient_reject_code - has_lrm=`$POSTCONF -c $config_directory -n local_recipient_maps` - has_lrjc=`$POSTCONF -c $config_directory -n $unknown_local` - - if [ -z "$has_lrm" -a -z "$has_lrjc" ] - then - echo SAFETY: editing main.cf, setting $unknown_local=450. - echo See the LOCAL_RECIPIENT_README file for details. - $POSTCONF -c $config_directory -e "$unknown_local = 450" || exit 1 - fi +# unknown_local=unknown_local_recipient_reject_code +# has_lrm=`$POSTCONF -c $config_directory -n local_recipient_maps` +# has_lrjc=`$POSTCONF -c $config_directory -n $unknown_local` +# +# if [ -z "$has_lrm" -a -z "$has_lrjc" ] +# then +# echo SAFETY: editing main.cf, setting $unknown_local=450. +# echo See the LOCAL_RECIPIENT_README file for details. +# $POSTCONF -c $config_directory -e "$unknown_local = 450" || exit 1 +# fi # Add missing proxymap service to master.cf. @@ -659,6 +659,15 @@ tlsmgr unix - - n 1000? 1 tlsmgr EOF } + # Add missing retry service to master.cf. + + grep '^retry.*error' $config_directory/master.cf >/dev/null || { + echo Editing $config_directory/master.cf, adding missing entry for retry service + cat >>$config_directory/master.cf <

When a spammer or worm sends mail with forged sender addresses, innocent sites are flooded with undeliverable mail notifications. -This is called backscatter mail, and if your system is flooded then -you will find out soon enough.

+This is called backscatter mail. With Postfix, you know that you're +a backscatter victim when your logfile goes on and on like this: +

+ +
+
+Dec  4 04:30:09 hostname postfix/smtpd[58549]: NOQUEUE: reject:
+RCPT from xxxxxxx[x.x.x.x]: 550 5.1.1 <yyyyyy@your.domain.here>:
+Recipient address rejected: User unknown; from=<>
+to=<yyyyyy@your.domain.here> proto=ESMTP helo=<zzzzzz>
+
+
+ +

What you see are lots of "user unknown" errors with "from=<>". +These are error reports from MAILER-DAEMONs elsewhere on the Internet. +

How do I block backscatter mail to random recipient addresses?

@@ -77,6 +91,9 @@ stress then it should not waste time.

/etc/postfix/main.cf: # Not needed with Postfix 2.1 and later. smtpd_error_sleep_time = 0 + + # Not needed with Postfix 2.4 and later. + unknown_local_recipient_reject_code = 550 diff --git a/postfix/html/bounce.5.html b/postfix/html/bounce.5.html index 7d5ebac31..e6c23320c 100644 --- a/postfix/html/bounce.5.html +++ b/postfix/html/bounce.5.html @@ -81,7 +81,7 @@ BOUNCE(5) BOUNCE(5) I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below. - For further assistance, please send mail to <postmaster> + For further assistance, please send mail to postmaster. If you do so, please include this problem report. You can delete your own text from the attached returned message. diff --git a/postfix/html/error.8.html b/postfix/html/error.8.html index 5cf408609..849a55641 100644 --- a/postfix/html/error.8.html +++ b/postfix/html/error.8.html @@ -7,7 +7,7 @@ ERROR(8) ERROR(8) NAME - error - Postfix error mail delivery agent + error - Postfix error/retry mail delivery agent SYNOPSIS error [generic Postfix daemon options] @@ -21,19 +21,18 @@ ERROR(8) ERROR(8) 3463-compatible detail code. This program expects to be run from the master(8) process manager. - The error(8) delivery agent bounces all recipients in the - delivery request using the "next-hop" domain or host - information as the reason for non-delivery, updates the - queue file and marks recipients as finished or informs the - queue manager that delivery should be tried again at a - later time. + Depending on the service name in master.cf, error or + retry, the server bounces or defers all recipients in the + delivery request using the "next-hop" information as the + reason for non-delivery. The retry service name is sup- + ported as of Postfix 2.4. - Delivery status reports are sent to the bounce(8), + Delivery status reports are sent to the bounce(8), defer(8) or trace(8) daemon as appropriate. SECURITY The error(8) mailer is not security-sensitive. It does not - talk to the network, and can be run chrooted at fixed low + talk to the network, and can be run chrooted at fixed low privilege. STANDARDS @@ -42,39 +41,39 @@ ERROR(8) ERROR(8) DIAGNOSTICS Problems and transactions are logged to syslogd(8). - Depending on the setting of the notify_classes parameter, - the postmaster is notified of bounces and of other trou- + Depending on the setting of the notify_classes parameter, + the postmaster is notified of bounces and of other trou- ble. CONFIGURATION PARAMETERS Changes to main.cf are picked up automatically as error(8) - processes run for only a limited amount of time. Use the + processes run for only a limited amount of time. Use the command "postfix reload" to speed up a change. - The text below provides only a parameter summary. See + The text below provides only a parameter summary. See postconf(5) for more details including examples. 2bounce_notice_recipient (postmaster) - The recipient of undeliverable mail that cannot be + The recipient of undeliverable mail that cannot be returned to the sender. bounce_notice_recipient (postmaster) - The recipient of postmaster notifications with the + The recipient of postmaster notifications with the message headers of mail that Postfix did not - deliver and of SMTP conversation transcripts of + deliver and of SMTP conversation transcripts of mail that Postfix did not receive. config_directory (see 'postconf -d' output) - The default location of the Postfix main.cf and + The default location of the Postfix main.cf and master.cf configuration files. daemon_timeout (18000s) - How much time a Postfix daemon process may take to - handle a request before it is terminated by a + How much time a Postfix daemon process may take to + handle a request before it is terminated by a built-in watchdog timer. delay_logging_resolution_limit (2) - The maximal number of digits after the decimal + The maximal number of digits after the decimal point when logging sub-second delay values. double_bounce_sender (double-bounce) @@ -86,36 +85,36 @@ ERROR(8) ERROR(8) over an internal communication channel. max_idle (100s) - The maximum amount of time that an idle Postfix - daemon process waits for the next service request + The maximum amount of time that an idle Postfix + daemon process waits for the next service request before exiting. max_use (100) - The maximal number of connection requests before a + The maximal number of connection requests before a Postfix daemon process terminates. notify_classes (resource, software) - The list of error classes that are reported to the + The list of error classes that are reported to the postmaster. process_id (read-only) - The process ID of a Postfix command or daemon + The process ID of a Postfix command or daemon process. process_name (read-only) - The process name of a Postfix command or daemon + The process name of a Postfix command or daemon process. queue_directory (see 'postconf -d' output) - The location of the Postfix top-level queue direc- + The location of the Postfix top-level queue direc- tory. syslog_facility (mail) The syslog facility of Postfix logging. syslog_name (postfix) - The mail system name that is prepended to the - process name in syslog records, so that "smtpd" + The mail system name that is prepended to the + process name in syslog records, so that "smtpd" becomes, for example, "postfix/smtpd". SEE ALSO @@ -128,7 +127,7 @@ ERROR(8) ERROR(8) syslogd(8), system logging LICENSE - The Secure Mailer license must be distributed with this + The Secure Mailer license must be distributed with this software. AUTHOR(S) diff --git a/postfix/html/postconf.5.html b/postfix/html/postconf.5.html index 49ac96961..0468636f9 100644 --- a/postfix/html/postconf.5.html +++ b/postfix/html/postconf.5.html @@ -1848,7 +1848,7 @@ non-address DSN status (e.g., 4.0.0).

default_recipient_limit -(default: 10000)
+(default: 20000)

The default per-transport upper limit on the number of in-memory @@ -1859,6 +1859,34 @@ and qmgr_message_recipi

+
+ +
default_recipient_refill_delay +(default: 5s)
+ +

+The default per-transport maximum delay between recipients refills. +When not all message recipients fit into the memory at once, keep loading +more of them at least once every this many seconds. This is used to +make sure the recipients are refilled in timely manner even when +$default_recipient_refill_limit is too high for too slow deliveries. +

+ + +
+ +
default_recipient_refill_limit +(default: 100)
+ +

+The default per-transport limit on the number of recipients refilled at +once. When not all message recipients fit into the memory at once, keep +loading more of them in batches of at least this many at a time. See also +$default_recipient_refill_delay, which may result in recipient batches +lower than this when this limit is too high for too slow deliveries. +

+ +
default_transport diff --git a/postfix/html/qmgr.8.html b/postfix/html/qmgr.8.html index 9c14ce638..e3c75a45d 100644 --- a/postfix/html/qmgr.8.html +++ b/postfix/html/qmgr.8.html @@ -216,7 +216,7 @@ QMGR(8) QMGR(8) The minimal number of in-memory recipients for any message. - default_recipient_limit (10000) + default_recipient_limit (20000) The default per-transport upper limit on the number of in-memory recipients. @@ -231,6 +231,24 @@ QMGR(8) QMGR(8) ent_limit) Idem, for delivery via the named message transport. + Available in Postfix version 2.4 and later: + + default_recipient_refill_limit (100) + The default per-transport limit on the number of + recipients refilled at once. + + transport_recipient_refill_limit ($default_recipi- + ent_refill_limit) + Idem, for delivery via the named message transport. + + default_recipient_refill_delay (5s) + The default per-transport maximum delay between + recipients refills. + + transport_recipient_refill_delay ($default_recipi- + ent_refill_delay) + Idem, for delivery via the named message transport. + DELIVERY CONCURRENCY CONTROLS initial_destination_concurrency (5) The initial per-destination concurrency level for diff --git a/postfix/html/smtp-sink.1.html b/postfix/html/smtp-sink.1.html index ec907ebdf..92e298c9a 100644 --- a/postfix/html/smtp-sink.1.html +++ b/postfix/html/smtp-sink.1.html @@ -47,6 +47,13 @@ SMTP-SINK(1) SMTP-SINK(1) -a Do not announce SASL authentication support. + -A delay + Wait delay seconds after responding to DATA, then + abort prematurely with a 550 reply status. Do not + read further input from the client; this is an + attempt to block the client before it sends ".". + Specify a zero delay value to abort immediately. + -c Display running counters that are updated whenever an SMTP session ends, a QUIT command is executed, or when "." is received. diff --git a/postfix/makedefs b/postfix/makedefs index 595ef9dc5..e4ed531a2 100644 --- a/postfix/makedefs +++ b/postfix/makedefs @@ -162,9 +162,9 @@ case "$SYSTEM.$RELEASE" in case $RELEASE in 5.[0-7]|5.[0-7].*) CCARGS="$CCARGS -DNO_IPV6";; esac - # Solaris 9 added closefrom() and /dev/*random + # Solaris 9 added closefrom(), futimesat() and /dev/*random case $RELEASE in - 5.[0-8]|5.[0-8].*) CCARGS="$CCARGS -DNO_CLOSEFROM -DNO_DEV_URANDOM";; + 5.[0-8]|5.[0-8].*) CCARGS="$CCARGS -DNO_CLOSEFROM -DNO_DEV_URANDOM -DNO_FUTIMESAT";; esac # Work around broken str*casecmp(). Do it all here instead # of having half the solution in the sys_defs.h file. diff --git a/postfix/man/man1/smtp-sink.1 b/postfix/man/man1/smtp-sink.1 index c14515afd..b03f7ad92 100644 --- a/postfix/man/man1/smtp-sink.1 +++ b/postfix/man/man1/smtp-sink.1 @@ -45,6 +45,12 @@ Postfix is built without IPv6 support. Do not announce 8BITMIME support. .IP \fB-a\fR Do not announce SASL authentication support. +.IP "\fB-A \fIdelay\fR" +Wait \fIdelay\fR seconds after responding to DATA, then +abort prematurely with a 550 reply status. Do not read +further input from the client; this is an attempt to block +the client before it sends ".". Specify a zero delay value +to abort immediately. .IP \fB-c\fR Display running counters that are updated whenever an SMTP session ends, a QUIT command is executed, or when "." is diff --git a/postfix/man/man5/bounce.5 b/postfix/man/man5/bounce.5 index ce3d03735..4c6215ab3 100644 --- a/postfix/man/man5/bounce.5 +++ b/postfix/man/man5/bounce.5 @@ -91,7 +91,7 @@ This is the mail system at host $myhostname. I'm sorry to have to inform you that your message could not be delivered to one or more recipients. It's attached below. -For further assistance, please send mail to +For further assistance, please send mail to postmaster. If you do so, please include this problem report. You can delete your own text from the attached returned message. diff --git a/postfix/man/man5/postconf.5 b/postfix/man/man5/postconf.5 index 688476ed8..4ae39dea1 100644 --- a/postfix/man/man5/postconf.5 +++ b/postfix/man/man5/postconf.5 @@ -1010,12 +1010,24 @@ When rejecting non-address information (such as the HELO command argument or the client hostname/address), the Postfix SMTP server will transform a sender or recipient DSN status into a generic non-address DSN status (e.g., 4.0.0). -.SH default_recipient_limit (default: 10000) +.SH default_recipient_limit (default: 20000) The default per-transport upper limit on the number of in-memory recipients. These limits take priority over the global qmgr_message_recipient_limit after the message has been assigned to the respective transports. See also default_extra_recipient_limit and qmgr_message_recipient_minimum. +.SH default_recipient_refill_delay (default: 5s) +The default per-transport maximum delay between recipients refills. +When not all message recipients fit into the memory at once, keep loading +more of them at least once every this many seconds. This is used to +make sure the recipients are refilled in timely manner even when +$default_recipient_refill_limit is too high for too slow deliveries. +.SH default_recipient_refill_limit (default: 100) +The default per-transport limit on the number of recipients refilled at +once. When not all message recipients fit into the memory at once, keep +loading more of them in batches of at least this many at a time. See also +$default_recipient_refill_delay, which may result in recipient batches +lower than this when this limit is too high for too slow deliveries. .SH default_transport (default: smtp) The default mail delivery transport and next-hop destination for destinations that do not match $mydestination, $inet_interfaces, diff --git a/postfix/man/man8/error.8 b/postfix/man/man8/error.8 index f5a19cf91..854837691 100644 --- a/postfix/man/man8/error.8 +++ b/postfix/man/man8/error.8 @@ -4,7 +4,7 @@ .SH NAME error \- -Postfix error mail delivery agent +Postfix error/retry mail delivery agent .SH "SYNOPSIS" .na .nf @@ -21,11 +21,11 @@ The reason may be prefixed with an RFC 3463-compatible detail code. This program expects to be run from the \fBmaster\fR(8) process manager. -The \fBerror\fR(8) delivery agent bounces all recipients in the delivery -request using the "next-hop" -domain or host information as the reason for non-delivery, updates -the queue file and marks recipients as finished or informs the -queue manager that delivery should be tried again at a later time. +Depending on the service name in master.cf, \fBerror\fR +or \fBretry\fR, the server bounces or defers all recipients +in the delivery request using the "next-hop" information +as the reason for non-delivery. The \fBretry\fR service name is +supported as of Postfix 2.4. Delivery status reports are sent to the \fBbounce\fR(8), \fBdefer\fR(8) or \fBtrace\fR(8) daemon as appropriate. diff --git a/postfix/man/man8/qmgr.8 b/postfix/man/man8/qmgr.8 index bca8945e3..9ad381d9b 100644 --- a/postfix/man/man8/qmgr.8 +++ b/postfix/man/man8/qmgr.8 @@ -201,7 +201,7 @@ queue manager, and the maximal size of the size of the short-term, in-memory "dead" destination status cache. .IP "\fBqmgr_message_recipient_minimum (10)\fR" The minimal number of in-memory recipients for any message. -.IP "\fBdefault_recipient_limit (10000)\fR" +.IP "\fBdefault_recipient_limit (20000)\fR" The default per-transport upper limit on the number of in-memory recipients. .IP "\fItransport\fB_recipient_limit ($default_recipient_limit)\fR" @@ -211,6 +211,17 @@ The default value for the extra per-transport limit imposed on the number of in-memory recipients. .IP "\fItransport\fB_extra_recipient_limit ($default_extra_recipient_limit)\fR" Idem, for delivery via the named message \fItransport\fR. +.PP +Available in Postfix version 2.4 and later: +.IP "\fBdefault_recipient_refill_limit (100)\fR" +The default per-transport limit on the number of recipients refilled at +once. +.IP "\fItransport\fB_recipient_refill_limit ($default_recipient_refill_limit)\fR" +Idem, for delivery via the named message \fItransport\fR. +.IP "\fBdefault_recipient_refill_delay (5s)\fR" +The default per-transport maximum delay between recipients refills. +.IP "\fItransport\fB_recipient_refill_delay ($default_recipient_refill_delay)\fR" +Idem, for delivery via the named message \fItransport\fR. .SH "DELIVERY CONCURRENCY CONTROLS" .na .nf diff --git a/postfix/mantools/postlink b/postfix/mantools/postlink index 4caf4f4c7..10ace1b3b 100755 --- a/postfix/mantools/postlink +++ b/postfix/mantools/postlink @@ -134,6 +134,8 @@ while (<>) { s;\bdefault_privs\b;$&;g; s;\bdefault_process_limit\b;$&;g; s;\bdefault_rbl_reply\b;$&;g; + s;\bdefault_recipient_refill_limit\b;$&;g; + s;\bdefault_recipient_refill_delay\b;$&;g; s;\bdefault_recip[-]*\n* *[]*ient_limit\b;$&;g; s;\bdefault_transport\b;$&;g; s;\bdefault_verp_delimiters\b;$&;g; diff --git a/postfix/proto/BACKSCATTER_README.html b/postfix/proto/BACKSCATTER_README.html index 4c7dd3677..af7659d24 100644 --- a/postfix/proto/BACKSCATTER_README.html +++ b/postfix/proto/BACKSCATTER_README.html @@ -57,8 +57,22 @@ scanners

When a spammer or worm sends mail with forged sender addresses, innocent sites are flooded with undeliverable mail notifications. -This is called backscatter mail, and if your system is flooded then -you will find out soon enough.

+This is called backscatter mail. With Postfix, you know that you're +a backscatter victim when your logfile goes on and on like this: +

+ +
+
+Dec  4 04:30:09 hostname postfix/smtpd[58549]: NOQUEUE: reject:
+RCPT from xxxxxxx[x.x.x.x]: 550 5.1.1 <yyyyyy@your.domain.here>:
+Recipient address rejected: User unknown; from=<>
+to=<yyyyyy@your.domain.here> proto=ESMTP helo=<zzzzzz>
+
+
+ +

What you see are lots of "user unknown" errors with "from=<>". +These are error reports from MAILER-DAEMONs elsewhere on the Internet. +

How do I block backscatter mail to random recipient addresses?

@@ -77,6 +91,9 @@ stress then it should not waste time.

/etc/postfix/main.cf: # Not needed with Postfix 2.1 and later. smtpd_error_sleep_time = 0 + + # Not needed with Postfix 2.4 and later. + unknown_local_recipient_reject_code = 550 diff --git a/postfix/proto/bounce b/postfix/proto/bounce index f770a5e33..842cad9b8 100644 --- a/postfix/proto/bounce +++ b/postfix/proto/bounce @@ -81,7 +81,7 @@ # I'm sorry to have to inform you that your message could not # be delivered to one or more recipients. It's attached below. # -# For further assistance, please send mail to +# For further assistance, please send mail to postmaster. # # If you do so, please include this problem report. You can # delete your own text from the attached returned message. diff --git a/postfix/proto/postconf.proto b/postfix/proto/postconf.proto index 67f86b46f..3d395e3f0 100644 --- a/postfix/proto/postconf.proto +++ b/postfix/proto/postconf.proto @@ -1086,7 +1086,7 @@ The smtpd_expansion_filter configuration parameter controls what characters may appear in $name expansions.

-%PARAM default_recipient_limit 10000 +%PARAM default_recipient_limit 20000

The default per-transport upper limit on the number of in-memory @@ -1096,6 +1096,26 @@ to the respective transports. See also default_extra_recipient_limit and qmgr_message_recipient_minimum.

+%PARAM default_recipient_refill_limit 100 + +

+The default per-transport limit on the number of recipients refilled at +once. When not all message recipients fit into the memory at once, keep +loading more of them in batches of at least this many at a time. See also +$default_recipient_refill_delay, which may result in recipient batches +lower than this when this limit is too high for too slow deliveries. +

+ +%PARAM default_recipient_refill_delay 5s + +

+The default per-transport maximum delay between recipients refills. +When not all message recipients fit into the memory at once, keep loading +more of them at least once every this many seconds. This is used to +make sure the recipients are refilled in timely manner even when +$default_recipient_refill_limit is too high for too slow deliveries. +

+ %PARAM default_transport smtp

diff --git a/postfix/src/bounce/bounce_cleanup.c b/postfix/src/bounce/bounce_cleanup.c index a68850a5b..9fb900bb0 100644 --- a/postfix/src/bounce/bounce_cleanup.c +++ b/postfix/src/bounce/bounce_cleanup.c @@ -129,7 +129,7 @@ static void bounce_cleanup_sig(int sig) */ if (bounce_cleanup_path) (void) unlink(vstring_str(bounce_cleanup_path)); - exit(sig); + _exit(sig); } /* bounce_cleanup_register - register logfile to clean up */ diff --git a/postfix/src/bounce/bounce_templates.c b/postfix/src/bounce/bounce_templates.c index d3e53c3b3..cdea765d2 100644 --- a/postfix/src/bounce/bounce_templates.c +++ b/postfix/src/bounce/bounce_templates.c @@ -98,7 +98,7 @@ static const char *def_bounce_failure_body[] = { "I'm sorry to have to inform you that your message could not", "be delivered to one or more recipients. It's attached below.", "", - "For further assistance, please send mail to <" MAIL_ADDR_POSTMASTER ">", + "For further assistance, please send mail to " MAIL_ADDR_POSTMASTER ".", "", "If you do so, please include this problem report. You can", "delete your own text from the attached returned message.", @@ -134,7 +134,7 @@ static const char *def_bounce_delay_body[] = { , "It will be retried until it is $maximal_queue_lifetime_days day(s) old.", "", - "For further assistance, please send mail to <" MAIL_ADDR_POSTMASTER ">", + "For further assistance, please send mail to " MAIL_ADDR_POSTMASTER ".", "", "If you do so, please include this problem report. You can", "delete your own text from the attached returned message.", diff --git a/postfix/src/cleanup/cleanup_bounce.c b/postfix/src/cleanup/cleanup_bounce.c index bf8f52368..93012f0c2 100644 --- a/postfix/src/cleanup/cleanup_bounce.c +++ b/postfix/src/cleanup/cleanup_bounce.c @@ -135,7 +135,7 @@ int cleanup_bounce(CLEANUP_STATE *state) * stream error flags to avoid false alarms. */ if (state->errs & CLEANUP_STAT_SIZE) { - (void) vstream_fpurge(state->dst); + (void) vstream_fpurge(state->dst, VSTREAM_PURGE_BOTH); vstream_clearerr(state->dst); } if (vstream_fseek(state->dst, 0L, SEEK_SET) < 0) diff --git a/postfix/src/error/Makefile.in b/postfix/src/error/Makefile.in index 84ececa7e..8e1d097c5 100644 --- a/postfix/src/error/Makefile.in +++ b/postfix/src/error/Makefile.in @@ -59,12 +59,15 @@ depend: $(MAKES) # do not edit below this line - it is generated by 'make depend' error.o: ../../include/attr.h error.o: ../../include/bounce.h +error.o: ../../include/defer.h error.o: ../../include/deliver_completed.h error.o: ../../include/deliver_request.h error.o: ../../include/dsn.h error.o: ../../include/dsn_buf.h error.o: ../../include/dsn_util.h error.o: ../../include/flush_clnt.h +error.o: ../../include/iostuff.h +error.o: ../../include/mail_proto.h error.o: ../../include/mail_queue.h error.o: ../../include/mail_server.h error.o: ../../include/msg.h diff --git a/postfix/src/error/error.c b/postfix/src/error/error.c index 2aeb98f03..77ae72ca8 100644 --- a/postfix/src/error/error.c +++ b/postfix/src/error/error.c @@ -2,7 +2,7 @@ /* NAME /* error 8 /* SUMMARY -/* Postfix error mail delivery agent +/* Postfix error/retry mail delivery agent /* SYNOPSIS /* \fBerror\fR [generic Postfix daemon options] /* DESCRIPTION @@ -15,11 +15,11 @@ /* This program expects to be run from the \fBmaster\fR(8) process /* manager. /* -/* The \fBerror\fR(8) delivery agent bounces all recipients in the delivery -/* request using the "next-hop" -/* domain or host information as the reason for non-delivery, updates -/* the queue file and marks recipients as finished or informs the -/* queue manager that delivery should be tried again at a later time. +/* Depending on the service name in master.cf, \fBerror\fR +/* or \fBretry\fR, the server bounces or defers all recipients +/* in the delivery request using the "next-hop" information +/* as the reason for non-delivery. The \fBretry\fR service name is +/* supported as of Postfix 2.4. /* /* Delivery status reports are sent to the \fBbounce\fR(8), /* \fBdefer\fR(8) or \fBtrace\fR(8) daemon as appropriate. @@ -120,10 +120,12 @@ #include #include #include +#include #include #include #include #include +#include /* Single server skeleton. */ @@ -131,7 +133,9 @@ /* deliver_message - deliver message with extreme prejudice */ -static int deliver_message(DELIVER_REQUEST *request) +static int deliver_message(DELIVER_REQUEST *request, const char *def_dsn, + int (*append) (int, const char *, MSG_STATS *, RECIPIENT *, + const char *, DSN *)) { const char *myname = "deliver_message"; VSTREAM *src; @@ -168,17 +172,17 @@ static int deliver_message(DELIVER_REQUEST *request) msg_info("%s: file %s", myname, VSTREAM_PATH(src)); /* - * Bounce all recipients. + * Bounce/defer/whatever all recipients. */ #define BOUNCE_FLAGS(request) DEL_REQ_TRACE_FLAGS(request->flags) - dsn_split(&dp, "5.0.0", request->nexthop); + dsn_split(&dp, def_dsn, request->nexthop); (void) DSN_SIMPLE(&dsn, DSN_STATUS(dp.dsn), dp.text); for (nrcpt = 0; nrcpt < request->rcpt_list.len; nrcpt++) { rcpt = request->rcpt_list.info + nrcpt; if (rcpt->offset >= 0) { - status = bounce_append(BOUNCE_FLAGS(request), request->queue_id, - &request->msg_stats, rcpt, "none", &dsn); + status = append(BOUNCE_FLAGS(request), request->queue_id, + &request->msg_stats, rcpt, "none", &dsn); if (status == 0) deliver_completed(src, rcpt->offset); result |= status; @@ -196,7 +200,7 @@ static int deliver_message(DELIVER_REQUEST *request) /* error_service - perform service for client */ -static void error_service(VSTREAM *client_stream, char *unused_service, char **argv) +static void error_service(VSTREAM *client_stream, char *service, char **argv) { DELIVER_REQUEST *request; int status; @@ -216,7 +220,12 @@ static void error_service(VSTREAM *client_stream, char *unused_service, char **a * in single_server.c. */ if ((request = deliver_request_read(client_stream)) != 0) { - status = deliver_message(request); + if (strcmp(service, MAIL_SERVICE_ERROR) == 0) + status = deliver_message(request, "5.0.0", bounce_append); + else if (strcmp(service, MAIL_SERVICE_RETRY) == 0) + status = deliver_message(request, "4.0.0", defer_append); + else + msg_fatal("bad error service name: %s", service); deliver_request_done(client_stream, request, status); } } diff --git a/postfix/src/global/Makefile.in b/postfix/src/global/Makefile.in index 778e3c5c1..1c2fc1ae3 100644 --- a/postfix/src/global/Makefile.in +++ b/postfix/src/global/Makefile.in @@ -1515,6 +1515,7 @@ recdump.o: rec_streamlf.h recdump.o: rec_type.h recdump.o: recdump.c recdump.o: record.h +recipient_list.o: ../../include/msg.h recipient_list.o: ../../include/mymalloc.h recipient_list.o: ../../include/sys_defs.h recipient_list.o: recipient_list.c diff --git a/postfix/src/global/mail_conf.h b/postfix/src/global/mail_conf.h index 29a2830f1..aa0e5d4aa 100644 --- a/postfix/src/global/mail_conf.h +++ b/postfix/src/global/mail_conf.h @@ -53,7 +53,7 @@ extern char *get_mail_conf_raw(const char *, const char *, int, int); extern int get_mail_conf_int2(const char *, const char *, int, int, int); extern long get_mail_conf_long2(const char *, const char *, long, long, long); -extern int get_mail_conf_time2(const char *, const char *, const char *, int, int); +extern int get_mail_conf_time2(const char *, const char *, int, int, int, int); /* * Lookup with function-call defaults. @@ -73,6 +73,7 @@ extern void set_mail_conf_int(const char *, int); extern void set_mail_conf_long(const char *, long); extern void set_mail_conf_bool(const char *, int); extern void set_mail_conf_time(const char *, const char *); +extern void set_mail_conf_time_int(const char *, int); /* * Tables that allow us to selectively copy values from the global diff --git a/postfix/src/global/mail_conf_time.c b/postfix/src/global/mail_conf_time.c index c90beebce..09926c76f 100644 --- a/postfix/src/global/mail_conf_time.c +++ b/postfix/src/global/mail_conf_time.c @@ -16,12 +16,17 @@ /* const char *name; /* const char *value; /* +/* void set_mail_conf_time_int(name, value) +/* const char *name; +/* int value; +/* /* void get_mail_conf_time_table(table) /* CONFIG_TIME_TABLE *table; /* AUXILIARY FUNCTIONS -/* int get_mail_conf_time2(name1, name2, defval, min, max); +/* int get_mail_conf_time2(name1, name2, defval, def_unit, min, max); /* const char *name1; /* const char *name2; +/* int defval; /* int def_unit; /* int min; /* int max; @@ -146,16 +151,14 @@ int get_mail_conf_time(const char *name, const char *defval, int min, int ma /* get_mail_conf_time2 - evaluate integer-valued configuration variable */ int get_mail_conf_time2(const char *name1, const char *name2, - const char *defval, int min, int max) + int defval, int def_unit, int min, int max) { int intval; char *name; - int def_unit; name = concatenate(name1, name2, (char *) 0); - def_unit = get_def_time_unit(name, defval); if (convert_mail_conf_time(name, &intval, def_unit) == 0) - set_mail_conf_time(name, defval); + set_mail_conf_time_int(name, defval); if (convert_mail_conf_time(name, &intval, def_unit) == 0) msg_panic("get_mail_conf_time2: parameter not found: %s", name); check_mail_conf_time(name, intval, min, max); @@ -170,6 +173,16 @@ void set_mail_conf_time(const char *name, const char *value) mail_conf_update(name, value); } +/* set_mail_conf_time_int - update integer-valued configuration dictionary entry */ + +void set_mail_conf_time_int(const char *name, int value) +{ + char buf[BUFSIZ]; /* yeah! crappy code! */ + + sprintf(buf, "%ds", value); /* yeah! more crappy code! */ + mail_conf_update(name, buf); +} + /* get_mail_conf_time_table - look up table of integers */ void get_mail_conf_time_table(CONFIG_TIME_TABLE *table) diff --git a/postfix/src/global/mail_params.h b/postfix/src/global/mail_params.h index fb77b7fed..34c2ed951 100644 --- a/postfix/src/global/mail_params.h +++ b/postfix/src/global/mail_params.h @@ -675,7 +675,7 @@ extern int var_qmgr_msg_rcpt_limit; #define VAR_XPORT_RCPT_LIMIT "default_recipient_limit" #define _XPORT_RCPT_LIMIT "_recipient_limit" -#define DEF_XPORT_RCPT_LIMIT 10000 +#define DEF_XPORT_RCPT_LIMIT 20000 extern int var_xport_rcpt_limit; #define VAR_STACK_RCPT_LIMIT "default_extra_recipient_limit" @@ -683,6 +683,16 @@ extern int var_xport_rcpt_limit; #define DEF_STACK_RCPT_LIMIT 1000 extern int var_stack_rcpt_limit; +#define VAR_XPORT_REFILL_LIMIT "default_recipient_refill_limit" +#define _XPORT_REFILL_LIMIT "_recipient_refill_limit" +#define DEF_XPORT_REFILL_LIMIT 100 +extern int var_xport_refill_limit; + +#define VAR_XPORT_REFILL_DELAY "default_recipient_refill_delay" +#define _XPORT_REFILL_DELAY "_recipient_refill_delay" +#define DEF_XPORT_REFILL_DELAY "5s" +extern int var_xport_refill_delay; + /* * Queue manager: default job scheduler parameters. */ diff --git a/postfix/src/global/mail_proto.h b/postfix/src/global/mail_proto.h index de83b6cc6..0d5a174af 100644 --- a/postfix/src/global/mail_proto.h +++ b/postfix/src/global/mail_proto.h @@ -50,6 +50,7 @@ #define MAIL_SERVICE_SMTPD "smtpd" #define MAIL_SERVICE_SHOWQ "showq" #define MAIL_SERVICE_ERROR "error" +#define MAIL_SERVICE_RETRY "retry" #define MAIL_SERVICE_FLUSH "flush" #define MAIL_SERVICE_VERIFY "verify" #define MAIL_SERVICE_TRACE "trace" diff --git a/postfix/src/global/mail_stream.c b/postfix/src/global/mail_stream.c index ca9613e86..ec5f3adb8 100644 --- a/postfix/src/global/mail_stream.c +++ b/postfix/src/global/mail_stream.c @@ -156,6 +156,58 @@ void mail_stream_cleanup(MAIL_STREAM *info) myfree((char *) info); } +#if defined(HAS_FUTIMES_AT) +#define CAN_STAMP_BY_STREAM + +/* stamp_stream - update open file [am]time stamp */ + +static int stamp_stream(VSTREAM *fp, time_t when) +{ + struct timeval tv; + + if (when != 0) { + tv.tv_sec = when; + tv.tv_usec = 0; + return (futimesat(vstream_fileno(fp), (char *) 0, &tv)); + } else { + return (futimesat(vstream_fileno(fp), (char *) 0, (struct timeval *) 0)); + } +} + +#elif defined(HAS_FUTIMES) +#define CAN_STAMP_BY_STREAM + +/* stamp_stream - update open file [am]time stamp */ + +static int stamp_stream(VSTREAM *fp, time_t when) +{ + struct timeval tv; + + if (when != 0) { + tv.tv_sec = when; + tv.tv_usec = 0; + return (futimes(vstream_fileno(fp), &tv)); + } else { + return (futimes(vstream_fileno(fp), (struct timeval *) 0)); + } +} + +#endif + +/* stamp_path - update file [am]time stamp by pathname */ + +static int stamp_path(const char *path, time_t when) +{ + struct utimbuf tbuf; + + if (when != 0) { + tbuf.actime = tbuf.modtime = when; + return (utime(path, &tbuf)); + } else { + return (utime(path, (struct utimbuf *) 0)); + } +} + /* mail_stream_finish_file - finish file mail stream */ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) @@ -163,13 +215,13 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) int status = CLEANUP_STAT_OK; static char wakeup[] = {TRIGGER_REQ_WAKEUP}; struct stat st; - time_t now; - struct utimbuf tbuf; char *path_to_reset = 0; static int incoming_fs_clock_ok = 0; static int incoming_clock_warned = 0; int check_incoming_fs_clock; int err; + time_t want_stamp; + time_t expect_stamp; /* * Make sure the message makes it to file. Set the execute bit when no @@ -186,7 +238,7 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) * Attempt to detect file system clocks that are ahead of local time, but * don't check the file system clock all the time. The effect of file * system clock drift can be difficult to understand (Postfix ignores new - * mail until the next queue run). + * mail until the local clock catches up with the file mtime stamp). * * This clock drift detection code may not work with file systems that work * on a local copy of the file and that update the server only after the @@ -204,6 +256,9 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) * creates a race even with local filesystems. But Wietse is not * confident that utime() before fsync() and close() will work reliably * with remote file systems. + * + * XXX Don't run the clock skew tests with Postfix sendmail submissions. + * Don't whine against unsuspecting users or applications. */ check_incoming_fs_clock = (!incoming_fs_clock_ok && !strcmp(info->queue, MAIL_QUEUE_INCOMING)); @@ -212,12 +267,29 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) if (strcmp(info->queue, MAIL_QUEUE_DEFERRED) != 0) info->delay = 0; if (info->delay > 0) - tbuf.actime = tbuf.modtime = time(&now) + info->delay; + want_stamp = time((time_t *) 0) + info->delay; + else #endif + want_stamp = 0; + /* + * If we can cheaply set the file time stamp (no pathname lookup) do it + * anyway, so that we can avoid whining later about file server/client + * clock skew. + * + * Otherwise, if we must set the file time stamp for delayed delivery, use + * whatever means we have to get the job done, no matter if it is + * expensive. + * + * XXX Unfortunately, Linux futimes() is not usable because it uses /proc. + * This may not be available because of chroot, or because of access + * restrictions after a process changes privileges. + */ if (vstream_fflush(info->stream) -#ifdef DELAY_ACTION - || (info->delay > 0 && utime(VSTREAM_PATH(info->stream), &tbuf)) +#ifdef CAN_STAMP_BY_STREAM + || stamp_stream(info->stream, want_stamp) +#else + || (want_stamp && stamp_path(VSTREAM_PATH(info->stream), want_stamp)) #endif || fchmod(vstream_fileno(info->stream), 0700 | info->mode) #ifdef HAS_FSYNC @@ -227,25 +299,32 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) && fstat(vstream_fileno(info->stream), &st) < 0) ) status = (errno == EFBIG ? CLEANUP_STAT_SIZE : CLEANUP_STAT_WRITE); - #ifdef TEST st.st_mtime += 10; #endif /* - * Work around file system clocks that are ahead of local time. + * Work around file system clock skew. If the file system clock is ahead + * of the local clock, Postfix won't deliver mail immediately, which is + * bad for performance. If the file system clock falls behind the local + * clock, it just looks silly in mail headers. */ if (status == CLEANUP_STAT_OK && check_incoming_fs_clock) { - if (st.st_mtime <= time(&now)) { - incoming_fs_clock_ok = 1; - } else { + /* Do NOT use time() result from before fsync(). */ + expect_stamp = want_stamp ? want_stamp : time((time_t *) 0); + if (st.st_mtime > expect_stamp) { path_to_reset = mystrdup(VSTREAM_PATH(info->stream)); if (incoming_clock_warned == 0) { msg_warn("file system clock is %d seconds ahead of local clock", - (int) (st.st_mtime - now)); + (int) (st.st_mtime - expect_stamp)); msg_warn("resetting file time stamps - this hurts performance"); incoming_clock_warned = 1; } + } else { + if (st.st_mtime < expect_stamp - 100) + msg_warn("file system clock is %d seconds behind local clock", + (int) (expect_stamp - st.st_mtime)); + incoming_fs_clock_ok = 1; } } @@ -267,8 +346,7 @@ static int mail_stream_finish_file(MAIL_STREAM *info, VSTRING *unused_why) */ if (path_to_reset != 0) { if (status == CLEANUP_STAT_OK) { - tbuf.actime = tbuf.modtime = now; - if (utime(path_to_reset, &tbuf) < 0 && errno != ENOENT) + if (stamp_path(path_to_reset, expect_stamp) < 0 && errno != ENOENT) msg_fatal("%s: update file time stamps: %m", info->id); } myfree(path_to_reset); diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index bc2d237b6..6461cc4d4 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20061203" +#define MAIL_RELEASE_DATE "20061209" #define MAIL_VERSION_NUMBER "2.4" #ifdef SNAPSHOT diff --git a/postfix/src/global/pipe_command.c b/postfix/src/global/pipe_command.c index 0fac5f7a8..d305f79d0 100644 --- a/postfix/src/global/pipe_command.c +++ b/postfix/src/global/pipe_command.c @@ -633,7 +633,7 @@ int pipe_command(VSTREAM *src, DSN_BUF *why,...) "Command died with signal %d: \"%s\"%s%s", WTERMSIG(wait_status), args.command, log_len ? ". Command output: " : "", log_buf); - return (PIPE_STAT_BOUNCE); + return (PIPE_STAT_DEFER); } /* Use "D.S.N text" command output. XXX What diagnostic code? */ else if (dsn_valid(log_buf) > 0) { diff --git a/postfix/src/global/recipient_list.c b/postfix/src/global/recipient_list.c index a39979241..4813748af 100644 --- a/postfix/src/global/recipient_list.c +++ b/postfix/src/global/recipient_list.c @@ -43,6 +43,10 @@ /* const char *orig_rcpt; /* const char *recipient; /* +/* void recipient_list_swap(a, b) +/* RECIPIENT_LIST *a; +/* RECIPIENT_LIST *b; +/* /* void recipient_list_free(list) /* RECIPIENT_LIST *list; /* @@ -72,6 +76,9 @@ /* recipient_list_add() adds a recipient to the specified list. /* Recipient address information is copied with mystrdup(). /* +/* recipient_list_swap() swaps the recipients between +/* the given two recipient lists. +/* /* recipient_list_free() releases memory for the specified list /* of recipient structures. /* @@ -111,6 +118,7 @@ /* Utility library. */ #include +#include /* Global library. */ @@ -154,6 +162,20 @@ void recipient_list_add(RECIPIENT_LIST *list, long offset, list->len++; } +/* recipient_list_swap - swap recipients between the two recipient lists */ + +void recipient_list_swap(RECIPIENT_LIST *a, RECIPIENT_LIST *b) +{ + if (b->variant != a->variant) + msg_panic("recipient_lists_swap: incompatible recipient list variants"); + +#define SWAP(t, x) do { t x = b->x; b->x = a->x ; a->x = x; } while (0) + + SWAP(RECIPIENT *, info); + SWAP(int, len); + SWAP(int, avail); +} + /* recipient_list_free - release memory for in-core recipient structure */ void recipient_list_free(RECIPIENT_LIST *list) diff --git a/postfix/src/global/recipient_list.h b/postfix/src/global/recipient_list.h index 6e9617218..8fc5d58b0 100644 --- a/postfix/src/global/recipient_list.h +++ b/postfix/src/global/recipient_list.h @@ -16,7 +16,7 @@ * tells us the position of the REC_TYPE_RCPT byte in the message queue * file, This byte is replaced by REC_TYPE_DONE when the delivery status to * that recipient is established. - * + * * Rather than bothering with subclasses that extend this structure with * application-specific fields we just add them here. */ @@ -30,7 +30,7 @@ typedef struct RECIPIENT { int status; /* SMTP client */ struct QMGR_QUEUE *queue; /* Queue manager */ const char *addr_type; /* DSN */ - } u; + } u; } RECIPIENT; #define RECIPIENT_ASSIGN(rcpt, offs, orcpt, notify, orig, addr) do { \ @@ -55,6 +55,7 @@ typedef struct RECIPIENT_LIST { extern void recipient_list_init(RECIPIENT_LIST *, int); extern void recipient_list_add(RECIPIENT_LIST *, long, const char *, int, const char *, const char *); +extern void recipient_list_swap(RECIPIENT_LIST *, RECIPIENT_LIST *); extern void recipient_list_free(RECIPIENT_LIST *); #define RCPT_LIST_INIT_STATUS 1 diff --git a/postfix/src/master/master_sig.c b/postfix/src/master/master_sig.c index fc76efe0d..0f7d2e6e1 100644 --- a/postfix/src/master/master_sig.c +++ b/postfix/src/master/master_sig.c @@ -45,6 +45,7 @@ #include #include +#include /* Application-specific. */ @@ -173,11 +174,9 @@ static void master_sigdeath(int sig) pid_t pid = getpid(); /* - * XXX We're running from a signal handler, and really should not call - * any msg() routines at all, but it would be even worse to silently - * terminate without informing the sysadmin. + * Set alarm clock here for suicide after 5s. */ - msg_info("terminating on signal %d", sig); + killme_after(5); /* * Terminate all processes in our process group, except ourselves. @@ -190,6 +189,14 @@ static void master_sigdeath(int sig) if (kill(-pid, SIGTERM) < 0) msg_fatal("%s: kill process group: %m", myname); + /* + * XXX We're running from a signal handler, and should not call complex + * routines at all, but it would be even worse to silently terminate + * without informing the sysadmin. For this reason, msg(3) was made safe + * for usage by signal handlers that terminate the process. + */ + msg_info("terminating on signal %d", sig); + /* * Deliver the signal to ourselves and clean up. XXX We're running as a * signal handler and really should not be doing complicated things... diff --git a/postfix/src/oqmgr/Makefile.in b/postfix/src/oqmgr/Makefile.in index 1420a87e5..136bd72e6 100644 --- a/postfix/src/oqmgr/Makefile.in +++ b/postfix/src/oqmgr/Makefile.in @@ -1,10 +1,10 @@ SHELL = /bin/sh SRCS = qmgr.c qmgr_active.c qmgr_transport.c qmgr_queue.c qmgr_entry.c \ qmgr_message.c qmgr_deliver.c qmgr_move.c \ - qmgr_defer.c qmgr_enable.c qmgr_scan.c qmgr_bounce.c + qmgr_defer.c qmgr_enable.c qmgr_scan.c qmgr_bounce.c qmgr_error.c OBJS = qmgr.o qmgr_active.o qmgr_transport.o qmgr_queue.o qmgr_entry.o \ qmgr_message.o qmgr_deliver.o qmgr_move.o \ - qmgr_defer.o qmgr_enable.o qmgr_scan.o qmgr_bounce.o + qmgr_defer.o qmgr_enable.o qmgr_scan.o qmgr_bounce.o qmgr_error.o HDRS = qmgr.h TESTSRC = DEFS = -I. -I$(INC_DIR) -D$(SYSTYPE) @@ -132,6 +132,8 @@ qmgr_defer.o: ../../include/defer.h qmgr_defer.o: ../../include/deliver_request.h qmgr_defer.o: ../../include/dsn.h qmgr_defer.o: ../../include/dsn_buf.h +qmgr_defer.o: ../../include/iostuff.h +qmgr_defer.o: ../../include/mail_proto.h qmgr_defer.o: ../../include/msg.h qmgr_defer.o: ../../include/msg_stats.h qmgr_defer.o: ../../include/recipient_list.h @@ -193,6 +195,17 @@ qmgr_entry.o: ../../include/vstream.h qmgr_entry.o: ../../include/vstring.h qmgr_entry.o: qmgr.h qmgr_entry.o: qmgr_entry.c +qmgr_error.o: ../../include/dsn.h +qmgr_error.o: ../../include/mymalloc.h +qmgr_error.o: ../../include/recipient_list.h +qmgr_error.o: ../../include/scan_dir.h +qmgr_error.o: ../../include/stringops.h +qmgr_error.o: ../../include/sys_defs.h +qmgr_error.o: ../../include/vbuf.h +qmgr_error.o: ../../include/vstream.h +qmgr_error.o: ../../include/vstring.h +qmgr_error.o: qmgr.h +qmgr_error.o: qmgr_error.c qmgr_message.o: ../../include/argv.h qmgr_message.o: ../../include/attr.h qmgr_message.o: ../../include/bounce.h diff --git a/postfix/src/oqmgr/qmgr.h b/postfix/src/oqmgr/qmgr.h index 383f0ec24..19f54b33c 100644 --- a/postfix/src/oqmgr/qmgr.h +++ b/postfix/src/oqmgr/qmgr.h @@ -135,6 +135,8 @@ extern void qmgr_transport_unthrottle(QMGR_TRANSPORT *); extern QMGR_TRANSPORT *qmgr_transport_create(const char *); extern QMGR_TRANSPORT *qmgr_transport_find(const char *); +#define QMGR_TRANSPORT_THROTTLED(t) ((t)->flags & QMGR_TRANSPORT_STAT_DEAD) + /* * Each next hop (e.g., a domain name) has its own queue of pending message * transactions. The "todo" queue contains messages that are to be delivered @@ -177,6 +179,8 @@ extern void qmgr_queue_throttle(QMGR_QUEUE *, DSN *); extern void qmgr_queue_unthrottle(QMGR_QUEUE *); extern QMGR_QUEUE *qmgr_queue_find(QMGR_TRANSPORT *, const char *); +#define QMGR_QUEUE_THROTTLED(q) ((q)->window <= 0) + /* * Structure of one next-hop queue entry. In order to save some copying * effort we allow multiple recipients per transaction. @@ -191,6 +195,7 @@ struct QMGR_ENTRY { extern QMGR_ENTRY *qmgr_entry_select(QMGR_QUEUE *); extern void qmgr_entry_unselect(QMGR_QUEUE *, QMGR_ENTRY *); +extern void qmgr_entry_move_todo(QMGR_QUEUE *, QMGR_ENTRY *); extern void qmgr_entry_done(QMGR_ENTRY *, int); extern QMGR_ENTRY *qmgr_entry_create(QMGR_QUEUE *, QMGR_MESSAGE *); @@ -321,6 +326,13 @@ extern QMGR_SCAN *qmgr_scan_create(const char *); extern void qmgr_scan_request(QMGR_SCAN *, int); extern char *qmgr_scan_next(QMGR_SCAN *); + /* + * qmgr_error.c + */ +extern QMGR_TRANSPORT *qmgr_error_transport(const char *); +extern QMGR_QUEUE *qmgr_error_queue(const char *, DSN *); +extern char *qmgr_error_nexthop(DSN *); + /* LICENSE /* .ad /* .fi diff --git a/postfix/src/oqmgr/qmgr_defer.c b/postfix/src/oqmgr/qmgr_defer.c index 51eaf8643..dc0319e77 100644 --- a/postfix/src/oqmgr/qmgr_defer.c +++ b/postfix/src/oqmgr/qmgr_defer.c @@ -15,7 +15,7 @@ /* QMGR_QUEUE *queue; /* DSN *dsn; /* -/* QMGR_QUEUE *qmgr_defer_transport(transport, dsn) +/* void qmgr_defer_transport(transport, dsn) /* QMGR_TRANSPORT *transport; /* DSN *dsn; /* DESCRIPTION @@ -71,6 +71,7 @@ /* Global library. */ +#include #include /* Application-specific. */ @@ -106,6 +107,7 @@ void qmgr_defer_todo(QMGR_QUEUE *queue, DSN *dsn) QMGR_MESSAGE *message; RECIPIENT *recipient; int nrcpt; + QMGR_QUEUE *retry_queue; /* * Sanity checks. @@ -115,10 +117,22 @@ void qmgr_defer_todo(QMGR_QUEUE *queue, DSN *dsn) queue->name, dsn->status, dsn->reason); /* - * Proceed carefully. Queue entries will disappear as a side effect. + * See if we can redirect the deliveries to the retry(8) delivery agent, + * so that they can be handled asynchronously. If the retry(8) service is + * unavailable, use the synchronous defer(8) server. With a large todo + * queue, this blocks the queue manager for a significant time. + */ + retry_queue = qmgr_error_queue(MAIL_SERVICE_RETRY, dsn); + + /* + * Proceed carefully. Queue entries may disappear as a side effect. */ for (entry = queue->todo.next; entry != 0; entry = next) { next = entry->peers.next; + if (retry_queue != 0) { + qmgr_entry_move_todo(retry_queue, entry); + continue; + } message = entry->message; for (nrcpt = 0; nrcpt < entry->rcpt_list.len; nrcpt++) { recipient = entry->rcpt_list.info + nrcpt; diff --git a/postfix/src/oqmgr/qmgr_deliver.c b/postfix/src/oqmgr/qmgr_deliver.c index 352939c93..8d3b682ab 100644 --- a/postfix/src/oqmgr/qmgr_deliver.c +++ b/postfix/src/oqmgr/qmgr_deliver.c @@ -214,8 +214,16 @@ static void qmgr_deliver_update(int unused_event, char *context) QMGR_MESSAGE *message = entry->message; static DSN_BUF *dsb; int status; - RECIPIENT *recipient; - int nrcpt; + + /* + * Release the delivery agent from a "hot" queue entry. + */ +#define QMGR_DELIVER_RELEASE_AGENT(entry) do { \ + event_disable_readwrite(vstream_fileno(entry->stream)); \ + (void) vstream_fclose(entry->stream); \ + entry->stream = 0; \ + qmgr_deliver_concurrency--; \ + } while (0) if (dsb == 0) dsb = dsb_create(); @@ -264,16 +272,18 @@ static void qmgr_deliver_update(int unused_event, char *context) * recipient. This omission was already present in the first queue * manager implementation of 199703, and was fixed 200511. * - * Don't move this queue entry back to the todo queue so that - * qmgr_defer_transport() can update the defer log. The queue entry - * is still hot, and making it cold would involve duplicating most - * but not all code at the end of this routine. That's too tricky. + * To avoid the synchronous qmgr_defer_recipient() operation for each + * recipient of this queue entry, release the delivery process and + * move the entry back to the todo queue. Let qmgr_defer_transport() + * log the recipient asynchronously if possible, and get out of here. + * Note: if asynchronous logging is not possible, + * qmgr_defer_transport() eventually invokes qmgr_entry_done() and + * the entry becomes a dangling pointer. */ - for (nrcpt = 0; nrcpt < entry->rcpt_list.len; nrcpt++) { - recipient = entry->rcpt_list.info + nrcpt; - qmgr_defer_recipient(message, recipient, &dsb->dsn); - } + QMGR_DELIVER_RELEASE_AGENT(entry); + qmgr_entry_unselect(queue, entry); qmgr_defer_transport(transport, &dsb->dsn); + return; } /* @@ -318,11 +328,7 @@ static void qmgr_deliver_update(int unused_event, char *context) * to be delivered. When all recipients for a message have been tried, * decide what to do next with this message: defer, bounce, delete. */ - event_disable_readwrite(vstream_fileno(entry->stream)); - if (vstream_fclose(entry->stream) != 0) - msg_warn("qmgr_deliver_update: close delivery stream: %m"); - entry->stream = 0; - qmgr_deliver_concurrency--; + QMGR_DELIVER_RELEASE_AGENT(entry); qmgr_entry_done(entry, QMGR_QUEUE_BUSY); } @@ -341,7 +347,7 @@ void qmgr_deliver(QMGR_TRANSPORT *transport, VSTREAM *stream) * routine runs in response to an external event, so it does not run * while some other queue manipulation is happening. */ - if (qmgr_deliver_initial_reply(stream) != 0) { + if (stream == 0 || qmgr_deliver_initial_reply(stream) != 0) { #if 0 whatsup = concatenate(transport->name, " mail transport unavailable", (char *) 0); @@ -354,7 +360,8 @@ void qmgr_deliver(QMGR_TRANSPORT *transport, VSTREAM *stream) "mail transport unavailable")); #endif qmgr_defer_transport(transport, &dsn); - (void) vstream_fclose(stream); + if (stream) + (void) vstream_fclose(stream); return; } diff --git a/postfix/src/oqmgr/qmgr_entry.c b/postfix/src/oqmgr/qmgr_entry.c index ccffdb0b0..fda4e5790 100644 --- a/postfix/src/oqmgr/qmgr_entry.c +++ b/postfix/src/oqmgr/qmgr_entry.c @@ -20,6 +20,10 @@ /* void qmgr_entry_unselect(queue, entry) /* QMGR_QUEUE *queue; /* QMGR_ENTRY *entry; +/* +/* void qmgr_entry_move_todo(dst, entry) +/* QMGR_QUEUE *dst; +/* QMGR_ENTRY *entry; /* DESCRIPTION /* These routines add/delete/manipulate per-site message /* delivery requests. @@ -55,6 +59,9 @@ /* qmgr_entry_unselect() takes the named entry off the named /* per-site queue's `busy' list and moves it to the queue's /* `todo' list. +/* +/* qmgr_entry_move_todo() moves the specified "todo" queue entry +/* to the specified "todo" queue. /* DIAGNOSTICS /* Panic: interface violations, internal inconsistencies. /* LICENSE @@ -169,6 +176,38 @@ void qmgr_entry_unselect(QMGR_QUEUE *queue, QMGR_ENTRY *entry) queue->todo_refcount++; } +/* qmgr_entry_move_todo - move entry between todo queues */ + +void qmgr_entry_move_todo(QMGR_QUEUE *dst, QMGR_ENTRY *entry) +{ + const char *myname = "qmgr_entry_move_todo"; + QMGR_MESSAGE *message = entry->message; + QMGR_QUEUE *src = entry->queue; + QMGR_ENTRY *new_entry; + + if (entry->stream != 0) + msg_panic("%s: queue %s entry is busy", myname, src->name); + if (QMGR_QUEUE_THROTTLED(dst)) + msg_panic("%s: destination queue %s is throttled", myname, dst->name); + if (QMGR_TRANSPORT_THROTTLED(dst->transport)) + msg_panic("%s: destination transport %s is throttled", + myname, dst->transport->name); + + /* + * Create new entry, swap the recipients between the old and new entries, + * then dispose of the old entry. This gives us any end-game actions that + * are implemented by qmgr_entry_done(), so we don't have to duplicate + * those actions here. + * + * XXX This does not enforce the per-entry recipient limit, but that is not + * a problem as long as qmgr_entry_move_todo() is called only to bounce + * or defer mail. + */ + new_entry = qmgr_entry_create(dst, message); + recipient_list_swap(&entry->rcpt_list, &new_entry->rcpt_list); + qmgr_entry_done(entry, QMGR_QUEUE_TODO); +} + /* qmgr_entry_done - dispose of queue entry */ void qmgr_entry_done(QMGR_ENTRY *entry, int which) @@ -210,6 +249,8 @@ void qmgr_entry_done(QMGR_ENTRY *entry, int which) * not dead, discard the in-core queue. When this site is dead, but the * number of in-core queues exceeds some threshold, get rid of this * in-core queue anyway, in order to avoid running out of memory. + * + * See also: qmgr_entry_move_todo(). */ if (queue->todo.next == 0 && queue->busy.next == 0) { if (queue->window == 0 && qmgr_queue_count > 2 * var_qmgr_rcpt_limit) diff --git a/postfix/src/oqmgr/qmgr_error.c b/postfix/src/oqmgr/qmgr_error.c new file mode 100644 index 000000000..6d07b3bd8 --- /dev/null +++ b/postfix/src/oqmgr/qmgr_error.c @@ -0,0 +1,121 @@ +/*++ +/* NAME +/* qmgr_error 3 +/* SUMMARY +/* look up/create error/retry queue +/* SYNOPSIS +/* #include "qmgr.h" +/* +/* QMGR_TRANSPORT *qmgr_error_transport(service) +/* const char *service; +/* +/* QMGR_QUEUE *qmgr_error_queue(service, dsn) +/* const char *service; +/* DSN *dsn; +/* +/* char *qmgr_error_nexthop(dsn) +/* DSN *dsn; +/* DESCRIPTION +/* qmgr_error_transport() looks up the error transport for the +/* specified service. The result is null if the transport is +/* not available. +/* +/* qmgr_error_queue() looks up an error queue for the specified +/* service and problem. The result is null if the queue is not +/* availabe. +/* +/* qmgr_error_nexthop() computes the next-hop information for +/* the specified problem. The result must be passed to myfree(). +/* +/* Arguments: +/* .IP dsn +/* See dsn(3). +/* .IP service +/* One of MAIL_SERVICE_ERROR or MAIL_SERVICE_RETRY. +/* DIAGNOSTICS +/* Panic: consistency check failure. Fatal: out of memory. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +/* System library. */ + +#include + +/* Utility library. */ + +#include +#include + +/* Global library. */ + +/* Application-specific. */ + +#include "qmgr.h" + +/* qmgr_error_transport - look up error transport for specified service */ + +QMGR_TRANSPORT *qmgr_error_transport(const char *service) +{ + QMGR_TRANSPORT *transport; + + /* + * Find or create retry transport. + */ + if ((transport = qmgr_transport_find(service)) == 0) + transport = qmgr_transport_create(service); + if (QMGR_TRANSPORT_THROTTLED(transport)) + return (0); + + /* + * Done. + */ + return (transport); +} + +/* qmgr_error_queue - look up error queue for specified service and problem */ + +QMGR_QUEUE *qmgr_error_queue(const char *service, DSN *dsn) +{ + QMGR_TRANSPORT *transport; + QMGR_QUEUE *queue; + char *nexthop; + + /* + * Find or create transport. + */ + if ((transport = qmgr_error_transport(service)) == 0) + return (0); + + /* + * Find or create queue. + */ + nexthop = qmgr_error_nexthop(dsn); + if ((queue = qmgr_queue_find(transport, nexthop)) == 0) + queue = qmgr_queue_create(transport, nexthop, nexthop); + myfree(nexthop); + if (QMGR_QUEUE_THROTTLED(queue)) + return (0); + + /* + * Done. + */ + return (queue); +} + +/* qmgr_error_nexthop - compute next-hop information from problem description */ + +char *qmgr_error_nexthop(DSN *dsn) +{ + char *nexthop; + + nexthop = concatenate(dsn->status, " ", dsn->reason, (char *) 0); + return (nexthop); +} diff --git a/postfix/src/oqmgr/qmgr_message.c b/postfix/src/oqmgr/qmgr_message.c index 33706b6d0..cf1bb6f7e 100644 --- a/postfix/src/oqmgr/qmgr_message.c +++ b/postfix/src/oqmgr/qmgr_message.c @@ -877,22 +877,24 @@ static void qmgr_message_sort(QMGR_MESSAGE *message) static int qmgr_resolve_one(QMGR_MESSAGE *message, RECIPIENT *recipient, const char *addr, RESOLVE_REPLY *reply) { - DSN dsn; +#define QMGR_REDIRECT(rp, tp, np) do { \ + (rp)->flags = 0; \ + vstring_strcpy((rp)->transport, (tp)); \ + vstring_strcpy((rp)->nexthop, (np)); \ + } while (0) if ((message->tflags & DEL_REQ_FLAG_MTA_VRFY) == 0) resolve_clnt_query_from(message->sender, addr, reply); else resolve_clnt_verify_from(message->sender, addr, reply); if (reply->flags & RESOLVE_FLAG_FAIL) { - qmgr_defer_recipient(message, recipient, - DSN_SIMPLE(&dsn, "4.3.0", - "address resolver failure")); - return (-1); + QMGR_REDIRECT(reply, MAIL_SERVICE_RETRY, + "4.3.0 address resolver failure"); + return (0); } else if (reply->flags & RESOLVE_FLAG_ERROR) { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "bad address syntax")); - return (-1); + QMGR_REDIRECT(reply, MAIL_SERVICE_ERROR, + "5.1.3 bad address syntax"); + return (0); } else { return (0); } @@ -916,6 +918,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) int status; DSN dsn; MSG_STATS stats; + DSN *saved_dsn; #define STREQ(x,y) (strcmp(x,y) == 0) #define STR vstring_str @@ -926,8 +929,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) for (recipient = list.info; recipient < list.info + list.len; recipient++) { /* - * Redirect overrides all else. But only once (per batch of - * recipients). For consistency with the remainder of Postfix, + * Redirect overrides all else. But only once (per entire + * message). For consistency with the remainder of Postfix, * rewrite the address to canonical form before resolving it. */ if (message->redirect_addr) { @@ -935,6 +938,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) recipient->u.queue = 0; continue; } + message->rcpt_offset = 0; rewrite_clnt_internal(REWRITE_CANON, message->redirect_addr, reply.recipient); RECIPIENT_UPDATE(recipient->address, STR(reply.recipient)); @@ -982,10 +986,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) * the queue manager process does not help. */ if (recipient->address[0] == 0) { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "null recipient address")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_ERROR, + "5.1.3 null recipient address"); } /* @@ -1000,10 +1002,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) * where it cannot be bypassed. */ if (var_allow_min_user == 0 && recipient->address[0] == '-') { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "bad address syntax")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_ERROR, + "5.1.3 bad address syntax"); } /* @@ -1047,10 +1047,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) if (strcmp(*cpp, STR(reply.transport)) == 0) break; if (*cpp) { - qmgr_defer_recipient(message, recipient, - DSN_SIMPLE(&dsn, "4.3.2", - "deferred transport")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_RETRY, + "4.3.2 deferred transport"); } } @@ -1066,9 +1064,17 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) /* * This transport is dead. Defer delivery to this recipient. */ - if ((transport->flags & QMGR_TRANSPORT_STAT_DEAD) != 0) { - qmgr_defer_recipient(message, recipient, transport->dsn); - continue; + if (QMGR_TRANSPORT_THROTTLED(transport)) { + saved_dsn = transport->dsn; + if ((transport = qmgr_error_transport(MAIL_SERVICE_RETRY)) != 0) { + nexthop = qmgr_error_nexthop(saved_dsn); + vstring_strcpy(reply.nexthop, nexthop); + myfree(nexthop); + queue = 0; + } else { + qmgr_defer_recipient(message, recipient, saved_dsn); + continue; + } } /* @@ -1106,6 +1112,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) */ vstring_strcpy(queue_name, STR(reply.nexthop)); if (strcmp(transport->name, MAIL_SERVICE_ERROR) != 0 + && strcmp(transport->name, MAIL_SERVICE_RETRY) != 0 && transport->recipient_limit == 1) { /* Copy the recipient localpart. */ at = strrchr(STR(reply.recipient), '@'); @@ -1133,9 +1140,12 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) /* * This queue is dead. Defer delivery to this recipient. */ - if (queue->window == 0) { - qmgr_defer_recipient(message, recipient, queue->dsn); - continue; + if (QMGR_QUEUE_THROTTLED(queue)) { + saved_dsn = queue->dsn; + if ((queue = qmgr_error_queue(MAIL_SERVICE_RETRY, saved_dsn)) == 0) { + qmgr_defer_recipient(message, recipient, saved_dsn); + continue; + } } /* diff --git a/postfix/src/oqmgr/qmgr_transport.c b/postfix/src/oqmgr/qmgr_transport.c index df522f376..65ae935b3 100644 --- a/postfix/src/oqmgr/qmgr_transport.c +++ b/postfix/src/oqmgr/qmgr_transport.c @@ -193,7 +193,8 @@ static void qmgr_transport_event(int unused_event, char *context) /* * Disable further read events that end up calling this function. */ - event_disable_readwrite(vstream_fileno(alloc->stream)); + if (alloc->stream) + event_disable_readwrite(vstream_fileno(alloc->stream)); alloc->transport->flags &= ~QMGR_TRANSPORT_STAT_BUSY; /* @@ -259,7 +260,6 @@ void qmgr_transport_alloc(QMGR_TRANSPORT *transport, QMGR_TRANSPORT_ALLOC_NOT { QMGR_TRANSPORT_ALLOC *alloc; VSTREAM *stream; - DSN dsn; /* * Sanity checks. @@ -286,18 +286,28 @@ void qmgr_transport_alloc(QMGR_TRANSPORT *transport, QMGR_TRANSPORT_ALLOC_NOT #define EVENT_HANDLER qmgr_transport_event #endif + /* + * When the connection to the delivery agent cannot be completed, notify + * the event handler so that it can throttle the transport and defer the + * todo queues, just like it does when communication fails *after* + * connection completion. + * + * Before Postfix 2.4, the event handler was not invoked, and mail was not + * deferred. Because of this, mail would be stuck in the active queue + * after triggering a "connection refused" condition. + */ if ((stream = mail_connect(MAIL_CLASS_PRIVATE, transport->name, BLOCK_MODE)) == 0) { msg_warn("connect to transport %s: %m", transport->name); - qmgr_transport_throttle(transport, - DSN_SIMPLE(&dsn, "4.3.0", - "mail transport unavailable")); - return; } alloc = (QMGR_TRANSPORT_ALLOC *) mymalloc(sizeof(*alloc)); alloc->stream = stream; alloc->transport = transport; alloc->notify = notify; transport->flags |= QMGR_TRANSPORT_STAT_BUSY; + if (alloc->stream == 0) { + event_request_timer(qmgr_transport_event, (char *) alloc, 0); + return; + } ENABLE_EVENTS(vstream_fileno(alloc->stream), EVENT_HANDLER, (char *) alloc); /* diff --git a/postfix/src/qmgr/Makefile.in b/postfix/src/qmgr/Makefile.in index 0f5489ebe..75b86bf91 100644 --- a/postfix/src/qmgr/Makefile.in +++ b/postfix/src/qmgr/Makefile.in @@ -2,11 +2,11 @@ SHELL = /bin/sh SRCS = qmgr.c qmgr_active.c qmgr_transport.c qmgr_queue.c qmgr_entry.c \ qmgr_message.c qmgr_deliver.c qmgr_move.c \ qmgr_job.c qmgr_peer.c \ - qmgr_defer.c qmgr_enable.c qmgr_scan.c qmgr_bounce.c + qmgr_defer.c qmgr_enable.c qmgr_scan.c qmgr_bounce.c qmgr_error.c OBJS = qmgr.o qmgr_active.o qmgr_transport.o qmgr_queue.o qmgr_entry.o \ qmgr_message.o qmgr_deliver.o qmgr_move.o \ qmgr_job.o qmgr_peer.o \ - qmgr_defer.o qmgr_enable.o qmgr_scan.o qmgr_bounce.o + qmgr_defer.o qmgr_enable.o qmgr_scan.o qmgr_bounce.o qmgr_error.o HDRS = qmgr.h TESTSRC = DEFS = -I. -I$(INC_DIR) -D$(SYSTYPE) @@ -134,6 +134,8 @@ qmgr_defer.o: ../../include/defer.h qmgr_defer.o: ../../include/deliver_request.h qmgr_defer.o: ../../include/dsn.h qmgr_defer.o: ../../include/dsn_buf.h +qmgr_defer.o: ../../include/iostuff.h +qmgr_defer.o: ../../include/mail_proto.h qmgr_defer.o: ../../include/msg.h qmgr_defer.o: ../../include/msg_stats.h qmgr_defer.o: ../../include/recipient_list.h @@ -195,6 +197,17 @@ qmgr_entry.o: ../../include/vstream.h qmgr_entry.o: ../../include/vstring.h qmgr_entry.o: qmgr.h qmgr_entry.o: qmgr_entry.c +qmgr_error.o: ../../include/dsn.h +qmgr_error.o: ../../include/mymalloc.h +qmgr_error.o: ../../include/recipient_list.h +qmgr_error.o: ../../include/scan_dir.h +qmgr_error.o: ../../include/stringops.h +qmgr_error.o: ../../include/sys_defs.h +qmgr_error.o: ../../include/vbuf.h +qmgr_error.o: ../../include/vstream.h +qmgr_error.o: ../../include/vstring.h +qmgr_error.o: qmgr.h +qmgr_error.o: qmgr_error.c qmgr_job.o: ../../include/dsn.h qmgr_job.o: ../../include/htable.h qmgr_job.o: ../../include/msg.h diff --git a/postfix/src/qmgr/qmgr.c b/postfix/src/qmgr/qmgr.c index 1c43299a3..7a386f6a2 100644 --- a/postfix/src/qmgr/qmgr.c +++ b/postfix/src/qmgr/qmgr.c @@ -173,7 +173,7 @@ /* in-memory "dead" destination status cache. /* .IP "\fBqmgr_message_recipient_minimum (10)\fR" /* The minimal number of in-memory recipients for any message. -/* .IP "\fBdefault_recipient_limit (10000)\fR" +/* .IP "\fBdefault_recipient_limit (20000)\fR" /* The default per-transport upper limit on the number of in-memory /* recipients. /* .IP "\fItransport\fB_recipient_limit ($default_recipient_limit)\fR" @@ -183,6 +183,17 @@ /* number of in-memory recipients. /* .IP "\fItransport\fB_extra_recipient_limit ($default_extra_recipient_limit)\fR" /* Idem, for delivery via the named message \fItransport\fR. +/* .PP +/* Available in Postfix version 2.4 and later: +/* .IP "\fBdefault_recipient_refill_limit (100)\fR" +/* The default per-transport limit on the number of recipients refilled at +/* once. +/* .IP "\fItransport\fB_recipient_refill_limit ($default_recipient_refill_limit)\fR" +/* Idem, for delivery via the named message \fItransport\fR. +/* .IP "\fBdefault_recipient_refill_delay (5s)\fR" +/* The default per-transport maximum delay between recipients refills. +/* .IP "\fItransport\fB_recipient_refill_delay ($default_recipient_refill_delay)\fR" +/* Idem, for delivery via the named message \fItransport\fR. /* DELIVERY CONCURRENCY CONTROLS /* .ad /* .fi @@ -361,6 +372,8 @@ int var_qmgr_rcpt_limit; int var_qmgr_msg_rcpt_limit; int var_xport_rcpt_limit; int var_stack_rcpt_limit; +int var_xport_refill_limit; +int var_xport_refill_delay; int var_delivery_slot_cost; int var_delivery_slot_loan; int var_delivery_slot_discount; @@ -595,6 +608,7 @@ int main(int argc, char **argv) VAR_DSN_QUEUE_TIME, DEF_DSN_QUEUE_TIME, &var_dsn_queue_time, 0, 8640000, VAR_XPORT_RETRY_TIME, DEF_XPORT_RETRY_TIME, &var_transport_retry_time, 1, 0, VAR_QMGR_CLOG_WARN_TIME, DEF_QMGR_CLOG_WARN_TIME, &var_qmgr_clog_warn_time, 0, 0, + VAR_XPORT_REFILL_DELAY, DEF_XPORT_REFILL_DELAY, &var_xport_refill_delay, 1, 0, 0, }; static CONFIG_INT_TABLE int_table[] = { @@ -603,6 +617,7 @@ int main(int argc, char **argv) VAR_QMGR_MSG_RCPT_LIMIT, DEF_QMGR_MSG_RCPT_LIMIT, &var_qmgr_msg_rcpt_limit, 1, 0, VAR_XPORT_RCPT_LIMIT, DEF_XPORT_RCPT_LIMIT, &var_xport_rcpt_limit, 0, 0, VAR_STACK_RCPT_LIMIT, DEF_STACK_RCPT_LIMIT, &var_stack_rcpt_limit, 0, 0, + VAR_XPORT_REFILL_LIMIT, DEF_XPORT_REFILL_LIMIT, &var_xport_refill_limit, 1, 0, VAR_DELIVERY_SLOT_COST, DEF_DELIVERY_SLOT_COST, &var_delivery_slot_cost, 0, 0, VAR_DELIVERY_SLOT_LOAN, DEF_DELIVERY_SLOT_LOAN, &var_delivery_slot_loan, 0, 0, VAR_DELIVERY_SLOT_DISCOUNT, DEF_DELIVERY_SLOT_DISCOUNT, &var_delivery_slot_discount, 0, 100, diff --git a/postfix/src/qmgr/qmgr.h b/postfix/src/qmgr/qmgr.h index b9ec6830e..03e1fa5cc 100644 --- a/postfix/src/qmgr/qmgr.h +++ b/postfix/src/qmgr/qmgr.h @@ -138,6 +138,8 @@ struct QMGR_TRANSPORT { int rcpt_per_stack; /* extra slots reserved for jobs put * on the job stack */ int rcpt_unused; /* available in-core recipient slots */ + int refill_limit; /* recipient batch size for message refill */ + int refill_delay; /* delay before message refill */ int slot_cost; /* cost of new preemption slot (# of * selected entries) */ int slot_loan; /* preemption boost offset and */ @@ -173,6 +175,8 @@ extern void qmgr_transport_unthrottle(QMGR_TRANSPORT *); extern QMGR_TRANSPORT *qmgr_transport_create(const char *); extern QMGR_TRANSPORT *qmgr_transport_find(const char *); +#define QMGR_TRANSPORT_THROTTLED(t) ((t)->flags & QMGR_TRANSPORT_STAT_DEAD) + /* * Each next hop (e.g., a domain name) has its own queue of pending message * transactions. The "todo" queue contains messages that are to be delivered @@ -213,6 +217,8 @@ extern void qmgr_queue_throttle(QMGR_QUEUE *, DSN *); extern void qmgr_queue_unthrottle(QMGR_QUEUE *); extern QMGR_QUEUE *qmgr_queue_find(QMGR_TRANSPORT *, const char *); +#define QMGR_QUEUE_THROTTLED(q) ((q)->window <= 0) + /* * Structure of one next-hop queue entry. In order to save some copying * effort we allow multiple recipients per transaction. @@ -229,6 +235,7 @@ struct QMGR_ENTRY { extern QMGR_ENTRY *qmgr_entry_select(QMGR_PEER *); extern void qmgr_entry_unselect(QMGR_ENTRY *); +extern void qmgr_entry_move_todo(QMGR_QUEUE *, QMGR_ENTRY *); extern void qmgr_entry_done(QMGR_ENTRY *, int); extern QMGR_ENTRY *qmgr_entry_create(QMGR_PEER *, QMGR_MESSAGE *); @@ -252,6 +259,7 @@ struct QMGR_MESSAGE { struct timeval active_time; /* time of entry into active queue */ time_t queued_time; /* sanitized time when moved to the * active queue */ + time_t refill_time; /* sanitized time of last message refill */ long warn_offset; /* warning bounce flag offset */ time_t warn_time; /* time next warning to be sent */ long data_offset; /* data seek offset */ @@ -359,6 +367,7 @@ extern void qmgr_job_move_limits(QMGR_JOB *); extern QMGR_PEER *qmgr_peer_create(QMGR_JOB *, QMGR_QUEUE *); extern QMGR_PEER *qmgr_peer_find(QMGR_JOB *, QMGR_QUEUE *); +extern QMGR_PEER *qmgr_peer_obtain(QMGR_JOB *, QMGR_QUEUE *); extern void qmgr_peer_free(QMGR_PEER *); /* @@ -423,6 +432,13 @@ extern QMGR_SCAN *qmgr_scan_create(const char *); extern void qmgr_scan_request(QMGR_SCAN *, int); extern char *qmgr_scan_next(QMGR_SCAN *); + /* + * qmgr_error.c + */ +extern QMGR_TRANSPORT *qmgr_error_transport(const char *); +extern QMGR_QUEUE *qmgr_error_queue(const char *, DSN *); +extern char *qmgr_error_nexthop(DSN *); + /* LICENSE /* .ad /* .fi diff --git a/postfix/src/qmgr/qmgr_defer.c b/postfix/src/qmgr/qmgr_defer.c index 4c70eef76..1c6801730 100644 --- a/postfix/src/qmgr/qmgr_defer.c +++ b/postfix/src/qmgr/qmgr_defer.c @@ -15,7 +15,7 @@ /* QMGR_QUEUE *queue; /* DSN *dsn; /* -/* QMGR_QUEUE *qmgr_defer_transport(transport, dsn) +/* void qmgr_defer_transport(transport, dsn) /* QMGR_TRANSPORT *transport; /* DSN *dsn; /* DESCRIPTION @@ -76,6 +76,7 @@ /* Global library. */ +#include #include /* Application-specific. */ @@ -111,6 +112,7 @@ void qmgr_defer_todo(QMGR_QUEUE *queue, DSN *dsn) QMGR_MESSAGE *message; RECIPIENT *recipient; int nrcpt; + QMGR_QUEUE *retry_queue; /* * Sanity checks. @@ -120,10 +122,22 @@ void qmgr_defer_todo(QMGR_QUEUE *queue, DSN *dsn) queue->name, dsn->status, dsn->reason); /* - * Proceed carefully. Queue entries will disappear as a side effect. + * See if we can redirect the deliveries to the retry(8) delivery agent, + * so that they can be handled asynchronously. If the retry(8) service is + * unavailable, use the synchronous defer(8) server. With a large todo + * queue, this blocks the queue manager for a significant time. + */ + retry_queue = qmgr_error_queue(MAIL_SERVICE_RETRY, dsn); + + /* + * Proceed carefully. Queue entries may disappear as a side effect. */ for (entry = queue->todo.next; entry != 0; entry = next) { next = entry->queue_peers.next; + if (retry_queue != 0) { + qmgr_entry_move_todo(retry_queue, entry); + continue; + } message = entry->message; for (nrcpt = 0; nrcpt < entry->rcpt_list.len; nrcpt++) { recipient = entry->rcpt_list.info + nrcpt; diff --git a/postfix/src/qmgr/qmgr_deliver.c b/postfix/src/qmgr/qmgr_deliver.c index 19dce9614..c0589e16d 100644 --- a/postfix/src/qmgr/qmgr_deliver.c +++ b/postfix/src/qmgr/qmgr_deliver.c @@ -219,8 +219,16 @@ static void qmgr_deliver_update(int unused_event, char *context) QMGR_MESSAGE *message = entry->message; static DSN_BUF *dsb; int status; - RECIPIENT *recipient; - int nrcpt; + + /* + * Release the delivery agent from a "hot" queue entry. + */ +#define QMGR_DELIVER_RELEASE_AGENT(entry) do { \ + event_disable_readwrite(vstream_fileno(entry->stream)); \ + (void) vstream_fclose(entry->stream); \ + entry->stream = 0; \ + qmgr_deliver_concurrency--; \ + } while (0) if (dsb == 0) dsb = dsb_create(); @@ -269,16 +277,18 @@ static void qmgr_deliver_update(int unused_event, char *context) * recipient. This omission was already present in the first queue * manager implementation of 199703, and was fixed 200511. * - * Don't move this queue entry back to the todo queue so that - * qmgr_defer_transport() can update the defer log. The queue entry - * is still hot, and making it cold would involve duplicating most - * but not all code at the end of this routine. That's too tricky. + * To avoid the synchronous qmgr_defer_recipient() operation for each + * recipient of this queue entry, release the delivery process and + * move the entry back to the todo queue. Let qmgr_defer_transport() + * log the recipient asynchronously if possible, and get out of here. + * Note: if asynchronous logging is not possible, + * qmgr_defer_transport() eventually invokes qmgr_entry_done() and + * the entry becomes a dangling pointer. */ - for (nrcpt = 0; nrcpt < entry->rcpt_list.len; nrcpt++) { - recipient = entry->rcpt_list.info + nrcpt; - qmgr_defer_recipient(message, recipient, &dsb->dsn); - } + QMGR_DELIVER_RELEASE_AGENT(entry); + qmgr_entry_unselect(entry); qmgr_defer_transport(transport, &dsb->dsn); + return; } /* @@ -323,11 +333,7 @@ static void qmgr_deliver_update(int unused_event, char *context) * to be delivered. When all recipients for a message have been tried, * decide what to do next with this message: defer, bounce, delete. */ - event_disable_readwrite(vstream_fileno(entry->stream)); - if (vstream_fclose(entry->stream) != 0) - msg_warn("qmgr_deliver_update: close delivery stream: %m"); - entry->stream = 0; - qmgr_deliver_concurrency--; + QMGR_DELIVER_RELEASE_AGENT(entry); qmgr_entry_done(entry, QMGR_QUEUE_BUSY); } @@ -345,7 +351,7 @@ void qmgr_deliver(QMGR_TRANSPORT *transport, VSTREAM *stream) * routine runs in response to an external event, so it does not run * while some other queue manipulation is happening. */ - if (qmgr_deliver_initial_reply(stream) != 0) { + if (stream == 0 || qmgr_deliver_initial_reply(stream) != 0) { #if 0 whatsup = concatenate(transport->name, " mail transport unavailable", (char *) 0); @@ -358,7 +364,8 @@ void qmgr_deliver(QMGR_TRANSPORT *transport, VSTREAM *stream) "mail transport unavailable")); #endif qmgr_defer_transport(transport, &dsn); - (void) vstream_fclose(stream); + if (stream) + (void) vstream_fclose(stream); return; } diff --git a/postfix/src/qmgr/qmgr_entry.c b/postfix/src/qmgr/qmgr_entry.c index 58c278bfd..6e4477be4 100644 --- a/postfix/src/qmgr/qmgr_entry.c +++ b/postfix/src/qmgr/qmgr_entry.c @@ -20,6 +20,10 @@ /* void qmgr_entry_unselect(queue, entry) /* QMGR_QUEUE *queue; /* QMGR_ENTRY *entry; +/* +/* void qmgr_entry_move_todo(dst, entry) +/* QMGR_QUEUE *dst; +/* QMGR_ENTRY *entry; /* DESCRIPTION /* These routines add/delete/manipulate per-site message /* delivery requests. @@ -57,7 +61,10 @@ /* /* qmgr_entry_unselect() takes the named entry off the named /* per-site queue's `busy' list and moves it to the queue's -/* `todo' list. The entry is also appended to its peer list again. +/* `todo' list. The entry is also prepended to its peer list again. +/* +/* qmgr_entry_move_todo() moves the specified "todo" queue entry +/* to the specified "todo" queue. /* DIAGNOSTICS /* Panic: interface violations, internal inconsistencies. /* LICENSE @@ -178,14 +185,66 @@ void qmgr_entry_unselect(QMGR_ENTRY *entry) QMGR_PEER *peer = entry->peer; QMGR_QUEUE *queue = entry->queue; + /* + * Move the entry back to the todo lists. In case of the peer list, + * put it back to the beginning, so the select()/unselect() does + * not reorder entries. We use this in qmgr_message_assign() + * to put recipients into existing entries when possible. + */ QMGR_LIST_UNLINK(queue->busy, QMGR_ENTRY *, entry, queue_peers); queue->busy_refcount--; QMGR_LIST_APPEND(queue->todo, entry, queue_peers); queue->todo_refcount++; - QMGR_LIST_APPEND(peer->entry_list, entry, peer_peers); + QMGR_LIST_PREPEND(peer->entry_list, entry, peer_peers); peer->job->selected_entries--; } +/* qmgr_entry_move_todo - move entry between todo queues */ + +void qmgr_entry_move_todo(QMGR_QUEUE *dst_queue, QMGR_ENTRY *entry) +{ + const char *myname = "qmgr_entry_move_todo"; + QMGR_TRANSPORT *dst_transport = dst_queue->transport; + QMGR_MESSAGE *message = entry->message; + QMGR_QUEUE *src_queue = entry->queue; + QMGR_PEER *dst_peer, *src_peer = entry->peer; + QMGR_JOB *dst_job, *src_job = src_peer->job; + QMGR_ENTRY *new_entry; + int rcpt_count = entry->rcpt_list.len; + + if (entry->stream != 0) + msg_panic("%s: queue %s entry is busy", myname, src_queue->name); + if (QMGR_QUEUE_THROTTLED(dst_queue)) + msg_panic("%s: destination queue %s is throttled", myname, dst_queue->name); + if (QMGR_TRANSPORT_THROTTLED(dst_transport)) + msg_panic("%s: destination transport %s is throttled", + myname, dst_transport->name); + + /* + * Create new entry, swap the recipients between the two entries, + * adjusting the job counters accordingly, then dispose of the old entry. + * + * Note that qmgr_entry_done() will also take care of adjusting the + * recipient limits of all the message jobs, so we do not have to do that + * explicitly for the new job here. + * + * XXX This does not enforce the per-entry recipient limit, but that is not + * a problem as long as qmgr_entry_move_todo() is called only to bounce + * or defer mail. + */ + dst_job = qmgr_job_obtain(message, dst_transport); + dst_peer = qmgr_peer_obtain(dst_job, dst_queue); + + new_entry = qmgr_entry_create(dst_peer, message); + + recipient_list_swap(&entry->rcpt_list, &new_entry->rcpt_list); + + src_job->rcpt_count -= rcpt_count; + dst_job->rcpt_count += rcpt_count; + + qmgr_entry_done(entry, QMGR_QUEUE_TODO); +} + /* qmgr_entry_done - dispose of queue entry */ void qmgr_entry_done(QMGR_ENTRY *entry, int which) @@ -193,8 +252,7 @@ void qmgr_entry_done(QMGR_ENTRY *entry, int which) QMGR_QUEUE *queue = entry->queue; QMGR_MESSAGE *message = entry->message; QMGR_PEER *peer = entry->peer; - QMGR_JOB *sponsor, - *job = peer->job; + QMGR_JOB *sponsor, *job = peer->job; QMGR_TRANSPORT *transport = job->transport; /* @@ -327,6 +385,7 @@ QMGR_ENTRY *qmgr_entry_create(QMGR_PEER *peer, QMGR_MESSAGE *message) entry->queue = queue; QMGR_LIST_APPEND(queue->todo, entry, queue_peers); queue->todo_refcount++; + peer->job->read_entries++; /* * Warn if a destination is falling behind while the active queue diff --git a/postfix/src/qmgr/qmgr_error.c b/postfix/src/qmgr/qmgr_error.c new file mode 100644 index 000000000..6d07b3bd8 --- /dev/null +++ b/postfix/src/qmgr/qmgr_error.c @@ -0,0 +1,121 @@ +/*++ +/* NAME +/* qmgr_error 3 +/* SUMMARY +/* look up/create error/retry queue +/* SYNOPSIS +/* #include "qmgr.h" +/* +/* QMGR_TRANSPORT *qmgr_error_transport(service) +/* const char *service; +/* +/* QMGR_QUEUE *qmgr_error_queue(service, dsn) +/* const char *service; +/* DSN *dsn; +/* +/* char *qmgr_error_nexthop(dsn) +/* DSN *dsn; +/* DESCRIPTION +/* qmgr_error_transport() looks up the error transport for the +/* specified service. The result is null if the transport is +/* not available. +/* +/* qmgr_error_queue() looks up an error queue for the specified +/* service and problem. The result is null if the queue is not +/* availabe. +/* +/* qmgr_error_nexthop() computes the next-hop information for +/* the specified problem. The result must be passed to myfree(). +/* +/* Arguments: +/* .IP dsn +/* See dsn(3). +/* .IP service +/* One of MAIL_SERVICE_ERROR or MAIL_SERVICE_RETRY. +/* DIAGNOSTICS +/* Panic: consistency check failure. Fatal: out of memory. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +/* System library. */ + +#include + +/* Utility library. */ + +#include +#include + +/* Global library. */ + +/* Application-specific. */ + +#include "qmgr.h" + +/* qmgr_error_transport - look up error transport for specified service */ + +QMGR_TRANSPORT *qmgr_error_transport(const char *service) +{ + QMGR_TRANSPORT *transport; + + /* + * Find or create retry transport. + */ + if ((transport = qmgr_transport_find(service)) == 0) + transport = qmgr_transport_create(service); + if (QMGR_TRANSPORT_THROTTLED(transport)) + return (0); + + /* + * Done. + */ + return (transport); +} + +/* qmgr_error_queue - look up error queue for specified service and problem */ + +QMGR_QUEUE *qmgr_error_queue(const char *service, DSN *dsn) +{ + QMGR_TRANSPORT *transport; + QMGR_QUEUE *queue; + char *nexthop; + + /* + * Find or create transport. + */ + if ((transport = qmgr_error_transport(service)) == 0) + return (0); + + /* + * Find or create queue. + */ + nexthop = qmgr_error_nexthop(dsn); + if ((queue = qmgr_queue_find(transport, nexthop)) == 0) + queue = qmgr_queue_create(transport, nexthop, nexthop); + myfree(nexthop); + if (QMGR_QUEUE_THROTTLED(queue)) + return (0); + + /* + * Done. + */ + return (queue); +} + +/* qmgr_error_nexthop - compute next-hop information from problem description */ + +char *qmgr_error_nexthop(DSN *dsn) +{ + char *nexthop; + + nexthop = concatenate(dsn->status, " ", dsn->reason, (char *) 0); + return (nexthop); +} diff --git a/postfix/src/qmgr/qmgr_job.c b/postfix/src/qmgr/qmgr_job.c index 6318893f4..ad727f43a 100644 --- a/postfix/src/qmgr/qmgr_job.c +++ b/postfix/src/qmgr/qmgr_job.c @@ -26,7 +26,7 @@ /* /* qmgr_job_obtain() finds an existing job for named message and /* transport combination. New empty job is created if no existing can -/* be found. In either case, the job is prepared for assignement of +/* be found. In either case, the job is prepared for assignment of /* (more) message recipients. /* /* qmgr_job_free() disposes of a per-transport job after all @@ -138,9 +138,9 @@ static void qmgr_job_link(QMGR_JOB *job) /* * Traverse the time list and the scheduler list from the end and stop - * when we found job older than the one beeing linked. + * when we found job older than the one being linked. * - * During the traversals keep track if we have come accross either the + * During the traversals keep track if we have come across either the * current job or the first unread job on the job list. If this is the * case, these pointers will be adjusted below as required. * @@ -575,7 +575,7 @@ static QMGR_JOB *qmgr_job_preempt(QMGR_JOB *current) /* * Suppress preempting completely if the current job is not big enough to - * accumulate even the mimimal number of slots required. + * accumulate even the minimal number of slots required. * * Also, don't look for better job candidate if there are no available slots * yet (the count can get negative due to the slot loans below). @@ -733,7 +733,7 @@ static void qmgr_job_pop(QMGR_JOB *job) job->stack_level = 0; /* - * Explicitely reset the candidate cache. It's not worth trying to skip + * Explicitly reset the candidate cache. It's not worth trying to skip * this under some complicated conditions - in most cases the popped job * is the current job so we would have to reset it anyway. */ @@ -764,34 +764,51 @@ static QMGR_PEER *qmgr_job_peer_select(QMGR_JOB *job) QMGR_MESSAGE *message = job->message; /* - * Workaround to prevent queue manager starvation until the last slow - * batch of multi-recipient mail is finished. Postfix 2.4 has a final - * solution, but that involves major changes. + * Try reading in more recipients. We do that as soon as possible + * (almost, see below), to make sure there is enough new blood pouring + * in. Otherwise single recipient for slow destination might starve the + * entire message delivery, leaving lot of fast destination recipients + * sitting idle in the queue file. + * + * Ideally we would like to read in recipients whenever there is a + * space, but to prevent excessive I/O, we read them only when enough + * time has passed or we can read enough of them at once. + * + * Note that even if we read the recipients few at a time, the message + * loading code tries to put them to existing recipient entries whenever + * possible, so the per-destination recipient grouping is not grossly + * affected. + * + * XXX Workaround for logic mismatch. The message->refcount test needs + * explanation. If the refcount is zero, it means that qmgr_active_done() + * is being completed asynchronously. In such case, we can't read in + * more recipients as bad things would happen after qmgr_active_done() + * continues processing. Note that this results in the given job being + * stalled for some time, but fortunately this particular situation is so + * rare that it is not critical. Still we seek for better solution. */ if (message->rcpt_offset != 0 - && message->rcpt_limit > message->rcpt_count + 100 - && message->refcount > 0) { + && message->refcount > 0 + && (message->rcpt_limit - message->rcpt_count >= job->transport->refill_limit + || (message->rcpt_limit > message->rcpt_count + && sane_time() - message->refill_time >= job->transport->refill_delay))) qmgr_message_realloc(message); - } + + /* + * Get the next suitable peer, if there is any. + */ if (HAS_ENTRIES(job) && (peer = qmgr_peer_select(job)) != 0) return (peer); /* - * Try reading in more recipients. Note that we do not try to read them - * as soon as possible as that would decrease the chance of per-site - * recipient grouping. We waited until reading more is really necessary. + * There is no suitable peer in-core, so try reading in more recipients if possible. + * This is our last chance to get suitable peer before giving up on this job for now. * - * XXX Workaround for logic mismatch. The message->refcount test needs - * explanation. If the refcount is zero, it means that qmgr_active_done() - * is beeing completed asynchronously. In such case, we can't read in - * more recipients as bad things would happen after qmgr_active_done() - * continues processing. Note that this results in the given job beeing - * stalled for some time, but fortunately this particular situation is so - * rare that it is not critical. Still we seek for better solution. + * XXX For message->refcount, see above. */ if (message->rcpt_offset != 0 - && message->rcpt_limit > message->rcpt_count - && message->refcount > 0) { + && message->refcount > 0 + && message->rcpt_limit > message->rcpt_count) { qmgr_message_realloc(message); if (HAS_ENTRIES(job)) return (qmgr_peer_select(job)); diff --git a/postfix/src/qmgr/qmgr_message.c b/postfix/src/qmgr/qmgr_message.c index dcb796df0..b2567460d 100644 --- a/postfix/src/qmgr/qmgr_message.c +++ b/postfix/src/qmgr/qmgr_message.c @@ -169,6 +169,7 @@ static QMGR_MESSAGE *qmgr_message_create(const char *queue_name, message->create_time = 0; GETTIMEOFDAY(&message->active_time); message->queued_time = sane_time(); + message->refill_time = 0; message->data_offset = 0; message->queue_id = mystrdup(queue_id); message->queue_name = mystrdup(queue_name); @@ -366,6 +367,8 @@ static int qmgr_message_read(QMGR_MESSAGE *message) recipient_limit = 5000; if (recipient_limit <= 0) msg_panic("%s: no recipient slots available", message->queue_id); + if (msg_verbose) + msg_info("%s: recipient limit %d", message->queue_id, recipient_limit); /* * Read envelope records. XXX Rely on the front-end programs to enforce @@ -746,6 +749,12 @@ static int qmgr_message_read(QMGR_MESSAGE *message) message->queue_id, orig_rcpt); myfree(orig_rcpt); } + + /* + * Remember when we have read the last recipient batch. Note that we do + * it here after reading as reading might have used considerable amount of time. + */ + message->refill_time = sane_time(); /* * Avoid clumsiness elsewhere in the program. When sending data across an @@ -924,22 +933,24 @@ static void qmgr_message_sort(QMGR_MESSAGE *message) static int qmgr_resolve_one(QMGR_MESSAGE *message, RECIPIENT *recipient, const char *addr, RESOLVE_REPLY *reply) { - DSN dsn; +#define QMGR_REDIRECT(rp, tp, np) do { \ + (rp)->flags = 0; \ + vstring_strcpy((rp)->transport, (tp)); \ + vstring_strcpy((rp)->nexthop, (np)); \ + } while (0) if ((message->tflags & DEL_REQ_FLAG_MTA_VRFY) == 0) resolve_clnt_query_from(message->sender, addr, reply); else resolve_clnt_verify_from(message->sender, addr, reply); if (reply->flags & RESOLVE_FLAG_FAIL) { - qmgr_defer_recipient(message, recipient, - DSN_SIMPLE(&dsn, "4.3.0", - "address resolver failure")); - return (-1); + QMGR_REDIRECT(reply, MAIL_SERVICE_RETRY, + "4.3.0 address resolver failure"); + return (0); } else if (reply->flags & RESOLVE_FLAG_ERROR) { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "bad address syntax")); - return (-1); + QMGR_REDIRECT(reply, MAIL_SERVICE_ERROR, + "5.1.3 bad address syntax"); + return (0); } else { return (0); } @@ -963,6 +974,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) int status; DSN dsn; MSG_STATS stats; + DSN *saved_dsn; #define STREQ(x,y) (strcmp(x,y) == 0) #define STR vstring_str @@ -973,8 +985,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) for (recipient = list.info; recipient < list.info + list.len; recipient++) { /* - * Redirect overrides all else. But only once (per batch of - * recipients). For consistency with the remainder of Postfix, + * Redirect overrides all else. But only once (per entire + * message). For consistency with the remainder of Postfix, * rewrite the address to canonical form before resolving it. */ if (message->redirect_addr) { @@ -982,6 +994,10 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) recipient->u.queue = 0; continue; } + + message->rcpt_offset = 0; + message->rcpt_unread = 0; + rewrite_clnt_internal(REWRITE_CANON, message->redirect_addr, reply.recipient); RECIPIENT_UPDATE(recipient->address, STR(reply.recipient)); @@ -1029,10 +1045,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) * the queue manager process does not help. */ if (recipient->address[0] == 0) { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "null recipient address")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_ERROR, + "5.1.3 null recipient address"); } /* @@ -1047,10 +1061,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) * where it cannot be bypassed. */ if (var_allow_min_user == 0 && recipient->address[0] == '-') { - qmgr_bounce_recipient(message, recipient, - DSN_SIMPLE(&dsn, "5.1.3", - "bad address syntax")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_ERROR, + "5.1.3 bad address syntax"); } /* @@ -1094,10 +1106,8 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) if (strcmp(*cpp, STR(reply.transport)) == 0) break; if (*cpp) { - qmgr_defer_recipient(message, recipient, - DSN_SIMPLE(&dsn, "4.3.2", - "deferred transport")); - continue; + QMGR_REDIRECT(&reply, MAIL_SERVICE_RETRY, + "4.3.2 deferred transport"); } } @@ -1113,9 +1123,17 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) /* * This transport is dead. Defer delivery to this recipient. */ - if ((transport->flags & QMGR_TRANSPORT_STAT_DEAD) != 0) { - qmgr_defer_recipient(message, recipient, transport->dsn); - continue; + if (QMGR_TRANSPORT_THROTTLED(transport)) { + saved_dsn = transport->dsn; + if ((transport = qmgr_error_transport(MAIL_SERVICE_RETRY)) != 0) { + nexthop = qmgr_error_nexthop(saved_dsn); + vstring_strcpy(reply.nexthop, nexthop); + myfree(nexthop); + queue = 0; + } else { + qmgr_defer_recipient(message, recipient, saved_dsn); + continue; + } } /* @@ -1153,6 +1171,7 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) */ vstring_strcpy(queue_name, STR(reply.nexthop)); if (strcmp(transport->name, MAIL_SERVICE_ERROR) != 0 + && strcmp(transport->name, MAIL_SERVICE_RETRY) != 0 && transport->recipient_limit == 1) { /* Copy the recipient localpart. */ at = strrchr(STR(reply.recipient), '@'); @@ -1180,9 +1199,12 @@ static void qmgr_message_resolve(QMGR_MESSAGE *message) /* * This queue is dead. Defer delivery to this recipient. */ - if (queue->window == 0) { - qmgr_defer_recipient(message, recipient, queue->dsn); - continue; + if (QMGR_QUEUE_THROTTLED(queue)) { + saved_dsn = queue->dsn; + if ((queue = qmgr_error_queue(MAIL_SERVICE_RETRY, saved_dsn)) == 0) { + qmgr_defer_recipient(message, recipient, saved_dsn); + continue; + } } /* @@ -1207,55 +1229,55 @@ static void qmgr_message_assign(QMGR_MESSAGE *message) /* * Try to bundle as many recipients in a delivery request as we can. When - * the recipient resolves to the same site and transport as the previous + * the recipient resolves to the same site and transport as an existing * recipient, do not create a new queue entry, just move that recipient * to the recipient list of the existing queue entry. All this provided * that we do not exceed the transport-specific limit on the number of - * recipients per transaction. Skip recipients with a dead transport or - * destination. + * recipients per transaction. */ #define LIMIT_OK(limit, count) ((limit) == 0 || ((count) < (limit))) for (recipient = list.info; recipient < list.info + list.len; recipient++) { - if ((queue = recipient->u.queue) != 0) { - if (message->single_rcpt || entry == 0 || entry->queue != queue - || !LIMIT_OK(queue->transport->recipient_limit, - entry->rcpt_list.len)) { - - /* - * Lookup or instantiate the message job if necessary. - */ - if (job == 0 || queue->transport != job->transport) { - job = qmgr_job_obtain(message, queue->transport); - peer = 0; - } - /* - * Lookup or instantiate job peer if necessary. - */ - if (peer == 0 || queue != peer->queue) { - if ((peer = qmgr_peer_find(job, queue)) == 0) - peer = qmgr_peer_create(job, queue); - } + /* + * Skip recipients with a dead transport or destination. + */ + if ((queue = recipient->u.queue) == 0) + continue; + + /* + * Lookup or instantiate the message job if necessary. + */ + if (job == 0 || queue->transport != job->transport) { + job = qmgr_job_obtain(message, queue->transport); + peer = 0; + } - /* - * Create new peer entry. - */ - entry = qmgr_entry_create(peer, message); - job->read_entries++; - } + /* + * Lookup or instantiate job peer if necessary. + */ + if (peer == 0 || queue != peer->queue) + peer = qmgr_peer_obtain(job, queue); + + /* + * Lookup old or instantiate new recipient entry. We try to reuse + * the last existing entry whenever the recipient limit permits. + */ + entry = peer->entry_list.prev; + if (message->single_rcpt || entry == 0 + || !LIMIT_OK(queue->transport->recipient_limit, entry->rcpt_list.len)) + entry = qmgr_entry_create(peer, message); - /* - * Add the recipient to the current entry and increase all those - * recipient counters accordingly. - */ - recipient_list_add(&entry->rcpt_list, recipient->offset, - recipient->dsn_orcpt, recipient->dsn_notify, - recipient->orig_addr, recipient->address); - job->rcpt_count++; - message->rcpt_count++; - qmgr_recipient_count++; - } + /* + * Add the recipient to the current entry and increase all those + * recipient counters accordingly. + */ + recipient_list_add(&entry->rcpt_list, recipient->offset, + recipient->dsn_orcpt, recipient->dsn_notify, + recipient->orig_addr, recipient->address); + job->rcpt_count++; + message->rcpt_count++; + qmgr_recipient_count++; } /* diff --git a/postfix/src/qmgr/qmgr_peer.c b/postfix/src/qmgr/qmgr_peer.c index 75ee14d5b..2ac020cb0 100644 --- a/postfix/src/qmgr/qmgr_peer.c +++ b/postfix/src/qmgr/qmgr_peer.c @@ -14,6 +14,10 @@ /* QMGR_JOB *job; /* QMGR_QUEUE *queue; /* +/* QMGR_PEER *qmgr_peer_obtain(job, queue) +/* QMGR_JOB *job; +/* QMGR_QUEUE *queue; +/* /* void qmgr_peer_free(peer) /* QMGR_PEER *peer; /* @@ -22,7 +26,7 @@ /* /* DESCRIPTION /* These routines add/delete/manipulate per-job peers. -/* Each queue corresponds to a specific job and destination. +/* Each peer corresponds to a specific job and destination. /* It is similar to per-transport queue structure, but groups /* only the entries of the given job. /* @@ -34,6 +38,9 @@ /* for the named job. A null result means that the peer /* was not found. /* +/* qmgr_peer_obtain() looks up the peer for the named destination +/* for the named job. If it doesn't exist yet, it creates it. +/* /* qmgr_peer_free() disposes of a per-job peer after all /* its entries have been taken care of. It is an error to dispose /* of a peer still in use. @@ -110,6 +117,17 @@ QMGR_PEER *qmgr_peer_find(QMGR_JOB *job, QMGR_QUEUE *queue) return ((QMGR_PEER *) htable_find(job->peer_byname, queue->name)); } +/* qmgr_peer_obtain - find/create peer associated with given job and queue */ + +QMGR_PEER *qmgr_peer_obtain(QMGR_JOB *job, QMGR_QUEUE *queue) +{ + QMGR_PEER *peer; + + if ((peer = qmgr_peer_find(job, queue)) == 0) + peer = qmgr_peer_create(job, queue); + return (peer); +} + /* qmgr_peer_select - select next peer suitable for delivery within given job */ QMGR_PEER *qmgr_peer_select(QMGR_JOB *job) diff --git a/postfix/src/qmgr/qmgr_transport.c b/postfix/src/qmgr/qmgr_transport.c index ce4de744f..d7e54f116 100644 --- a/postfix/src/qmgr/qmgr_transport.c +++ b/postfix/src/qmgr/qmgr_transport.c @@ -198,7 +198,8 @@ static void qmgr_transport_event(int unused_event, char *context) /* * Disable further read events that end up calling this function. */ - event_disable_readwrite(vstream_fileno(alloc->stream)); + if (alloc->stream) + event_disable_readwrite(vstream_fileno(alloc->stream)); alloc->transport->flags &= ~QMGR_TRANSPORT_STAT_BUSY; /* @@ -264,7 +265,6 @@ void qmgr_transport_alloc(QMGR_TRANSPORT *transport, QMGR_TRANSPORT_ALLOC_NOT { QMGR_TRANSPORT_ALLOC *alloc; VSTREAM *stream; - DSN dsn; /* * Sanity checks. @@ -291,18 +291,28 @@ void qmgr_transport_alloc(QMGR_TRANSPORT *transport, QMGR_TRANSPORT_ALLOC_NOT #define EVENT_HANDLER qmgr_transport_event #endif + /* + * When the connection to the delivery agent cannot be completed, notify + * the event handler so that it can throttle the transport and defer the + * todo queues, just like it does when communication fails *after* + * connection completion. + * + * Before Postfix 2.4, the event handler was not invoked, and mail was not + * deferred. Because of this, mail would be stuck in the active queue + * after triggering a "connection refused" condition. + */ if ((stream = mail_connect(MAIL_CLASS_PRIVATE, transport->name, BLOCK_MODE)) == 0) { msg_warn("connect to transport %s: %m", transport->name); - qmgr_transport_throttle(transport, - DSN_SIMPLE(&dsn, "4.3.0", - "mail transport unavailable")); - return; } alloc = (QMGR_TRANSPORT_ALLOC *) mymalloc(sizeof(*alloc)); alloc->stream = stream; alloc->transport = transport; alloc->notify = notify; transport->flags |= QMGR_TRANSPORT_STAT_BUSY; + if (alloc->stream == 0) { + event_request_timer(qmgr_transport_event, (char *) alloc, 0); + return; + } ENABLE_EVENTS(vstream_fileno(alloc->stream), EVENT_HANDLER, (char *) alloc); /* @@ -353,6 +363,10 @@ QMGR_TRANSPORT *qmgr_transport_create(const char *name) var_xport_rcpt_limit, 0, 0); transport->rcpt_per_stack = get_mail_conf_int2(name, _STACK_RCPT_LIMIT, var_stack_rcpt_limit, 0, 0); + transport->refill_limit = get_mail_conf_int2(name, _XPORT_REFILL_LIMIT, + var_xport_refill_limit, 1, 0); + transport->refill_delay = get_mail_conf_time2(name, _XPORT_REFILL_DELAY, + var_xport_refill_delay, 's', 1, 0); transport->queue_byname = htable_create(0); QMGR_LIST_INIT(transport->queue_list); diff --git a/postfix/src/smtp/smtp_chat.c b/postfix/src/smtp/smtp_chat.c index 82ea27564..252924f92 100644 --- a/postfix/src/smtp/smtp_chat.c +++ b/postfix/src/smtp/smtp_chat.c @@ -305,7 +305,8 @@ SMTP_RESP *smtp_chat_resp(SMTP_SESSION *session) */ session->error_mask |= MAIL_ERROR_PROTOCOL; if (session->features & SMTP_FEATURE_PIPELINING) { - msg_warn("non-%s response from %s: %.100s", + msg_warn("%s: non-%s response from %s: %.100s", + session->state->request->queue_id, (session->state->misc_flags & SMTP_MISC_FLAG_USE_LMTP) ? "LMTP" : "ESMTP", session->namaddrport, STR(session->buffer)); diff --git a/postfix/src/smtp/smtp_proto.c b/postfix/src/smtp/smtp_proto.c index bdd46366c..1f4912bf0 100644 --- a/postfix/src/smtp/smtp_proto.c +++ b/postfix/src/smtp/smtp_proto.c @@ -775,7 +775,7 @@ static int smtp_start_tls(SMTP_STATE *state) /* * We must avoid further I/O, the peer is in an undefined state. */ - (void) vstream_fpurge(session->stream); + (void) vstream_fpurge(session->stream, VSTREAM_PURGE_BOTH); DONT_USE_DEAD_SESSION; /* @@ -1041,7 +1041,7 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, } \ } while (0) - /* Caution: changes to RETURN() also affect code outside the main loop. */ + /* Caution: changes to RETURN() also affect code outside the main loop. */ #define RETURN(x) do { \ if (recv_state != SMTP_STATE_LAST) \ @@ -1382,12 +1382,36 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, /* * Receive the next server response. Use the proper timeout, * and log the proper client state in case of trouble. + * + * XXX If we lose the connection before sending end-of-data, + * find out if the server sent a premature end-of-data reply. + * If this read attempt fails, report "lost connection while + * sending message body", not "lost connection while sending + * end-of-data". + * + * "except" becomes zero just above the protocol loop, and stays + * zero or triggers an early return from the loop. In just + * one case: loss of the connection when sending the message + * body, we record the exception, and keep processing in the + * hope of detecting a premature 5XX. We must be careful to + * not clobber this non-zero value once it is set. The + * variable need not survive longjmp() calls, since the only + * setjmp() which does not return early is the one sets this + * condition, subquent failures always return early. */ +#define LOST_CONNECTION_INSIDE_DATA (except == SMTP_ERR_EOF) + smtp_timeout_setup(session->stream, *xfer_timeouts[recv_state]); - if ((except = vstream_setjmp(session->stream)) != 0) - RETURN(SENDING_MAIL ? smtp_stream_except(state, except, + if (LOST_CONNECTION_INSIDE_DATA) { + if (vstream_setjmp(session->stream) != 0) + RETURN(smtp_stream_except(state, SMTP_ERR_EOF, + "sending message body")); + } else { + if ((except = vstream_setjmp(session->stream)) != 0) + RETURN(SENDING_MAIL ? smtp_stream_except(state, except, xfer_states[recv_state]) : -1); + } resp = smtp_chat_resp(session); /* @@ -1569,8 +1593,11 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, * otherwise the sender and receiver loops get out of * sync. The caller will call smtp_quit() if appropriate. */ - recv_state = (var_skip_quit_resp || THIS_SESSION_IS_CACHED ? - SMTP_STATE_LAST : SMTP_STATE_QUIT); + if (var_skip_quit_resp || THIS_SESSION_IS_CACHED + || LOST_CONNECTION_INSIDE_DATA) + recv_state = SMTP_STATE_LAST; + else + recv_state = SMTP_STATE_QUIT; break; /* @@ -1658,98 +1685,127 @@ static int smtp_loop(SMTP_STATE *state, NOCLOBBER int send_state, * transaction in progress. */ if (send_state == SMTP_STATE_DOT && nrcpt > 0) { - downgrading = SMTP_MIME_DOWNGRADE(session, request); - /* XXX Don't downgrade just because generic_maps is turned on. */ - if (downgrading || smtp_generic_maps) - session->mime_state = mime_state_alloc(downgrading ? - MIME_OPT_DOWNGRADE + + smtp_timeout_setup(session->stream, + var_smtp_data1_tmout); + + if ((except = vstream_setjmp(session->stream)) == 0) { + + if (vstream_fseek(state->src, request->data_offset, SEEK_SET) < 0) + msg_fatal("seek queue file: %m"); + + downgrading = SMTP_MIME_DOWNGRADE(session, request); + + /* + * XXX Don't downgrade just because generic_maps is turned + * on. + */ + if (downgrading || smtp_generic_maps) + session->mime_state = mime_state_alloc(downgrading ? + MIME_OPT_DOWNGRADE | MIME_OPT_REPORT_NESTING : - MIME_OPT_REPORT_NESTING, - smtp_generic_maps ? + MIME_OPT_DISABLE_MIME, + smtp_generic_maps ? smtp_header_rewrite : - smtp_header_out, + smtp_header_out, (MIME_STATE_ANY_END) 0, - smtp_text_out, + smtp_text_out, (MIME_STATE_ANY_END) 0, (MIME_STATE_ERR_PRINT) 0, - (void *) state); - state->space_left = var_smtp_line_limit; - smtp_timeout_setup(session->stream, - var_smtp_data1_tmout); - if ((except = vstream_setjmp(session->stream)) != 0) - RETURN(smtp_stream_except(state, except, - "sending message body")); + (void *) state); + state->space_left = var_smtp_line_limit; - if (vstream_fseek(state->src, request->data_offset, SEEK_SET) < 0) - msg_fatal("seek queue file: %m"); + while ((rec_type = rec_get(state->src, session->scratch, 0)) > 0) { + if (rec_type != REC_TYPE_NORM && rec_type != REC_TYPE_CONT) + break; + if (session->mime_state == 0) { + smtp_text_out((void *) state, rec_type, + vstring_str(session->scratch), + VSTRING_LEN(session->scratch), + (off_t) 0); + } else { + mime_errs = + mime_state_update(session->mime_state, rec_type, + vstring_str(session->scratch), + VSTRING_LEN(session->scratch)); + if (mime_errs) { + smtp_mime_fail(state, mime_errs); + RETURN(0); + } + } + prev_type = rec_type; + } - while ((rec_type = rec_get(state->src, session->scratch, 0)) > 0) { - if (rec_type != REC_TYPE_NORM && rec_type != REC_TYPE_CONT) - break; - if (session->mime_state == 0) { - smtp_text_out((void *) state, rec_type, - vstring_str(session->scratch), - VSTRING_LEN(session->scratch), - (off_t) 0); - } else { + if (session->mime_state) { + + /* + * The cleanup server normally ends MIME content with a + * normal text record. The following code is needed to + * flush an internal buffer when someone submits 8-bit + * mail not ending in newline via /usr/sbin/sendmail + * while MIME input processing is turned off, and MIME + * 8bit->7bit conversion is requested upon delivery. + * + * Or some error while doing generic address mapping. + */ mime_errs = - mime_state_update(session->mime_state, rec_type, - vstring_str(session->scratch), - VSTRING_LEN(session->scratch)); + mime_state_update(session->mime_state, rec_type, "", 0); if (mime_errs) { smtp_mime_fail(state, mime_errs); RETURN(0); } + } else if (prev_type == REC_TYPE_CONT) /* missing newline */ + smtp_fputs("", 0, session->stream); + if ((session->features & SMTP_FEATURE_PIX_DELAY_DOTCRLF) != 0 + && request->msg_stats.incoming_arrival.tv_sec + <= vstream_ftime(session->stream) - var_smtp_pix_thresh) { + smtp_flush(session->stream);/* hurts performance */ + sleep(var_smtp_pix_delay); /* not to mention this */ } - prev_type = rec_type; - } - - if (session->mime_state) { + if (vstream_ferror(state->src)) + msg_fatal("queue file read error"); + if (rec_type != REC_TYPE_XTRA) { + msg_warn("%s: bad record type: %d in message content", + request->queue_id, rec_type); + fail_status = smtp_mesg_fail(state, DSN_BY_LOCAL_MTA, + SMTP_RESP_FAKE(&fake, "5.3.0"), + "unreadable mail queue entry"); + if (fail_status == 0) + (void) mark_corrupt(state->src); + RETURN(fail_status); + } + } else { + if (!LOST_CONNECTION_INSIDE_DATA) + RETURN(smtp_stream_except(state, except, + "sending message body")); /* - * The cleanup server normally ends MIME content with a - * normal text record. The following code is needed to flush - * an internal buffer when someone submits 8-bit mail not - * ending in newline via /usr/sbin/sendmail while MIME input - * processing is turned off, and MIME 8bit->7bit conversion - * is requested upon delivery. + * We will clear the stream error flag to try and read a + * premature 5XX response, so it is important to flush any + * unwritten data. Otherwise, we will try to flush it again + * before reading, which may incur an unnecessary delay and + * will prevent the reading of any response that is not + * already buffered (bundled with the DATA 354 response). * - * Or some error while doing generic address mapping. + * Not much point in sending QUIT at this point, skip right to + * SMTP_STATE_LAST. The read engine above will likewise avoid + * looking for a QUIT response. */ - mime_errs = - mime_state_update(session->mime_state, rec_type, "", 0); - if (mime_errs) { - smtp_mime_fail(state, mime_errs); - RETURN(0); - } - } else if (prev_type == REC_TYPE_CONT) /* missing newline */ - smtp_fputs("", 0, session->stream); - if ((session->features & SMTP_FEATURE_PIX_DELAY_DOTCRLF) != 0 - && request->msg_stats.incoming_arrival.tv_sec - <= vstream_ftime(session->stream) - var_smtp_pix_thresh) { - smtp_flush(session->stream); /* hurts performance */ - sleep(var_smtp_pix_delay); /* not to mention this */ - } - if (vstream_ferror(state->src)) - msg_fatal("queue file read error"); - if (rec_type != REC_TYPE_XTRA) { - msg_warn("%s: bad record type: %d in message content", - request->queue_id, rec_type); - fail_status = smtp_mesg_fail(state, DSN_BY_LOCAL_MTA, - SMTP_RESP_FAKE(&fake, "5.3.0"), - "unreadable mail queue entry"); - if (fail_status == 0) - (void) mark_corrupt(state->src); - RETURN(fail_status); + (void) vstream_fpurge(session->stream, VSTREAM_PURGE_WRITE); + next_state = SMTP_STATE_LAST; } } /* * Copy the next command to the buffer and update the sender state. */ - if (sndbuffree > 0) - sndbuffree -= VSTRING_LEN(next_command) + 2; - smtp_chat_cmd(session, "%s", vstring_str(next_command)); + if (except == 0) { + if (sndbuffree > 0) + sndbuffree -= VSTRING_LEN(next_command) + 2; + smtp_chat_cmd(session, "%s", vstring_str(next_command)); + } else { + DONT_CACHE_THIS_SESSION; + } send_state = next_state; send_rcpt = next_rcpt; } while (recv_state != SMTP_STATE_LAST); diff --git a/postfix/src/smtpstone/Makefile.in b/postfix/src/smtpstone/Makefile.in index fd86e89e1..6de57d175 100644 --- a/postfix/src/smtpstone/Makefile.in +++ b/postfix/src/smtpstone/Makefile.in @@ -114,19 +114,14 @@ qmqp-source.o: ../../include/vbuf.h qmqp-source.o: ../../include/vstream.h qmqp-source.o: ../../include/vstring.h qmqp-source.o: qmqp-source.c -smtp-sink.o: ../../include/chroot_uid.h smtp-sink.o: ../../include/events.h smtp-sink.o: ../../include/get_hostname.h smtp-sink.o: ../../include/inet_proto.h smtp-sink.o: ../../include/iostuff.h smtp-sink.o: ../../include/listen.h -smtp-sink.o: ../../include/mail_date.h -smtp-sink.o: ../../include/make_dirs.h smtp-sink.o: ../../include/msg.h smtp-sink.o: ../../include/msg_vstream.h -smtp-sink.o: ../../include/myaddrinfo.h smtp-sink.o: ../../include/mymalloc.h -smtp-sink.o: ../../include/myrand.h smtp-sink.o: ../../include/sane_accept.h smtp-sink.o: ../../include/smtp_stream.h smtp-sink.o: ../../include/stringops.h diff --git a/postfix/src/smtpstone/smtp-sink.c b/postfix/src/smtpstone/smtp-sink.c index f89fae7e7..feb8e13bb 100644 --- a/postfix/src/smtpstone/smtp-sink.c +++ b/postfix/src/smtpstone/smtp-sink.c @@ -39,6 +39,12 @@ /* Do not announce 8BITMIME support. /* .IP \fB-a\fR /* Do not announce SASL authentication support. +/* .IP "\fB-A \fIdelay\fR" +/* Wait \fIdelay\fR seconds after responding to DATA, then +/* abort prematurely with a 550 reply status. Do not read +/* further input from the client; this is an attempt to block +/* the client before it sends ".". Specify a zero delay value +/* to abort immediately. /* .IP \fB-c\fR /* Display running counters that are updated whenever an SMTP /* session ends, a QUIT command is executed, or when "." is @@ -254,6 +260,7 @@ #include #include #include +#include /* Global library. */ @@ -301,6 +308,8 @@ static char *var_myhostname; static int command_read(SINK_STATE *); static int data_read(SINK_STATE *); static void disconnect(SINK_STATE *); +static void read_timeout(int, char *); +static void read_event(int, char *); static int count; static int sess_count; static int quit_count; @@ -319,6 +328,7 @@ static int disable_enh_status; static int max_client_count = DEF_MAX_CLIENT_COUNT; static int client_count; static int sock; +static int abort_delay = -1; static char *single_template; /* individual template */ static char *shared_template; /* shared template */ @@ -659,6 +669,17 @@ static void rcpt_response(SINK_STATE *state, const char *args) } } +/* abort_event - delayed abort after DATA command */ + +static void abort_event(int unused_event, char *context) +{ + SINK_STATE *state = (SINK_STATE *) context; + + smtp_printf(state->stream, "550 This violates SMTP"); + smtp_flush(state->stream); + disconnect(state); +} + /* data_response - respond to DATA command */ static void data_response(SINK_STATE *state, const char *unused_args) @@ -672,7 +693,14 @@ static void data_response(SINK_STATE *state, const char *unused_args) state->data_state = ST_CR_LF; smtp_printf(state->stream, "354 End data with ."); smtp_flush(state->stream); - state->read_fn = data_read; + if (abort_delay < 0) { + state->read_fn = data_read; + } else { + /* Stop reading, send premature 550, and disconnect. */ + event_disable_readwrite(vstream_fileno(state->stream)); + event_cancel_timer(read_event, (char *) state); + event_request_timer(abort_event, (char *) state, abort_delay); + } if (state->dump_file) mail_file_finish_header(state); } @@ -684,6 +712,9 @@ static void data_event(int unused_event, char *context) SINK_STATE *state = (SINK_STATE *) context; data_response(state, ""); + /* Resume input event handling after the delayed DATA response. */ + event_enable_read(vstream_fileno(state->stream), read_event, (char *) state); + event_request_timer(read_timeout, (char *) state, var_tmout); } /* dot_resp_hard - hard error response to . command */ @@ -917,6 +948,9 @@ static int command_resp(SINK_STATE *state, SINK_COMMAND *cmdp, return (0); } if (cmdp->response == data_response && fixed_delay > 0) { + /* Suspend input event handling while delaying the DATA response. */ + event_disable_readwrite(vstream_fileno(state->stream)); + event_cancel_timer(read_timeout, (char *) state); event_request_timer(data_event, (char *) state, fixed_delay); } else { cmdp->response(state, args); @@ -1158,7 +1192,13 @@ static void connect_event(int unused_event, char *unused_context) state->in_mail = 0; state->rcpts = 0; /* Initialize file capture attributes. */ - state->addr_prefix = (sa.sa_family == AF_INET6 ? "ipv6:" : ""); +#ifdef AF_INET6 + if (sa.sa_family == AF_INET6) + state->addr_prefix = "ipv6:"; + else +#endif + state->addr_prefix = ""; + state->helo_args = 0; state->client_proto = enable_lmtp ? "LMTP" : "SMTP"; state->start_time = 0; @@ -1202,7 +1242,7 @@ static void connect_event(int unused_event, char *unused_context) static void usage(char *myname) { - msg_fatal("usage: %s [-468acCeEFLpPv] [-f commands] [-h hostname] [-m max_concurrency] [-n quit_count] [-q commands] [-r commands] [-s commands] [-w delay] [-d dump-template] [-D dump-template] [-R root-dir] [-S start-string] [-u user_privs] [host]:port backlog", myname); + msg_fatal("usage: %s [-468acCeEFLpPv] [-A abort_delay] [-f commands] [-h hostname] [-m max_concurrency] [-n quit_count] [-q commands] [-r commands] [-s commands] [-w delay] [-d dump-template] [-D dump-template] [-R root-dir] [-S start-string] [-u user_privs] [host]:port backlog", myname); } int main(int argc, char **argv) @@ -1227,7 +1267,7 @@ int main(int argc, char **argv) /* * Parse JCL. */ - while ((ch = GETOPT(argc, argv, "468acCd:D:eEf:Fh:Ln:m:pPq:r:R:s:S:t:u:vw:")) > 0) { + while ((ch = GETOPT(argc, argv, "468aA:cCd:D:eEf:Fh:Ln:m:pPq:r:R:s:S:t:u:vw:")) > 0) { switch (ch) { case '4': protocols = INET_PROTO_NAME_IPV4; @@ -1241,6 +1281,10 @@ int main(int argc, char **argv) case 'a': disable_saslauth = 1; break; + case 'A': + if (!alldig(optarg) || (abort_delay = atoi(optarg)) < 0) + usage(argv[0]); + break; case 'c': count++; break; diff --git a/postfix/src/util/Makefile.in b/postfix/src/util/Makefile.in index 9a9d8e730..707f47cb5 100644 --- a/postfix/src/util/Makefile.in +++ b/postfix/src/util/Makefile.in @@ -30,7 +30,7 @@ SRCS = alldig.c allprint.c argv.c argv_split.c attr_clnt.c attr_print0.c \ username.c valid_hostname.c vbuf.c vbuf_print.c vstream.c \ vstream_popen.c vstring.c vstring_vstream.c watchdog.c writable.c \ write_buf.c write_wait.c sane_basename.c format_tv.c allspace.c \ - allascii.c load_file.c + allascii.c load_file.c killme_after.c OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ attr_print64.o attr_print_plain.o attr_scan0.o attr_scan64.o \ attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \ @@ -62,7 +62,7 @@ OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ username.o valid_hostname.o vbuf.o vbuf_print.o vstream.o \ vstream_popen.o vstring.o vstring_vstream.o watchdog.o writable.o \ write_buf.o write_wait.o sane_basename.o format_tv.o allspace.o \ - allascii.o load_file.o + allascii.o load_file.o killme_after.o HDRS = argv.h attr.h attr_clnt.h auto_clnt.h base64_code.h binhash.h \ chroot_uid.h cidr_match.h clean_env.h connect.h ctable.h dict.h \ dict_cdb.h dict_cidr.h dict_db.h dict_dbm.h dict_env.h dict_ht.h \ @@ -81,7 +81,7 @@ HDRS = argv.h attr.h attr_clnt.h auto_clnt.h base64_code.h binhash.h \ sigdelay.h sock_addr.h spawn_command.h split_at.h stat_as.h \ stringops.h sys_defs.h timed_connect.h timed_wait.h trigger.h \ username.h valid_hostname.h vbuf.h vbuf_print.h vstream.h vstring.h \ - vstring_vstream.h watchdog.h format_tv.h load_file.h + vstring_vstream.h watchdog.h format_tv.h load_file.h killme_after.h TESTSRC = fifo_open.c fifo_rdwr_bug.c fifo_rdonly_bug.c select_bug.c \ stream_test.c dup2_pass_on_exec.c DEFS = -I. -D$(SYSTYPE) @@ -1122,6 +1122,9 @@ inet_trigger.o: msg.h inet_trigger.o: mymalloc.h inet_trigger.o: sys_defs.h inet_trigger.o: trigger.h +killme_after.o: killme_after.c +killme_after.o: killme_after.h +killme_after.o: sys_defs.h line_wrap.o: line_wrap.c line_wrap.o: line_wrap.h line_wrap.o: sys_defs.h @@ -1611,6 +1614,7 @@ vstring_vstream.o: vstream.h vstring_vstream.o: vstring.h vstring_vstream.o: vstring_vstream.c vstring_vstream.o: vstring_vstream.h +watchdog.o: killme_after.h watchdog.o: msg.h watchdog.o: mymalloc.h watchdog.o: posix_signals.h diff --git a/postfix/src/util/killme_after.c b/postfix/src/util/killme_after.c new file mode 100644 index 000000000..1ce06d675 --- /dev/null +++ b/postfix/src/util/killme_after.c @@ -0,0 +1,58 @@ +/*++ +/* NAME +/* killme_after 3 +/* SUMMARY +/* programmed death +/* SYNOPSIS +/* #include +/* +/* void killme_after(seconds) +/* unsigned int seconds; +/* DESCRIPTION +/* The killme_after() function does a best effort to terminate +/* the process after the specified time, should it still exist. +/* It is meant to be used in a signal handler, as an insurance +/* against getting stuck somewhere while preparing for exit. +/* DIAGNOSTICS +/* None. This routine does a best effort, damn the torpedoes. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +/* System library. */ + +#include +#include +#include + +/* Utility library. */ + +#include + +/* killme_after - self-assured death */ + +void killme_after(unsigned int seconds) +{ + struct sigaction sig_action; + + /* + * Schedule an ALARM signal, and make sure the signal will be delivered + * even if we are being called from a signal handler and SIGALRM delivery + * is blocked. + */ + alarm(0); + sigemptyset(&sig_action.sa_mask); + sig_action.sa_flags = 0; + sig_action.sa_handler = SIG_DFL; + sigaction(SIGALRM, &sig_action, (struct sigaction *) 0); + alarm(seconds); + sigaddset(&sig_action.sa_mask, SIGALRM); + sigprocmask(SIG_UNBLOCK, &sig_action.sa_mask, (sigset_t *) 0); +} diff --git a/postfix/src/util/killme_after.h b/postfix/src/util/killme_after.h new file mode 100644 index 000000000..9505b2d60 --- /dev/null +++ b/postfix/src/util/killme_after.h @@ -0,0 +1,30 @@ +#ifndef _KILLME_AFTER_H_INCLUDED_ +#define _KILLME_AFTER_H_INCLUDED_ + +/*++ +/* NAME +/* killme_after 3h +/* SUMMARY +/* programmed death +/* SYNOPSIS +/* #include "killme_after.h" +/* DESCRIPTION +/* .nf + + /* + * External interface. + */ +extern void killme_after(unsigned int); + +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +#endif diff --git a/postfix/src/util/msg.c b/postfix/src/util/msg.c index 3aac01976..eeb6953f9 100644 --- a/postfix/src/util/msg.c +++ b/postfix/src/util/msg.c @@ -39,12 +39,31 @@ /* to the standard error stream, but the disposition can be changed /* by the user. See the hints below in the SEE ALSO section. /* -/* msg_info(), msg_warn(), msg_error(), msg_fatal() and msg_panic() +/* msg_info(), msg_warn(), msg_error(), msg_fatal*() and msg_panic() /* produce a one-line record with the program name, a severity code /* (except for msg_info()), and an informative message. The program /* name must have been set by calling one of the msg_XXX_init() /* functions (see the SEE ALSO section). /* +/* The aforementioned logging routines are protected against +/* ordinary recursive calls and against re-entry by a signal +/* handler. +/* +/* Protection against re-entry by signal handlers requires +/* that: +/* .IP \(bu +/* The signal handler must never return. In other words, the +/* signal handler must either call _exit(), kill itself with +/* a signal, or do both. +/* .IP \(bu +/* The signal handler must not execute before the msg_XXX_init() +/* functions complete initialization. +/* .PP +/* When re-entrancy is detected, the requested logging and +/* optional cleanup operations are skipped. Skipping the logging +/* operation prevents deadlock on Linux releases that use +/* mutexes within system library routines such as syslog(). +/* /* msg_error() reports a recoverable error and increments the error /* counter. When the error count exceeds a pre-set limit (default: 13) /* the program terminates by calling msg_fatal(). @@ -64,6 +83,12 @@ /* current function pointer. Specify a null argument to disable /* this feature. /* +/* Note: each msg_cleanup() call-back function, and each Postfix +/* or system function called by that call-back function, either +/* protects itself against recursive calls and re-entry by a +/* terminating signal handler, or is called exclusively by +/* functions in the msg(3) module. +/* /* msg_error_limit() sets the error message count limit, and returns. /* the old limit. /* @@ -118,23 +143,28 @@ int msg_verbose = 0; /* - * Private state. The msg_exiting flag prevents us from recursively - * reporting an error. + * Private state. */ static MSG_CLEANUP_FN msg_cleanup_fn = 0; -static int msg_exiting = 0; static int msg_error_count = 0; static int msg_error_bound = 13; + /* + * Global scope, to discourage the compiler from doing smart things. + */ +volatile int msg_exiting = 0; + /* msg_info - report informative message */ void msg_info(const char *fmt,...) { va_list ap; + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); va_start(ap, fmt); msg_vprintf(MSG_INFO, fmt, ap); va_end(ap); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); } /* msg_warn - report warning message */ @@ -143,9 +173,11 @@ void msg_warn(const char *fmt,...) { va_list ap; + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); va_start(ap, fmt); msg_vprintf(MSG_WARN, fmt, ap); va_end(ap); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); } /* msg_error - report recoverable error */ @@ -154,9 +186,11 @@ void msg_error(const char *fmt,...) { va_list ap; + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); va_start(ap, fmt); msg_vprintf(MSG_ERROR, fmt, ap); va_end(ap); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); if (++msg_error_count >= msg_error_bound) msg_fatal("too many errors - program terminated"); } @@ -167,15 +201,16 @@ NORETURN msg_fatal(const char *fmt,...) { va_list ap; - if (msg_exiting++ == 0) { - va_start(ap, fmt); - msg_vprintf(MSG_FATAL, fmt, ap); - va_end(ap); - if (msg_cleanup_fn) - msg_cleanup_fn(); - } + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); + va_start(ap, fmt); + msg_vprintf(MSG_FATAL, fmt, ap); + va_end(ap); + if (msg_cleanup_fn) + msg_cleanup_fn(); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); sleep(1); - exit(1); + /* In case we're running as a signal handler. */ + _exit(1); } /* msg_fatal_status - report error and terminate gracefully */ @@ -184,15 +219,16 @@ NORETURN msg_fatal_status(int status, const char *fmt,...) { va_list ap; - if (msg_exiting++ == 0) { - va_start(ap, fmt); - msg_vprintf(MSG_FATAL, fmt, ap); - va_end(ap); - if (msg_cleanup_fn) - msg_cleanup_fn(); - } + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); + va_start(ap, fmt); + msg_vprintf(MSG_FATAL, fmt, ap); + va_end(ap); + if (msg_cleanup_fn) + msg_cleanup_fn(); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); sleep(1); - exit(status); + /* In case we're running as a signal handler. */ + _exit(status); } /* msg_panic - report error and dump core */ @@ -201,14 +237,15 @@ NORETURN msg_panic(const char *fmt,...) { va_list ap; - if (msg_exiting++ == 0) { - va_start(ap, fmt); - msg_vprintf(MSG_PANIC, fmt, ap); - va_end(ap); - } + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); + va_start(ap, fmt); + msg_vprintf(MSG_PANIC, fmt, ap); + va_end(ap); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_exiting); sleep(1); abort(); /* Die! */ - exit(1); /* DIE!! */ + /* In case we're running as a signal handler. */ + _exit(1); /* DIE!! */ } /* msg_cleanup - specify cleanup routine */ diff --git a/postfix/src/util/msg_output.c b/postfix/src/util/msg_output.c index 889206bdb..b97b9c5ca 100644 --- a/postfix/src/util/msg_output.c +++ b/postfix/src/util/msg_output.c @@ -25,7 +25,27 @@ /* const char *text; /* DESCRIPTION /* This module implements low-level output management for the -/* msg(3) diagnostics interface. +/* msg(3) diagnostics interface. The output routines are +/* protected against ordinary recursive calls and against +/* re-entry by a signal handler. +/* +/* Protection against re-entry by a signal handler requires +/* that: +/* .IP \(bu +/* The signal handler never returns. +/* .IP \(bu +/* The signal handler does not execute before msg_output() +/* completes initialization. +/* .IP \(bu +/* Each msg_output() call-back function, and each Postfix or +/* system function called by that call-back function, either +/* protects itself against recursive calls and re-entry by a +/* terminating signal handler, or is called exclusively by +/* functions in the msg_output(3) module. +/* .PP +/* When re-entrancy is detected, the requested output operation +/* is skipped. This prevents deadlock on Linux releases that +/* use mutexes within system library routines such as syslog(). /* /* msg_output() registers an output handler for the diagnostics /* interface. An application can register multiple output handlers. @@ -67,6 +87,12 @@ #include #include + /* + * Global scope, to discourage the compiler from doing smart things. + */ +volatile int msg_vp_lock; +volatile int msg_txt_lock; + /* * Private state. */ @@ -79,6 +105,12 @@ static VSTRING *msg_buffer = 0; void msg_output(MSG_OUTPUT_FN output_fn) { + /* + * Allocate all resources during initialization. + */ + if (msg_buffer == 0) + msg_buffer = vstring_alloc(100); + /* * We're not doing this often, so avoid complexity and allocate memory * for an exact fit. @@ -106,10 +138,10 @@ void msg_printf(int level, const char *format,...) void msg_vprintf(int level, const char *format, va_list ap) { - if (msg_buffer == 0) - msg_buffer = vstring_alloc(100); + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_vp_lock); vstring_vsprintf(msg_buffer, percentm(format, errno), ap); msg_text(level, vstring_str(msg_buffer)); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_vp_lock); } /* msg_text - sanitize and log pre-formatted text */ @@ -121,13 +153,14 @@ void msg_text(int level, const char *text) /* * Sanitize the text. Use a private copy if necessary. */ - if (msg_buffer == 0) - msg_buffer = vstring_alloc(100); + BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_txt_lock); if (text != vstring_str(msg_buffer)) vstring_strcpy(msg_buffer, text); printable(vstring_str(msg_buffer), '?'); + /* On-the-fly initialization for debugging test programs only. */ if (msg_output_fn_count == 0) msg_vstream_init("unknown", VSTREAM_ERR); for (i = 0; i < msg_output_fn_count; i++) msg_output_fn[i] (level, vstring_str(msg_buffer)); + END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(msg_txt_lock); } diff --git a/postfix/src/util/sys_defs.h b/postfix/src/util/sys_defs.h index 513d25223..93cda277d 100644 --- a/postfix/src/util/sys_defs.h +++ b/postfix/src/util/sys_defs.h @@ -89,6 +89,7 @@ #if __FreeBSD_version >= 300000 #define HAS_ISSETUGID +#define HAS_FUTIMES #endif #if __FreeBSD_version >= 400000 @@ -98,6 +99,10 @@ /* OpenBSD version is year+month */ +#if OpenBSD >= 199805 /* XXX */ +#define HAS_FUTIMES /* XXX maybe earlier */ +#endif + #if OpenBSD >= 200000 /* XXX */ #define HAS_ISSETUGID #define HAS_DEV_URANDOM /* XXX probably earlier */ @@ -133,6 +138,10 @@ #define HAS_CLOSEFROM #endif +#if (defined(__NetBSD_Version__) && __NetBSD_Version__ >= 102000000) +#define HAS_FUTIMES +#endif + #if (defined(__NetBSD_Version__) && __NetBSD_Version__ >= 105000000) \ || (defined(__FreeBSD__) && __FreeBSD__ >= 4) \ || (defined(OpenBSD) && OpenBSD >= 200003) \ @@ -179,6 +188,7 @@ # define HAS_IPV6 # define HAVE_GETIFADDRS #endif +#define HAS_FUTIMES /* XXX Guessing */ #define NATIVE_SENDMAIL_PATH "/usr/sbin/sendmail" #define NATIVE_MAILQ_PATH "/usr/bin/mailq" #define NATIVE_NEWALIAS_PATH "/usr/bin/newaliases" @@ -380,6 +390,9 @@ extern int opterr; #ifndef NO_DEV_URANDOM # define HAS_DEV_URANDOM #endif +#ifndef NO_FUTIMESAT +# define HAS_FUTIMESAT +#endif /* * Allow build environment to override paths. @@ -1414,6 +1427,28 @@ typedef int pid_t; */ extern int REMOVE(const char *); + /* + * Enter, or skip, a critical region. This is used in fatal run-time error + * handlers. + * + * It is OK if a terminating signal handler hijacks control before the next + * statement executes. The interrupted thread will never be resumed; when + * the signal handler leaves the critical region, the state of the + * "critical" variable can safely be left in an inconsistent state. We + * explicitly give the critical variable global scope, to discourage the + * compiler from trying to do clever things. + */ +#define BEGIN_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(name) \ + if (name == 0) { \ + name = 1; + + /* + * Leave critical region. + */ +#define END_PROTECT_AGAINST_RECURSION_OR_TERMINATING_SIGNAL_HANDLER(name) \ + name = 0; \ + } + /* LICENSE /* .ad /* .fi diff --git a/postfix/src/util/vstream.c b/postfix/src/util/vstream.c index 1a3373a8e..602d659d3 100644 --- a/postfix/src/util/vstream.c +++ b/postfix/src/util/vstream.c @@ -58,8 +58,9 @@ /* int vstream_fflush(stream) /* VSTREAM *stream; /* -/* int vstream_fpurge(stream) +/* int vstream_fpurge(stream, direction) /* VSTREAM *stream; +/* int direction; /* /* ssize_t vstream_fread(stream, buf, len) /* VSTREAM *stream; @@ -160,6 +161,7 @@ /* /* vstream_fclose() closes the named buffered stream. The result /* is 0 in case of success, VSTREAM_EOF in case of problems. +/* vstream_fclose() reports the same errors as vstream_ferror(). /* /* vstream_fdclose() leaves the file(s) open but is otherwise /* identical to vstream_fclose(). @@ -215,12 +217,15 @@ /* opened in read-write or write-only mode. /* vstream_fflush() returns 0 in case of success, VSTREAM_EOF in /* case of problems. It is an error to flush a read-only stream. +/* vstream_fflush() reports the same errors as vstream_ferror(). /* /* vstream_fpurge() discards the contents of the stream buffer. -/* In the case of a double-buffered stream, it discards the -/* content of both the read and write buffers. -/* vstream_fpurge() returns 0 in case of success, VSTREAM_EOF in -/* case of problems. +/* If direction is VSTREAM_PURGE_READ, it discards unread data, +/* else if direction is VSTREAM_PURGE_WRITE, it discards unwritten +/* data. In the case of a double-buffered stream, if direction is +/* VSTREAM_PURGE_BOTH, it discards the content of both the read +/* and write buffers. vstream_fpurge() returns 0 in case of success, +/* VSTREAM_EOF in case of problems. /* /* vstream_fread() and vstream_fwrite() perform unformatted I/O /* on the named stream. The result value is the number of bytes @@ -280,18 +285,20 @@ /* /* vstream_feof() returns non-zero when a previous operation on the /* specified stream caused an end-of-file condition. -/* Further read requests after EOF may complete succesfully, -/* even when vstream_clearerr() is not called for that stream. +/* Although further read requests after EOF may complete +/* succesfully, vstream_feof() will keep returning non-zero +/* until vstream_clearerr() is called for that stream. /* /* vstream_ferror() returns non-zero when a previous operation on the /* specified stream caused a non-EOF error condition, including timeout. -/* After a non-EOF, non-timeout, error on a stream, no I/O request will +/* After a non-EOF, non-timeout, error on a stream, no I/O request will /* complete until after vstream_clearerr() is called for that stream. /* /* vstream_ftimeout() returns non-zero when a previous operation on the /* specified stream caused a timeout error condition. -/* Further I/O requests after timeout may complete succesfully, -/* even when vstream_clearerr() is not called for that stream. +/* Although further I/O requests after timeout may complete +/* succesfully, vstream_ftimeout() will keep returning non-zero +/* until vstream_clearerr() is called for that stream. /* /* vstream_clearerr() resets the timeout, error and end-of-file indication /* of the specified stream, and returns no useful result. @@ -824,11 +831,16 @@ static int vstream_buf_space(VBUF *bp, ssize_t want) /* vstream_fpurge - discard unread or unwritten content */ -int vstream_fpurge(VSTREAM *stream) +int vstream_fpurge(VSTREAM *stream, int direction) { const char *myname = "vstream_fpurge"; VBUF *bp = &stream->buf; +#define VSTREAM_MAYBE_PURGE_WRITE(d, b) if ((d) & VSTREAM_PURGE_WRITE) \ + VSTREAM_BUF_AT_START((b)) +#define VSTREAM_MAYBE_PURGE_READ(d, b) if ((d) & VSTREAM_PURGE_READ) \ + VSTREAM_BUF_AT_END((b)) + /* * To discard all unread contents, position the read buffer at its end, * so that we skip over any unread data, and so that the next read @@ -840,20 +852,20 @@ int vstream_fpurge(VSTREAM *stream) */ switch (bp->flags & (VSTREAM_FLAG_READ_DOUBLE | VSTREAM_FLAG_WRITE)) { case VSTREAM_FLAG_READ_DOUBLE: - VSTREAM_BUF_AT_START(&stream->write_buf); + VSTREAM_MAYBE_PURGE_WRITE(direction, &stream->write_buf); /* FALLTHROUGH */ case VSTREAM_FLAG_READ: - VSTREAM_BUF_AT_END(bp); + VSTREAM_MAYBE_PURGE_READ(direction, bp); break; case VSTREAM_FLAG_DOUBLE: - VSTREAM_BUF_AT_START(&stream->write_buf); - VSTREAM_BUF_AT_END(&stream->read_buf); + VSTREAM_MAYBE_PURGE_WRITE(direction, &stream->write_buf); + VSTREAM_MAYBE_PURGE_READ(direction, &stream->read_buf); break; case VSTREAM_FLAG_WRITE_DOUBLE: - VSTREAM_BUF_AT_END(&stream->read_buf); + VSTREAM_MAYBE_PURGE_READ(direction, &stream->read_buf); /* FALLTHROUGH */ case VSTREAM_FLAG_WRITE: - VSTREAM_BUF_AT_START(bp); + VSTREAM_MAYBE_PURGE_WRITE(direction, bp); break; case VSTREAM_FLAG_READ_DOUBLE | VSTREAM_FLAG_WRITE: case VSTREAM_FLAG_READ | VSTREAM_FLAG_WRITE: diff --git a/postfix/src/util/vstream.h b/postfix/src/util/vstream.h index 37c9efba9..625309298 100644 --- a/postfix/src/util/vstream.h +++ b/postfix/src/util/vstream.h @@ -30,7 +30,7 @@ * Simple buffered stream. The members of this structure are not part of the * official interface and can change without prior notice. */ -typedef ssize_t(*VSTREAM_FN) (int, void *, size_t, int, void *); +typedef ssize_t (*VSTREAM_FN) (int, void *, size_t, int, void *); typedef int (*VSTREAM_WAITPID_FN) (pid_t, WAIT_STATUS_T *, int); typedef struct VSTREAM { @@ -70,13 +70,17 @@ extern VSTREAM vstream_fstd[]; /* pre-defined streams */ #define VSTREAM_FLAG_NSEEK (1<<11) /* can't seek this file */ #define VSTREAM_FLAG_DOUBLE (1<<12) /* double buffer */ +#define VSTREAM_PURGE_READ (1<<0) /* flush unread data */ +#define VSTREAM_PURGE_WRITE (1<<1) /* flush unwritten data */ +#define VSTREAM_PURGE_BOTH (VSTREAM_PURGE_READ|VSTREAM_PURGE_WRITE) + #define VSTREAM_BUFSIZE 4096 extern VSTREAM *vstream_fopen(const char *, int, mode_t); extern int vstream_fclose(VSTREAM *); extern off_t vstream_fseek(VSTREAM *, off_t, int); extern off_t vstream_ftell(VSTREAM *); -extern int vstream_fpurge(VSTREAM *); +extern int vstream_fpurge(VSTREAM *, int); extern int vstream_fflush(VSTREAM *); extern int vstream_fputs(const char *, VSTREAM *); extern VSTREAM *vstream_fdopen(int, int); diff --git a/postfix/src/util/watchdog.c b/postfix/src/util/watchdog.c index d69e210e1..a1ab0cf1e 100644 --- a/postfix/src/util/watchdog.c +++ b/postfix/src/util/watchdog.c @@ -87,6 +87,7 @@ #include #include +#include #include /* Application-specific. */ @@ -128,7 +129,8 @@ static void watchdog_event(int unused_sig) /* * This routine runs as a signal handler. We should not do anything that * could involve memory allocation/deallocation, but exiting without - * proper explanation would be unacceptable. + * proper explanation would be unacceptable. For this reason, msg(3) was + * made safe for usage by signal handlers that terminate the process. */ if ((wp = watchdog_curr) == 0) msg_panic("%s: no instance", myname); @@ -139,8 +141,13 @@ static void watchdog_event(int unused_sig) } else { if (wp->action) wp->action(wp, wp->context); - else + else { + killme_after(5); +#ifdef TEST + pause(); +#endif msg_fatal("watchdog timeout"); + } } }