From: Wietse Venema Date: Thu, 28 Jul 2016 05:00:00 +0000 (-0500) Subject: postfix-3.2-20160728 X-Git-Tag: v3.2.0-RC1~22 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=98b90b32f0cdbb94ae9549e4245fa44585570d8a;p=thirdparty%2Fpostfix.git postfix-3.2-20160728 --- diff --git a/postfix/.indent.pro b/postfix/.indent.pro index 5502f64db..06435a2a6 100644 --- a/postfix/.indent.pro +++ b/postfix/.indent.pro @@ -253,6 +253,7 @@ -TPOST_MAIL_STATE -TPRIVATE_STR_TABLE -TPSC_CALL_BACK_ENTRY +-TPSC_CLIENT_INFO -TPSC_DNSBL_HEAD -TPSC_DNSBL_SCORE -TPSC_DNSBL_SITE diff --git a/postfix/HISTORY b/postfix/HISTORY index 7246e3529..741ec1c24 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -1,8 +1,4 @@ In addition to the names listed below, the following people provided - -20160618 - - Bugfix: useful inputs on many occasions: Paul D. Robertson, Simon J. Mudd. Apologies for any names omitted. @@ -22384,7 +22380,7 @@ Apologies for any names omitted. 20160618 - Bugfix(introduced: 20091121): with the introduction of + Bugfix (introduced: 20091121): with the introduction of sender_dependent_default_transport_maps, the SMTP daemon was not updated. This resulted in false rejects with sender-dependent "error" transports. Based on a fix by @@ -22411,3 +22407,15 @@ Apologies for any names omitted. xsasl/xsasl_server.c. Cleanup: dnsblog manpage. File: dnsblog/dnsblog.c. + +20160728 + + Bugfix (introduced: 20090614): with concurrent connections + from the same client IP address, and after-220 tests enabled, + postscreen could overwrite the cached "all tests completed" + result of one connection that completed the after-220 tests, + with the "some tests not completed" result of a concurrent + connection where the client hung up before completing the + after-220 tests. Files: postscreen_misc.c, postscreen_state.c, + postscreen.h, postscreen_tests.c, postscreen.c, postscreen_smtpd.c, + postscreen_early.c. diff --git a/postfix/README_FILES/XCLIENT_README b/postfix/README_FILES/XCLIENT_README index 439dd4aa7..89b11bff6 100644 --- a/postfix/README_FILES/XCLIENT_README +++ b/postfix/README_FILES/XCLIENT_README @@ -50,7 +50,8 @@ are in fact case insensitive. xclient-command = XCLIENT 1*( SP attribute-name"="attribute-value ) - attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN ) + attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN | DESTADDR | + DESTPORT ) attribute-value = xtext diff --git a/postfix/RELEASE_NOTES b/postfix/RELEASE_NOTES index 94fab09e3..1fe8d8331 100644 --- a/postfix/RELEASE_NOTES +++ b/postfix/RELEASE_NOTES @@ -16,6 +16,29 @@ specifies the release date of a stable release or snapshot release. If you upgrade from Postfix 3.0 or earlier, read RELEASE_NOTES-3.1 before proceeding. +Major changes with snapshot 20160625 +==================================== + +Support in the Postfix SMTP server for propagating the local SMTP +server IP address and port. This affects the following Postfix +interfaces: + +- Policy delegation. The server address and port are available as +"server_address" and "server_port". See SMTPD_POLICY_README for an +overview of available attributes. + +- Milter applications. The server address and port are available +as "{daemon_addr}" and "{daemon_port}". See MILTER_README for a +table of available attributes. + +- Cyrus SASL. The server address and port are now passed to the +sasl_server_new() function as "ipaddress;port". + +- XCLIENT protocol. The server address and port can be specified +as "DESTADDR" and "DESTPORT". See XCLIENT_README for a description +of the attribute syntax. The new attributes may be of interest for +nxginx. + Major changes with snapshot 20160527 ==================================== diff --git a/postfix/WISHLIST b/postfix/WISHLIST index c9f6558cd..1df78719e 100644 --- a/postfix/WISHLIST +++ b/postfix/WISHLIST @@ -6,6 +6,9 @@ Wish list: Disable -DSNAPSHOT and -DNONPROD in makedefs. + Why does postqueue show UTC time, even if TZ is set in + the import_environment setting? + Propagate SMTPD_PEER_CODE_XXX from smtpd(8) to cleanup(8), so that {client_resolve} and {_} produce consistent results. @@ -13,7 +16,8 @@ Wish list: Modeline support in config files to enable/disable trailing #comment, and to give hints about how to handle an LHS or - RHS. + RHS. This will not preserve trailing comments in lines that + are modified with "postconf -e" and the like. The cleanup daemon searches canonical_maps and virtual_alias_maps with quoted address forms. The address local part should diff --git a/postfix/html/XCLIENT_README.html b/postfix/html/XCLIENT_README.html index 60724dd36..a58a32140 100644 --- a/postfix/html/XCLIENT_README.html +++ b/postfix/html/XCLIENT_README.html @@ -80,7 +80,7 @@ names are shown in upper case, they are in fact case insensitive. xclient-command = XCLIENT 1*( SP attribute-name"="attribute-value )

- attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN ) + attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN | DESTADDR | DESTPORT )

attribute-value = xtext diff --git a/postfix/html/postconf.5.html b/postfix/html/postconf.5.html index 35ae0188c..1134ae35f 100644 --- a/postfix/html/postconf.5.html +++ b/postfix/html/postconf.5.html @@ -17335,8 +17335,8 @@ errors while accessing the Postfix main.cf configu (default: see "postconf -d" output)

-The mail system name that is prepended to the process name in syslog -records, so that "smtpd" becomes, for example, "postfix/smtpd". +A prefix that is prepended to the process name in syslog +records, so that, for example, "smtpd" becomes "prefix/smtpd".

diff --git a/postfix/man/man5/postconf.5 b/postfix/man/man5/postconf.5 index e0dd9d1db..a22a09927 100644 --- a/postfix/man/man5/postconf.5 +++ b/postfix/man/man5/postconf.5 @@ -11886,8 +11886,8 @@ process initialization will be logged with the default facility. Examples are errors while parsing the command line arguments, and errors while accessing the Postfix main.cf configuration file. .SH syslog_name (default: see "postconf \-d" output) -The mail system name that is prepended to the process name in syslog -records, so that "smtpd" becomes, for example, "postfix/smtpd". +A prefix that is prepended to the process name in syslog +records, so that, for example, "smtpd" becomes "prefix/smtpd". .PP Warning: a non\-default syslog_name setting takes effect only after a Postfix process has completed initialization. Errors during diff --git a/postfix/proto/XCLIENT_README.html b/postfix/proto/XCLIENT_README.html index 5873662f7..e4e2f0a1f 100644 --- a/postfix/proto/XCLIENT_README.html +++ b/postfix/proto/XCLIENT_README.html @@ -80,7 +80,7 @@ names are shown in upper case, they are in fact case insensitive. xclient-command = XCLIENT 1*( SP attribute-name"="attribute-value )

- attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN ) + attribute-name = ( NAME | ADDR | PORT | PROTO | HELO | LOGIN | DESTADDR | DESTPORT )

attribute-value = xtext diff --git a/postfix/proto/postconf.proto b/postfix/proto/postconf.proto index 9ead6ea9e..eb62ab0ea 100644 --- a/postfix/proto/postconf.proto +++ b/postfix/proto/postconf.proto @@ -6746,8 +6746,8 @@ errors while accessing the Postfix main.cf configuration file. %PARAM syslog_name see "postconf -d" output

-The mail system name that is prepended to the process name in syslog -records, so that "smtpd" becomes, for example, "postfix/smtpd". +A prefix that is prepended to the process name in syslog +records, so that, for example, "smtpd" becomes "prefix/smtpd".

diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index f26efffd3..f723684bf 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20160625" +#define MAIL_RELEASE_DATE "20160728" #define MAIL_VERSION_NUMBER "3.2" #ifdef SNAPSHOT diff --git a/postfix/src/postscreen/postscreen.c b/postfix/src/postscreen/postscreen.c index 188d28423..4b766d4f0 100644 --- a/postfix/src/postscreen/postscreen.c +++ b/postfix/src/postscreen/postscreen.c @@ -592,6 +592,9 @@ static void psc_drain(char *unused_service, char **unused_argv) * * XXX Some Berkeley DB versions break with close-after-fork. Every new * version is an improvement over its predecessor. + * + * XXX Don't assume that it is OK to share the same LMDB lockfile descriptor + * between different processes. */ if (psc_cache_map != 0 /* XXX && psc_cache_map requires locking */ ) { @@ -766,6 +769,7 @@ static void psc_endpt_lookup_done(int endpt_status, * valid. */ if ((state->flags & PSC_STATE_MASK_ANY_FAIL) == 0 + && state->client_info->concurrency == 1 && psc_cache_map != 0 && (stamp_str = psc_cache_lookup(psc_cache_map, state->smtp_client_addr)) != 0) { saved_flags = state->flags; @@ -779,6 +783,13 @@ static void psc_endpt_lookup_done(int endpt_status, psc_conclude(state); return; } + } else if (state->client_info->concurrency > 1) { + saved_flags = state->flags; + psc_todo_tests(state, event_time()); + state->flags |= saved_flags; + if (msg_verbose) + msg_info("%s: new + recent flags: %s", + myname, psc_print_state_flags(state->flags, myname)); } else { saved_flags = state->flags; psc_new_tests(state); @@ -830,6 +841,7 @@ static int psc_cache_validator(const char *client_addr, void *unused_context) { PSC_STATE dummy; + PSC_CLIENT_INFO dummy_client_info; /* * This function is called by the cache cleanup pseudo thread. @@ -839,6 +851,7 @@ static int psc_cache_validator(const char *client_addr, * silly logging we remove the cache entry only after all tests have * expired longer ago than the cache retention time. */ + dummy.client_info = &dummy_client_info; psc_parse_tests(&dummy, stamp_str, event_time() - var_psc_cache_ret); return ((dummy.flags & PSC_STATE_MASK_ANY_TODO) == 0); } diff --git a/postfix/src/postscreen/postscreen.h b/postfix/src/postscreen/postscreen.h index 2e4585b5c..a02fa7fac 100644 --- a/postfix/src/postscreen/postscreen.h +++ b/postfix/src/postscreen/postscreen.h @@ -54,6 +54,15 @@ #define PSC_TINDX_BYTNAME(tname) (PSC_TINDX_ ## tname) + /* + * Per-client shared state. + */ +typedef struct { + int concurrency; /* per-client */ + int pass_new_count; /* per-client */ + time_t expire_time[PSC_TINDX_COUNT]; /* per-test expiration */ +} PSC_CLIENT_INFO; + /* * Per-session state. */ @@ -66,13 +75,12 @@ typedef struct { char *smtp_client_port; /* client port */ char *smtp_server_addr; /* server address */ char *smtp_server_port; /* server port */ - int client_concurrency; /* per-client */ const char *final_reply; /* cause for hanging up */ VSTRING *send_buf; /* pending output */ /* Test context. */ struct timeval start_time; /* start of current test */ const char *test_name; /* name of current test */ - time_t expire_time[PSC_TINDX_COUNT]; /* per-test expiration */ + PSC_CLIENT_INFO *client_info; /* shared client state */ VSTRING *dnsbl_reply; /* dnsbl reject text */ int dnsbl_score; /* saved DNSBL score */ int dnsbl_ttl; /* saved DNSBL TTL */ @@ -95,11 +103,14 @@ typedef struct { * Emulate legacy ad-hoc variables on top of indexable time stamps. This * avoids massive scar tissue during initial feature development. */ -#define pregr_stamp expire_time[PSC_TINDX_PREGR] -#define dnsbl_stamp expire_time[PSC_TINDX_DNSBL] -#define pipel_stamp expire_time[PSC_TINDX_PIPEL] -#define nsmtp_stamp expire_time[PSC_TINDX_NSMTP] -#define barlf_stamp expire_time[PSC_TINDX_BARLF] +#define pregr_stamp client_info->expire_time[PSC_TINDX_PREGR] +#define dnsbl_stamp client_info->expire_time[PSC_TINDX_DNSBL] +#define pipel_stamp client_info->expire_time[PSC_TINDX_PIPEL] +#define nsmtp_stamp client_info->expire_time[PSC_TINDX_NSMTP] +#define barlf_stamp client_info->expire_time[PSC_TINDX_BARLF] + + /* Minize the patch size for stable releases. */ +#define client_concurrency client_info->concurrency /* * Special expiration time values. @@ -489,17 +500,21 @@ extern int psc_dnsbl_request(const char *, void (*) (int, void *), void *); #define PSC_INIT_TESTS(dst) do { \ time_t *_it_stamp_p; \ (dst)->flags = 0; \ - for (_it_stamp_p = (dst)->expire_time; \ - _it_stamp_p < (dst)->expire_time + PSC_TINDX_COUNT; \ + for (_it_stamp_p = (dst)->client_info->expire_time; \ + _it_stamp_p < (dst)->client_info->expire_time + PSC_TINDX_COUNT; \ _it_stamp_p++) \ *_it_stamp_p = PSC_TIME_STAMP_INVALID; \ } while (0) +#define PSC_INIT_TEST_FLAGS_ONLY(dst) do { \ + (dst)->flags = 0; \ + } while (0) #define PSC_BEGIN_TESTS(state, name) do { \ (state)->test_name = (name); \ GETTIMEOFDAY(&(state)->start_time); \ } while (0) extern void psc_new_tests(PSC_STATE *); extern void psc_parse_tests(PSC_STATE *, const char *, time_t); +extern void psc_todo_tests(PSC_STATE *, time_t); extern char *psc_print_tests(VSTRING *, PSC_STATE *); extern char *psc_print_grey_key(VSTRING *, const char *, const char *, const char *, const char *); diff --git a/postfix/src/postscreen/postscreen_early.c b/postfix/src/postscreen/postscreen_early.c index 36c3d5cd6..b5bb50e14 100644 --- a/postfix/src/postscreen/postscreen_early.c +++ b/postfix/src/postscreen/postscreen_early.c @@ -92,8 +92,8 @@ static void psc_whitelist_non_dnsbl(PSC_STATE *state) state->flags |= PSC_STATE_FLAG_BYTINDX_PASS(tindx); } /* Update expiration even if the test was completed or disabled. */ - if (state->expire_time[tindx] < now + state->dnsbl_ttl) - state->expire_time[tindx] = now + state->dnsbl_ttl; + if (state->client_info->expire_time[tindx] < now + state->dnsbl_ttl) + state->client_info->expire_time[tindx] = now + state->dnsbl_ttl; } } } diff --git a/postfix/src/postscreen/postscreen_misc.c b/postfix/src/postscreen/postscreen_misc.c index 57a1bb67d..95cd6e9a8 100644 --- a/postfix/src/postscreen/postscreen_misc.c +++ b/postfix/src/postscreen/postscreen_misc.c @@ -105,7 +105,8 @@ void psc_conclude(PSC_STATE *state) if ((state->flags & PSC_STATE_MASK_ANY_PASS) != 0 && (state->flags & PSC_STATE_MASK_ANY_PASS) == PSC_STATE_FLAGS_TODO_TO_PASS(state->flags & PSC_STATE_MASK_ANY_TODO)) - msg_info("PASS %s [%s]:%s", (state->flags & PSC_STATE_FLAG_NEW) == 0 ? + msg_info("PASS %s [%s]:%s", (state->flags & PSC_STATE_FLAG_NEW) == 0 + || state->client_info->pass_new_count++ > 0 ? "OLD" : "NEW", PSC_CLIENT_ADDR_PORT(state)); /* @@ -114,7 +115,7 @@ void psc_conclude(PSC_STATE *state) * that client does not "fail" any test. */ if ((state->flags & PSC_STATE_MASK_ANY_UPDATE) != 0 - && psc_cache_map != 0) { + && psc_cache_map != 0 && state->client_info->pass_new_count <= 1) { psc_print_tests(psc_temp, state); psc_cache_update(psc_cache_map, state->smtp_client_addr, STR(psc_temp)); } diff --git a/postfix/src/postscreen/postscreen_state.c b/postfix/src/postscreen/postscreen_state.c index 946afd6fa..b84f6656b 100644 --- a/postfix/src/postscreen/postscreen_state.c +++ b/postfix/src/postscreen/postscreen_state.c @@ -61,7 +61,8 @@ /* psc_new_session_state() creates a new session state object /* for the specified client stream, and increments the /* psc_check_queue_length counter. The flags and per-test time -/* stamps are initialized with PSC_INIT_TESTS(). The addr and +/* stamps are initialized with PSC_INIT_TESTS(), or for concurrent +/* sessions, with PSC_INIT_TEST_FLAGS_ONLY(). The addr and /* port arguments are null-terminated strings with the remote /* SMTP client endpoint. The _reply members are set to /* polite "try again" SMTP replies. The protocol member is set @@ -149,10 +150,8 @@ PSC_STATE *psc_new_session_state(VSTREAM *stream, const char *server_port) { PSC_STATE *state; - HTABLE_INFO *ht; state = (PSC_STATE *) mymalloc(sizeof(*state)); - PSC_INIT_TESTS(state); if ((state->smtp_client_stream = stream) != 0) psc_check_queue_length++; state->smtp_server_fd = (-1); @@ -188,10 +187,19 @@ PSC_STATE *psc_new_session_state(VSTREAM *stream, /* * Update the per-client session count. */ - if ((ht = htable_locate(psc_client_concurrency, client_addr)) == 0) - ht = htable_enter(psc_client_concurrency, client_addr, (void *) 0); - ht->value += 1; - state->client_concurrency = CAST_ANY_PTR_TO_INT(ht->value); + if ((state->client_info = (PSC_CLIENT_INFO *) + htable_find(psc_client_concurrency, client_addr)) == 0) { + state->client_info = (PSC_CLIENT_INFO *) + mymalloc(sizeof(state->client_info[0])); + (void) htable_enter(psc_client_concurrency, client_addr, + (void *) state->client_info); + PSC_INIT_TESTS(state); + state->client_info->concurrency = 1; + state->client_info->pass_new_count = 0; + } else { + PSC_INIT_TEST_FLAGS_ONLY(state); + state->client_info->concurrency += 1; + } return (state); } @@ -210,9 +218,8 @@ void psc_free_session_state(PSC_STATE *state) state->smtp_client_addr)) == 0) msg_panic("%s: unknown client address: %s", myname, state->smtp_client_addr); - if (--(ht->value) == 0) - htable_delete(psc_client_concurrency, state->smtp_client_addr, - (void (*) (void *)) 0); + if (--(state->client_info->concurrency) == 0) + htable_delete(psc_client_concurrency, state->smtp_client_addr, myfree); if (state->smtp_client_stream != 0) { event_server_disconnect(state->smtp_client_stream); diff --git a/postfix/src/postscreen/postscreen_tests.c b/postfix/src/postscreen/postscreen_tests.c index 17b7a3926..f8b92a988 100644 --- a/postfix/src/postscreen/postscreen_tests.c +++ b/postfix/src/postscreen/postscreen_tests.c @@ -17,6 +17,11 @@ /* const char *stamp_text; /* time_t time_value; /* +/* void psc_todo_tests(state, time_value) +/* PSC_STATE *state; +/* const char *stamp_text; +/* time_t time_value; +/* /* char *psc_print_tests(buffer, state) /* VSTRING *buffer; /* PSC_STATE *state; @@ -41,13 +46,18 @@ /* zeroes all the flags bits. These values are not meant to /* be stored into the postscreen(8) cache. /* +/* PSC_INIT_TEST_FLAGS_ONLY() zeroes all the flag bits. It +/* should be used when the time stamps are already initialized. +/* /* psc_new_tests() sets all test expiration time stamps to -/* PSC_TIME_STAMP_NEW, and overwrites all flags bits. Only -/* enabled tests are flagged with PSC_STATE_FLAG_TODO; the -/* object is flagged with PSC_STATE_FLAG_NEW. +/* PSC_TIME_STAMP_NEW, and invokes psc_todo_tests(). +/* +/* psc_parse_tests() parses a cache file record and invokes +/* psc_todo_tests(). /* -/* psc_parse_tests() parses a cache file record and overwrites -/* all flags bits. Tests are considered "expired" when they +/* psc_todo_tests() overwrites all per-session flag bits, and +/* populates the flags based on test expiration time stamp +/* information. Tests are considered "expired" when they /* would be expired at the specified time value. Only enabled /* tests are flagged as "expired"; the object is flagged as /* "new" if some enabled tests have "new" time stamps. @@ -124,11 +134,6 @@ void psc_new_tests(PSC_STATE *state) { - /* - * We know this client is brand new. - */ - state->flags = PSC_STATE_FLAG_NEW; - /* * Give all tests a PSC_TIME_STAMP_NEW time stamp, so that we can later * recognize cache entries that haven't passed all enabled tests. When we @@ -142,19 +147,9 @@ void psc_new_tests(PSC_STATE *state) state->barlf_stamp = PSC_TIME_STAMP_NEW; /* - * Don't flag disabled tests as "todo", because there would be no way to - * make those bits go away. + * Determine what tests need to be completed. */ - if (PSC_PREGR_TEST_ENABLE()) - state->flags |= PSC_STATE_FLAG_PREGR_TODO; - if (PSC_DNSBL_TEST_ENABLE()) - state->flags |= PSC_STATE_FLAG_DNSBL_TODO; - if (var_psc_pipel_enable) - state->flags |= PSC_STATE_FLAG_PIPEL_TODO; - if (var_psc_nsmtp_enable) - state->flags |= PSC_STATE_FLAG_NSMTP_TODO; - if (var_psc_barlf_enable) - state->flags |= PSC_STATE_FLAG_BARLF_TODO; + psc_todo_tests(state, PSC_TIME_STAMP_NEW + 1); } /* psc_parse_tests - parse test results from cache */ @@ -165,29 +160,18 @@ void psc_parse_tests(PSC_STATE *state, { const char *start = stamp_str; char *cp; - time_t *time_stamps = state->expire_time; + time_t *time_stamps = state->client_info->expire_time; time_t *sp; - /* - * We don't know what tests have expired or have never passed. - */ - state->flags = 0; - /* * Parse the cache entry, and allow for older postscreen versions that * implemented fewer tests. We pretend that the newer tests were disabled * at the time that the cache entry was written. - * - * Flag the cache entry as "new" when the cache entry has fields for all - * enabled tests, but the remote SMTP client has not yet passed all those - * tests. */ for (sp = time_stamps; sp < time_stamps + PSC_TINDX_COUNT; sp++) { *sp = strtoul(start, &cp, 10); if (*start == 0 || (*cp != '\0' && *cp != ';') || errno == ERANGE) *sp = PSC_TIME_STAMP_DISABLED; - if (*sp == PSC_TIME_STAMP_NEW) - state->flags |= PSC_STATE_FLAG_NEW; if (msg_verbose) msg_info("%s -> %lu", start, (unsigned long) *sp); if (*cp == ';') @@ -196,6 +180,34 @@ void psc_parse_tests(PSC_STATE *state, start = cp; } + /* + * Determine what tests need to be completed. + */ + psc_todo_tests(state, time_value); +} + +/* psc_todo_tests - determine what tests to perform */ + +void psc_todo_tests(PSC_STATE *state, time_t time_value) +{ + time_t *time_stamps = state->client_info->expire_time; + time_t *sp; + + /* + * Reset all per-session flags. + */ + state->flags = 0; + + /* + * Flag the tests as "new" when the cache entry has fields for all + * enabled tests, but the remote SMTP client has not yet passed all those + * tests. + */ + for (sp = time_stamps; sp < time_stamps + PSC_TINDX_COUNT; sp++) { + if (*sp == PSC_TIME_STAMP_NEW) + state->flags |= PSC_STATE_FLAG_NEW; + } + /* * Don't flag disabled tests as "todo", because there would be no way to * make those bits go away.