instead of overstrike sequences. To make matters worse, it
uses the ESC[0m sequence sometimes for end-of-bold and
sometimes for end-of-italic. File: mantools/man2html.
+
+20130714
+
+ Cleanup: added smtpd_relay_restrictions entries to the
+ default master.cf file, so that main.cf settings won't
+ affect the submission and smtps services. Simon Matter.
+ File: conf/master.cf.
+
+20130728
+
+ Cleanup: wrong function name in error message. John Fawcett.
+ File: util/vstring_vstream.c.
+
+20130801
+
+ Cleanup: with ``make makefiles CCARGS="-DHAS_DB...'', the
+ makedefs script no longer tries to locate the Linux Berkeley
+ DB include and library files. Instead it assumes that the
+ locations are given on the command line, as shown in the
+ DB_README examples. Leo Baltus. File: makedefs.
+
+20130805
+
+ Documentation: clarified reject_non_fqdn_helo_hostname.
+ File: proto/postconf.proto.
+
+20130809
+
+ Cleanup: the lmdb_map_size parameter is now a long integer.
+ Howard Chu. Files: global/mail_params.[hc].
+
+20130815
+
+ Documentation: added pointer to Dovecot 2 configuration.
+ File: proto/SASL_README.html
+
+20130818
+
+ Update: LMDB client updated to LMDB 0.9.7, which hopefully
+ fixes the unrecoverable "transaction full" error. With a
+ new MDB_MAP_FULL workaround by Howard Chu that ensures that
+ postfix will make progress as long as the disk is not full.
+ File: util/dict_lmdb.c.
I\bIn\bnt\btr\bro\bod\bdu\buc\bct\bti\bio\bon\bn
-Note:
- Postfix support for LMDB databases is suspended due to the existence of a
- hard limit (an "out of storage" failure mode that cannot be resolved by
- increasing the database size).
-
- Postfix may support LMDB again when it no longer limits the size of Postfix
- transactions, whether the limit is built into LMDB itself, or implicit by
- requiring an unbounded amount of memory to handle a large transaction.
-
Postfix uses databases of various kinds to store and look up information.
Postfix databases are specified as "type:name". OpenLDAP LMDB implements the
Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is
the name of the database file without the ".lmdb" suffix. OpenLDAP LMDB
-databases are maintained with the postmap(1) command.
+databases are maintained with the postmap(1) and postalias(1) commands.
This document describes:
Postfix provides a configuration parameter that controls how large an OpenLDAP
LMDB database may grow.
- * lmdb_map_size (default: 10 MBytes per table). This setting controls how
+ * lmdb_map_size (default: 16 MBytes per table). This setting controls how
large any OpenLDAP LMDB database may grow. It must be set large enough to
accommodate the largest table that Postfix will use.
U\bUn\bne\bex\bxp\bpe\bec\bct\bte\bed\bd p\bpo\bos\bst\btm\bma\bap\bp(\b(1\b1)\b)/\b/p\bpo\bos\bst\bta\bal\bli\bia\bas\bs(\b(1\b1)\b) "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
-Problem:
- The "postmap lmdb:filename" command fails with an MDB_TXN_FULL error. This
- problem does not exist with other Postfix databases.
-
-Background:
- The LMDB implementation has a hard limit on the total transaction size.
- This limit is independent of the LMDB database size. Therefore, the problem
- cannot be resolved by increasing the lmdb_map_size value.
-
- This symptom is indicative of a flawed design. All LMDB data structures
- should share the same storage pool so that they can scale with the database
- size, and so that all "out of storage" errors are resolved by increasing
- the database size.
-
Problem:
The "postmap lmdb:filename" command fails with an MDB_MAP_FULL error. This
problem does not exist with other Postfix databases.
still exists in the database.
Mitigation:
+ When the postmap(1) or postalias(1) command fails with an MDB_MAP_FULL
+ error, it expands the database file size to the current LMDB map size limit
+ before terminating.
+
When the postmap(1) or postalias(1) command opens an LMDB file larger than
lmdb_map_size/3, it logs a warning and uses a larger size limit instead:
warning: filename.lmdb: file size 15024128 >= (lmdb map size limit
16777216)/3 -- using a larger map size limit
- This can be used to automate recovery and avoid the need for human
- intervention. Just keep running "postmap lmdb:filename". After each failure
- it will use a 3x larger size limit, and eventually the "database full"
- error will disappear.
+ The two steps above can be used to automate recovery and avoid the need for
+ human intervention. Just repeat "postmap lmdb:filename" (up to some limit).
+ After each failure it will use a 3x larger size limit, and eventually the
+ "database full" error will disappear.
Prevention:
Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest
Background:
The Postfix LMDB database client does not truncate the database file.
- Instead it attempts to create a transaction for a "drop" request and
+ Instead it attempts to create a transaction for a "drop" request plus
subsequent "store" requests. That is obviously not possible with a
corrupted database file.
Prevention:
Arrange your file systems such that they never run out of free space.
+ Use ECC memory to detect and correct silent corruption of in-memory file
+ system data and metadata.
+
+ Use a file system such as ZFS to detect and correct silent corruption of
+ on-disk file system data and metadata.
+
The following example assumes that the Postfix queue is under /var/spool/
postfix/.
+Note: the example uses Dovecot 1 syntax, See http://www.dovecot.org/ for newer
+syntax.
+
1 /etc/dovecot.conf:
2 auth default {
3 mechanisms = plain login
The following examples assume that Postfix should communicate with Dovecot on
TCP port 12345.
+Note: the examples use Dovecot 1 syntax, See http://www.dovecot.org/ for newer
+syntax.
+
1 /etc/dovecot/conf.d/10-auth.conf:
2 auth_mechanisms = plain login
Things to do after the stable release:
+ Why does postlink no longer hyperlink static:all?
+
Begin code revision, after DANE support stabilizes. This
should be one pass that changes only names and no code.
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
-# -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
+# -o smtpd_recipient_restrictions=
+# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#smtps inet n - n - - smtpd
# -o syslog_name=postfix/smtps
# -o smtpd_client_restrictions=$mua_client_restrictions
# -o smtpd_helo_restrictions=$mua_helo_restrictions
# -o smtpd_sender_restrictions=$mua_sender_restrictions
-# -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
+# -o smtpd_recipient_restrictions=
+# -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
# -o milter_macro_daemon_name=ORIGINATING
#628 inet n - n - - qmqpd
pickup unix n - n 60 1 pickup
<h2>Introduction</h2>
+<!--
+
<dl> <dt> Note: </dt> <dd> <p> Postfix support for LMDB databases
is suspended due to the existence of a hard limit (an "out of
storage" failure mode that cannot be resolved by increasing the
is built into LMDB itself, or implicit by requiring an unbounded
amount of memory to handle a large transaction. </p> </dd> </dl>
+-->
+
<p> Postfix uses databases of various kinds to store and look up
information. Postfix databases are specified as "type:name".
OpenLDAP LMDB implements the Postfix database type "lmdb".
The name of a Postfix OpenLDAP LMDB database is the name
of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases
-are maintained with the <a href="postmap.1.html">postmap(1)</a> command. </p>
+are maintained with the <a href="postmap.1.html">postmap(1)</a> and <a href="postalias.1.html">postalias(1)</a> commands. </p>
<p> This document describes: </p>
<ul>
-<li> <p> <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> (default: 10 MBytes per
-table). This setting controls how large any OpenLDAP LMDB database
-may grow. It must be set large enough to accommodate the largest
-table that Postfix will use. </p>
+<li> <p> <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> (default: 16 MBytes per table). This setting
+controls how large any OpenLDAP LMDB database may grow. It must be
+set large enough to accommodate the largest table that Postfix will
+use. </p>
</ul>
<dl>
+<!--
+
<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
fails with an MDB_TXN_FULL error. This problem does not exist with
other Postfix databases. </p> </dd>
with the database size, and so that all "out of storage" errors are
resolved by increasing the database size. </p> </dd>
+-->
+
<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
fails with an MDB_MAP_FULL error. This problem does not exist with
other Postfix databases. </p> </dd>
availability, because the old data still exists in the database.
</p> </dd>
-<dt> Mitigation: </dt> <dd> <p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>
-command opens an LMDB file larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a
-warning and uses a larger size limit instead: </p>
+<dt> Mitigation: </dt> <dd>
+
+<p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command fails with an
+MDB_MAP_FULL error, it expands the database file size to the current
+LMDB map size limit before terminating. </p>
+
+<p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command opens an LMDB file
+larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a warning and uses a larger
+size limit instead: </p>
<p> <tt> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>: file size 15024128 ≥
(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
</p>
-<p> This can be used to automate recovery and avoid the need for
-human intervention. Just keep running "postmap <a href="LMDB_README.html">lmdb</a>:filename".
-After each failure it will use a 3x larger size limit, and eventually
-the "database full" error will disappear. </p>
+<p> The two steps above can be used to automate recovery and avoid
+the need for human intervention. Just repeat "postmap <a href="LMDB_README.html">lmdb</a>:filename"
+(up to some limit). After each failure it will use a 3x larger
+size limit, and eventually the "database full" error will disappear.
+</p>
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p>
sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p>
</dd> </dl>
-</dl>
-
<p> <strong>Non-obvious recovery with <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a>/<a href="tlsmgr.8.html">tlsmgr(8)</a>
from a corrupted database. </strong></p>
<dt> Background: </dt> <dd> <p> The Postfix LMDB database client
does not truncate the database file. Instead it attempts to create
-a transaction for a "drop" request and subsequent "store" requests.
+a transaction for a "drop" request plus subsequent "store" requests.
That is obviously not possible with a corrupted database file. </p>
</dd>
or wait until the <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts automatically. </p>
</dd>
-<dt> Prevention: </dt> <dd> <p> Arrange your file systems such that
-they never run out of free space. </p> </dd> </dl>
+<dt> Prevention: </dt> <dd>
+
+<p> Arrange your file systems such that they never run out of free
+space. </p>
-</dl>
+<p> Use ECC memory to detect and correct silent corruption of
+in-memory file system data and metadata. </p>
+
+<p> Use a file system such as ZFS to detect and correct silent
+corruption of on-disk file system data and metadata. </p>
+
+</dd> </dl>
<p> The following example assumes that the Postfix queue is under
<code>/var/spool/postfix/</code>. </p>
+<p> Note: the example uses Dovecot 1 syntax, See <a href="http://www.dovecot.org/">http://www.dovecot.org/</a>
+for newer syntax. </p>
+
<blockquote>
<pre>
1 /etc/dovecot.conf:
<p> The following examples assume that Postfix should communicate
with Dovecot on TCP port 12345. </p>
+<p> Note: the examples use Dovecot 1 syntax, See <a href="http://www.dovecot.org/">http://www.dovecot.org/</a>
+for newer syntax. </p>
+
<blockquote>
<pre>
1 /etc/dovecot/conf.d/10-auth.conf:
port may be given in symbolic form (host or
service name) or in numeric form (IP address
or port number). Host information may be
- enclosed inside "[]", but this form is not
- necessary.
+ enclosed inside "[]"; this form is necessary
+ only with IPv6 addresses.
Examples: a service named <b>127.0.0.1:smtp</b> or
<b>::1:smtp</b> receives mail via the loopback
in the <a href="postconf.5.html">main.cf</a> configuration file. See
<a href="DEBUG_README.html">DEBUG_README</a> for hints and tips.
- <b>-o</b> <i>name</i>=<i>value</i>
+ <b>-o <i></b>name</i>=<i>value</i>
Override the named <a href="postconf.5.html">main.cf</a> configuration
parameter. The parameter value can refer to
other parameters as <i>$name</i> etc., just like in
pcre_table - format of Postfix PCRE tables
<b>SYNOPSIS</b>
- <b>postmap -q "</b><i>string</i><b>" <a href="pcre_table.5.html">pcre</a>:/etc/postfix/</b><i>filename</i>
+ <b>postmap -q "<i></b>string</i><b>" <a href="pcre_table.5.html">pcre</a>:/etc/postfix/<i></b>filename\e[0m
- <b>postmap -q - <a href="pcre_table.5.html">pcre</a>:/etc/postfix/</b><i>filename</i> <<i>inputfile</i>
+ <b>postmap -q - <a href="pcre_table.5.html">pcre</a>:/etc/postfix/<i></b>filename</i> <<i>inputfile</i>
<b>DESCRIPTION</b>
The Postfix mail system uses optional tables for address
<b>TABLE FORMAT</b>
The general form of a PCRE table is:
- <b>/</b><i>pattern</i><b>/</b><i>flags result</i>
+ <b>/<i></b>pattern</i><b>/<i></b>flags result</i>
When <i>pattern</i> matches the input string, use the cor-
responding <i>result</i> value.
- <b>!/</b><i>pattern</i><b>/</b><i>flags result</i>
+ <b>!/<i></b>pattern</i><b>/<i></b>flags result</i>
When <i>pattern</i> does <b>not</b> match the input string, use
the corresponding <i>result</i> value.
- <b>if /</b><i>pattern</i><b>/</b><i>flags</i>
+ <b>if /<i></b>pattern</i><b>/<i></b>flags\e[0m
<b>endif</b> Match the input string against the patterns between
<b>if</b> and <b>endif</b>, if and only if that same input string
This feature is available in Postfix 2.1 and later.
- <b>if !/</b><i>pattern</i><b>/</b><i>flags</i>
+ <b>if !/<i></b>pattern</i><b>/<i></b>flags\e[0m
<b>endif</b> Match the input string against the patterns between
<b>if</b> and <b>endif</b>, if and only if that same input string
a whitespace character as part of the pattern,
escape it with backslash.
- Note: do not use <b>#</b><i>comment</i> after patterns.
+ Note: do not use <b>#<i></b>comment</i> after patterns.
<b>A</b> (default: off)
Toggles the PCRE_ANCHORED flag. When this flag is
nor is <i>user+foo</i> broken up into <i>user</i> and <i>foo</i>.
<b>TEXT SUBSTITUTION</b>
- Substitution of substrings from the matched expression
- into the result string is possible using the conventional
- perl syntax ($1, $2, etc.); specify $$ to produce a $
- character as output. The macros in the result string may
- need to be written as ${n} or $(n) if they aren't followed
- by whitespace.
+ Substitution of substrings (text that matches patterns
+ inside "()") from the matched expression into the result
+ string is requested with $1, $2, etc.; specify $$ to pro-
+ duce a $ character as output. The macros in the result
+ string may need to be written as ${n} or $(n) if they
+ aren't followed by whitespace.
Note: since negated patterns (those preceded by <b>!</b>) return
a result when the expression does not match, substitutions
<dt><b><a name="reject_non_fqdn_helo_hostname">reject_non_fqdn_helo_hostname</a></b> (with Postfix < 2.3: reject_non_fqdn_hostname)</dt>
<dd>Reject the request when the HELO or EHLO hostname is not in
-fully-qualified domain form, as required by the RFC. Note: specify
+fully-qualified domain or address literal form, as required by the
+RFC. Note: specify
"<a href="postconf.5.html#smtpd_helo_required">smtpd_helo_required</a> = yes" to fully enforce this restriction
(without "<a href="postconf.5.html#smtpd_helo_required">smtpd_helo_required</a> = yes", a client can simply skip
<a href="postconf.5.html#reject_non_fqdn_helo_hostname">reject_non_fqdn_helo_hostname</a> by not sending HELO or EHLO). <br>
regexp_table - format of Postfix regular expression tables
<b>SYNOPSIS</b>
- <b>postmap -q "</b><i>string</i><b>" <a href="regexp_table.5.html">regexp</a>:/etc/postfix/</b><i>filename</i>
+ <b>postmap -q "<i></b>string</i><b>" <a href="regexp_table.5.html">regexp</a>:/etc/postfix/<i></b>filename\e[0m
- <b>postmap -q - <a href="regexp_table.5.html">regexp</a>:/etc/postfix/</b><i>filename</i> <<i>inputfile</i>
+ <b>postmap -q - <a href="regexp_table.5.html">regexp</a>:/etc/postfix/<i></b>filename</i> <<i>inputfile</i>
<b>DESCRIPTION</b>
The Postfix mail system uses optional tables for address
<b>TABLE FORMAT</b>
The general form of a Postfix regular expression table is:
- <b>/</b><i>pattern</i><b>/</b><i>flags result</i>
+ <b>/<i></b>pattern</i><b>/<i></b>flags result</i>
When <i>pattern</i> matches the input string, use the cor-
responding <i>result</i> value.
- <b>!/</b><i>pattern</i><b>/</b><i>flags result</i>
+ <b>!/<i></b>pattern</i><b>/<i></b>flags result</i>
When <i>pattern</i> does <b>not</b> match the input string, use
the corresponding <i>result</i> value.
- <b>if /</b><i>pattern</i><b>/</b><i>flags</i>
+ <b>if /<i></b>pattern</i><b>/<i></b>flags\e[0m
<b>endif</b> Match the input string against the patterns between
<b>if</b> and <b>endif</b>, if and only if that same input string
This feature is available in Postfix 2.1 and later.
- <b>if !/</b><i>pattern</i><b>/</b><i>flags</i>
+ <b>if !/<i></b>pattern</i><b>/<i></b>flags\e[0m
<b>endif</b> Match the input string against the patterns between
<b>if</b> and <b>endif</b>, if and only if that same input string
nor is <i>user+foo</i> broken up into <i>user</i> and <i>foo</i>.
<b>TEXT SUBSTITUTION</b>
- Substitution of substrings from the matched expression
- into the result string is possible using $1, $2, etc.;
- specify $$ to produce a $ character as output. The macros
- in the result string may need to be written as ${n} or
- $(n) if they aren't followed by whitespace.
-
- Note: since negated patterns (those preceded by <b>!</b>) return
+ Substitution of substrings (text that matches patterns
+ inside "()") from the matched expression into the result
+ string is requested with $1, $2, etc.; specify $$ to pro-
+ duce a $ character as output. The macros in the result
+ string may need to be written as ${n} or $(n) if they
+ aren't followed by whitespace.
+
+ Note: since negated patterns (those preceded by <b>!</b>) return
a result when the expression does not match, substitutions
are not available for negated patterns.
Linux.1*) SYSTYPE=LINUX1
case "$CCARGS" in
*-DNO_DB*) ;;
+ *-DHAS_DB*) ;;
*) SYSLIBS="-ldb";;
esac
;;
Linux.2*) SYSTYPE=LINUX2
case "$CCARGS" in
*-DNO_DB*) ;;
+ *-DHAS_DB*) ;;
*) if [ -f /usr/include/db.h ]
then
: we are all set
Linux.3*) SYSTYPE=LINUX3
case "$CCARGS" in
*-DNO_DB*) ;;
+ *-DHAS_DB*) ;;
*) if [ -f /usr/include/db.h ]
then
: we are all set
accepted. The host part (and colon) may be omitted. Either
host or port may be given in symbolic form (host or service
name) or in numeric form (IP address or port number).
-Host information may be enclosed inside "[]", but this form
-is not necessary.
+Host information may be enclosed inside "[]"; this form
+is necessary only with IPv6 addresses.
.sp
Examples: a service named \fB127.0.0.1:smtp\fR or \fB::1:smtp\fR
receives
.nf
.ad
.fi
-Substitution of substrings from the matched expression into the result
-string is possible using the conventional perl syntax ($1, $2, etc.);
-specify $$ to produce a $ character as output.
-The macros in the result string may need to be written as ${n}
-or $(n) if they aren't followed by whitespace.
+Substitution of substrings (text that matches patterns
+inside "()") from the matched expression into the result
+string is requested with $1, $2, etc.; specify $$ to produce
+a $ character as output.
+The macros in the result string may need to be written as
+${n} or $(n) if they aren't followed by whitespace.
Note: since negated patterns (those preceded by \fB!\fR) return a
result when the expression does not match, substitutions are not
.br
.IP "\fBreject_non_fqdn_helo_hostname\fR (with Postfix < 2.3: reject_non_fqdn_hostname)"
Reject the request when the HELO or EHLO hostname is not in
-fully-qualified domain form, as required by the RFC. Note: specify
+fully-qualified domain or address literal form, as required by the
+RFC. Note: specify
"smtpd_helo_required = yes" to fully enforce this restriction
(without "smtpd_helo_required = yes", a client can simply skip
reject_non_fqdn_helo_hostname by not sending HELO or EHLO).
.nf
.ad
.fi
-Substitution of substrings from the matched expression into the result
-string is possible using $1, $2, etc.;
-specify $$ to produce a $ character as output.
-The macros in the result string
-may need to be written as ${n} or $(n) if they aren't followed
-by whitespace.
+Substitution of substrings (text that matches patterns
+inside "()") from the matched expression into the result
+string is requested with $1, $2, etc.; specify $$ to produce
+a $ character as output.
+The macros in the result string may need to be written as
+${n} or $(n) if they aren't followed by whitespace.
Note: since negated patterns (those preceded by \fB!\fR) return a
result when the expression does not match, substitutions are not
<h2>Introduction</h2>
+<!--
+
<dl> <dt> Note: </dt> <dd> <p> Postfix support for LMDB databases
is suspended due to the existence of a hard limit (an "out of
storage" failure mode that cannot be resolved by increasing the
is built into LMDB itself, or implicit by requiring an unbounded
amount of memory to handle a large transaction. </p> </dd> </dl>
+-->
+
<p> Postfix uses databases of various kinds to store and look up
information. Postfix databases are specified as "type:name".
OpenLDAP LMDB implements the Postfix database type "lmdb".
The name of a Postfix OpenLDAP LMDB database is the name
of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases
-are maintained with the postmap(1) command. </p>
+are maintained with the postmap(1) and postalias(1) commands. </p>
<p> This document describes: </p>
<ul>
-<li> <p> lmdb_map_size (default: 10 MBytes per
-table). This setting controls how large any OpenLDAP LMDB database
-may grow. It must be set large enough to accommodate the largest
-table that Postfix will use. </p>
+<li> <p> lmdb_map_size (default: 16 MBytes per table). This setting
+controls how large any OpenLDAP LMDB database may grow. It must be
+set large enough to accommodate the largest table that Postfix will
+use. </p>
</ul>
<dl>
+<!--
+
<dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command
fails with an MDB_TXN_FULL error. This problem does not exist with
other Postfix databases. </p> </dd>
with the database size, and so that all "out of storage" errors are
resolved by increasing the database size. </p> </dd>
+-->
+
<dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command
fails with an MDB_MAP_FULL error. This problem does not exist with
other Postfix databases. </p> </dd>
availability, because the old data still exists in the database.
</p> </dd>
-<dt> Mitigation: </dt> <dd> <p> When the postmap(1) or postalias(1)
-command opens an LMDB file larger than lmdb_map_size/3, it logs a
-warning and uses a larger size limit instead: </p>
+<dt> Mitigation: </dt> <dd>
+
+<p> When the postmap(1) or postalias(1) command fails with an
+MDB_MAP_FULL error, it expands the database file size to the current
+LMDB map size limit before terminating. </p>
+
+<p> When the postmap(1) or postalias(1) command opens an LMDB file
+larger than lmdb_map_size/3, it logs a warning and uses a larger
+size limit instead: </p>
<p> <tt> warning: <i>filename</i>.lmdb: file size 15024128 ≥
(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
</p>
-<p> This can be used to automate recovery and avoid the need for
-human intervention. Just keep running "postmap lmdb:filename".
-After each failure it will use a 3x larger size limit, and eventually
-the "database full" error will disappear. </p>
+<p> The two steps above can be used to automate recovery and avoid
+the need for human intervention. Just repeat "postmap lmdb:filename"
+(up to some limit). After each failure it will use a 3x larger
+size limit, and eventually the "database full" error will disappear.
+</p>
<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
sure that lmdb_map_size > 3x the largest LMDB file size. </p>
sure that lmdb_map_size > 3x the largest LMDB file size. </p>
</dd> </dl>
-</dl>
-
<p> <strong>Non-obvious recovery with postmap(1)/postalias(1)/tlsmgr(8)
from a corrupted database. </strong></p>
<dt> Background: </dt> <dd> <p> The Postfix LMDB database client
does not truncate the database file. Instead it attempts to create
-a transaction for a "drop" request and subsequent "store" requests.
+a transaction for a "drop" request plus subsequent "store" requests.
That is obviously not possible with a corrupted database file. </p>
</dd>
or wait until the tlsmgr(8) daemon restarts automatically. </p>
</dd>
-<dt> Prevention: </dt> <dd> <p> Arrange your file systems such that
-they never run out of free space. </p> </dd> </dl>
+<dt> Prevention: </dt> <dd>
+
+<p> Arrange your file systems such that they never run out of free
+space. </p>
-</dl>
+<p> Use ECC memory to detect and correct silent corruption of
+in-memory file system data and metadata. </p>
+
+<p> Use a file system such as ZFS to detect and correct silent
+corruption of on-disk file system data and metadata. </p>
+
+</dd> </dl>
<p> The following example assumes that the Postfix queue is under
<code>/var/spool/postfix/</code>. </p>
+<p> Note: the example uses Dovecot 1 syntax, See http://www.dovecot.org/
+for newer syntax. </p>
+
<blockquote>
<pre>
1 /etc/dovecot.conf:
<p> The following examples assume that Postfix should communicate
with Dovecot on TCP port 12345. </p>
+<p> Note: the examples use Dovecot 1 syntax, See http://www.dovecot.org/
+for newer syntax. </p>
+
<blockquote>
<pre>
1 /etc/dovecot/conf.d/10-auth.conf:
# accepted. The host part (and colon) may be omitted. Either
# host or port may be given in symbolic form (host or service
# name) or in numeric form (IP address or port number).
-# Host information may be enclosed inside "[]", but this form
-# is not necessary.
+# Host information may be enclosed inside "[]"; this form
+# is necessary only with IPv6 addresses.
# .sp
# Examples: a service named \fB127.0.0.1:smtp\fR or \fB::1:smtp\fR
# receives
# TEXT SUBSTITUTION
# .ad
# .fi
-# Substitution of substrings from the matched expression into the result
-# string is possible using the conventional perl syntax ($1, $2, etc.);
-# specify $$ to produce a $ character as output.
-# The macros in the result string may need to be written as ${n}
-# or $(n) if they aren't followed by whitespace.
+# Substitution of substrings (text that matches patterns
+# inside "()") from the matched expression into the result
+# string is requested with $1, $2, etc.; specify $$ to produce
+# a $ character as output.
+# The macros in the result string may need to be written as
+# ${n} or $(n) if they aren't followed by whitespace.
#
# Note: since negated patterns (those preceded by \fB!\fR) return a
# result when the expression does not match, substitutions are not
<dt><b><a name="reject_non_fqdn_helo_hostname">reject_non_fqdn_helo_hostname</a></b> (with Postfix < 2.3: reject_non_fqdn_hostname)</dt>
<dd>Reject the request when the HELO or EHLO hostname is not in
-fully-qualified domain form, as required by the RFC. Note: specify
+fully-qualified domain or address literal form, as required by the
+RFC. Note: specify
"smtpd_helo_required = yes" to fully enforce this restriction
(without "smtpd_helo_required = yes", a client can simply skip
reject_non_fqdn_helo_hostname by not sending HELO or EHLO). <br>
# TEXT SUBSTITUTION
# .ad
# .fi
-# Substitution of substrings from the matched expression into the result
-# string is possible using $1, $2, etc.;
-# specify $$ to produce a $ character as output.
-# The macros in the result string
-# may need to be written as ${n} or $(n) if they aren't followed
-# by whitespace.
+# Substitution of substrings (text that matches patterns
+# inside "()") from the matched expression into the result
+# string is requested with $1, $2, etc.; specify $$ to produce
+# a $ character as output.
+# The macros in the result string may need to be written as
+# ${n} or $(n) if they aren't followed by whitespace.
#
# Note: since negated patterns (those preceded by \fB!\fR) return a
# result when the expression does not match, substitutions are not
/* char *var_proxywrite_service;
/* int var_db_create_buf;
/* int var_db_read_buf;
-/* int var_lmdb_map_size;
+/* long var_lmdb_map_size;
/* int var_mime_maxdepth;
/* int var_mime_bound_len;
/* int var_header_limit;
char *var_proxywrite_service;
int var_db_create_buf;
int var_db_read_buf;
-int var_lmdb_map_size;
+long var_lmdb_map_size;
int var_mime_maxdepth;
int var_mime_bound_len;
int var_header_limit;
VAR_FAULT_INJ_CODE, DEF_FAULT_INJ_CODE, &var_fault_inj_code, 0, 0,
VAR_DB_CREATE_BUF, DEF_DB_CREATE_BUF, &var_db_create_buf, 1, 0,
VAR_DB_READ_BUF, DEF_DB_READ_BUF, &var_db_read_buf, 1, 0,
- VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 1, 0,
VAR_HEADER_LIMIT, DEF_HEADER_LIMIT, &var_header_limit, 1, 0,
VAR_TOKEN_LIMIT, DEF_TOKEN_LIMIT, &var_token_limit, 1, 0,
VAR_MIME_MAXDEPTH, DEF_MIME_MAXDEPTH, &var_mime_maxdepth, 1, 0,
};
static const CONFIG_LONG_TABLE long_defaults[] = {
VAR_MESSAGE_LIMIT, DEF_MESSAGE_LIMIT, &var_message_limit, 0, 0,
+ VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 1, 0,
0,
};
static const CONFIG_TIME_TABLE time_defaults[] = {
*/
#define VAR_LMDB_MAP_SIZE "lmdb_map_size"
#define DEF_LMDB_MAP_SIZE (16 * 1024 *1024)
-extern int var_lmdb_map_size;
+extern long var_lmdb_map_size;
/*
* Named queue file attributes.
* Patches change both the patchlevel and the release date. Snapshots have no
* patchlevel; they change the release date only.
*/
-#define MAIL_RELEASE_DATE "20130710"
+#define MAIL_RELEASE_DATE "20130818"
#define MAIL_VERSION_NUMBER "2.11"
#ifdef SNAPSHOT
/* file size, so that there is room for the file to grow. This
/* ensures that a process can recover from a "table full" error
/* with a simple terminate-and-restart.
+/*
+/* As a second safety measure, when an update or delete operation
+/* runs into an MDB_MAP_FULL error, Postfix will extend the
+/* database file to the current ".lmdb" file size, so that the
+/* above workaround will be triggered the next time the database
+/* is opened.
/* DIAGNOSTICS
/* Fatal errors: cannot open file, file write error, out of memory.
/* SEE ALSO
unsigned int dict_lmdb_max_readers = 216; /* 200 postfix processes,
* plus some extra */
+ /*
+ * Out-of-space safety net. When an update or delete operation fails with
+ * MDB_MAP_FULL, extend the database file size so that the next
+ * dict_lmdb_open() call will force a 3x over-allocation.
+ *
+ * XXX This strategy assumes that a bogus file size will not affect LMDB
+ * operation. In private communication on August 18, 2013, Howard Chu
+ * confirmed this as follows: "It will have no effect. LMDB internally
+ * accounts for the last used page#, the filesystem's notion of filesize
+ * isn't used for any purpose."
+ *
+ * We make no assumptions about which LMDB operations may fail with
+ * MDB_MAP_FULL. Instead we wrap all LMDB operations inside a Postfix
+ * function that may change a database.
+ */
+#define DICT_LMDB_WRAPPER(dict, status, operation) \
+ ((status = operation) == MDB_MAP_FULL ? \
+ (dict_lmdb_grow(dict), status) : status)
+
+/* dict_lmdb_grow - grow the DB file if the last txn failed to grow it */
+
+static void dict_lmdb_grow(DICT *dict)
+{
+ struct stat st;
+ char *mdb_path = concatenate(dict->name, "." DICT_TYPE_LMDB, (char *) 0);
+
+ /*
+ * After MDB_MAP_FULL error, expand the file size to trigger the 3x size
+ * limit workaround on the next open() attempt.
+ */
+ if (stat(mdb_path, &st) == 0 && st.st_size < dict_lmdb_map_size
+ && truncate(mdb_path, dict_lmdb_map_size) < 0)
+ msg_warn("dict_lmdb_grow: cannot grow database file %s:%s: %m",
+ dict->type, dict->name);
+ myfree(mdb_path);
+}
+
/* dict_lmdb_lookup - find database entry */
static const char *dict_lmdb_lookup(DICT *dict, const char *name)
*/
if (dict_lmdb->txn)
txn = dict_lmdb->txn;
- else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
+ else if (DICT_LMDB_WRAPPER(dict, status,
+ mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
/*
* Do the update.
*/
- status = mdb_put(txn, dict_lmdb->dbi, &mdb_key, &mdb_value,
- (dict->flags & DICT_FLAG_DUP_REPLACE) ? 0 : MDB_NOOVERWRITE);
+ (void) DICT_LMDB_WRAPPER(dict, status,
+ mdb_put(txn, dict_lmdb->dbi, &mdb_key, &mdb_value,
+ (dict->flags & DICT_FLAG_DUP_REPLACE) ? 0 : MDB_NOOVERWRITE));
if (status) {
if (status == MDB_KEYEXIST) {
if (dict->flags & DICT_FLAG_DUP_IGNORE)
/*
* Commit the transaction if it's not the global txn.
*/
- if (!dict_lmdb->txn && ((status = mdb_txn_commit(txn))))
+ if (!dict_lmdb->txn && DICT_LMDB_WRAPPER(dict, status, mdb_txn_commit(txn)))
msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(status));
return (status);
*/
if (dict_lmdb->txn)
txn = dict_lmdb->txn;
- else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
+ else if (DICT_LMDB_WRAPPER(dict, status,
+ mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
/*
if (dict->flags & DICT_FLAG_TRY1NULL) {
mdb_key.mv_data = (void *) name;
mdb_key.mv_size = klen + 1;
- status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL);
+ (void) DICT_LMDB_WRAPPER(dict, status,
+ mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL));
if (status) {
if (status == MDB_NOTFOUND)
status = 1;
if (status > 0 && (dict->flags & DICT_FLAG_TRY0NULL)) {
mdb_key.mv_data = (void *) name;
mdb_key.mv_size = klen;
- status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL);
+ (void) DICT_LMDB_WRAPPER(dict, status,
+ mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL));
if (status) {
if (status == MDB_NOTFOUND)
status = 1;
/*
* Commit the transaction if it's not the global txn.
*/
- if (!dict_lmdb->txn && ((rc = mdb_txn_commit(txn))))
+ if (!dict_lmdb->txn && DICT_LMDB_WRAPPER(dict, rc, mdb_txn_commit(txn)))
msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(rc));
return (status);
DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
if (dict_lmdb->txn) {
- int status = mdb_txn_commit(dict_lmdb->txn);
+ int status;
+ (void) DICT_LMDB_WRAPPER(dict, status, mdb_txn_commit(dict_lmdb->txn));
if (status)
msg_fatal("%s: closing dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
dict_lmdb->cursor = NULL;
char *mdb_path;
int env_flags, status;
- mdb_path = concatenate(path, ".lmdb", (char *) 0);
+ mdb_path = concatenate(path, "." DICT_TYPE_LMDB, (char *) 0);
env_flags = MDB_NOSUBDIR;
if (open_flags == O_RDONLY)
int c;
if (bound <= 0)
- msg_panic("vstring_get_nonl_bound: invalid bound %ld", (long) bound);
+ msg_panic("vstring_get_null_bound: invalid bound %ld", (long) bound);
VSTRING_RESET(vp);
while (bound-- > 0 && (c = VSTREAM_GETC(fp)) != VSTREAM_EOF && c != 0)
projects / thirdparty / postfix.git / commitdiff
