20130315
- Feature: preliminary LMDB (memory-mapped persistent file)
- support by Howard Chu. This feature has unexpected limitations
- that don't exist with other Postfix databases, and is
- therefore "snapshot only", i.e. it will not be part of a
- stable release without further changes to the Postfix LMDB
- client or the Postfix dictionary API. Files: proto/postconf.proto,
- proto/LMDB_README.html, proto/DATABASE_README.html,
- proto/INSTALL.html util/dict_lmdb.[hc], util/dict_open.c,
- global/mkmap_lmdb.[hc], global/mkmap_open.c, postconf/postconf.c.
+ Feature: LMDB (memory-mapped persistent file) support by
+ Howard Chu. This implementation has unexpected failure modes
+ that don't exist with other Postfix databases, so don't
+ just yet abandon CDB. See LMDB_README for details. Files:
+ proto/postconf.proto, proto/LMDB_README.html,
+ proto/DATABASE_README.html, proto/INSTALL.html util/dict_lmdb.[hc],
+ util/dict_open.c, global/mkmap_lmdb.[hc], global/mkmap_open.c,
+ postconf/postconf.c.
+
+20130316
+
+ Cleanup: new Postfix dictionary API flag to control the use
+ of (LMDB) bulk database transactions. With this, LMDB
+ databases no longer fail to commit any transactions with
+ tlsmgr(8), and LMDB databases no longer perform glacially
+ slow with postmap -i/postalias -i. Files: util/dict.h,
+ util/dict_lmdb.c, postmap/postmap.c, postalias/postalias.c.
+
+20130317
+
+ Debugging: generalized setting of dictionary API flags.
+ File: util/dict.[hc], util/dict_test.c.
+
+ Robustness: Postfix programs can now recover from LMDB
+ "database full" errors without requiring human intervention.
+ When a program opens an LMDB file larger than lmdb_map_size/3,
+ it logs a warning and uses a larger size limit instead.
+ Files: util/dict_lmdb.c, proto/LMDB_README.html.
3. Missing pthread library trouble.
-Caution:
- The current Postfix LMDB client has unexpected limitations that don't exist
- with other Postfix databases. For this reason, LMDB support will not be
- part of the stable Postfix release without further changes to the Postfix
- LMDB client or the Postfix dictionary API.
+Note:
+ The Postfix LMDB client implementation introduces unexpected failure modes
+ that don't exist with other Postfix databases. Don't just yet abandon CDB.
B\bBu\bui\bil\bld\bdi\bin\bng\bg P\bPo\bos\bst\btf\bfi\bix\bx w\bwi\bit\bth\bh O\bOp\bpe\ben\bnL\bLD\bDA\bAP\bP L\bLM\bMD\bDB\bB s\bsu\bup\bpp\bpo\bor\brt\bt
Source code for OpenLDAP LMDB is available at http://www.openldap.org. More
information is available at http://highlandsun.com/hyc/mdb/.
-L\bLi\bim\bmi\bit\bta\bat\bti\bio\bon\bns\bs o\bof\bf P\bPo\bos\bst\btf\bfi\bix\bx L\bLM\bMD\bDB\bB d\bda\bat\bta\bab\bba\bas\bse\bes\bs.\b.
+U\bUn\bne\bex\bxp\bpe\bec\bct\bte\bed\bd f\bfa\bai\bil\blu\bur\bre\be m\bmo\bod\bde\bes\bs o\bof\bf P\bPo\bos\bst\btf\bfi\bix\bx L\bLM\bMD\bDB\bB d\bda\bat\bta\bab\bba\bas\bse\bes\bs.\b.
+
+As documented below, conversion to LMDB introduces a number of failure modes
+that don't exist with other Postfix databases.
U\bUn\bne\bex\bxp\bpe\bec\bct\bte\bed\bd p\bpo\bos\bst\btm\bma\bap\bp(\b(1\b1)\b)/\b/p\bpo\bos\bst\bta\bal\bli\bia\bas\bs(\b(1\b1)\b) "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
Problem:
- Even if the "postmap lmdb:filename" command succeeds, the exact same
- command (with the exact same input data) may fail subsequently with an
- MDB_MAP_FULL error. This problem does not exist with other Postfix
- databases.
+ The "postmap lmdb:filename" command fails with an MDB_MAP_FULL error. This
+ problem does not exist with other Postfix databases.
Background:
LMDB databases have a hard size limit (configured with the lmdb_map_size
configuration parameter).
When executing "postmap lmdb:filename", the Postfix LMDB database client
- does not truncate the database file. Instead it saves the "drop" request
- and subsequent "store" requests to a transaction (which takes up space in
- addition to the existing data), and commits the transaction when it closes
- the database. Only then can the space for old data be reused.
+ stores the new data in a transaction which takes up space in addition to
+ the existing data, and commits the transaction when it closes the database.
+ Only then can the space for old data be reused.
Impact:
This failure does not affect Postfix availability, because the old data
still exists in the database.
-Recovery:
- Increase the lmdb_map_size limit in main.cf, and retry the postmap(1) or
- postalias(1) command.
+Mitigation:
+ When the postmap(1) or postalias(1) command opens an LMDB file larger than
+ lmdb_map_size/3, it logs a warning and uses a larger size limit instead:
-P\bPo\bos\bst\btf\bfi\bix\bx d\bda\bae\bem\bmo\bon\bn "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
+ warning: filename.lmdb: file size 15024128 >= (lmdb map size limit
+ 16777216)/3 -- using a larger map size limit
-Problem:
- "database full" errors with daemon programs such as postscreen(8), tlsmgr
- (8) or verify(8). This problem does not exist with other Postfix databases.
-
-Impact:
- Postfix does not process mail until someone fixes the problem.
+ This can be used to automate recovery and avoid the need for human
+ intervention. Just keep running "postmap lmdb:filename". After each failure
+ it will use a 3x larger size limit, and eventually the "database full"
+ error will disappear.
-Recovery:
- Increase the lmdb_map_size limit in main.cf, and "reload" Postfix.
+Prevention:
+ Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest
+ LMDB file size.
-N\bNo\bon\bn-\b-o\bob\bbv\bvi\bio\bou\bus\bs r\bre\bec\bco\bov\bve\ber\bry\by w\bwi\bit\bth\bh p\bpo\bos\bst\btm\bma\bap\bp(\b(1\b1)\b)/\b/p\bpo\bos\bst\bta\bal\bli\bia\bas\bs(\b(1\b1)\b) f\bfr\bro\bom\bm a\ba c\bco\bor\brr\bru\bup\bpt\bte\bed\bd d\bda\bat\bta\bab\bba\bas\bse\be.\b.
+U\bUn\bne\bex\bxp\bpe\bec\bct\bte\bed\bd P\bPo\bos\bst\btf\bfi\bix\bx d\bda\bae\bem\bmo\bon\bn "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
Problem:
- You cannot rebuild a corrupted LMDB database simply by running postmap(1)
- or postalias(1). This problem does not exist with other Postfix databases.
+ Postfix daemon programs fail with "database full" errors, such as
+ postscreen(8), tlsmgr(8) or verify(8). This problem does not exist with
+ other Postfix databases.
-Background:
- The reason for this limitation is that the Postfix LMDB database client
- does not truncate the database file. Instead it attempts to save the "drop"
- request and subsequent "store" requests to a transaction for later
- processing. That is obviously not possible with a corrupted database file.
+Impact:
+ This failure temporarily affects Postfix availability. The daemon restarts
+ automatically and tries to open the database again as described next.
-Recovery:
- First delete the ".lmdb" file by hand, then rebuild the file with the
- postmap(1) or postalias(1) command.
+Mitigation:
+ When a Postfix daemon opens an LMDB file larger than lmdb_map_size/3, it
+ logs a warning and uses a larger size limit instead:
-I\bIn\bnc\bco\bom\bmp\bpa\bat\bti\bib\bbi\bil\bli\bit\bty\by w\bwi\bit\bth\bh t\btl\bls\bsm\bmg\bgr\br(\b(8\b8)\b).\b.
+ warning: filename.lmdb: file size 15024128 >= (lmdb map size limit
+ 16777216)/3 -- using a larger map size limit
-Problem:
- The Postfix LMDB database client never commits any tlsmgr(8) transaction.
- This problem does not exist with other Postfix databases.
+ This can be used to automate recovery and avoid the need for human
+ intervention. Each time the daemon runs into a "database full" error, it
+ restarts and uses a 3x larger size limit. The "database full" error will
+ disappear, at least for a while.
-Background:
- Instead, it creates a single transaction that accumulates a "drop" request
- and all tlsmgr(8) "store" requests that are made during the lifetime of the
- process.
+Prevention:
+ Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest
+ LMDB file size.
-Solution:
- This requires changes to the Postfix dictionary API, or to the Postfix LMDB
- database client.
+N\bNo\bon\bn-\b-o\bob\bbv\bvi\bio\bou\bus\bs r\bre\bec\bco\bov\bve\ber\bry\by w\bwi\bit\bth\bh p\bpo\bos\bst\btm\bma\bap\bp(\b(1\b1)\b)/\b/p\bpo\bos\bst\bta\bal\bli\bia\bas\bs(\b(1\b1)\b)/\b/t\btl\bls\bsm\bmg\bgr\br(\b(8\b8)\b) f\bfr\bro\bom\bm a\ba c\bco\bor\brr\bru\bup\bpt\bte\bed\bd
+d\bda\bat\bta\bab\bba\bas\bse\be.\b.
Problem:
- The Postfix LMDB database client breaks how tlsmgr(8) automatically
- recovers from a corrupted database file. This problem does not exist with
- other Postfix databases.
+ You cannot rebuild a corrupted LMDB database simply by running postmap(1)
+ or postalias(1), or by waiting until the tlsmgr(8) daemon restarts
+ automatically. This problem does not exist with other Postfix databases.
Background:
The Postfix LMDB database client does not truncate the database file.
- Instead it attempts to create a transaction which obviously is not possible
- when the database file is corrupted.
+ Instead it attempts to create a transaction for a "drop" request and
+ subsequent "store" requests. That is obviously not possible with a
+ corrupted database file.
Impact:
- The tlsmgr(8) process will keep crashing until someone removes the ".lmdb"
- file.
+ Postfix does not process mail until someone fixes the problem.
Recovery:
- Remove the the ".lmdb" file by hand, and wait until the tlsmgr(8) process
- restarts.
+ First delete the ".lmdb" file by hand, then rebuild the file with the
+ postmap(1) or postalias(1) command, or wait until the tlsmgr(8) daemon
+ restarts automatically.
+
+Prevention:
+ Arrange your file systems such that they never run out of free space.
Major changes with snapshot 20130315
====================================
-Preliminary LMDB support by Howard Chu. This implementation has
-unexpected limitations that don't exist with other Postfix databases,
-and therefore the code is "snapshot only", i.e. it will not be part
-of the stable release without further changes to the Postfix LMDB
-client or the Postfix dictionary API. See LMDB_README for details.
+LMDB support by Howard Chu. This implementation has unexpected
+failure modes that don't exist with other Postfix databases, so
+don't just yet abandon CDB. See LMDB_README for details.
</ol>
-<dl> <dt> Caution: </dt> <dd> <p> The current Postfix LMDB client
-has <a href="#limitations">unexpected limitations</a> that don't
-exist with other Postfix databases. For this reason, LMDB support
-will not be part of the stable Postfix release without further
-changes to the Postfix LMDB client or the Postfix dictionary API.
-</p> </dd> </dl>
+<dl> <dt> Note: </dt> <dd> <p> The Postfix LMDB client implementation
+introduces <a href="#limitations">unexpected failure modes</a> that
+don't exist with other Postfix databases. Don't just yet abandon
+CDB. </p> </dd> </dl>
<h2><a name="with_lmdb">Building Postfix with OpenLDAP LMDB support</a></h2>
More information is available at
<a href="http://highlandsun.com/hyc/mdb/">http://highlandsun.com/hyc/mdb/</a>. </p>
-<h2><a name="limitations">Limitations of Postfix LMDB databases.
-</a> </h2>
+<h2><a name="limitations">Unexpected failure modes of Postfix LMDB
+databases. </a> </h2>
-<p> <strong>Unexpected <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a> "database full" errors.
-</strong></p>
+<p> As documented below, conversion to LMDB introduces a number of
+failure modes that don't exist with other Postfix databases. </p>
+
+<p> <strong>Unexpected <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a> "database full"
+errors. </strong></p>
<dl>
-<dt> Problem: </dt> <dd> <p> Even if the "postmap <a href="LMDB_README.html">lmdb</a>:filename"
-command succeeds, the exact same command (with the exact same input
-data) may fail subsequently with an MDB_MAP_FULL error. This problem
-does not exist with other Postfix databases. </p> </dd>
+<dt> Problem: </dt> <dd> <p> The "postmap <a href="LMDB_README.html">lmdb</a>:filename" command
+fails with an MDB_MAP_FULL error. This problem does not exist with
+other Postfix databases. </p> </dd>
-<dt> Background: </dt>
+<dt> Background: </dt>
-<dd>
+<dd>
<p> LMDB databases have a hard size limit (configured with the
<a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> configuration parameter). </p>
<p> When executing "postmap <a href="LMDB_README.html">lmdb</a>:filename", the Postfix LMDB database
-client does not truncate the database file. Instead it saves the
-"drop" request and subsequent "store" requests to a transaction
-(which takes up space in addition to the existing data), and commits
-the transaction when it closes the database. Only then can the
-space for old data be reused. </p>
+client stores the new data in a transaction which takes up space
+in addition to the existing data, and commits the transaction when
+it closes the database. Only then can the space for old data be
+reused. </p>
</dd>
availability, because the old data still exists in the database.
</p> </dd>
-<dt> Recovery: </dt> <dd> <p> Increase the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> limit in
-<a href="postconf.5.html">main.cf</a>, and retry the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. </p>
-</dd>
+<dt> Mitigation: </dt> <dd> <p> When the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>
+command opens an LMDB file larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a
+warning and uses a larger size limit instead: </p>
-</dl>
+<p> <tt> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>: file size 15024128 ≥
+(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
+</p>
-<p> <strong>Postfix daemon "database full" errors. </strong></p>
+<p> This can be used to automate recovery and avoid the need for
+human intervention. Just keep running "postmap <a href="LMDB_README.html">lmdb</a>:filename".
+After each failure it will use a 3x larger size limit, and eventually
+the "database full" error will disappear. </p>
-<dl>
-
-<dt> Problem: </dt> <dd> <p> "database full" errors with daemon
-programs such as <a href="postscreen.8.html">postscreen(8)</a>, <a href="tlsmgr.8.html">tlsmgr(8)</a> or <a href="verify.8.html">verify(8)</a>. This problem
-does not exist with other Postfix databases. </p> </dd>
+<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
+sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p>
+</dd> </dl>
-<dt> Impact: </dt> <dd> <p> Postfix does not process mail until
-someone fixes the problem. </p> </dd>
+<p> <strong>Unexpected Postfix daemon "database full" errors.
+</strong></p>
-<dt> Recovery: </dt> <dd> <p> Increase the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> limit in
-<a href="postconf.5.html">main.cf</a>, and "reload" Postfix. </p> </dd>
+<dl>
-</dl>
+<dt> Problem: </dt> <dd> <p> Postfix daemon programs fail with
+"database full" errors, such as <a href="postscreen.8.html">postscreen(8)</a>, <a href="tlsmgr.8.html">tlsmgr(8)</a> or <a href="verify.8.html">verify(8)</a>.
+This problem does not exist with other Postfix databases. </p>
+</dd>
-<p> <strong>Non-obvious recovery with <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a>
-from a corrupted database. </strong></p>
+<dt> Impact: </dt> <dd> <p> This failure temporarily affects Postfix
+availability. The daemon restarts automatically and tries to open
+the database again as described next. </p> </dd>
-<dl>
+<dt> Mitigation: </dt> <dd> <p> When a Postfix daemon opens an LMDB
+file larger than <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a>/3, it logs a warning and uses a
+larger size limit instead: </p>
-<dt> Problem: </dt> <dd> <p> You cannot rebuild a corrupted LMDB
-database simply by running <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>. This problem
-does not exist with other Postfix databases. </p> </dd>
+<p> <tt> warning: <i>filename</i>.<a href="LMDB_README.html">lmdb</a>: file size 15024128 ≥
+(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
+</p>
-<dt> Background: </dt> <dd> <p> The reason for this limitation is
-that the Postfix LMDB database client does not truncate the database
-file. Instead it attempts to save the "drop" request and subsequent
-"store" requests to a transaction for later processing. That is
-obviously not possible with a corrupted database file. </p> </dd>
+<p> This can be used to automate recovery and avoid the need for
+human intervention. Each time the daemon runs into a "database full"
+error, it restarts and uses a 3x larger size limit. The "database
+full" error will disappear, at least for a while. </p>
-<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand,
-then rebuild the file with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command.
-</p> </dd>
+<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
+sure that <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> > 3x the largest LMDB file size. </p>
+</dd> </dl>
</dl>
-<p> <strong>Incompatibility with <a href="tlsmgr.8.html">tlsmgr(8)</a>. </strong></p>
+<p> <strong>Non-obvious recovery with <a href="postmap.1.html">postmap(1)</a>/<a href="postalias.1.html">postalias(1)</a>/<a href="tlsmgr.8.html">tlsmgr(8)</a>
+from a corrupted database. </strong></p>
<dl>
-<dt> Problem: </dt> <dd> <p> The Postfix LMDB database client never
-commits any <a href="tlsmgr.8.html">tlsmgr(8)</a> transaction. This problem does not exist with
-other Postfix databases. </p> </dd>
-
-<dt> Background: </dt> <dd> <p> Instead, it creates a single
-transaction that accumulates a "drop" request and all <a href="tlsmgr.8.html">tlsmgr(8)</a>
-"store" requests that are made during the lifetime of the process.
-</p> </dd>
-
-<dt> Solution: </dt> <dd> <p> This requires changes to the Postfix
-dictionary API, or to the Postfix LMDB database client. </p> </dd>
-
-<dt> Problem: </dt> <dd> <p> The Postfix LMDB database client breaks
-how <a href="tlsmgr.8.html">tlsmgr(8)</a> automatically recovers from a corrupted database file.
-This problem does not exist with other Postfix databases. <p> </dd>
+<dt> Problem: </dt> <dd> <p> You cannot rebuild a corrupted LMDB
+database simply by running <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>, or by waiting
+until the <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts automatically. This problem
+does not exist with other Postfix databases. </p> </dd>
<dt> Background: </dt> <dd> <p> The Postfix LMDB database client
does not truncate the database file. Instead it attempts to create
-a transaction which obviously is not possible when the database
-file is corrupted. </p> </dd>
-
-<dt> Impact: </dt> <dd> <p> The <a href="tlsmgr.8.html">tlsmgr(8)</a> process will keep crashing
-until someone removes the ".lmdb" file. </p> </dd>
+a transaction for a "drop" request and subsequent "store" requests.
+That is obviously not possible with a corrupted database file. </p>
+</dd>
-<dt> Recovery: </dt> <dd> <p> Remove the the ".lmdb" file by hand,
-and wait until the <a href="tlsmgr.8.html">tlsmgr(8)</a> process restarts. </p> </dd>
+<dt> Impact: </dt> <dd> <p> Postfix does not process mail until
+someone fixes the problem. </p> </dd>
-</dl>
+<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand,
+then rebuild the file with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command,
+or wait until the <a href="tlsmgr.8.html">tlsmgr(8)</a> daemon restarts automatically. </p>
+</dd>
-</body>
+<dt> Prevention: </dt> <dd> <p> Arrange your file systems such that
+they never run out of free space. </p> </dd> </dl>
-</html>
+</dl>
</ol>
-<dl> <dt> Caution: </dt> <dd> <p> The current Postfix LMDB client
-has <a href="#limitations">unexpected limitations</a> that don't
-exist with other Postfix databases. For this reason, LMDB support
-will not be part of the stable Postfix release without further
-changes to the Postfix LMDB client or the Postfix dictionary API.
-</p> </dd> </dl>
+<dl> <dt> Note: </dt> <dd> <p> The Postfix LMDB client implementation
+introduces <a href="#limitations">unexpected failure modes</a> that
+don't exist with other Postfix databases. Don't just yet abandon
+CDB. </p> </dd> </dl>
<h2><a name="with_lmdb">Building Postfix with OpenLDAP LMDB support</a></h2>
More information is available at
http://highlandsun.com/hyc/mdb/. </p>
-<h2><a name="limitations">Limitations of Postfix LMDB databases.
-</a> </h2>
+<h2><a name="limitations">Unexpected failure modes of Postfix LMDB
+databases. </a> </h2>
-<p> <strong>Unexpected postmap(1)/postalias(1) "database full" errors.
-</strong></p>
+<p> As documented below, conversion to LMDB introduces a number of
+failure modes that don't exist with other Postfix databases. </p>
+
+<p> <strong>Unexpected postmap(1)/postalias(1) "database full"
+errors. </strong></p>
<dl>
-<dt> Problem: </dt> <dd> <p> Even if the "postmap lmdb:filename"
-command succeeds, the exact same command (with the exact same input
-data) may fail subsequently with an MDB_MAP_FULL error. This problem
-does not exist with other Postfix databases. </p> </dd>
+<dt> Problem: </dt> <dd> <p> The "postmap lmdb:filename" command
+fails with an MDB_MAP_FULL error. This problem does not exist with
+other Postfix databases. </p> </dd>
-<dt> Background: </dt>
+<dt> Background: </dt>
-<dd>
+<dd>
<p> LMDB databases have a hard size limit (configured with the
lmdb_map_size configuration parameter). </p>
<p> When executing "postmap lmdb:filename", the Postfix LMDB database
-client does not truncate the database file. Instead it saves the
-"drop" request and subsequent "store" requests to a transaction
-(which takes up space in addition to the existing data), and commits
-the transaction when it closes the database. Only then can the
-space for old data be reused. </p>
+client stores the new data in a transaction which takes up space
+in addition to the existing data, and commits the transaction when
+it closes the database. Only then can the space for old data be
+reused. </p>
</dd>
availability, because the old data still exists in the database.
</p> </dd>
-<dt> Recovery: </dt> <dd> <p> Increase the lmdb_map_size limit in
-main.cf, and retry the postmap(1) or postalias(1) command. </p>
-</dd>
+<dt> Mitigation: </dt> <dd> <p> When the postmap(1) or postalias(1)
+command opens an LMDB file larger than lmdb_map_size/3, it logs a
+warning and uses a larger size limit instead: </p>
-</dl>
+<p> <tt> warning: <i>filename</i>.lmdb: file size 15024128 ≥
+(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
+</p>
-<p> <strong>Postfix daemon "database full" errors. </strong></p>
+<p> This can be used to automate recovery and avoid the need for
+human intervention. Just keep running "postmap lmdb:filename".
+After each failure it will use a 3x larger size limit, and eventually
+the "database full" error will disappear. </p>
-<dl>
-
-<dt> Problem: </dt> <dd> <p> "database full" errors with daemon
-programs such as postscreen(8), tlsmgr(8) or verify(8). This problem
-does not exist with other Postfix databases. </p> </dd>
+<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
+sure that lmdb_map_size > 3x the largest LMDB file size. </p>
+</dd> </dl>
-<dt> Impact: </dt> <dd> <p> Postfix does not process mail until
-someone fixes the problem. </p> </dd>
+<p> <strong>Unexpected Postfix daemon "database full" errors.
+</strong></p>
-<dt> Recovery: </dt> <dd> <p> Increase the lmdb_map_size limit in
-main.cf, and "reload" Postfix. </p> </dd>
+<dl>
-</dl>
+<dt> Problem: </dt> <dd> <p> Postfix daemon programs fail with
+"database full" errors, such as postscreen(8), tlsmgr(8) or verify(8).
+This problem does not exist with other Postfix databases. </p>
+</dd>
-<p> <strong>Non-obvious recovery with postmap(1)/postalias(1)
-from a corrupted database. </strong></p>
+<dt> Impact: </dt> <dd> <p> This failure temporarily affects Postfix
+availability. The daemon restarts automatically and tries to open
+the database again as described next. </p> </dd>
-<dl>
+<dt> Mitigation: </dt> <dd> <p> When a Postfix daemon opens an LMDB
+file larger than lmdb_map_size/3, it logs a warning and uses a
+larger size limit instead: </p>
-<dt> Problem: </dt> <dd> <p> You cannot rebuild a corrupted LMDB
-database simply by running postmap(1) or postalias(1). This problem
-does not exist with other Postfix databases. </p> </dd>
+<p> <tt> warning: <i>filename</i>.lmdb: file size 15024128 ≥
+(lmdb map size limit 16777216)/3 -- using a larger map size limit</tt>
+</p>
-<dt> Background: </dt> <dd> <p> The reason for this limitation is
-that the Postfix LMDB database client does not truncate the database
-file. Instead it attempts to save the "drop" request and subsequent
-"store" requests to a transaction for later processing. That is
-obviously not possible with a corrupted database file. </p> </dd>
+<p> This can be used to automate recovery and avoid the need for
+human intervention. Each time the daemon runs into a "database full"
+error, it restarts and uses a 3x larger size limit. The "database
+full" error will disappear, at least for a while. </p>
-<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand,
-then rebuild the file with the postmap(1) or postalias(1) command.
-</p> </dd>
+<dt> Prevention: </dt> <dd> <p> Monitor your LMDB files and make
+sure that lmdb_map_size > 3x the largest LMDB file size. </p>
+</dd> </dl>
</dl>
-<p> <strong>Incompatibility with tlsmgr(8). </strong></p>
+<p> <strong>Non-obvious recovery with postmap(1)/postalias(1)/tlsmgr(8)
+from a corrupted database. </strong></p>
<dl>
-<dt> Problem: </dt> <dd> <p> The Postfix LMDB database client never
-commits any tlsmgr(8) transaction. This problem does not exist with
-other Postfix databases. </p> </dd>
-
-<dt> Background: </dt> <dd> <p> Instead, it creates a single
-transaction that accumulates a "drop" request and all tlsmgr(8)
-"store" requests that are made during the lifetime of the process.
-</p> </dd>
-
-<dt> Solution: </dt> <dd> <p> This requires changes to the Postfix
-dictionary API, or to the Postfix LMDB database client. </p> </dd>
-
-<dt> Problem: </dt> <dd> <p> The Postfix LMDB database client breaks
-how tlsmgr(8) automatically recovers from a corrupted database file.
-This problem does not exist with other Postfix databases. <p> </dd>
+<dt> Problem: </dt> <dd> <p> You cannot rebuild a corrupted LMDB
+database simply by running postmap(1) or postalias(1), or by waiting
+until the tlsmgr(8) daemon restarts automatically. This problem
+does not exist with other Postfix databases. </p> </dd>
<dt> Background: </dt> <dd> <p> The Postfix LMDB database client
does not truncate the database file. Instead it attempts to create
-a transaction which obviously is not possible when the database
-file is corrupted. </p> </dd>
-
-<dt> Impact: </dt> <dd> <p> The tlsmgr(8) process will keep crashing
-until someone removes the ".lmdb" file. </p> </dd>
+a transaction for a "drop" request and subsequent "store" requests.
+That is obviously not possible with a corrupted database file. </p>
+</dd>
-<dt> Recovery: </dt> <dd> <p> Remove the the ".lmdb" file by hand,
-and wait until the tlsmgr(8) process restarts. </p> </dd>
+<dt> Impact: </dt> <dd> <p> Postfix does not process mail until
+someone fixes the problem. </p> </dd>
-</dl>
+<dt> Recovery: </dt> <dd> <p> First delete the ".lmdb" file by hand,
+then rebuild the file with the postmap(1) or postalias(1) command,
+or wait until the tlsmgr(8) daemon restarts automatically. </p>
+</dd>
-</body>
+<dt> Prevention: </dt> <dd> <p> Arrange your file systems such that
+they never run out of free space. </p> </dd> </dl>
-</html>
+</dl>
* Patches change both the patchlevel and the release date. Snapshots have no
* patchlevel; they change the release date only.
*/
-#define MAIL_RELEASE_DATE "20130316"
+#define MAIL_RELEASE_DATE "20130317"
#define MAIL_VERSION_NUMBER "2.11"
#ifdef SNAPSHOT
#include <myflock.h>
#include <warn_stat.h>
-#if defined(HAS_LMDB) && defined(SNAPSHOT)
+#ifdef HAS_LMDB
#ifdef PATH_LMDB_H
#include PATH_LMDB_H
#else
DICT_TYPE_HASH, mkmap_hash_open,
DICT_TYPE_BTREE, mkmap_btree_open,
#endif
-#if defined(HAS_LMDB) && defined(SNAPSHOT)
+#ifdef HAS_LMDB
DICT_TYPE_LMDB, mkmap_lmdb_open,
#endif
DICT_TYPE_FAIL, mkmap_fail_open,
key_buffer = vstring_alloc(100);
value_buffer = vstring_alloc(100);
if ((open_flags & O_TRUNC) == 0) {
+ /* Incremental mode. */
source_fp = VSTREAM_IN;
vstream_control(source_fp, VSTREAM_CTL_PATH, "stdin", VSTREAM_CTL_END);
- } else if (strcmp(map_type, DICT_TYPE_PROXY) == 0) {
- msg_fatal("can't create maps via the proxy service");
- } else if ((source_fp = vstream_fopen(path_name, O_RDONLY, 0)) == 0) {
- msg_fatal("open %s: %m", path_name);
+ } else {
+ /* Create database. */
+ if (strcmp(map_type, DICT_TYPE_PROXY) == 0)
+ msg_fatal("can't create maps via the proxy service");
+ if ((source_fp = vstream_fopen(path_name, O_RDONLY, 0)) == 0)
+ msg_fatal("open %s: %m", path_name);
}
+ dict_flags |= DICT_FLAG_BULK_UPDATE;
if (fstat(vstream_fileno(source_fp), &st) < 0)
msg_fatal("fstat %s: %m", path_name);
*/
line_buffer = vstring_alloc(100);
if ((open_flags & O_TRUNC) == 0) {
+ /* Incremental mode. */
source_fp = VSTREAM_IN;
vstream_control(source_fp, VSTREAM_CTL_PATH, "stdin", VSTREAM_CTL_END);
- } else if (strcmp(map_type, DICT_TYPE_PROXY) == 0) {
- msg_fatal("can't create maps via the proxy service");
- } else if ((source_fp = vstream_fopen(path_name, O_RDONLY, 0)) == 0) {
- msg_fatal("open %s: %m", path_name);
+ } else {
+ /* Create database. */
+ if (strcmp(map_type, DICT_TYPE_PROXY) == 0)
+ msg_fatal("can't create maps via the proxy service");
+ if ((source_fp = vstream_fopen(path_name, O_RDONLY, 0)) == 0)
+ msg_fatal("open %s: %m", path_name);
}
+ dict_flags |= DICT_FLAG_BULK_UPDATE;
if (fstat(vstream_fileno(source_fp), &st) < 0)
msg_fatal("fstat %s: %m", path_name);
* Delete behind. This is a no-op if an expired cache entry was updated
* in the mean time. Use the saved lookup criteria so that the "delete
* behind" operation works as promised.
+ *
+ * The delete-behind strategy assumes that all updates are made by a single
+ * process. Otherwise, delete-behind may remove an entry that was updated
+ * after it was scheduled for deletion.
*/
if (cp->flags & TLS_SCACHE_FLAG_DEL_SAVED_CURSOR) {
cp->flags &= ~TLS_SCACHE_FLAG_DEL_SAVED_CURSOR;
/*
/* const char *dict_flags_str(dict_flags)
/* int dict_flags;
+/*
+/* int dict_flags_mask(names)
+/* const char *names;
/* DESCRIPTION
/* This module maintains a collection of name-value dictionaries.
/* Each dictionary has its own name and has its own methods to read
/* dict_flags_str() returns a printable representation of the
/* specified dictionary flags. The result is overwritten upon
/* each call.
+/*
+/* dict_flags_mask() returns the bitmask for the specified
+/* comma/space-separated dictionary flag names.
/* SEE ALSO
/* htable(3)
/* BUGS
"fold_fix", DICT_FLAG_FOLD_FIX, /* case-fold with fixed-case key map */
"fold_mul", DICT_FLAG_FOLD_MUL, /* case-fold with multi-case key map */
"open_lock", DICT_FLAG_OPEN_LOCK, /* permanent lock upon open */
+ "bulk_update", DICT_FLAG_BULK_UPDATE, /* bulk update if supported */
0,
};
-/* dict_flags_str - convert mask to string for debugging purposes */
+/* dict_flags_str - convert bitmask to symbolic flag names */
const char *dict_flags_str(int dict_flags)
{
return (str_name_mask_opt(buf, "dictionary flags", dict_mask, dict_flags,
NAME_MASK_NUMBER | NAME_MASK_PIPE));
}
+
+/* dict_flags_mask - convert symbolic flag names to bitmask */
+
+int dict_flags_mask(const char *names)
+{
+ return (name_mask("dictionary flags", dict_mask, names));
+}
#define DICT_FLAG_FOLD_MUL (1<<15) /* case-fold key with multi-case map */
#define DICT_FLAG_FOLD_ANY (DICT_FLAG_FOLD_FIX | DICT_FLAG_FOLD_MUL)
#define DICT_FLAG_OPEN_LOCK (1<<16) /* perm lock if not multi-writer safe */
+#define DICT_FLAG_BULK_UPDATE (1<<17) /* optimize for bulk updates */
/* IMPORTANT: Update the dict_mask[] table when the above changes */
extern int dict_changed(void);
extern const char *dict_changed_name(void);
extern const char *dict_flags_str(int);
+extern int dict_flags_mask(const char *);
/*
* Driver for interactive or scripted tests.
/* until a cache cleanup run is completed. Some entries may
/* never be removed when the process max_idle time is less
/* than the time needed to make a full pass over the cache.
+/*
+/* The delete-behind strategy assumes that all updates are
+/* made by a single process. Otherwise, delete-behind may
+/* remove an entry that was updated after it was scheduled for
+/* deletion.
/* LICENSE
/* .ad
/* .fi
/* SYNOPSIS
/* #include <dict_lmdb.h>
/*
+/* size_t dict_lmdb_map_size;
+/*
/* DICT *dict_lmdb_open(path, open_flags, dict_flags)
/* const char *name;
/* const char *path;
/* dict_lmdb_open() opens the named LMDB database and makes it available
/* via the generic interface described in dict_open(3).
/*
-/* The dict_lmdb_map_size variable specifies a non-default per-table
-/* memory map size. The map size is 10MB. The map size is also the
-/* maximum size the table can grow to, so it must be set large enough
+/* The dict_lmdb_map_size variable specifies a non-default
+/* per-table memory map size. The map size is also the maximum
+/* size the table can grow to, so it must be set large enough
/* to accomodate the largest tables in use.
+/*
+/* As a safety measure, when Postfix opens an LMDB database it
+/* will set the memory size limit to at least 3x the
+/* ".lmdb" file size, so that there is room for the file to
+/* grow. This ensures continued availability of Postfix daemon
+/* processes.
/* DIAGNOSTICS
/* Fatal errors: cannot open file, file write error, out of memory.
/* SEE ALSO
#include "sys_defs.h"
-#if defined(HAS_LMDB) && defined(SNAPSHOT)
+#ifdef HAS_LMDB
/* System library. */
#include <sys/stat.h>
#include <string.h>
#include <unistd.h>
+#include <limits.h>
#ifdef PATH_LMDB_H
#include PATH_LMDB_H
DICT dict; /* generic members */
MDB_env *env; /* LMDB environment */
MDB_dbi dbi; /* database handle */
- MDB_txn *txn; /* write transaction for O_TRUNC */
+ MDB_txn *txn; /* bulk update transaction */
MDB_cursor *cursor; /* for sequence ops */
VSTRING *key_buf; /* key buffer */
VSTRING *val_buf; /* result buffer */
if ((status = mdb_env_create(&env)))
msg_fatal("env_create %s: %s", mdb_path, mdb_strerror(status));
+ /*
+ * For continued availability, try to ensure that the LMDB size limit is
+ * at least 3x the current LMDB file size. This should be sufficient for
+ * short-lived Postfix daemon processes.
+ */
+#ifdef SIZE_T_MAX
+#define SIZE_T_MAX __MAXINT__(size_t)
+#endif
+
+ if (stat(mdb_path, &st) == 0 && st.st_size >= dict_lmdb_map_size / 2) {
+ msg_warn("%s: file size %lu >= (%s map size limit %ld)/3 -- "
+ "using a larger map size limit",
+ mdb_path, (unsigned long) st.st_size,
+ DICT_TYPE_LMDB, (long) dict_lmdb_map_size);
+ dict_lmdb_map_size = 3 * st.st_size;
+ if (dict_lmdb_map_size / 3 != st.st_size)
+ dict_lmdb_map_size = SIZE_T_MAX;
+ }
if ((status = mdb_env_set_mapsize(env, dict_lmdb_map_size)))
msg_fatal("env_set_mapsize %s: %s", mdb_path, mdb_strerror(status));
/*
* mdb_open requires a txn, but since the default DB always exists in an
- * LMDB environment, we don't need to do anything else with the txn.
+ * LMDB environment, we usually don't need to do anything else with the
+ * txn.
*/
if ((status = mdb_open(txn, NULL, 0, &dbi)))
msg_fatal("mdb_open %s: %s", mdb_path, mdb_strerror(status));
/*
- * However, if O_TRUNC was specified, we need to do it now. Also with
- * O_TRUNC we keep this write txn for as long as the database is open,
- * since we'll probably be doing a bulk import immediately after.
+ * Cases where we use the mdb_open transaction:
+ *
+ * - With O_TRUNC we make the "drop" request before populating the database.
+ *
+ * - With DICT_FLAG_BULK_UPDATE we commit the transaction when the database
+ * is closed.
*/
if (open_flags & O_TRUNC) {
if ((status = mdb_drop(txn, dbi, 0)))
msg_fatal("truncate %s: %s", mdb_path, mdb_strerror(status));
- } else {
+ if ((dict_flags & DICT_FLAG_BULK_UPDATE) == 0) {
+ if ((status = mdb_txn_commit(txn)))
+ msg_fatal("truncate %s: %s", mdb_path, mdb_strerror(status));
+ txn = NULL;
+ }
+ } else if ((env_flags & MDB_RDONLY) != 0
+ || (dict_flags & DICT_FLAG_BULK_UPDATE) == 0) {
mdb_txn_abort(txn);
txn = NULL;
}
-
dict_lmdb = (DICT_LMDB *) dict_alloc(DICT_TYPE_LMDB, path, sizeof(*dict_lmdb));
dict_lmdb->dict.lookup = dict_lmdb_lookup;
dict_lmdb->dict.update = dict_lmdb_update;
dict_lmdb->dict.sequence = dict_lmdb_sequence;
dict_lmdb->dict.close = dict_lmdb_close;
dict_lmdb->dict.lock = dict_lmdb_lock;
- dict_lmdb->dict.stat_fd = open(mdb_path, O_RDONLY);
+ if ((dict_lmdb->dict.stat_fd = open(mdb_path, O_RDONLY)) < 0)
+ msg_fatal("dict_lmdb_open: %s: %m", mdb_path);
if (fstat(dict_lmdb->dict.stat_fd, &st) < 0)
msg_fatal("dict_lmdb_open: fstat: %m");
dict_lmdb->dict.mtime = st.st_mtime;
DICT_TYPE_HASH, dict_hash_open,
DICT_TYPE_BTREE, dict_btree_open,
#endif
-#if defined(HAS_LMDB) && defined(SNAPSHOT)
+#ifdef HAS_LMDB
DICT_TYPE_LMDB, dict_lmdb_open,
#endif
#ifdef HAS_NIS
static NORETURN usage(char *myname)
{
- msg_fatal("usage: %s type:file read|write|create [fold] [sync]", myname);
+ msg_fatal("usage: %s type:file read|write|create [flags...]", myname);
}
void dict_test(int argc, char **argv)
const char *key;
const char *value;
int ch;
- int dict_flags = DICT_FLAG_LOCK | DICT_FLAG_DUP_REPLACE;
+ int dict_flags = 0;
int n;
int rc;
open_flags = O_RDONLY;
else
msg_fatal("unknown access mode: %s", argv[2]);
- for (n = 2; argv[optind + n]; n++) {
- if (strcasecmp(argv[optind + 2], "fold") == 0)
- dict_flags |= DICT_FLAG_FOLD_ANY;
- else if (strcasecmp(argv[optind + 2], "sync") == 0)
- dict_flags |= DICT_FLAG_SYNC_UPDATE;
- else if (strcasecmp(argv[optind + 2], "open_lock") == 0) {
- dict_flags |= DICT_FLAG_OPEN_LOCK;
- dict_flags &= ~DICT_FLAG_LOCK;
- } else
- usage(argv[0]);
- }
+ for (n = 2; argv[optind + n]; n++)
+ dict_flags |= dict_flags_mask(argv[optind + 2]);
+ if ((dict_flags & DICT_FLAG_OPEN_LOCK) == 0)
+ dict_flags |= DICT_FLAG_LOCK;
+ if ((dict_flags & (DICT_FLAG_DUP_WARN | DICT_FLAG_DUP_IGNORE)) == 0)
+ dict_flags |= DICT_FLAG_DUP_REPLACE;
vstream_fflush(VSTREAM_OUT);
dict_name = argv[optind];
dict_allow_surrogate = 1;
projects / thirdparty / postfix.git / commitdiff
