-TDICT_FAIL
-TDICT_HT
-TDICT_LDAP
+-TDICT_LMDB
-TDICT_MC
-TDICT_MYSQL
-TDICT_NI
Bugfix: an error handler for smtp_tls_policy_maps lookups
was never invoked. File: smtp/smtp_session.c.
+
+20130315
+
+ Feature: preliminary LMDB (memory-mapped persistent file)
+ support by Howard Chu. This feature has a number of unexpected
+ limitations and is therefore "snapshot only", i.e. it will
+ not be part of a stable release. Files: proto/postconf.proto,
+ proto/LMDB_README.html, proto/DATABASE_README.html,
+ util/dict_lmdb.[hc], util/dict_open.c, global/mkmap_lmdb.[hc],
+ global/mkmap_open.c, postconf/postconf.c.
i\bin\bnt\bte\ber\brn\bna\bal\bl
A non-shared, in-memory hash table. Its content are lost when a process
terminates.
+ l\blm\bmd\bdb\bb
+ The OpenLDAP LMDB database (a memory-mapped, persistent file). Database
+ files are created with the postmap(1) or postalias(1) command. The
+ database name as used in "lmdb:table" is the database file name without
+ the ".lmdb" suffix. This database type has unexpected limitations and
+ is therefore not part of the stable Postfix release.
l\bld\bda\bap\bp (read-only)
Perform lookups using the LDAP protocol. Configuration details are
given in the ldap_table(5).
and so on. In some cases, optimization is turned off automatically.
-4\b4.\b.3\b3 -\b- B\bBu\bui\bil\bld\bdi\bin\bng\bg w\bwi\bit\bth\bh o\bop\bpt\bti\bio\bon\bna\bal\bl e\bex\bxt\bte\ben\bns\bsi\bio\bon\bns\bs
+4\b4.\b.3\b3 -\b- B\bBu\bui\bil\bld\bdi\bin\bng\bg w\bwi\bit\bth\bh o\bop\bpt\bti\bio\bon\bna\bal\bl f\bfe\bea\bat\btu\bur\bre\bes\bs
By default, Postfix builds as a mail system with relatively few bells and
whistles. Support for third-party databases etc. must be configured when
Postfix is compiled. The following documents describe how to build Postfix with
-support for extensions:
+support for optional features:
_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b
- |P\bPo\bos\bst\btf\bfi\bix\bx e\bex\bxt\bte\ben\bns\bsi\bio\bon\bn |D\bDo\boc\bcu\bum\bme\ben\bnt\bt |A\bAv\bva\bai\bil\bla\bab\bbi\bil\bli\bit\bty\by|
+ |O\bOp\bpt\bti\bio\bon\bna\bal\bl f\bfe\bea\bat\btu\bur\bre\be |D\bDo\boc\bcu\bum\bme\ben\bnt\bt |A\bAv\bva\bai\bil\bla\bab\bbi\bil\bli\bit\bty\by|
|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b |
|Berkeley DB database |DB_README |Postfix 1.0 |
|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b |
+ |LMDB database |LMDB_README |Postfix 2.11|
+ |_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b |
|LDAP database |LDAP_README |Postfix 1.0 |
|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b|_\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b _\b |
|MySQL database |MYSQL_README |Postfix 1.0 |
--- /dev/null
+P\bPo\bos\bst\btf\bfi\bix\bx O\bOp\bpe\ben\bnL\bLD\bDA\bAP\bP L\bLM\bMD\bDB\bB H\bHo\bow\bwt\bto\bo
+
+-------------------------------------------------------------------------------
+
+I\bIn\bnt\btr\bro\bod\bdu\buc\bct\bti\bio\bon\bn
+
+Postfix uses databases of various kinds to store and look up information.
+Postfix databases are specified as "type:name". OpenLDAP LMDB implements the
+Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is
+the name of the database file without the ".lmdb" suffix. OpenLDAP LMDB
+databases are maintained with the postmap(1) command.
+
+This document describes:
+
+ 1. How to build Postfix with OpenLDAP LMDB support.
+
+ 2. How to configure LMDB settings.
+
+ 3. Missing pthread library trouble.
+
+ Caution: the current Postfix LMDB client has unexpected limitations. For
+ this reason, LMDB support is not part of the stable Postfix release.
+
+B\bBu\bui\bil\bld\bdi\bin\bng\bg P\bPo\bos\bst\btf\bfi\bix\bx w\bwi\bit\bth\bh O\bOp\bpe\ben\bnL\bLD\bDA\bAP\bP L\bLM\bMD\bDB\bB s\bsu\bup\bpp\bpo\bor\brt\bt
+
+Postfix normally does not enable OpenLDAP LMDB support. To build Postfix after
+you installed OpenLDAP LMDB from source code, use something like:
+
+ % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-L/usr/local/lib -llmdb"
+ % make
+
+Solaris needs this:
+
+ % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
+ % make
+
+The exact pathnames depend on how OpenLDAP LMDB was installed.
+
+C\bCo\bon\bnf\bfi\big\bgu\bur\bre\be L\bLM\bMD\bDB\bB s\bse\bet\btt\bti\bin\bng\bgs\bs
+
+Postfix provides a configuration parameter that controls how large an OpenLDAP
+LMDB database may grow.
+
+ * lmdb_map_size (default: 10 MBytes per table). This setting controls how
+ large any OpenLDAP LMDB database may grow. It must be set large enough to
+ accommodate the largest table that Postfix will use.
+
+M\bMi\bis\bss\bsi\bin\bng\bg p\bpt\bth\bhr\bre\bea\bad\bd l\bli\bib\bbr\bra\bar\bry\by t\btr\bro\bou\bub\bbl\ble\be
+
+When building Postfix fails with:
+
+ undefined reference to `pthread_mutexattr_destroy'
+ undefined reference to `pthread_mutexattr_init'
+ undefined reference to `pthread_mutex_lock'
+
+Add the "-lpthread" library to the "make makefiles" command.
+
+ % make makefiles .... AUXLIBS="... -lpthread"
+
+Source code for OpenLDAP LMDB is available at http://www.openldap.org. More
+information is available at http://highlandsun.com/hyc/mdb/.
+
+L\bLi\bim\bmi\bit\bta\bat\bti\bio\bon\bns\bs o\bof\bf P\bPo\bos\bst\btf\bfi\bix\bx L\bLM\bMD\bDB\bB d\bda\bat\bta\bab\bba\bas\bse\bes\bs.\b.
+
+ * U\bUn\bne\bex\bxp\bpe\bec\bct\bte\bed\bd p\bpo\bos\bst\btm\bma\bap\bp/\b/p\bpo\bos\bst\bta\bal\bli\bia\bas\bs "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
+
+ Even if the "postmap filename" command succeeds, the exact same command,
+ with the exact same input data, may fail subsequently with an MDB_MAP_FULL
+ error. The reason is that unlike other Postfix databases such as "hash" or
+ "btree",
+
+ o LMDB databases have a hard size limit (configured with the
+ lmdb_map_size configuration parameter).
+
+ o The Postfix LMDB database client does not implement the "truncate"
+ operation. Instead it saves all store requests to a transaction (which
+ takes up space in addition to the existing data), and commits the
+ transaction when the database is closed. Only then can the space for
+ old data be reused.
+
+ This postmap(1) or postalias(1) command failure does not affect Postfix
+ availability, because the old data still exists in the database.
+
+ To recover, increase the lmdb_map_size limit in main.cf, and retry the
+ postmap(1) or postalias(1) command.
+
+ * P\bPo\bos\bst\btf\bfi\bix\bx d\bda\bae\bem\bmo\bon\bn "\b"d\bda\bat\bta\bab\bba\bas\bse\be f\bfu\bul\bll\bl"\b" e\ber\brr\bro\bor\brs\bs.\b.
+
+ Unfortunately, "database full" problems will affect Postfix availability
+ with daemon programs such as postscreen(8), tlsmgr(8) or verify(8). These
+ daemon programs will continue to fail until someone increases the
+ lmdb_map_size parameter value. Meanwhile, mail processing comes to a halt.
+
+ * N\bNo\bon\bn-\b-o\bob\bbv\bvi\bio\bou\bus\bs r\bre\bec\bco\bov\bve\ber\bry\by f\bfr\bro\bom\bm a\ba c\bco\bor\brr\bru\bup\bpt\bte\bed\bd d\bda\bat\bta\bab\bba\bas\bse\be.\b.
+
+ Unlike other Postfix databases such as "hash" or "btree", you can't rebuild
+ a corrupted LMDB database simply by running postmap(1) or postalias(1), as
+ those commands will crash, too.
+
+ The reason for this limitation is that the Postfix LMDB database client
+ does not implement the database "truncate" operation. Instead it tries to
+ save all store requests to a transaction for later processing. That is
+ obviously not possible with a corrupted database file.
+
+ To recover, you must first delete the ".lmdb" file by hand, and only then
+ you can retry the postmap(1) or postalias(1) command.
+
+ * I\bIn\bnc\bco\bom\bmp\bpa\bat\bti\bib\bbi\bil\bli\bit\bty\by w\bwi\bit\bth\bh t\btl\bls\bsm\bmg\bgr\br(\b(8\b8)\b).\b.
+
+ The Postfix LMDB database client breaks tlsmgr(8) TLS session cache
+ management. Specifically, it breaks how tlsmgr(8) clears its TLS session
+ cache databases upon start-up, it breaks how tlsmgr(8) looks up new TLS
+ session cache entries, and it breaks how tlsmgr(8) automatically recovers
+ from a corrupted database file.
+
+ The reason for these limitations is that the Postfix LMDB database client
+ does not implement the database "truncate" operation. Instead it saves all
+ store requests to a transaction which it commits only when the database is
+ closed. Therefore, tlsmgr(8) will never find any entries that it stores
+ after opening its TLS session cache databases. And when the database
+ becomes corrupted, tlsmgr(8) will keep crashing until someone removes the
+ file ".lmdb" file by hand.
+
If you upgrade from Postfix 2.9 or earlier, read RELEASE_NOTES-2.10
before proceeding.
+
+Major changes with snapshot 20130315
+====================================
+
+Perliminary LMDB support by Howard Chu. This implementation has
+unexpected limitations that make it a poor fit, and for this reason
+the code is "snapshot only", i.e. it will not be part of the stable
+release. See LMDB_README for details.
<dd> A non-shared, in-memory hash table. Its content are lost when
a process terminates. </dd>
+<dt> <b>lmdb</b> </dt>
+
+<dd> The OpenLDAP LMDB database (a memory-mapped, persistent file).
+Database files are created with the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a>
+command. The database name as used in "<a href="LMDB_README.html">lmdb</a>:table" is the database
+file name without the ".lmdb" suffix. This database type has
+unexpected limitations and is therefore not part of the stable
+Postfix release.
+
<dt> <b>ldap</b> (read-only) </dt>
<dd> Perform lookups using the LDAP protocol. Configuration details
<p> and so on. In some cases, optimization is turned off automatically. </p>
-<h3>4.3 - Building with optional extensions</h3>
+<h3>4.3 - Building with optional features</h3>
By default, Postfix builds as a mail system with relatively few
bells and whistles. Support for third-party databases etc.
-must be configured when Postfix is compiled. The following documents describe how to build Postfix with support for extensions:
+must be configured when Postfix is compiled. The following documents
+describe how to build Postfix with support for optional features:
<blockquote>
<table border="1">
-<tr> <th>Postfix extension </th> <th>Document </th> <th>Availability</th>
+<tr> <th>Optional feature </th> <th>Document </th> <th>Availability</th>
</tr>
<tr> <td> Berkeley DB database</td> <td><a href="DB_README.html">DB_README</a></td> <td> Postfix
1.0 </td> </tr>
+<tr> <td> LMDB database</td> <td><a href="LMDB_README.html">LMDB_README</a></td> <td> Postfix
+2.11 </td> </tr>
+
<tr> <td> LDAP database</td> <td><a href="LDAP_README.html">LDAP_README</a></td> <td> Postfix
1.0 </td> </tr>
--- /dev/null
+<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
+ "http://www.w3.org/TR/html4/loose.dtd">
+
+<html>
+
+<head>
+
+<title>Postfix OpenLDAP LMDB Howto</title>
+
+<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
+
+</head>
+
+<body>
+
+<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix OpenLDAP LMDB Howto</h1>
+
+<hr>
+
+<h2>Introduction</h2>
+
+<p> Postfix uses databases of various kinds to store and look up
+information. Postfix databases are specified as "type:name".
+OpenLDAP LMDB implements the Postfix database type "lmdb".
+The name of a Postfix OpenLDAP LMDB database is the name
+of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases
+are maintained with the <a href="postmap.1.html">postmap(1)</a> command. </p>
+
+<p> This document describes: </p>
+
+<ol>
+
+<li> <p> How to build Postfix <a href="#with_lmdb">with OpenLDAP
+LMDB support</a>. </p>
+
+<li> <p> How to <a href="#configure">configure</a> LMDB settings. </p>
+
+<li> <p> Missing <a href="#pthread">pthread</a> library trouble. </p>
+
+</ol>
+
+<blockquote> <p> Caution: the current Postfix LMDB client has <a
+href="#limitations">unexpected limitations</a>. For this reason,
+LMDB support is not part of the stable Postfix release. </p>
+</blockquote>
+
+<h2><a name="with_lmdb">Building Postfix with OpenLDAP LMDB support</a></h2>
+
+<p> Postfix normally does not enable OpenLDAP LMDB support.
+To build Postfix after you installed OpenLDAP LMDB from
+source code, use something like: </p>
+
+<blockquote>
+<pre>
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-L/usr/local/lib -llmdb"
+% make
+</pre>
+</blockquote>
+
+<p> Solaris needs this: </p>
+
+<blockquote>
+<pre>
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
+% make
+</pre>
+</blockquote>
+
+<p> The exact pathnames depend on how OpenLDAP LMDB was installed. </p>
+
+<h2><a name="configure">Configure LMDB settings</a></h2>
+
+<p> Postfix provides a configuration parameter that controls how
+large an OpenLDAP LMDB database may grow. </p>
+
+<ul>
+
+<li> <p> <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> (default: 10 MBytes per
+table). This setting controls how large any OpenLDAP LMDB database
+may grow. It must be set large enough to accommodate the largest
+table that Postfix will use. </p>
+
+</ul>
+
+<h2><a name="pthread">Missing pthread library trouble</a></h2>
+
+<p> When building Postfix fails with: </p>
+
+<blockquote>
+<pre>
+undefined reference to `pthread_mutexattr_destroy'
+undefined reference to `pthread_mutexattr_init'
+undefined reference to `pthread_mutex_lock'
+</pre>
+</blockquote>
+
+<p> Add the "-lpthread" library to the "make makefiles" command. </p>
+
+<blockquote>
+<pre>
+% make makefiles .... AUXLIBS="... -lpthread"
+</pre>
+</blockquote>
+
+<p> Source code for OpenLDAP LMDB is available at
+<a href="http://www.openldap.org">http://www.openldap.org</a>.
+More information is available at
+<a href="http://highlandsun.com/hyc/mdb/">http://highlandsun.com/hyc/mdb/</a>. </p>
+
+<h2><a name="limitations">Limitations of Postfix LMDB databases. </a> </h2>
+
+<ul>
+
+<li> <p> <strong>Unexpected postmap/postalias "database full" errors.
+</strong></p>
+
+<p> Even if the "postmap filename" command succeeds, the exact same
+command, with the exact same input data, may fail subsequently with
+an MDB_MAP_FULL error. The reason is that unlike other Postfix
+databases such as "hash" or "btree", </p>
+
+<ul>
+
+<li> <p> LMDB databases have a hard size limit (configured with the
+<a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> configuration parameter). </p>
+
+<li> <p> The Postfix LMDB database client does not implement the
+"truncate" operation. Instead it saves all store requests to a
+transaction (which takes up space in addition to the existing data),
+and commits the transaction when the database is closed. Only then
+can the space for old data be reused. </p>
+
+</ul>
+
+<p> This <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command failure does not affect
+Postfix availability, because the old data still exists in the
+database. </p>
+
+<p> To recover, increase the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> limit in <a href="postconf.5.html">main.cf</a>, and
+retry the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command. </p>
+
+<li> <p> <strong>Postfix daemon "database full" errors. </strong></p>
+
+<p> Unfortunately, "database full" problems will affect Postfix
+availability with daemon programs such as <a href="postscreen.8.html">postscreen(8)</a>, <a href="tlsmgr.8.html">tlsmgr(8)</a>
+or <a href="verify.8.html">verify(8)</a>. These daemon programs will continue to fail until
+someone increases the <a href="postconf.5.html#lmdb_map_size">lmdb_map_size</a> parameter value. Meanwhile,
+mail processing comes to a halt. </p>
+
+<li> <p> <strong>Non-obvious recovery from a corrupted database.
+</strong></p>
+
+<p> Unlike other Postfix databases such as "hash" or "btree", you
+can't rebuild a corrupted LMDB database simply by running <a href="postmap.1.html">postmap(1)</a>
+or <a href="postalias.1.html">postalias(1)</a>, as those commands will crash, too. </p>
+
+<p> The reason for this limitation is that the Postfix LMDB database
+client does not implement the database "truncate" operation. Instead
+it tries to save all store requests to a transaction for later
+processing. That is obviously not possible with a corrupted database
+file. </p>
+
+<p> To recover, you must first delete the ".lmdb" file by hand, and
+only then you can retry the <a href="postmap.1.html">postmap(1)</a> or <a href="postalias.1.html">postalias(1)</a> command.
+</p>
+
+<li> <p> <strong>Incompatibility with <a href="tlsmgr.8.html">tlsmgr(8)</a>. </strong></p>
+
+<p> The Postfix LMDB database client breaks <a href="tlsmgr.8.html">tlsmgr(8)</a> TLS session
+cache management. Specifically, it breaks how <a href="tlsmgr.8.html">tlsmgr(8)</a> clears its
+TLS session cache databases upon start-up, it breaks how <a href="tlsmgr.8.html">tlsmgr(8)</a>
+looks up new TLS session cache entries, and it breaks how <a href="tlsmgr.8.html">tlsmgr(8)</a>
+automatically recovers from a corrupted database file. <p>
+
+<p> The reason for these limitations is that the Postfix LMDB
+database client does not implement the database "truncate" operation.
+Instead it saves all store requests to a transaction which it commits
+only when the database is closed. Therefore, <a href="tlsmgr.8.html">tlsmgr(8)</a> will never
+find any entries that it stores after opening its TLS session cache
+databases. And when the database becomes corrupted, <a href="tlsmgr.8.html">tlsmgr(8)</a> will
+keep crashing until someone removes the file ".lmdb" file by hand.
+</p>
+
+</ul>
+
+</body>
+
+</html>
this length; upon delivery, long lines are reconstructed. </p>
+</DD>
+
+<DT><b><a name="lmdb_map_size">lmdb_map_size</a>
+(default: 10485760)</b></DT><DD>
+
+<p>
+The per-table size limit for programs that create OpenLDAP LMDB
+tables. Specify a byte count.
+</p>
+
+<p>
+This feature is available in Postfix 2.11 and later.
+</p>
+
+
</DD>
<DT><b><a name="lmtp_address_preference">lmtp_address_preference</a>
.SH line_length_limit (default: 2048)
Upon input, long lines are chopped up into pieces of at most
this length; upon delivery, long lines are reconstructed.
+.SH lmdb_map_size (default: 10485760)
+The per-table size limit for programs that create OpenLDAP LMDB
+tables. Specify a byte count.
+.PP
+This feature is available in Postfix 2.11 and later.
.SH lmtp_address_preference (default: ipv6)
The LMTP-specific version of the smtp_address_preference
configuration parameter. See there for details.
s;\bipc_timeout\b;<a href="postconf.5.html#ipc_timeout">$&</a>;g;
s;\bipc_ttl\b;<a href="postconf.5.html#ipc_ttl">$&</a>;g;
s;\bline_length_limit\b;<a href="postconf.5.html#line_length_limit">$&</a>;g;
+ s;\blmdb_map_size\b;<a href="postconf.5.html#lmdb_map_size">$&</a>;g;
s;\blmtp_bind_address\b;<a href="postconf.5.html#lmtp_bind_address">$&</a>;g;
s;\blmtp_bind_address6\b;<a href="postconf.5.html#lmtp_bind_address6">$&</a>;g;
s;\blmtp_assume_final\b;<a href="postconf.5.html#lmtp_assume_final">$&</a>;g;
s/\b(mysql):/<a href="mysql_table.5.html">$1<\/a>:/g;
s/\b(nisplus):/<a href="nisplus_table.5.html">$1<\/a>:/g;
s/\b(ldap):/<a href="ldap_table.5.html">$1<\/a>:/g;
+ s/\b(lmdb):/<a href="LMDB_README.html">$1<\/a>:/g;
s/\b(regexp):/<a href="regexp_table.5.html">$1<\/a>:/g;
s/\b(sqlite):/<a href="sqlite_table.5.html">$1<\/a>:/g;
s/\b(static):/<a href="DATABASE_README.html#types">$1<\/a>:/g;
<dd> A non-shared, in-memory hash table. Its content are lost when
a process terminates. </dd>
+<dt> <b>lmdb</b> </dt>
+
+<dd> The OpenLDAP LMDB database (a memory-mapped, persistent file).
+Database files are created with the postmap(1) or postalias(1)
+command. The database name as used in "lmdb:table" is the database
+file name without the ".lmdb" suffix. This database type has
+unexpected limitations and is therefore not part of the stable
+Postfix release.
+
<dt> <b>ldap</b> (read-only) </dt>
<dd> Perform lookups using the LDAP protocol. Configuration details
<p> and so on. In some cases, optimization is turned off automatically. </p>
-<h3>4.3 - Building with optional extensions</h3>
+<h3>4.3 - Building with optional features</h3>
By default, Postfix builds as a mail system with relatively few
bells and whistles. Support for third-party databases etc.
-must be configured when Postfix is compiled. The following documents describe how to build Postfix with support for extensions:
+must be configured when Postfix is compiled. The following documents
+describe how to build Postfix with support for optional features:
<blockquote>
<table border="1">
-<tr> <th>Postfix extension </th> <th>Document </th> <th>Availability</th>
+<tr> <th>Optional feature </th> <th>Document </th> <th>Availability</th>
</tr>
<tr> <td> Berkeley DB database</td> <td>DB_README</td> <td> Postfix
1.0 </td> </tr>
+<tr> <td> LMDB database</td> <td>LMDB_README</td> <td> Postfix
+2.11 </td> </tr>
+
<tr> <td> LDAP database</td> <td>LDAP_README</td> <td> Postfix
1.0 </td> </tr>
--- /dev/null
+<!doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN"
+ "http://www.w3.org/TR/html4/loose.dtd">
+
+<html>
+
+<head>
+
+<title>Postfix OpenLDAP LMDB Howto</title>
+
+<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
+
+</head>
+
+<body>
+
+<h1><img src="postfix-logo.jpg" width="203" height="98" ALT="">Postfix OpenLDAP LMDB Howto</h1>
+
+<hr>
+
+<h2>Introduction</h2>
+
+<p> Postfix uses databases of various kinds to store and look up
+information. Postfix databases are specified as "type:name".
+OpenLDAP LMDB implements the Postfix database type "lmdb".
+The name of a Postfix OpenLDAP LMDB database is the name
+of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases
+are maintained with the postmap(1) command. </p>
+
+<p> This document describes: </p>
+
+<ol>
+
+<li> <p> How to build Postfix <a href="#with_lmdb">with OpenLDAP
+LMDB support</a>. </p>
+
+<li> <p> How to <a href="#configure">configure</a> LMDB settings. </p>
+
+<li> <p> Missing <a href="#pthread">pthread</a> library trouble. </p>
+
+</ol>
+
+<blockquote> <p> Caution: the current Postfix LMDB client has <a
+href="#limitations">unexpected limitations</a>. For this reason,
+LMDB support is not part of the stable Postfix release. </p>
+</blockquote>
+
+<h2><a name="with_lmdb">Building Postfix with OpenLDAP LMDB support</a></h2>
+
+<p> Postfix normally does not enable OpenLDAP LMDB support.
+To build Postfix after you installed OpenLDAP LMDB from
+source code, use something like: </p>
+
+<blockquote>
+<pre>
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-L/usr/local/lib -llmdb"
+% make
+</pre>
+</blockquote>
+
+<p> Solaris needs this: </p>
+
+<blockquote>
+<pre>
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+ AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
+% make
+</pre>
+</blockquote>
+
+<p> The exact pathnames depend on how OpenLDAP LMDB was installed. </p>
+
+<h2><a name="configure">Configure LMDB settings</a></h2>
+
+<p> Postfix provides a configuration parameter that controls how
+large an OpenLDAP LMDB database may grow. </p>
+
+<ul>
+
+<li> <p> lmdb_map_size (default: 10 MBytes per
+table). This setting controls how large any OpenLDAP LMDB database
+may grow. It must be set large enough to accommodate the largest
+table that Postfix will use. </p>
+
+</ul>
+
+<h2><a name="pthread">Missing pthread library trouble</a></h2>
+
+<p> When building Postfix fails with: </p>
+
+<blockquote>
+<pre>
+undefined reference to `pthread_mutexattr_destroy'
+undefined reference to `pthread_mutexattr_init'
+undefined reference to `pthread_mutex_lock'
+</pre>
+</blockquote>
+
+<p> Add the "-lpthread" library to the "make makefiles" command. </p>
+
+<blockquote>
+<pre>
+% make makefiles .... AUXLIBS="... -lpthread"
+</pre>
+</blockquote>
+
+<p> Source code for OpenLDAP LMDB is available at
+http://www.openldap.org.
+More information is available at
+http://highlandsun.com/hyc/mdb/. </p>
+
+<h2><a name="limitations">Limitations of Postfix LMDB databases. </a> </h2>
+
+<ul>
+
+<li> <p> <strong>Unexpected postmap/postalias "database full" errors.
+</strong></p>
+
+<p> Even if the "postmap filename" command succeeds, the exact same
+command, with the exact same input data, may fail subsequently with
+an MDB_MAP_FULL error. The reason is that unlike other Postfix
+databases such as "hash" or "btree", </p>
+
+<ul>
+
+<li> <p> LMDB databases have a hard size limit (configured with the
+lmdb_map_size configuration parameter). </p>
+
+<li> <p> The Postfix LMDB database client does not implement the
+"truncate" operation. Instead it saves all store requests to a
+transaction (which takes up space in addition to the existing data),
+and commits the transaction when the database is closed. Only then
+can the space for old data be reused. </p>
+
+</ul>
+
+<p> This postmap(1) or postalias(1) command failure does not affect
+Postfix availability, because the old data still exists in the
+database. </p>
+
+<p> To recover, increase the lmdb_map_size limit in main.cf, and
+retry the postmap(1) or postalias(1) command. </p>
+
+<li> <p> <strong>Postfix daemon "database full" errors. </strong></p>
+
+<p> Unfortunately, "database full" problems will affect Postfix
+availability with daemon programs such as postscreen(8), tlsmgr(8)
+or verify(8). These daemon programs will continue to fail until
+someone increases the lmdb_map_size parameter value. Meanwhile,
+mail processing comes to a halt. </p>
+
+<li> <p> <strong>Non-obvious recovery from a corrupted database.
+</strong></p>
+
+<p> Unlike other Postfix databases such as "hash" or "btree", you
+can't rebuild a corrupted LMDB database simply by running postmap(1)
+or postalias(1), as those commands will crash, too. </p>
+
+<p> The reason for this limitation is that the Postfix LMDB database
+client does not implement the database "truncate" operation. Instead
+it tries to save all store requests to a transaction for later
+processing. That is obviously not possible with a corrupted database
+file. </p>
+
+<p> To recover, you must first delete the ".lmdb" file by hand, and
+only then you can retry the postmap(1) or postalias(1) command.
+</p>
+
+<li> <p> <strong>Incompatibility with tlsmgr(8). </strong></p>
+
+<p> The Postfix LMDB database client breaks tlsmgr(8) TLS session
+cache management. Specifically, it breaks how tlsmgr(8) clears its
+TLS session cache databases upon start-up, it breaks how tlsmgr(8)
+looks up new TLS session cache entries, and it breaks how tlsmgr(8)
+automatically recovers from a corrupted database file. <p>
+
+<p> The reason for these limitations is that the Postfix LMDB
+database client does not implement the database "truncate" operation.
+Instead it saves all store requests to a transaction which it commits
+only when the database is closed. Therefore, tlsmgr(8) will never
+find any entries that it stores after opening its TLS session cache
+databases. And when the database becomes corrupted, tlsmgr(8) will
+keep crashing until someone removes the file ".lmdb" file by hand.
+</p>
+
+</ul>
+
+</body>
+
+</html>
../html/LDAP_README.html \
../html/LINUX_README.html \
../html/LOCAL_RECIPIENT_README.html ../html/MAILDROP_README.html \
+ ../html/LMDB_README.html \
../html/MEMCACHE_README.html \
../html/MILTER_README.html \
../html/MULTI_INSTANCE_README.html \
../README_FILES/LDAP_README \
../README_FILES/LINUX_README \
../README_FILES/LOCAL_RECIPIENT_README ../README_FILES/MAILDROP_README \
+ ../README_FILES/LMDB_README \
../README_FILES/MEMCACHE_README \
../README_FILES/MILTER_README \
../README_FILES/MULTI_INSTANCE_README \
../html/MAILDROP_README.html: MAILDROP_README.html
$(POSTLINK) $? >$@
+../html/LMDB_README.html: LMDB_README.html
+ $(POSTLINK) $? >$@
+
../html/MEMCACHE_README.html: MEMCACHE_README.html
$(POSTLINK) $? >$@
../README_FILES/MAILDROP_README: MAILDROP_README.html
$(HT2READ) $? >$@
+../README_FILES/LMDB_README: LMDB_README.html
+ $(HT2READ) $? >$@
+
../README_FILES/MEMCACHE_README: MEMCACHE_README.html
$(HT2READ) $? >$@
Specify 0 when mail delivery should be tried only once.
</p>
+%PARAM lmdb_map_size 10485760
+
+<p>
+The per-table size limit for programs that create OpenLDAP LMDB
+tables. Specify a byte count.
+</p>
+
+<p>
+This feature is available in Postfix 2.11 and later.
+</p>
+
%PARAM message_size_limit 10240000
<p>
mail_params.c mail_pathname.c mail_queue.c mail_run.c \
mail_scan_dir.c mail_stream.c mail_task.c mail_trigger.c maps.c \
mark_corrupt.c match_parent_style.c mbox_conf.c mbox_open.c \
- mime_state.c mkmap_cdb.c mkmap_db.c mkmap_dbm.c mkmap_open.c \
+ mime_state.c mkmap_cdb.c mkmap_db.c mkmap_dbm.c mkmap_lmdb.c mkmap_open.c \
mkmap_sdbm.c msg_stats_print.c msg_stats_scan.c mynetworks.c \
mypwd.c namadr_list.c off_cvt.c opened.c own_inet_addr.c \
pipe_command.c post_mail.c quote_821_local.c quote_822_local.c \
mail_params.o mail_pathname.o mail_queue.o mail_run.o \
mail_scan_dir.o mail_stream.o mail_task.o mail_trigger.o maps.o \
mark_corrupt.o match_parent_style.o mbox_conf.o mbox_open.o \
- mime_state.o mkmap_cdb.o mkmap_db.o mkmap_dbm.o mkmap_open.o \
+ mime_state.o mkmap_cdb.o mkmap_db.o mkmap_dbm.o mkmap_lmdb.o mkmap_open.o \
mkmap_sdbm.o msg_stats_print.o msg_stats_scan.o mynetworks.o \
mypwd.o namadr_list.o off_cvt.o opened.o own_inet_addr.o \
pipe_command.o post_mail.o quote_821_local.o quote_822_local.o \
data_redirect.o: ../../include/dict_cdb.h
data_redirect.o: ../../include/dict_db.h
data_redirect.o: ../../include/dict_dbm.h
+data_redirect.o: ../../include/dict_lmdb.h
data_redirect.o: ../../include/msg.h
data_redirect.o: ../../include/myflock.h
data_redirect.o: ../../include/name_code.h
mail_params.o: ../../include/attr.h
mail_params.o: ../../include/dict.h
mail_params.o: ../../include/dict_db.h
+mail_params.o: ../../include/dict_lmdb.h
mail_params.o: ../../include/get_hostname.h
mail_params.o: ../../include/inet_addr_list.h
mail_params.o: ../../include/inet_proto.h
mkmap_fail.o: ../../include/vstring.h
mkmap_fail.o: mkmap.h
mkmap_fail.o: mkmap_fail.c
+mkmap_lmdb.o: ../../include/argv.h
+mkmap_lmdb.o: ../../include/dict.h
+mkmap_lmdb.o: ../../include/dict_lmdb.h
+mkmap_lmdb.o: ../../include/msg.h
+mkmap_lmdb.o: ../../include/myflock.h
+mkmap_lmdb.o: ../../include/mymalloc.h
+mkmap_lmdb.o: ../../include/stringops.h
+mkmap_lmdb.o: ../../include/sys_defs.h
+mkmap_lmdb.o: ../../include/vbuf.h
+mkmap_lmdb.o: ../../include/vstream.h
+mkmap_lmdb.o: ../../include/vstring.h
+mkmap_lmdb.o: ../../include/warn_stat.h
+mkmap_lmdb.o: mail_conf.h
+mkmap_lmdb.o: mail_params.h
+mkmap_lmdb.o: mkmap.h
+mkmap_lmdb.o: mkmap_lmdb.c
mkmap_open.o: ../../include/argv.h
mkmap_open.o: ../../include/dict.h
mkmap_open.o: ../../include/dict_cdb.h
mkmap_open.o: ../../include/dict_db.h
mkmap_open.o: ../../include/dict_dbm.h
mkmap_open.o: ../../include/dict_fail.h
+mkmap_open.o: ../../include/dict_lmdb.h
mkmap_open.o: ../../include/dict_sdbm.h
mkmap_open.o: ../../include/msg.h
mkmap_open.o: ../../include/myflock.h
#include <dict_db.h>
#include <dict_dbm.h>
#include <dict_cdb.h>
+#include <dict_lmdb.h>
#include <warn_stat.h>
/* Global directory. */
DICT_TYPE_HASH, 1,
DICT_TYPE_BTREE, 1,
DICT_TYPE_DBM, 1,
+ DICT_TYPE_LMDB, 1,
DICT_TYPE_CDB, 1, /* not a read-write map type */
"sdbm", 1, /* legacy 3rd-party TLS */
"dbz", 1, /* just in case */
/* char *var_proxywrite_service;
/* int var_db_create_buf;
/* int var_db_read_buf;
+/* int var_lmdb_map_size;
/* int var_mime_maxdepth;
/* int var_mime_bound_len;
/* int var_header_limit;
#ifdef HAS_DB
#include <dict_db.h>
#endif
+#ifdef HAS_LMDB
+#include <dict_lmdb.h>
+#endif
#include <inet_proto.h>
#include <vstring_vstream.h>
#include <iostuff.h>
char *var_proxywrite_service;
int var_db_create_buf;
int var_db_read_buf;
+int var_lmdb_map_size;
int var_mime_maxdepth;
int var_mime_bound_len;
int var_header_limit;
VAR_FAULT_INJ_CODE, DEF_FAULT_INJ_CODE, &var_fault_inj_code, 0, 0,
VAR_DB_CREATE_BUF, DEF_DB_CREATE_BUF, &var_db_create_buf, 1, 0,
VAR_DB_READ_BUF, DEF_DB_READ_BUF, &var_db_read_buf, 1, 0,
+ VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 1, 0,
VAR_HEADER_LIMIT, DEF_HEADER_LIMIT, &var_header_limit, 1, 0,
VAR_TOKEN_LIMIT, DEF_TOKEN_LIMIT, &var_token_limit, 1, 0,
VAR_MIME_MAXDEPTH, DEF_MIME_MAXDEPTH, &var_mime_maxdepth, 1, 0,
check_overlap();
#ifdef HAS_DB
dict_db_cache_size = var_db_read_buf;
+#endif
+#ifdef HAS_LMDB
+ dict_lmdb_map_size = var_lmdb_map_size;
#endif
inet_windowsize = var_inet_windowsize;
#define DEF_DB_READ_BUF (128 *1024)
extern int var_db_read_buf;
+ /*
+ * OpenLDAP LMDB memory map size.
+ */
+#define VAR_LMDB_MAP_SIZE "lmdb_map_size"
+#define DEF_LMDB_MAP_SIZE (16 * 1024 *1024)
+extern int var_lmdb_map_size;
+
/*
* Named queue file attributes.
*/
* Patches change both the patchlevel and the release date. Snapshots have no
* patchlevel; they change the release date only.
*/
-#define MAIL_RELEASE_DATE "20130211"
+#define MAIL_RELEASE_DATE "20130315"
#define MAIL_VERSION_NUMBER "2.11"
#ifdef SNAPSHOT
extern MKMAP *mkmap_cdb_open(const char *);
extern MKMAP *mkmap_hash_open(const char *);
extern MKMAP *mkmap_btree_open(const char *);
+extern MKMAP *mkmap_lmdb_open(const char *);
extern MKMAP *mkmap_sdbm_open(const char *);
extern MKMAP *mkmap_proxy_open(const char *);
extern MKMAP *mkmap_fail_open(const char *);
--- /dev/null
+/*++
+/* NAME
+/* mkmap_lmdb 3
+/* SUMMARY
+/* create or open database, LMDB style
+/* SYNOPSIS
+/* #include <mkmap.h>
+/*
+/* MKMAP *mkmap_lmdb_open(path)
+/* const char *path;
+/*
+/* DESCRIPTION
+/* This module implements support for creating LMDB databases.
+/*
+/* mkmap_lmdb_open() takes a file name, appends the ".lmdb"
+/* suffix, and does whatever initialization is required
+/* before the OpenLDAP LMDB open routine is called.
+/*
+/* All errors are fatal.
+/* SEE ALSO
+/* dict_lmdb(3), LMDB dictionary interface.
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Howard Chu
+/* Symas Corporation
+/*--*/
+
+/* System library. */
+
+#include <sys_defs.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <errno.h>
+
+/* Utility library. */
+
+#include <msg.h>
+#include <mymalloc.h>
+#include <stringops.h>
+#include <dict.h>
+#include <dict_lmdb.h>
+#include <myflock.h>
+#include <warn_stat.h>
+
+#if defined(HAS_LMDB) && defined(SNAPSHOT)
+#ifdef PATH_LMDB_H
+#include PATH_LMDB_H
+#else
+#include <lmdb.h>
+#endif
+
+/* Global library. */
+
+#include <mail_conf.h>
+#include <mail_params.h>
+
+/* Application-specific. */
+
+#include "mkmap.h"
+
+int var_proc_limit;
+
+/* mkmap_lmdb_open */
+
+MKMAP *mkmap_lmdb_open(const char *path)
+{
+ MKMAP *mkmap = (MKMAP *) mymalloc(sizeof(*mkmap));
+ static const CONFIG_INT_TABLE int_table[] = {
+ VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0,
+ 0,
+ };
+
+ get_mail_conf_int_table(int_table);
+
+ /*
+ * XXX Why is this not set in mail_params.c (with proper #ifdefs)?
+ *
+ * Override the default per-table map size for map (re)builds.
+ *
+ * lmdb_map_size is defined in util/dict_lmdb.c and defaults to 10MB. It
+ * needs to be large enough to contain the largest tables in use.
+ *
+ * XXX This should be specified via the DICT interface so that the buffer
+ * size becomes an object property, instead of being specified by poking
+ * a global variable so that it becomes a class property.
+ *
+ * XXX Wietse disagrees: storage infrastructure that requires up-front
+ * max-size information is evil. This unlike Postfix (e.g. line length or
+ * process count) limits which are a defense against out-of-control or
+ * malicious external actors.
+ *
+ * XXX Need to check that an existing table can be rebuilt with a larger
+ * size limit than was used for the initial build.
+ */
+ dict_lmdb_map_size = var_lmdb_map_size;
+
+ /*
+ * XXX Why is this not set in mail_params.c (with proper #ifdefs)?
+ *
+ * Set the max number of concurrent readers per table. This is the
+ * maximum number of postfix processes, plus some extra for CLI users.
+ *
+ * XXX Postfix uses asynchronous or blocking I/O with single-threaded
+ * processes so this limit will never be reached, assuming that the limit
+ * is a per-client property, not a shared database property.
+ */
+ dict_lmdb_max_readers = var_proc_limit * 2 + 16;
+
+ /*
+ * Fill in the generic members.
+ */
+ mkmap->open = dict_lmdb_open;
+ mkmap->after_open = 0;
+ mkmap->after_close = 0;
+
+ /*
+ * LMDB uses MVCC so it needs no special lock management here.
+ */
+
+ return (mkmap);
+}
+
+#endif
#include <dict_db.h>
#include <dict_cdb.h>
#include <dict_dbm.h>
+#include <dict_lmdb.h>
#include <dict_sdbm.h>
#include <dict_proxy.h>
#include <dict_fail.h>
#ifdef HAS_DB
DICT_TYPE_HASH, mkmap_hash_open,
DICT_TYPE_BTREE, mkmap_btree_open,
+#endif
+#if defined(HAS_LMDB) && defined(SNAPSHOT)
+ DICT_TYPE_LMDB, mkmap_lmdb_open,
#endif
DICT_TYPE_FAIL, mkmap_fail_open,
0,
attr_scan_plain.c auto_clnt.c base64_code.c basename.c binhash.c \
chroot_uid.c cidr_match.c clean_env.c close_on_exec.c concatenate.c \
ctable.c dict.c dict_alloc.c dict_cdb.c dict_cidr.c dict_db.c \
- dict_dbm.c dict_debug.c dict_env.c dict_ht.c dict_ni.c dict_nis.c \
+ dict_dbm.c dict_debug.c dict_env.c dict_ht.c dict_lmdb.c dict_ni.c dict_nis.c \
dict_nisplus.c dict_open.c dict_pcre.c dict_regexp.c dict_sdbm.c \
dict_static.c dict_tcp.c dict_unix.c dir_forest.c doze.c dummy_read.c \
dummy_write.c duplex_pipe.c environ.c events.c exec_command.c \
attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \
chroot_uid.o cidr_match.o clean_env.o close_on_exec.o concatenate.o \
ctable.o dict.o dict_alloc.o dict_cdb.o dict_cidr.o dict_db.o \
- dict_dbm.o dict_debug.o dict_env.o dict_ht.o dict_ni.o dict_nis.o \
+ dict_dbm.o dict_debug.o dict_env.o dict_ht.o dict_lmdb.o dict_ni.o dict_nis.o \
dict_nisplus.o dict_open.o dict_pcre.o dict_regexp.o dict_sdbm.o \
dict_static.o dict_tcp.o dict_unix.o dir_forest.o doze.o dummy_read.o \
dummy_write.o duplex_pipe.o environ.o events.o exec_command.o \
HDRS = argv.h attr.h attr_clnt.h auto_clnt.h base64_code.h binhash.h \
chroot_uid.h cidr_match.h clean_env.h connect.h ctable.h dict.h \
dict_cdb.h dict_cidr.h dict_db.h dict_dbm.h dict_env.h dict_ht.h \
- dict_ni.h dict_nis.h dict_nisplus.h dict_pcre.h dict_regexp.h \
+ dict_lmdb.h dict_ni.h dict_nis.h dict_nisplus.h dict_pcre.h dict_regexp.h \
dict_sdbm.h dict_static.h dict_tcp.h dict_unix.h dir_forest.h \
events.h exec_command.h find_inet.h fsspace.h fullname.h \
get_domainname.h get_hostname.h hex_code.h hex_quote.h host_port.h \
dict_ht.o: vbuf.h
dict_ht.o: vstream.h
dict_ht.o: vstring.h
+dict_lmdb.o: argv.h
+dict_lmdb.o: dict.h
+dict_lmdb.o: dict_lmdb.c
+dict_lmdb.o: dict_lmdb.h
+dict_lmdb.o: htable.h
+dict_lmdb.o: iostuff.h
+dict_lmdb.o: msg.h
+dict_lmdb.o: myflock.h
+dict_lmdb.o: mymalloc.h
+dict_lmdb.o: stringops.h
+dict_lmdb.o: sys_defs.h
+dict_lmdb.o: vbuf.h
+dict_lmdb.o: vstream.h
+dict_lmdb.o: vstring.h
+dict_lmdb.o: warn_stat.h
dict_ni.o: dict_ni.c
dict_ni.o: sys_defs.h
dict_nis.o: argv.h
dict_open.o: dict_env.h
dict_open.o: dict_fail.h
dict_open.o: dict_ht.h
+dict_open.o: dict_lmdb.h
dict_open.o: dict_ni.h
dict_open.o: dict_nis.h
dict_open.o: dict_nisplus.h
--- /dev/null
+/*++
+/* NAME
+/* dict_lmdb 3
+/* SUMMARY
+/* dictionary manager interface to OpenLDAP LMDB files
+/* SYNOPSIS
+/* #include <dict_lmdb.h>
+/*
+/* DICT *dict_lmdb_open(path, open_flags, dict_flags)
+/* const char *name;
+/* const char *path;
+/* int open_flags;
+/* int dict_flags;
+/* DESCRIPTION
+/* dict_lmdb_open() opens the named LMDB database and makes it available
+/* via the generic interface described in dict_open(3).
+/*
+/* The dict_lmdb_map_size variable specifies a non-default per-table
+/* memory map size. The map size is 10MB. The map size is also the
+/* maximum size the table can grow to, so it must be set large enough
+/* to accomodate the largest tables in use.
+/* DIAGNOSTICS
+/* Fatal errors: cannot open file, file write error, out of memory.
+/* SEE ALSO
+/* dict(3) generic dictionary manager
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Howard Chu
+/* Symas Corporation
+/*--*/
+
+#include "sys_defs.h"
+
+#if defined(HAS_LMDB) && defined(SNAPSHOT)
+
+/* System library. */
+
+#include <sys/stat.h>
+#include <string.h>
+#include <unistd.h>
+
+#ifdef PATH_LMDB_H
+#include PATH_LMDB_H
+#else
+#include <lmdb.h>
+#endif
+
+/* Utility library. */
+
+#include "msg.h"
+#include "mymalloc.h"
+#include "htable.h"
+#include "iostuff.h"
+#include "vstring.h"
+#include "myflock.h"
+#include "stringops.h"
+#include "dict.h"
+#include "dict_lmdb.h"
+#include "warn_stat.h"
+
+/* Application-specific. */
+
+typedef struct {
+ DICT dict; /* generic members */
+ MDB_env *env; /* LMDB environment */
+ MDB_dbi dbi; /* database handle */
+ MDB_txn *txn; /* write transaction for O_TRUNC */
+ MDB_cursor *cursor; /* for sequence ops */
+ VSTRING *key_buf; /* key buffer */
+ VSTRING *val_buf; /* result buffer */
+} DICT_LMDB;
+
+#define SCOPY(buf, data, size) \
+ vstring_str(vstring_strncpy(buf ? buf : (buf = vstring_alloc(10)), data, size))
+
+size_t dict_lmdb_map_size = (10 * 1024 * 1024); /* 10MB default mmap
+ * size */
+unsigned int dict_lmdb_max_readers = 216; /* 200 postfix processes,
+ * plus some extra */
+
+/* dict_lmdb_lookup - find database entry */
+
+static const char *dict_lmdb_lookup(DICT *dict, const char *name)
+{
+ DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
+ MDB_val mdb_key;
+ MDB_val mdb_value;
+ MDB_txn *txn;
+ const char *result = 0;
+ int status, klen;
+
+ dict->error = 0;
+ klen = strlen(name);
+
+ /*
+ * Sanity check.
+ */
+ if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0)
+ msg_panic("dict_lmdb_lookup: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag");
+
+ /*
+ * Optionally fold the key.
+ */
+ if (dict->flags & DICT_FLAG_FOLD_FIX) {
+ if (dict->fold_buf == 0)
+ dict->fold_buf = vstring_alloc(10);
+ vstring_strcpy(dict->fold_buf, name);
+ name = lowercase(vstring_str(dict->fold_buf));
+ }
+
+ /*
+ * Start a read transaction if there's no global txn.
+ */
+ if (dict_lmdb->txn)
+ txn = dict_lmdb->txn;
+ else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, MDB_RDONLY, &txn)))
+ msg_fatal("%s: txn_begin(read) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+
+ /*
+ * See if this LMDB file was written with one null byte appended to key
+ * and value.
+ */
+ if (dict->flags & DICT_FLAG_TRY1NULL) {
+ mdb_key.mv_data = (void *) name;
+ mdb_key.mv_size = klen + 1;
+ status = mdb_get(txn, dict_lmdb->dbi, &mdb_key, &mdb_value);
+ if (!status) {
+ dict->flags &= ~DICT_FLAG_TRY0NULL;
+ result = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size);
+ }
+ }
+
+ /*
+ * See if this LMDB file was written with no null byte appended to key
+ * and value.
+ */
+ if (result == 0 && (dict->flags & DICT_FLAG_TRY0NULL)) {
+ mdb_key.mv_data = (void *) name;
+ mdb_key.mv_size = klen;
+ status = mdb_get(txn, dict_lmdb->dbi, &mdb_key, &mdb_value);
+ if (!status) {
+ dict->flags &= ~DICT_FLAG_TRY1NULL;
+ result = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size);
+ }
+ }
+
+ /*
+ * Close the read txn if it's not the global txn.
+ */
+ if (!dict_lmdb->txn)
+ mdb_txn_abort(txn);
+
+ return (result);
+}
+
+/* dict_lmdb_update - add or update database entry */
+
+static int dict_lmdb_update(DICT *dict, const char *name, const char *value)
+{
+ DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
+ MDB_val mdb_key;
+ MDB_val mdb_value;
+ MDB_txn *txn;
+ int status;
+
+ dict->error = 0;
+
+ /*
+ * Sanity check.
+ */
+ if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0)
+ msg_panic("dict_lmdb_update: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag");
+
+ /*
+ * Optionally fold the key.
+ */
+ if (dict->flags & DICT_FLAG_FOLD_FIX) {
+ if (dict->fold_buf == 0)
+ dict->fold_buf = vstring_alloc(10);
+ vstring_strcpy(dict->fold_buf, name);
+ name = lowercase(vstring_str(dict->fold_buf));
+ }
+ mdb_key.mv_data = (void *) name;
+ mdb_value.mv_data = (void *) value;
+ mdb_key.mv_size = strlen(name);
+ mdb_value.mv_size = strlen(value);
+
+ /*
+ * If undecided about appending a null byte to key and value, choose a
+ * default depending on the platform.
+ */
+ if ((dict->flags & DICT_FLAG_TRY1NULL)
+ && (dict->flags & DICT_FLAG_TRY0NULL)) {
+#ifdef LMDB_NO_TRAILING_NULL
+ dict->flags &= ~DICT_FLAG_TRY1NULL;
+#else
+ dict->flags &= ~DICT_FLAG_TRY0NULL;
+#endif
+ }
+
+ /*
+ * Optionally append a null byte to key and value.
+ */
+ if (dict->flags & DICT_FLAG_TRY1NULL) {
+ mdb_key.mv_size++;
+ mdb_value.mv_size++;
+ }
+
+ /*
+ * Start a write transaction if there's no global txn.
+ */
+ if (dict_lmdb->txn)
+ txn = dict_lmdb->txn;
+ else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
+ msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+
+ /*
+ * Do the update.
+ */
+ status = mdb_put(txn, dict_lmdb->dbi, &mdb_key, &mdb_value,
+ (dict->flags & DICT_FLAG_DUP_REPLACE) ? 0 : MDB_NOOVERWRITE);
+ if (status) {
+ if (status == MDB_KEYEXIST) {
+ if (dict->flags & DICT_FLAG_DUP_IGNORE)
+ /* void */ ;
+ else if (dict->flags & DICT_FLAG_DUP_WARN)
+ msg_warn("%s: duplicate entry: \"%s\"", dict_lmdb->dict.name, name);
+ else
+ msg_fatal("%s: duplicate entry: \"%s\"", dict_lmdb->dict.name, name);
+ } else {
+ msg_fatal("error writing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ }
+ }
+
+ /*
+ * Commit the transaction if it's not the global txn.
+ */
+ if (!dict_lmdb->txn && ((status = mdb_txn_commit(txn))))
+ msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(status));
+
+ return (status);
+}
+
+/* dict_lmdb_delete - delete one entry from the dictionary */
+
+static int dict_lmdb_delete(DICT *dict, const char *name)
+{
+ DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
+ MDB_val mdb_key;
+ MDB_txn *txn;
+ int status = 1, klen, rc;
+
+ dict->error = 0;
+ klen = strlen(name);
+
+ /*
+ * Sanity check.
+ */
+ if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0)
+ msg_panic("dict_lmdb_delete: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag");
+
+ /*
+ * Optionally fold the key.
+ */
+ if (dict->flags & DICT_FLAG_FOLD_FIX) {
+ if (dict->fold_buf == 0)
+ dict->fold_buf = vstring_alloc(10);
+ vstring_strcpy(dict->fold_buf, name);
+ name = lowercase(vstring_str(dict->fold_buf));
+ }
+
+ /*
+ * Start a write transaction if there's no global txn.
+ */
+ if (dict_lmdb->txn)
+ txn = dict_lmdb->txn;
+ else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn)))
+ msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+
+ /*
+ * See if this LMDB file was written with one null byte appended to key
+ * and value.
+ */
+ if (dict->flags & DICT_FLAG_TRY1NULL) {
+ mdb_key.mv_data = (void *) name;
+ mdb_key.mv_size = klen + 1;
+ status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL);
+ if (status) {
+ if (status == MDB_NOTFOUND)
+ status = 1;
+ else
+ msg_fatal("error deleting from %s: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ } else {
+ dict->flags &= ~DICT_FLAG_TRY0NULL; /* found */
+ }
+ }
+
+ /*
+ * See if this LMDB file was written with no null byte appended to key
+ * and value.
+ */
+ if (status > 0 && (dict->flags & DICT_FLAG_TRY0NULL)) {
+ mdb_key.mv_data = (void *) name;
+ mdb_key.mv_size = klen;
+ status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL);
+ if (status) {
+ if (status == MDB_NOTFOUND)
+ status = 1;
+ else
+ msg_fatal("error deleting from %s: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ } else {
+ dict->flags &= ~DICT_FLAG_TRY1NULL; /* found */
+ }
+ }
+
+ /*
+ * Commit the transaction if it's not the global txn.
+ */
+ if (!dict_lmdb->txn && ((rc = mdb_txn_commit(txn))))
+ msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(rc));
+
+ return (status);
+}
+
+/* traverse the dictionary */
+
+static int dict_lmdb_sequence(DICT *dict, int function,
+ const char **key, const char **value)
+{
+ const char *myname = "dict_lmdb_sequence";
+ DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
+ MDB_val mdb_key;
+ MDB_val mdb_value;
+ MDB_txn *txn;
+ MDB_cursor_op op;
+ int status;
+
+ dict->error = 0;
+
+ /*
+ * Determine the seek function.
+ */
+ switch (function) {
+ case DICT_SEQ_FUN_FIRST:
+ op = MDB_FIRST;
+ break;
+ case DICT_SEQ_FUN_NEXT:
+ op = MDB_NEXT;
+ break;
+ default:
+ msg_panic("%s: invalid function: %d", myname, function);
+ }
+
+ /*
+ * Open a read transaction and cursor if needed.
+ */
+ if (dict_lmdb->cursor == 0) {
+ if ((status = mdb_txn_begin(dict_lmdb->env, NULL, MDB_RDONLY, &txn)))
+ msg_fatal("%s: txn_begin(read) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ if ((status = mdb_cursor_open(txn, dict_lmdb->dbi, &dict_lmdb->cursor)))
+ msg_fatal("%s: cursor_open dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ }
+
+ /*
+ * Database lookup.
+ */
+ status = mdb_cursor_get(dict_lmdb->cursor, &mdb_key, &mdb_value, op);
+ if (status && status != MDB_NOTFOUND)
+ msg_fatal("%s: seeking dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+
+ if (status == MDB_NOTFOUND) {
+
+ /*
+ * Caller must read to end, to ensure cursor gets closed.
+ */
+ status = 1;
+ txn = mdb_cursor_txn(dict_lmdb->cursor);
+ mdb_cursor_close(dict_lmdb->cursor);
+ mdb_txn_abort(txn);
+ dict_lmdb->cursor = 0;
+ } else {
+
+ /*
+ * Copy the key so that it is guaranteed null terminated.
+ */
+ *key = SCOPY(dict_lmdb->key_buf, mdb_key.mv_data, mdb_key.mv_size);
+
+ if (mdb_value.mv_data != 0 && mdb_value.mv_size > 0) {
+
+ /*
+ * Copy the value so that it is guaranteed null terminated.
+ */
+ *value = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size);
+ status = 0;
+ }
+ }
+
+ return (status);
+}
+
+/* dict_lmdb_lock - noop lock handler */
+
+static int dict_lmdb_lock(DICT *dict, int unused_op)
+{
+ /* LMDB does its own concurrency control */
+ return 0;
+}
+
+/* dict_lmdb_close - disassociate from data base */
+
+static void dict_lmdb_close(DICT *dict)
+{
+ DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict;
+
+ if (dict_lmdb->txn) {
+ int status = mdb_txn_commit(dict_lmdb->txn);
+
+ if (status)
+ msg_fatal("%s: closing dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status));
+ dict_lmdb->cursor = NULL;
+ }
+ if (dict_lmdb->cursor) {
+ MDB_txn *txn = mdb_cursor_txn(dict_lmdb->cursor);
+
+ mdb_cursor_close(dict_lmdb->cursor);
+ mdb_txn_abort(txn);
+ }
+ if (dict_lmdb->dict.stat_fd >= 0)
+ close(dict_lmdb->dict.stat_fd);
+ mdb_env_close(dict_lmdb->env);
+ if (dict_lmdb->key_buf)
+ vstring_free(dict_lmdb->key_buf);
+ if (dict_lmdb->val_buf)
+ vstring_free(dict_lmdb->val_buf);
+ if (dict->fold_buf)
+ vstring_free(dict->fold_buf);
+ dict_free(dict);
+}
+
+/* dict_lmdb_open - open LMDB data base */
+
+DICT *dict_lmdb_open(const char *path, int open_flags, int dict_flags)
+{
+ DICT_LMDB *dict_lmdb;
+ struct stat st;
+ MDB_env *env;
+ MDB_txn *txn;
+ MDB_dbi dbi;
+ char *mdb_path;
+ int env_flags, status;
+
+ mdb_path = concatenate(path, ".lmdb", (char *) 0);
+
+ env_flags = MDB_NOSUBDIR;
+ if (open_flags == O_RDONLY)
+ env_flags |= MDB_RDONLY;
+
+ if ((status = mdb_env_create(&env)))
+ msg_fatal("env_create %s: %s", mdb_path, mdb_strerror(status));
+
+ if ((status = mdb_env_set_mapsize(env, dict_lmdb_map_size)))
+ msg_fatal("env_set_mapsize %s: %s", mdb_path, mdb_strerror(status));
+
+ if ((status = mdb_env_set_maxreaders(env, dict_lmdb_max_readers)))
+ msg_fatal("env_set_maxreaders %s: %s", mdb_path, mdb_strerror(status));
+
+ if ((status = mdb_env_open(env, mdb_path, env_flags, 0644)))
+ msg_fatal("env_open %s: %s", mdb_path, mdb_strerror(status));
+
+ if ((status = mdb_txn_begin(env, NULL, env_flags & MDB_RDONLY, &txn)))
+ msg_fatal("txn_begin %s: %s", mdb_path, mdb_strerror(status));
+
+ /*
+ * mdb_open requires a txn, but since the default DB always exists in an
+ * LMDB environment, we don't need to do anything else with the txn.
+ */
+ if ((status = mdb_open(txn, NULL, 0, &dbi)))
+ msg_fatal("mdb_open %s: %s", mdb_path, mdb_strerror(status));
+
+ /*
+ * However, if O_TRUNC was specified, we need to do it now. Also with
+ * O_TRUNC we keep this write txn for as long as the database is open,
+ * since we'll probably be doing a bulk import immediately after.
+ */
+ if (open_flags & O_TRUNC) {
+ if ((status = mdb_drop(txn, dbi, 0)))
+ msg_fatal("truncate %s: %s", mdb_path, mdb_strerror(status));
+ } else {
+ mdb_txn_abort(txn);
+ txn = NULL;
+ }
+
+ dict_lmdb = (DICT_LMDB *) dict_alloc(DICT_TYPE_LMDB, path, sizeof(*dict_lmdb));
+ dict_lmdb->dict.lookup = dict_lmdb_lookup;
+ dict_lmdb->dict.update = dict_lmdb_update;
+ dict_lmdb->dict.delete = dict_lmdb_delete;
+ dict_lmdb->dict.sequence = dict_lmdb_sequence;
+ dict_lmdb->dict.close = dict_lmdb_close;
+ dict_lmdb->dict.lock = dict_lmdb_lock;
+ dict_lmdb->dict.stat_fd = open(mdb_path, O_RDONLY);
+ if (fstat(dict_lmdb->dict.stat_fd, &st) < 0)
+ msg_fatal("dict_lmdb_open: fstat: %m");
+ dict_lmdb->dict.mtime = st.st_mtime;
+ dict_lmdb->dict.owner.uid = st.st_uid;
+ dict_lmdb->dict.owner.status = (st.st_uid != 0);
+
+ /*
+ * Warn if the source file is newer than the indexed file, except when
+ * the source file changed only seconds ago.
+ */
+ if ((dict_flags & DICT_FLAG_LOCK) != 0
+ && stat(path, &st) == 0
+ && st.st_mtime > dict_lmdb->dict.mtime
+ && st.st_mtime < time((time_t *) 0) - 100)
+ msg_warn("database %s is older than source file %s", mdb_path, path);
+
+ close_on_exec(dict_lmdb->dict.stat_fd, CLOSE_ON_EXEC);
+ dict_lmdb->dict.flags = dict_flags | DICT_FLAG_FIXED;
+ if ((dict_flags & (DICT_FLAG_TRY0NULL | DICT_FLAG_TRY1NULL)) == 0)
+ dict_lmdb->dict.flags |= (DICT_FLAG_TRY0NULL | DICT_FLAG_TRY1NULL);
+ if (dict_flags & DICT_FLAG_FOLD_FIX)
+ dict_lmdb->dict.fold_buf = vstring_alloc(10);
+ dict_lmdb->env = env;
+ dict_lmdb->dbi = dbi;
+
+ /* Save the write txn if we opened with O_TRUNC */
+ dict_lmdb->txn = txn;
+
+ dict_lmdb->cursor = 0;
+ dict_lmdb->key_buf = 0;
+ dict_lmdb->val_buf = 0;
+
+ myfree(mdb_path);
+
+ return (DICT_DEBUG (&dict_lmdb->dict));
+}
+
+#endif
--- /dev/null
+#ifndef _DICT_LMDB_H_INCLUDED_
+#define _DICT_LMDB_H_INCLUDED_
+
+/*++
+/* NAME
+/* dict_lmdb 3h
+/* SUMMARY
+/* dictionary manager interface to OpenLDAP LMDB files
+/* SYNOPSIS
+/* #include <dict_lmdb.h>
+/* DESCRIPTION
+/* .nf
+
+ /*
+ * Utility library.
+ */
+#include <dict.h>
+
+ /*
+ * External interface.
+ */
+#define DICT_TYPE_LMDB "lmdb"
+
+extern DICT *dict_lmdb_open(const char *, int, int);
+
+ /*
+ * XXX Should be part of the DICT interface.
+ */
+extern size_t dict_lmdb_map_size;
+extern unsigned int dict_lmdb_max_readers;
+
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Howard Chu
+/* Symas Corporation
+/*--*/
+
+#endif
#include <dict_sdbm.h>
#include <dict_dbm.h>
#include <dict_db.h>
+#include <dict_lmdb.h>
#include <dict_nis.h>
#include <dict_nisplus.h>
#include <dict_ni.h>
DICT_TYPE_HASH, dict_hash_open,
DICT_TYPE_BTREE, dict_btree_open,
#endif
+#if defined(HAS_LMDB) && defined(SNAPSHOT)
+ DICT_TYPE_LMDB, dict_lmdb_open,
+#endif
#ifdef HAS_NIS
DICT_TYPE_NIS, dict_nis_open,
#endif
projects / thirdparty / postfix.git / commitdiff
