From: Wietse Venema Date: Fri, 15 Mar 2013 05:00:00 +0000 (-0500) Subject: postfix-2.11-20130315 X-Git-Tag: v2.11.0-RC1~52 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=e24ec95ab0cf59900da5e878b1875351c5389156;p=thirdparty%2Fpostfix.git postfix-2.11-20130315 --- diff --git a/postfix/.indent.pro b/postfix/.indent.pro index 204f673d7..41daed59e 100644 --- a/postfix/.indent.pro +++ b/postfix/.indent.pro @@ -72,6 +72,7 @@ -TDICT_FAIL -TDICT_HT -TDICT_LDAP +-TDICT_LMDB -TDICT_MC -TDICT_MYSQL -TDICT_NI diff --git a/postfix/HISTORY b/postfix/HISTORY index 1793ebebb..d0d817ed3 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -18247,3 +18247,13 @@ Apologies for any names omitted. Bugfix: an error handler for smtp_tls_policy_maps lookups was never invoked. File: smtp/smtp_session.c. + +20130315 + + Feature: preliminary LMDB (memory-mapped persistent file) + support by Howard Chu. This feature has a number of unexpected + limitations and is therefore "snapshot only", i.e. it will + not be part of a stable release. Files: proto/postconf.proto, + proto/LMDB_README.html, proto/DATABASE_README.html, + util/dict_lmdb.[hc], util/dict_open.c, global/mkmap_lmdb.[hc], + global/mkmap_open.c, postconf/postconf.c. diff --git a/postfix/README_FILES/DATABASE_README b/postfix/README_FILES/DATABASE_README index 79181be35..ceb661b7d 100644 --- a/postfix/README_FILES/DATABASE_README +++ b/postfix/README_FILES/DATABASE_README @@ -210,6 +210,12 @@ To find out what database types your Postfix system supports, use the "ppooss iinntteerrnnaall A non-shared, in-memory hash table. Its content are lost when a process terminates. + llmmddbb + The OpenLDAP LMDB database (a memory-mapped, persistent file). Database + files are created with the postmap(1) or postalias(1) command. The + database name as used in "lmdb:table" is the database file name without + the ".lmdb" suffix. This database type has unexpected limitations and + is therefore not part of the stable Postfix release. llddaapp (read-only) Perform lookups using the LDAP protocol. Configuration details are given in the ldap_table(5). diff --git a/postfix/README_FILES/INSTALL b/postfix/README_FILES/INSTALL index 2ab58b4e9..9ed6a0541 100644 --- a/postfix/README_FILES/INSTALL +++ b/postfix/README_FILES/INSTALL @@ -148,18 +148,20 @@ compiler. Here are a few examples: and so on. In some cases, optimization is turned off automatically. -44..33 -- BBuuiillddiinngg wwiitthh ooppttiioonnaall eexxtteennssiioonnss +44..33 -- BBuuiillddiinngg wwiitthh ooppttiioonnaall ffeeaattuurreess By default, Postfix builds as a mail system with relatively few bells and whistles. Support for third-party databases etc. must be configured when Postfix is compiled. The following documents describe how to build Postfix with -support for extensions: +support for optional features: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - |PPoossttffiixx eexxtteennssiioonn |DDooccuummeenntt |AAvvaaiillaabbiilliittyy| + |OOppttiioonnaall ffeeaattuurree |DDooccuummeenntt |AAvvaaiillaabbiilliittyy| |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ | |Berkeley DB database |DB_README |Postfix 1.0 | |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ | + |LMDB database |LMDB_README |Postfix 2.11| + |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ | |LDAP database |LDAP_README |Postfix 1.0 | |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ _ _|_ _ _ _ _ _ _ _ _ _ _ _ | |MySQL database |MYSQL_README |Postfix 1.0 | diff --git a/postfix/README_FILES/LMDB_README b/postfix/README_FILES/LMDB_README new file mode 100644 index 000000000..7d422e0a3 --- /dev/null +++ b/postfix/README_FILES/LMDB_README @@ -0,0 +1,125 @@ +PPoossttffiixx OOppeennLLDDAAPP LLMMDDBB HHoowwttoo + +------------------------------------------------------------------------------- + +IInnttrroodduuccttiioonn + +Postfix uses databases of various kinds to store and look up information. +Postfix databases are specified as "type:name". OpenLDAP LMDB implements the +Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is +the name of the database file without the ".lmdb" suffix. OpenLDAP LMDB +databases are maintained with the postmap(1) command. + +This document describes: + + 1. How to build Postfix with OpenLDAP LMDB support. + + 2. How to configure LMDB settings. + + 3. Missing pthread library trouble. + + Caution: the current Postfix LMDB client has unexpected limitations. For + this reason, LMDB support is not part of the stable Postfix release. + +BBuuiillddiinngg PPoossttffiixx wwiitthh OOppeennLLDDAAPP LLMMDDBB ssuuppppoorrtt + +Postfix normally does not enable OpenLDAP LMDB support. To build Postfix after +you installed OpenLDAP LMDB from source code, use something like: + + % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ + AUXLIBS="-L/usr/local/lib -llmdb" + % make + +Solaris needs this: + + % make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \ + AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb" + % make + +The exact pathnames depend on how OpenLDAP LMDB was installed. + +CCoonnffiigguurree LLMMDDBB sseettttiinnggss + +Postfix provides a configuration parameter that controls how large an OpenLDAP +LMDB database may grow. + + * lmdb_map_size (default: 10 MBytes per table). This setting controls how + large any OpenLDAP LMDB database may grow. It must be set large enough to + accommodate the largest table that Postfix will use. + +MMiissssiinngg pptthhrreeaadd lliibbrraarryy ttrroouubbllee + +When building Postfix fails with: + + undefined reference to `pthread_mutexattr_destroy' + undefined reference to `pthread_mutexattr_init' + undefined reference to `pthread_mutex_lock' + +Add the "-lpthread" library to the "make makefiles" command. + + % make makefiles .... AUXLIBS="... -lpthread" + +Source code for OpenLDAP LMDB is available at http://www.openldap.org. More +information is available at http://highlandsun.com/hyc/mdb/. + +LLiimmiittaattiioonnss ooff PPoossttffiixx LLMMDDBB ddaattaabbaasseess.. + + * UUnneexxppeecctteedd ppoossttmmaapp//ppoossttaalliiaass ""ddaattaabbaassee ffuullll"" eerrrroorrss.. + + Even if the "postmap filename" command succeeds, the exact same command, + with the exact same input data, may fail subsequently with an MDB_MAP_FULL + error. The reason is that unlike other Postfix databases such as "hash" or + "btree", + + o LMDB databases have a hard size limit (configured with the + lmdb_map_size configuration parameter). + + o The Postfix LMDB database client does not implement the "truncate" + operation. Instead it saves all store requests to a transaction (which + takes up space in addition to the existing data), and commits the + transaction when the database is closed. Only then can the space for + old data be reused. + + This postmap(1) or postalias(1) command failure does not affect Postfix + availability, because the old data still exists in the database. + + To recover, increase the lmdb_map_size limit in main.cf, and retry the + postmap(1) or postalias(1) command. + + * PPoossttffiixx ddaaeemmoonn ""ddaattaabbaassee ffuullll"" eerrrroorrss.. + + Unfortunately, "database full" problems will affect Postfix availability + with daemon programs such as postscreen(8), tlsmgr(8) or verify(8). These + daemon programs will continue to fail until someone increases the + lmdb_map_size parameter value. Meanwhile, mail processing comes to a halt. + + * NNoonn--oobbvviioouuss rreeccoovveerryy ffrroomm aa ccoorrrruupptteedd ddaattaabbaassee.. + + Unlike other Postfix databases such as "hash" or "btree", you can't rebuild + a corrupted LMDB database simply by running postmap(1) or postalias(1), as + those commands will crash, too. + + The reason for this limitation is that the Postfix LMDB database client + does not implement the database "truncate" operation. Instead it tries to + save all store requests to a transaction for later processing. That is + obviously not possible with a corrupted database file. + + To recover, you must first delete the ".lmdb" file by hand, and only then + you can retry the postmap(1) or postalias(1) command. + + * IInnccoommppaattiibbiilliittyy wwiitthh ttllssmmggrr((88)).. + + The Postfix LMDB database client breaks tlsmgr(8) TLS session cache + management. Specifically, it breaks how tlsmgr(8) clears its TLS session + cache databases upon start-up, it breaks how tlsmgr(8) looks up new TLS + session cache entries, and it breaks how tlsmgr(8) automatically recovers + from a corrupted database file. + + The reason for these limitations is that the Postfix LMDB database client + does not implement the database "truncate" operation. Instead it saves all + store requests to a transaction which it commits only when the database is + closed. Therefore, tlsmgr(8) will never find any entries that it stores + after opening its TLS session cache databases. And when the database + becomes corrupted, tlsmgr(8) will keep crashing until someone removes the + file ".lmdb" file by hand. + diff --git a/postfix/RELEASE_NOTES b/postfix/RELEASE_NOTES index 41818732b..94a1a46ea 100644 --- a/postfix/RELEASE_NOTES +++ b/postfix/RELEASE_NOTES @@ -13,3 +13,11 @@ specifies the release date of a stable release or snapshot release. If you upgrade from Postfix 2.9 or earlier, read RELEASE_NOTES-2.10 before proceeding. + +Major changes with snapshot 20130315 +==================================== + +Perliminary LMDB support by Howard Chu. This implementation has +unexpected limitations that make it a poor fit, and for this reason +the code is "snapshot only", i.e. it will not be part of the stable +release. See LMDB_README for details. diff --git a/postfix/html/DATABASE_README.html b/postfix/html/DATABASE_README.html index bf1447b01..0c1169d99 100644 --- a/postfix/html/DATABASE_README.html +++ b/postfix/html/DATABASE_README.html @@ -311,6 +311,15 @@ name as used in "hash:table" is the database file name without the
A non-shared, in-memory hash table. Its content are lost when a process terminates.
+
lmdb
+ +
The OpenLDAP LMDB database (a memory-mapped, persistent file). +Database files are created with the postmap(1) or postalias(1) +command. The database name as used in "lmdb:table" is the database +file name without the ".lmdb" suffix. This database type has +unexpected limitations and is therefore not part of the stable +Postfix release. +
ldap (read-only)
Perform lookups using the LDAP protocol. Configuration details diff --git a/postfix/html/INSTALL.html b/postfix/html/INSTALL.html index 9356d7817..d4fd3a287 100644 --- a/postfix/html/INSTALL.html +++ b/postfix/html/INSTALL.html @@ -229,21 +229,25 @@ $ make

and so on. In some cases, optimization is turned off automatically.

-

4.3 - Building with optional extensions

+

4.3 - Building with optional features

By default, Postfix builds as a mail system with relatively few bells and whistles. Support for third-party databases etc. -must be configured when Postfix is compiled. The following documents describe how to build Postfix with support for extensions: +must be configured when Postfix is compiled. The following documents +describe how to build Postfix with support for optional features:
- + + + diff --git a/postfix/html/LMDB_README.html b/postfix/html/LMDB_README.html new file mode 100644 index 000000000..df4e32d83 --- /dev/null +++ b/postfix/html/LMDB_README.html @@ -0,0 +1,190 @@ + + + + + + +Postfix OpenLDAP LMDB Howto + + + + + + + +

Postfix OpenLDAP LMDB Howto

+ +
+ +

Introduction

+ +

Postfix uses databases of various kinds to store and look up +information. Postfix databases are specified as "type:name". +OpenLDAP LMDB implements the Postfix database type "lmdb". +The name of a Postfix OpenLDAP LMDB database is the name +of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases +are maintained with the postmap(1) command.

+ +

This document describes:

+ +
    + +
  1. How to build Postfix with OpenLDAP +LMDB support.

    + +
  2. How to configure LMDB settings.

    + +
  3. Missing pthread library trouble.

    + +
+ +

Caution: the current Postfix LMDB client has unexpected limitations. For this reason, +LMDB support is not part of the stable Postfix release.

+
+ +

Building Postfix with OpenLDAP LMDB support

+ +

Postfix normally does not enable OpenLDAP LMDB support. +To build Postfix after you installed OpenLDAP LMDB from +source code, use something like:

+ +
+
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+    AUXLIBS="-L/usr/local/lib -llmdb"
+% make
+
+
+ +

Solaris needs this:

+ +
+
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+    AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
+% make
+
+
+ +

The exact pathnames depend on how OpenLDAP LMDB was installed.

+ +

Configure LMDB settings

+ +

Postfix provides a configuration parameter that controls how +large an OpenLDAP LMDB database may grow.

+ +
    + +
  • lmdb_map_size (default: 10 MBytes per +table). This setting controls how large any OpenLDAP LMDB database +may grow. It must be set large enough to accommodate the largest +table that Postfix will use.

    + +
+ +

Missing pthread library trouble

+ +

When building Postfix fails with:

+ +
+
+undefined reference to `pthread_mutexattr_destroy'
+undefined reference to `pthread_mutexattr_init'
+undefined reference to `pthread_mutex_lock'
+
+
+ +

Add the "-lpthread" library to the "make makefiles" command.

+ +
+
+% make makefiles .... AUXLIBS="... -lpthread"
+
+
+ +

Source code for OpenLDAP LMDB is available at +http://www.openldap.org. +More information is available at +http://highlandsun.com/hyc/mdb/.

+ +

Limitations of Postfix LMDB databases.

+ +
    + +
  • Unexpected postmap/postalias "database full" errors. +

    + +

    Even if the "postmap filename" command succeeds, the exact same +command, with the exact same input data, may fail subsequently with +an MDB_MAP_FULL error. The reason is that unlike other Postfix +databases such as "hash" or "btree",

    + +
      + +
    • LMDB databases have a hard size limit (configured with the +lmdb_map_size configuration parameter).

      + +
    • The Postfix LMDB database client does not implement the +"truncate" operation. Instead it saves all store requests to a +transaction (which takes up space in addition to the existing data), +and commits the transaction when the database is closed. Only then +can the space for old data be reused.

      + +
    + +

    This postmap(1) or postalias(1) command failure does not affect +Postfix availability, because the old data still exists in the +database.

    + +

    To recover, increase the lmdb_map_size limit in main.cf, and +retry the postmap(1) or postalias(1) command.

    + +
  • Postfix daemon "database full" errors.

    + +

    Unfortunately, "database full" problems will affect Postfix +availability with daemon programs such as postscreen(8), tlsmgr(8) +or verify(8). These daemon programs will continue to fail until +someone increases the lmdb_map_size parameter value. Meanwhile, +mail processing comes to a halt.

    + +
  • Non-obvious recovery from a corrupted database. +

    + +

    Unlike other Postfix databases such as "hash" or "btree", you +can't rebuild a corrupted LMDB database simply by running postmap(1) +or postalias(1), as those commands will crash, too.

    + +

    The reason for this limitation is that the Postfix LMDB database +client does not implement the database "truncate" operation. Instead +it tries to save all store requests to a transaction for later +processing. That is obviously not possible with a corrupted database +file.

    + +

    To recover, you must first delete the ".lmdb" file by hand, and +only then you can retry the postmap(1) or postalias(1) command. +

    + +
  • Incompatibility with tlsmgr(8).

    + +

    The Postfix LMDB database client breaks tlsmgr(8) TLS session +cache management. Specifically, it breaks how tlsmgr(8) clears its +TLS session cache databases upon start-up, it breaks how tlsmgr(8) +looks up new TLS session cache entries, and it breaks how tlsmgr(8) +automatically recovers from a corrupted database file.

    + +

    The reason for these limitations is that the Postfix LMDB +database client does not implement the database "truncate" operation. +Instead it saves all store requests to a transaction which it commits +only when the database is closed. Therefore, tlsmgr(8) will never +find any entries that it stores after opening its TLS session cache +databases. And when the database becomes corrupted, tlsmgr(8) will +keep crashing until someone removes the file ".lmdb" file by hand. +

    + +
+ + + + diff --git a/postfix/html/postconf.5.html b/postfix/html/postconf.5.html index 0a8a9cf91..562f34e51 100644 --- a/postfix/html/postconf.5.html +++ b/postfix/html/postconf.5.html @@ -3773,6 +3773,21 @@ This feature is available in Postfix 2.1 and later. this length; upon delivery, long lines are reconstructed.

+ + +
lmdb_map_size +(default: 10485760)
+ +

+The per-table size limit for programs that create OpenLDAP LMDB +tables. Specify a byte count. +

+ +

+This feature is available in Postfix 2.11 and later. +

+ +
lmtp_address_preference diff --git a/postfix/man/man5/postconf.5 b/postfix/man/man5/postconf.5 index 9e1b3e9bb..f9706e676 100644 --- a/postfix/man/man5/postconf.5 +++ b/postfix/man/man5/postconf.5 @@ -2236,6 +2236,11 @@ This feature is available in Postfix 2.1 and later. .SH line_length_limit (default: 2048) Upon input, long lines are chopped up into pieces of at most this length; upon delivery, long lines are reconstructed. +.SH lmdb_map_size (default: 10485760) +The per-table size limit for programs that create OpenLDAP LMDB +tables. Specify a byte count. +.PP +This feature is available in Postfix 2.11 and later. .SH lmtp_address_preference (default: ipv6) The LMTP-specific version of the smtp_address_preference configuration parameter. See there for details. diff --git a/postfix/mantools/postlink b/postfix/mantools/postlink index 151866be1..e2a54e2f0 100755 --- a/postfix/mantools/postlink +++ b/postfix/mantools/postlink @@ -208,6 +208,7 @@ while (<>) { s;\bipc_timeout\b;$&;g; s;\bipc_ttl\b;$&;g; s;\bline_length_limit\b;$&;g; + s;\blmdb_map_size\b;$&;g; s;\blmtp_bind_address\b;$&;g; s;\blmtp_bind_address6\b;$&;g; s;\blmtp_assume_final\b;$&;g; @@ -1068,6 +1069,7 @@ while (<>) { s/\b(mysql):/$1<\/a>:/g; s/\b(nisplus):/$1<\/a>:/g; s/\b(ldap):/$1<\/a>:/g; + s/\b(lmdb):/$1<\/a>:/g; s/\b(regexp):/$1<\/a>:/g; s/\b(sqlite):/$1<\/a>:/g; s/\b(static):/$1<\/a>:/g; diff --git a/postfix/proto/DATABASE_README.html b/postfix/proto/DATABASE_README.html index f19bf3124..0de4f55a2 100644 --- a/postfix/proto/DATABASE_README.html +++ b/postfix/proto/DATABASE_README.html @@ -311,6 +311,15 @@ name as used in "hash:table" is the database file name without the
A non-shared, in-memory hash table. Its content are lost when a process terminates.
+
lmdb
+ +
The OpenLDAP LMDB database (a memory-mapped, persistent file). +Database files are created with the postmap(1) or postalias(1) +command. The database name as used in "lmdb:table" is the database +file name without the ".lmdb" suffix. This database type has +unexpected limitations and is therefore not part of the stable +Postfix release. +
ldap (read-only)
Perform lookups using the LDAP protocol. Configuration details diff --git a/postfix/proto/INSTALL.html b/postfix/proto/INSTALL.html index 6e26141a0..a4bb6ac46 100644 --- a/postfix/proto/INSTALL.html +++ b/postfix/proto/INSTALL.html @@ -229,21 +229,25 @@ $ make

and so on. In some cases, optimization is turned off automatically.

-

4.3 - Building with optional extensions

+

4.3 - Building with optional features

By default, Postfix builds as a mail system with relatively few bells and whistles. Support for third-party databases etc. -must be configured when Postfix is compiled. The following documents describe how to build Postfix with support for extensions: +must be configured when Postfix is compiled. The following documents +describe how to build Postfix with support for optional features:
Postfix extension Document Availability
Optional feature Document Availability
Berkeley DB database DB_README Postfix 1.0
LMDB database LMDB_README Postfix +2.11
LDAP database LDAP_README Postfix 1.0
- + + + diff --git a/postfix/proto/LMDB_README.html b/postfix/proto/LMDB_README.html new file mode 100644 index 000000000..2f7b4248f --- /dev/null +++ b/postfix/proto/LMDB_README.html @@ -0,0 +1,190 @@ + + + + + + +Postfix OpenLDAP LMDB Howto + + + + + + + +

Postfix OpenLDAP LMDB Howto

+ +
+ +

Introduction

+ +

Postfix uses databases of various kinds to store and look up +information. Postfix databases are specified as "type:name". +OpenLDAP LMDB implements the Postfix database type "lmdb". +The name of a Postfix OpenLDAP LMDB database is the name +of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases +are maintained with the postmap(1) command.

+ +

This document describes:

+ +
    + +
  1. How to build Postfix with OpenLDAP +LMDB support.

    + +
  2. How to configure LMDB settings.

    + +
  3. Missing pthread library trouble.

    + +
+ +

Caution: the current Postfix LMDB client has unexpected limitations. For this reason, +LMDB support is not part of the stable Postfix release.

+
+ +

Building Postfix with OpenLDAP LMDB support

+ +

Postfix normally does not enable OpenLDAP LMDB support. +To build Postfix after you installed OpenLDAP LMDB from +source code, use something like:

+ +
+
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+    AUXLIBS="-L/usr/local/lib -llmdb"
+% make
+
+
+ +

Solaris needs this:

+ +
+
+% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
+    AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
+% make
+
+
+ +

The exact pathnames depend on how OpenLDAP LMDB was installed.

+ +

Configure LMDB settings

+ +

Postfix provides a configuration parameter that controls how +large an OpenLDAP LMDB database may grow.

+ +
    + +
  • lmdb_map_size (default: 10 MBytes per +table). This setting controls how large any OpenLDAP LMDB database +may grow. It must be set large enough to accommodate the largest +table that Postfix will use.

    + +
+ +

Missing pthread library trouble

+ +

When building Postfix fails with:

+ +
+
+undefined reference to `pthread_mutexattr_destroy'
+undefined reference to `pthread_mutexattr_init'
+undefined reference to `pthread_mutex_lock'
+
+
+ +

Add the "-lpthread" library to the "make makefiles" command.

+ +
+
+% make makefiles .... AUXLIBS="... -lpthread"
+
+
+ +

Source code for OpenLDAP LMDB is available at +http://www.openldap.org. +More information is available at +http://highlandsun.com/hyc/mdb/.

+ +

Limitations of Postfix LMDB databases.

+ +
    + +
  • Unexpected postmap/postalias "database full" errors. +

    + +

    Even if the "postmap filename" command succeeds, the exact same +command, with the exact same input data, may fail subsequently with +an MDB_MAP_FULL error. The reason is that unlike other Postfix +databases such as "hash" or "btree",

    + +
      + +
    • LMDB databases have a hard size limit (configured with the +lmdb_map_size configuration parameter).

      + +
    • The Postfix LMDB database client does not implement the +"truncate" operation. Instead it saves all store requests to a +transaction (which takes up space in addition to the existing data), +and commits the transaction when the database is closed. Only then +can the space for old data be reused.

      + +
    + +

    This postmap(1) or postalias(1) command failure does not affect +Postfix availability, because the old data still exists in the +database.

    + +

    To recover, increase the lmdb_map_size limit in main.cf, and +retry the postmap(1) or postalias(1) command.

    + +
  • Postfix daemon "database full" errors.

    + +

    Unfortunately, "database full" problems will affect Postfix +availability with daemon programs such as postscreen(8), tlsmgr(8) +or verify(8). These daemon programs will continue to fail until +someone increases the lmdb_map_size parameter value. Meanwhile, +mail processing comes to a halt.

    + +
  • Non-obvious recovery from a corrupted database. +

    + +

    Unlike other Postfix databases such as "hash" or "btree", you +can't rebuild a corrupted LMDB database simply by running postmap(1) +or postalias(1), as those commands will crash, too.

    + +

    The reason for this limitation is that the Postfix LMDB database +client does not implement the database "truncate" operation. Instead +it tries to save all store requests to a transaction for later +processing. That is obviously not possible with a corrupted database +file.

    + +

    To recover, you must first delete the ".lmdb" file by hand, and +only then you can retry the postmap(1) or postalias(1) command. +

    + +
  • Incompatibility with tlsmgr(8).

    + +

    The Postfix LMDB database client breaks tlsmgr(8) TLS session +cache management. Specifically, it breaks how tlsmgr(8) clears its +TLS session cache databases upon start-up, it breaks how tlsmgr(8) +looks up new TLS session cache entries, and it breaks how tlsmgr(8) +automatically recovers from a corrupted database file.

    + +

    The reason for these limitations is that the Postfix LMDB +database client does not implement the database "truncate" operation. +Instead it saves all store requests to a transaction which it commits +only when the database is closed. Therefore, tlsmgr(8) will never +find any entries that it stores after opening its TLS session cache +databases. And when the database becomes corrupted, tlsmgr(8) will +keep crashing until someone removes the file ".lmdb" file by hand. +

    + +
+ + + + diff --git a/postfix/proto/Makefile.in b/postfix/proto/Makefile.in index d71454fc0..e9a69cd15 100644 --- a/postfix/proto/Makefile.in +++ b/postfix/proto/Makefile.in @@ -23,6 +23,7 @@ HTML = ../html/ADDRESS_CLASS_README.html \ ../html/LDAP_README.html \ ../html/LINUX_README.html \ ../html/LOCAL_RECIPIENT_README.html ../html/MAILDROP_README.html \ + ../html/LMDB_README.html \ ../html/MEMCACHE_README.html \ ../html/MILTER_README.html \ ../html/MULTI_INSTANCE_README.html \ @@ -64,6 +65,7 @@ README = ../README_FILES/ADDRESS_CLASS_README \ ../README_FILES/LDAP_README \ ../README_FILES/LINUX_README \ ../README_FILES/LOCAL_RECIPIENT_README ../README_FILES/MAILDROP_README \ + ../README_FILES/LMDB_README \ ../README_FILES/MEMCACHE_README \ ../README_FILES/MILTER_README \ ../README_FILES/MULTI_INSTANCE_README \ @@ -201,6 +203,9 @@ clobber: ../html/MAILDROP_README.html: MAILDROP_README.html $(POSTLINK) $? >$@ +../html/LMDB_README.html: LMDB_README.html + $(POSTLINK) $? >$@ + ../html/MEMCACHE_README.html: MEMCACHE_README.html $(POSTLINK) $? >$@ @@ -360,6 +365,9 @@ clobber: ../README_FILES/MAILDROP_README: MAILDROP_README.html $(HT2READ) $? >$@ +../README_FILES/LMDB_README: LMDB_README.html + $(HT2READ) $? >$@ + ../README_FILES/MEMCACHE_README: MEMCACHE_README.html $(HT2READ) $? >$@ diff --git a/postfix/proto/postconf.proto b/postfix/proto/postconf.proto index 18d27d10f..8ca20beed 100644 --- a/postfix/proto/postconf.proto +++ b/postfix/proto/postconf.proto @@ -2831,6 +2831,17 @@ The default time unit is d (days). Specify 0 when mail delivery should be tried only once.

+%PARAM lmdb_map_size 10485760 + +

+The per-table size limit for programs that create OpenLDAP LMDB +tables. Specify a byte count. +

+ +

+This feature is available in Postfix 2.11 and later. +

+ %PARAM message_size_limit 10240000

diff --git a/postfix/src/global/Makefile.in b/postfix/src/global/Makefile.in index 664d7726f..1b0e570fc 100644 --- a/postfix/src/global/Makefile.in +++ b/postfix/src/global/Makefile.in @@ -16,7 +16,7 @@ SRCS = abounce.c anvil_clnt.c been_here.c bounce.c bounce_log.c \ mail_params.c mail_pathname.c mail_queue.c mail_run.c \ mail_scan_dir.c mail_stream.c mail_task.c mail_trigger.c maps.c \ mark_corrupt.c match_parent_style.c mbox_conf.c mbox_open.c \ - mime_state.c mkmap_cdb.c mkmap_db.c mkmap_dbm.c mkmap_open.c \ + mime_state.c mkmap_cdb.c mkmap_db.c mkmap_dbm.c mkmap_lmdb.c mkmap_open.c \ mkmap_sdbm.c msg_stats_print.c msg_stats_scan.c mynetworks.c \ mypwd.c namadr_list.c off_cvt.c opened.c own_inet_addr.c \ pipe_command.c post_mail.c quote_821_local.c quote_822_local.c \ @@ -50,7 +50,7 @@ OBJS = abounce.o anvil_clnt.o been_here.o bounce.o bounce_log.o \ mail_params.o mail_pathname.o mail_queue.o mail_run.o \ mail_scan_dir.o mail_stream.o mail_task.o mail_trigger.o maps.o \ mark_corrupt.o match_parent_style.o mbox_conf.o mbox_open.o \ - mime_state.o mkmap_cdb.o mkmap_db.o mkmap_dbm.o mkmap_open.o \ + mime_state.o mkmap_cdb.o mkmap_db.o mkmap_dbm.o mkmap_lmdb.o mkmap_open.o \ mkmap_sdbm.o msg_stats_print.o msg_stats_scan.o mynetworks.o \ mypwd.o namadr_list.o off_cvt.o opened.o own_inet_addr.o \ pipe_command.o post_mail.o quote_821_local.o quote_822_local.o \ @@ -740,6 +740,7 @@ data_redirect.o: ../../include/dict.h data_redirect.o: ../../include/dict_cdb.h data_redirect.o: ../../include/dict_db.h data_redirect.o: ../../include/dict_dbm.h +data_redirect.o: ../../include/dict_lmdb.h data_redirect.o: ../../include/msg.h data_redirect.o: ../../include/myflock.h data_redirect.o: ../../include/name_code.h @@ -1434,6 +1435,7 @@ mail_params.o: ../../include/argv.h mail_params.o: ../../include/attr.h mail_params.o: ../../include/dict.h mail_params.o: ../../include/dict_db.h +mail_params.o: ../../include/dict_lmdb.h mail_params.o: ../../include/get_hostname.h mail_params.o: ../../include/inet_addr_list.h mail_params.o: ../../include/inet_proto.h @@ -1690,12 +1692,29 @@ mkmap_fail.o: ../../include/vstream.h mkmap_fail.o: ../../include/vstring.h mkmap_fail.o: mkmap.h mkmap_fail.o: mkmap_fail.c +mkmap_lmdb.o: ../../include/argv.h +mkmap_lmdb.o: ../../include/dict.h +mkmap_lmdb.o: ../../include/dict_lmdb.h +mkmap_lmdb.o: ../../include/msg.h +mkmap_lmdb.o: ../../include/myflock.h +mkmap_lmdb.o: ../../include/mymalloc.h +mkmap_lmdb.o: ../../include/stringops.h +mkmap_lmdb.o: ../../include/sys_defs.h +mkmap_lmdb.o: ../../include/vbuf.h +mkmap_lmdb.o: ../../include/vstream.h +mkmap_lmdb.o: ../../include/vstring.h +mkmap_lmdb.o: ../../include/warn_stat.h +mkmap_lmdb.o: mail_conf.h +mkmap_lmdb.o: mail_params.h +mkmap_lmdb.o: mkmap.h +mkmap_lmdb.o: mkmap_lmdb.c mkmap_open.o: ../../include/argv.h mkmap_open.o: ../../include/dict.h mkmap_open.o: ../../include/dict_cdb.h mkmap_open.o: ../../include/dict_db.h mkmap_open.o: ../../include/dict_dbm.h mkmap_open.o: ../../include/dict_fail.h +mkmap_open.o: ../../include/dict_lmdb.h mkmap_open.o: ../../include/dict_sdbm.h mkmap_open.o: ../../include/msg.h mkmap_open.o: ../../include/myflock.h diff --git a/postfix/src/global/data_redirect.c b/postfix/src/global/data_redirect.c index 096e58c14..7f497ca49 100644 --- a/postfix/src/global/data_redirect.c +++ b/postfix/src/global/data_redirect.c @@ -72,6 +72,7 @@ #include #include #include +#include #include /* Global directory. */ @@ -99,6 +100,7 @@ static const NAME_CODE data_redirect_map_types[] = { DICT_TYPE_HASH, 1, DICT_TYPE_BTREE, 1, DICT_TYPE_DBM, 1, + DICT_TYPE_LMDB, 1, DICT_TYPE_CDB, 1, /* not a read-write map type */ "sdbm", 1, /* legacy 3rd-party TLS */ "dbz", 1, /* just in case */ diff --git a/postfix/src/global/mail_params.c b/postfix/src/global/mail_params.c index 0e098b3db..94ea155d7 100644 --- a/postfix/src/global/mail_params.c +++ b/postfix/src/global/mail_params.c @@ -96,6 +96,7 @@ /* char *var_proxywrite_service; /* int var_db_create_buf; /* int var_db_read_buf; +/* int var_lmdb_map_size; /* int var_mime_maxdepth; /* int var_mime_bound_len; /* int var_header_limit; @@ -177,6 +178,9 @@ #ifdef HAS_DB #include #endif +#ifdef HAS_LMDB +#include +#endif #include #include #include @@ -284,6 +288,7 @@ char *var_proxymap_service; char *var_proxywrite_service; int var_db_create_buf; int var_db_read_buf; +int var_lmdb_map_size; int var_mime_maxdepth; int var_mime_bound_len; int var_header_limit; @@ -596,6 +601,7 @@ void mail_params_init() VAR_FAULT_INJ_CODE, DEF_FAULT_INJ_CODE, &var_fault_inj_code, 0, 0, VAR_DB_CREATE_BUF, DEF_DB_CREATE_BUF, &var_db_create_buf, 1, 0, VAR_DB_READ_BUF, DEF_DB_READ_BUF, &var_db_read_buf, 1, 0, + VAR_LMDB_MAP_SIZE, DEF_LMDB_MAP_SIZE, &var_lmdb_map_size, 1, 0, VAR_HEADER_LIMIT, DEF_HEADER_LIMIT, &var_header_limit, 1, 0, VAR_TOKEN_LIMIT, DEF_TOKEN_LIMIT, &var_token_limit, 1, 0, VAR_MIME_MAXDEPTH, DEF_MIME_MAXDEPTH, &var_mime_maxdepth, 1, 0, @@ -712,6 +718,9 @@ void mail_params_init() check_overlap(); #ifdef HAS_DB dict_db_cache_size = var_db_read_buf; +#endif +#ifdef HAS_LMDB + dict_lmdb_map_size = var_lmdb_map_size; #endif inet_windowsize = var_inet_windowsize; diff --git a/postfix/src/global/mail_params.h b/postfix/src/global/mail_params.h index 9c5fde004..53aa582cb 100644 --- a/postfix/src/global/mail_params.h +++ b/postfix/src/global/mail_params.h @@ -2727,6 +2727,13 @@ extern int var_db_create_buf; #define DEF_DB_READ_BUF (128 *1024) extern int var_db_read_buf; + /* + * OpenLDAP LMDB memory map size. + */ +#define VAR_LMDB_MAP_SIZE "lmdb_map_size" +#define DEF_LMDB_MAP_SIZE (16 * 1024 *1024) +extern int var_lmdb_map_size; + /* * Named queue file attributes. */ diff --git a/postfix/src/global/mail_version.h b/postfix/src/global/mail_version.h index 27b088b89..702a68240 100644 --- a/postfix/src/global/mail_version.h +++ b/postfix/src/global/mail_version.h @@ -20,7 +20,7 @@ * Patches change both the patchlevel and the release date. Snapshots have no * patchlevel; they change the release date only. */ -#define MAIL_RELEASE_DATE "20130211" +#define MAIL_RELEASE_DATE "20130315" #define MAIL_VERSION_NUMBER "2.11" #ifdef SNAPSHOT diff --git a/postfix/src/global/mkmap.h b/postfix/src/global/mkmap.h index d74a9125d..c131d4618 100644 --- a/postfix/src/global/mkmap.h +++ b/postfix/src/global/mkmap.h @@ -39,6 +39,7 @@ extern MKMAP *mkmap_dbm_open(const char *); extern MKMAP *mkmap_cdb_open(const char *); extern MKMAP *mkmap_hash_open(const char *); extern MKMAP *mkmap_btree_open(const char *); +extern MKMAP *mkmap_lmdb_open(const char *); extern MKMAP *mkmap_sdbm_open(const char *); extern MKMAP *mkmap_proxy_open(const char *); extern MKMAP *mkmap_fail_open(const char *); diff --git a/postfix/src/global/mkmap_lmdb.c b/postfix/src/global/mkmap_lmdb.c new file mode 100644 index 000000000..7ab120954 --- /dev/null +++ b/postfix/src/global/mkmap_lmdb.c @@ -0,0 +1,126 @@ +/*++ +/* NAME +/* mkmap_lmdb 3 +/* SUMMARY +/* create or open database, LMDB style +/* SYNOPSIS +/* #include +/* +/* MKMAP *mkmap_lmdb_open(path) +/* const char *path; +/* +/* DESCRIPTION +/* This module implements support for creating LMDB databases. +/* +/* mkmap_lmdb_open() takes a file name, appends the ".lmdb" +/* suffix, and does whatever initialization is required +/* before the OpenLDAP LMDB open routine is called. +/* +/* All errors are fatal. +/* SEE ALSO +/* dict_lmdb(3), LMDB dictionary interface. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Howard Chu +/* Symas Corporation +/*--*/ + +/* System library. */ + +#include +#include +#include +#include + +/* Utility library. */ + +#include +#include +#include +#include +#include +#include +#include + +#if defined(HAS_LMDB) && defined(SNAPSHOT) +#ifdef PATH_LMDB_H +#include PATH_LMDB_H +#else +#include +#endif + +/* Global library. */ + +#include +#include + +/* Application-specific. */ + +#include "mkmap.h" + +int var_proc_limit; + +/* mkmap_lmdb_open */ + +MKMAP *mkmap_lmdb_open(const char *path) +{ + MKMAP *mkmap = (MKMAP *) mymalloc(sizeof(*mkmap)); + static const CONFIG_INT_TABLE int_table[] = { + VAR_PROC_LIMIT, DEF_PROC_LIMIT, &var_proc_limit, 1, 0, + 0, + }; + + get_mail_conf_int_table(int_table); + + /* + * XXX Why is this not set in mail_params.c (with proper #ifdefs)? + * + * Override the default per-table map size for map (re)builds. + * + * lmdb_map_size is defined in util/dict_lmdb.c and defaults to 10MB. It + * needs to be large enough to contain the largest tables in use. + * + * XXX This should be specified via the DICT interface so that the buffer + * size becomes an object property, instead of being specified by poking + * a global variable so that it becomes a class property. + * + * XXX Wietse disagrees: storage infrastructure that requires up-front + * max-size information is evil. This unlike Postfix (e.g. line length or + * process count) limits which are a defense against out-of-control or + * malicious external actors. + * + * XXX Need to check that an existing table can be rebuilt with a larger + * size limit than was used for the initial build. + */ + dict_lmdb_map_size = var_lmdb_map_size; + + /* + * XXX Why is this not set in mail_params.c (with proper #ifdefs)? + * + * Set the max number of concurrent readers per table. This is the + * maximum number of postfix processes, plus some extra for CLI users. + * + * XXX Postfix uses asynchronous or blocking I/O with single-threaded + * processes so this limit will never be reached, assuming that the limit + * is a per-client property, not a shared database property. + */ + dict_lmdb_max_readers = var_proc_limit * 2 + 16; + + /* + * Fill in the generic members. + */ + mkmap->open = dict_lmdb_open; + mkmap->after_open = 0; + mkmap->after_close = 0; + + /* + * LMDB uses MVCC so it needs no special lock management here. + */ + + return (mkmap); +} + +#endif diff --git a/postfix/src/global/mkmap_open.c b/postfix/src/global/mkmap_open.c index d939d443d..e3e2d9595 100644 --- a/postfix/src/global/mkmap_open.c +++ b/postfix/src/global/mkmap_open.c @@ -65,6 +65,7 @@ #include #include #include +#include #include #include #include @@ -100,6 +101,9 @@ static const MKMAP_OPEN_INFO mkmap_types[] = { #ifdef HAS_DB DICT_TYPE_HASH, mkmap_hash_open, DICT_TYPE_BTREE, mkmap_btree_open, +#endif +#if defined(HAS_LMDB) && defined(SNAPSHOT) + DICT_TYPE_LMDB, mkmap_lmdb_open, #endif DICT_TYPE_FAIL, mkmap_fail_open, 0, diff --git a/postfix/src/util/Makefile.in b/postfix/src/util/Makefile.in index ea44c4ae5..14216c62b 100644 --- a/postfix/src/util/Makefile.in +++ b/postfix/src/util/Makefile.in @@ -4,7 +4,7 @@ SRCS = alldig.c allprint.c argv.c argv_split.c attr_clnt.c attr_print0.c \ attr_scan_plain.c auto_clnt.c base64_code.c basename.c binhash.c \ chroot_uid.c cidr_match.c clean_env.c close_on_exec.c concatenate.c \ ctable.c dict.c dict_alloc.c dict_cdb.c dict_cidr.c dict_db.c \ - dict_dbm.c dict_debug.c dict_env.c dict_ht.c dict_ni.c dict_nis.c \ + dict_dbm.c dict_debug.c dict_env.c dict_ht.c dict_lmdb.c dict_ni.c dict_nis.c \ dict_nisplus.c dict_open.c dict_pcre.c dict_regexp.c dict_sdbm.c \ dict_static.c dict_tcp.c dict_unix.c dir_forest.c doze.c dummy_read.c \ dummy_write.c duplex_pipe.c environ.c events.c exec_command.c \ @@ -41,7 +41,7 @@ OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \ chroot_uid.o cidr_match.o clean_env.o close_on_exec.o concatenate.o \ ctable.o dict.o dict_alloc.o dict_cdb.o dict_cidr.o dict_db.o \ - dict_dbm.o dict_debug.o dict_env.o dict_ht.o dict_ni.o dict_nis.o \ + dict_dbm.o dict_debug.o dict_env.o dict_ht.o dict_lmdb.o dict_ni.o dict_nis.o \ dict_nisplus.o dict_open.o dict_pcre.o dict_regexp.o dict_sdbm.o \ dict_static.o dict_tcp.o dict_unix.o dir_forest.o doze.o dummy_read.o \ dummy_write.o duplex_pipe.o environ.o events.o exec_command.o \ @@ -76,7 +76,7 @@ OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ HDRS = argv.h attr.h attr_clnt.h auto_clnt.h base64_code.h binhash.h \ chroot_uid.h cidr_match.h clean_env.h connect.h ctable.h dict.h \ dict_cdb.h dict_cidr.h dict_db.h dict_dbm.h dict_env.h dict_ht.h \ - dict_ni.h dict_nis.h dict_nisplus.h dict_pcre.h dict_regexp.h \ + dict_lmdb.h dict_ni.h dict_nis.h dict_nisplus.h dict_pcre.h dict_regexp.h \ dict_sdbm.h dict_static.h dict_tcp.h dict_unix.h dir_forest.h \ events.h exec_command.h find_inet.h fsspace.h fullname.h \ get_domainname.h get_hostname.h hex_code.h hex_quote.h host_port.h \ @@ -983,6 +983,21 @@ dict_ht.o: sys_defs.h dict_ht.o: vbuf.h dict_ht.o: vstream.h dict_ht.o: vstring.h +dict_lmdb.o: argv.h +dict_lmdb.o: dict.h +dict_lmdb.o: dict_lmdb.c +dict_lmdb.o: dict_lmdb.h +dict_lmdb.o: htable.h +dict_lmdb.o: iostuff.h +dict_lmdb.o: msg.h +dict_lmdb.o: myflock.h +dict_lmdb.o: mymalloc.h +dict_lmdb.o: stringops.h +dict_lmdb.o: sys_defs.h +dict_lmdb.o: vbuf.h +dict_lmdb.o: vstream.h +dict_lmdb.o: vstring.h +dict_lmdb.o: warn_stat.h dict_ni.o: dict_ni.c dict_ni.o: sys_defs.h dict_nis.o: argv.h @@ -1018,6 +1033,7 @@ dict_open.o: dict_dbm.h dict_open.o: dict_env.h dict_open.o: dict_fail.h dict_open.o: dict_ht.h +dict_open.o: dict_lmdb.h dict_open.o: dict_ni.h dict_open.o: dict_nis.h dict_open.o: dict_nisplus.h diff --git a/postfix/src/util/dict_lmdb.c b/postfix/src/util/dict_lmdb.c new file mode 100644 index 000000000..e60d44a79 --- /dev/null +++ b/postfix/src/util/dict_lmdb.c @@ -0,0 +1,541 @@ +/*++ +/* NAME +/* dict_lmdb 3 +/* SUMMARY +/* dictionary manager interface to OpenLDAP LMDB files +/* SYNOPSIS +/* #include +/* +/* DICT *dict_lmdb_open(path, open_flags, dict_flags) +/* const char *name; +/* const char *path; +/* int open_flags; +/* int dict_flags; +/* DESCRIPTION +/* dict_lmdb_open() opens the named LMDB database and makes it available +/* via the generic interface described in dict_open(3). +/* +/* The dict_lmdb_map_size variable specifies a non-default per-table +/* memory map size. The map size is 10MB. The map size is also the +/* maximum size the table can grow to, so it must be set large enough +/* to accomodate the largest tables in use. +/* DIAGNOSTICS +/* Fatal errors: cannot open file, file write error, out of memory. +/* SEE ALSO +/* dict(3) generic dictionary manager +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Howard Chu +/* Symas Corporation +/*--*/ + +#include "sys_defs.h" + +#if defined(HAS_LMDB) && defined(SNAPSHOT) + +/* System library. */ + +#include +#include +#include + +#ifdef PATH_LMDB_H +#include PATH_LMDB_H +#else +#include +#endif + +/* Utility library. */ + +#include "msg.h" +#include "mymalloc.h" +#include "htable.h" +#include "iostuff.h" +#include "vstring.h" +#include "myflock.h" +#include "stringops.h" +#include "dict.h" +#include "dict_lmdb.h" +#include "warn_stat.h" + +/* Application-specific. */ + +typedef struct { + DICT dict; /* generic members */ + MDB_env *env; /* LMDB environment */ + MDB_dbi dbi; /* database handle */ + MDB_txn *txn; /* write transaction for O_TRUNC */ + MDB_cursor *cursor; /* for sequence ops */ + VSTRING *key_buf; /* key buffer */ + VSTRING *val_buf; /* result buffer */ +} DICT_LMDB; + +#define SCOPY(buf, data, size) \ + vstring_str(vstring_strncpy(buf ? buf : (buf = vstring_alloc(10)), data, size)) + +size_t dict_lmdb_map_size = (10 * 1024 * 1024); /* 10MB default mmap + * size */ +unsigned int dict_lmdb_max_readers = 216; /* 200 postfix processes, + * plus some extra */ + +/* dict_lmdb_lookup - find database entry */ + +static const char *dict_lmdb_lookup(DICT *dict, const char *name) +{ + DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict; + MDB_val mdb_key; + MDB_val mdb_value; + MDB_txn *txn; + const char *result = 0; + int status, klen; + + dict->error = 0; + klen = strlen(name); + + /* + * Sanity check. + */ + if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0) + msg_panic("dict_lmdb_lookup: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag"); + + /* + * Optionally fold the key. + */ + if (dict->flags & DICT_FLAG_FOLD_FIX) { + if (dict->fold_buf == 0) + dict->fold_buf = vstring_alloc(10); + vstring_strcpy(dict->fold_buf, name); + name = lowercase(vstring_str(dict->fold_buf)); + } + + /* + * Start a read transaction if there's no global txn. + */ + if (dict_lmdb->txn) + txn = dict_lmdb->txn; + else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, MDB_RDONLY, &txn))) + msg_fatal("%s: txn_begin(read) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + + /* + * See if this LMDB file was written with one null byte appended to key + * and value. + */ + if (dict->flags & DICT_FLAG_TRY1NULL) { + mdb_key.mv_data = (void *) name; + mdb_key.mv_size = klen + 1; + status = mdb_get(txn, dict_lmdb->dbi, &mdb_key, &mdb_value); + if (!status) { + dict->flags &= ~DICT_FLAG_TRY0NULL; + result = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size); + } + } + + /* + * See if this LMDB file was written with no null byte appended to key + * and value. + */ + if (result == 0 && (dict->flags & DICT_FLAG_TRY0NULL)) { + mdb_key.mv_data = (void *) name; + mdb_key.mv_size = klen; + status = mdb_get(txn, dict_lmdb->dbi, &mdb_key, &mdb_value); + if (!status) { + dict->flags &= ~DICT_FLAG_TRY1NULL; + result = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size); + } + } + + /* + * Close the read txn if it's not the global txn. + */ + if (!dict_lmdb->txn) + mdb_txn_abort(txn); + + return (result); +} + +/* dict_lmdb_update - add or update database entry */ + +static int dict_lmdb_update(DICT *dict, const char *name, const char *value) +{ + DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict; + MDB_val mdb_key; + MDB_val mdb_value; + MDB_txn *txn; + int status; + + dict->error = 0; + + /* + * Sanity check. + */ + if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0) + msg_panic("dict_lmdb_update: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag"); + + /* + * Optionally fold the key. + */ + if (dict->flags & DICT_FLAG_FOLD_FIX) { + if (dict->fold_buf == 0) + dict->fold_buf = vstring_alloc(10); + vstring_strcpy(dict->fold_buf, name); + name = lowercase(vstring_str(dict->fold_buf)); + } + mdb_key.mv_data = (void *) name; + mdb_value.mv_data = (void *) value; + mdb_key.mv_size = strlen(name); + mdb_value.mv_size = strlen(value); + + /* + * If undecided about appending a null byte to key and value, choose a + * default depending on the platform. + */ + if ((dict->flags & DICT_FLAG_TRY1NULL) + && (dict->flags & DICT_FLAG_TRY0NULL)) { +#ifdef LMDB_NO_TRAILING_NULL + dict->flags &= ~DICT_FLAG_TRY1NULL; +#else + dict->flags &= ~DICT_FLAG_TRY0NULL; +#endif + } + + /* + * Optionally append a null byte to key and value. + */ + if (dict->flags & DICT_FLAG_TRY1NULL) { + mdb_key.mv_size++; + mdb_value.mv_size++; + } + + /* + * Start a write transaction if there's no global txn. + */ + if (dict_lmdb->txn) + txn = dict_lmdb->txn; + else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn))) + msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + + /* + * Do the update. + */ + status = mdb_put(txn, dict_lmdb->dbi, &mdb_key, &mdb_value, + (dict->flags & DICT_FLAG_DUP_REPLACE) ? 0 : MDB_NOOVERWRITE); + if (status) { + if (status == MDB_KEYEXIST) { + if (dict->flags & DICT_FLAG_DUP_IGNORE) + /* void */ ; + else if (dict->flags & DICT_FLAG_DUP_WARN) + msg_warn("%s: duplicate entry: \"%s\"", dict_lmdb->dict.name, name); + else + msg_fatal("%s: duplicate entry: \"%s\"", dict_lmdb->dict.name, name); + } else { + msg_fatal("error writing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(status)); + } + } + + /* + * Commit the transaction if it's not the global txn. + */ + if (!dict_lmdb->txn && ((status = mdb_txn_commit(txn)))) + msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(status)); + + return (status); +} + +/* dict_lmdb_delete - delete one entry from the dictionary */ + +static int dict_lmdb_delete(DICT *dict, const char *name) +{ + DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict; + MDB_val mdb_key; + MDB_txn *txn; + int status = 1, klen, rc; + + dict->error = 0; + klen = strlen(name); + + /* + * Sanity check. + */ + if ((dict->flags & (DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL)) == 0) + msg_panic("dict_lmdb_delete: no DICT_FLAG_TRY1NULL | DICT_FLAG_TRY0NULL flag"); + + /* + * Optionally fold the key. + */ + if (dict->flags & DICT_FLAG_FOLD_FIX) { + if (dict->fold_buf == 0) + dict->fold_buf = vstring_alloc(10); + vstring_strcpy(dict->fold_buf, name); + name = lowercase(vstring_str(dict->fold_buf)); + } + + /* + * Start a write transaction if there's no global txn. + */ + if (dict_lmdb->txn) + txn = dict_lmdb->txn; + else if ((status = mdb_txn_begin(dict_lmdb->env, NULL, 0, &txn))) + msg_fatal("%s: txn_begin(write) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + + /* + * See if this LMDB file was written with one null byte appended to key + * and value. + */ + if (dict->flags & DICT_FLAG_TRY1NULL) { + mdb_key.mv_data = (void *) name; + mdb_key.mv_size = klen + 1; + status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL); + if (status) { + if (status == MDB_NOTFOUND) + status = 1; + else + msg_fatal("error deleting from %s: %s", dict_lmdb->dict.name, mdb_strerror(status)); + } else { + dict->flags &= ~DICT_FLAG_TRY0NULL; /* found */ + } + } + + /* + * See if this LMDB file was written with no null byte appended to key + * and value. + */ + if (status > 0 && (dict->flags & DICT_FLAG_TRY0NULL)) { + mdb_key.mv_data = (void *) name; + mdb_key.mv_size = klen; + status = mdb_del(txn, dict_lmdb->dbi, &mdb_key, NULL); + if (status) { + if (status == MDB_NOTFOUND) + status = 1; + else + msg_fatal("error deleting from %s: %s", dict_lmdb->dict.name, mdb_strerror(status)); + } else { + dict->flags &= ~DICT_FLAG_TRY1NULL; /* found */ + } + } + + /* + * Commit the transaction if it's not the global txn. + */ + if (!dict_lmdb->txn && ((rc = mdb_txn_commit(txn)))) + msg_fatal("error committing LMDB database %s: %s", dict_lmdb->dict.name, mdb_strerror(rc)); + + return (status); +} + +/* traverse the dictionary */ + +static int dict_lmdb_sequence(DICT *dict, int function, + const char **key, const char **value) +{ + const char *myname = "dict_lmdb_sequence"; + DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict; + MDB_val mdb_key; + MDB_val mdb_value; + MDB_txn *txn; + MDB_cursor_op op; + int status; + + dict->error = 0; + + /* + * Determine the seek function. + */ + switch (function) { + case DICT_SEQ_FUN_FIRST: + op = MDB_FIRST; + break; + case DICT_SEQ_FUN_NEXT: + op = MDB_NEXT; + break; + default: + msg_panic("%s: invalid function: %d", myname, function); + } + + /* + * Open a read transaction and cursor if needed. + */ + if (dict_lmdb->cursor == 0) { + if ((status = mdb_txn_begin(dict_lmdb->env, NULL, MDB_RDONLY, &txn))) + msg_fatal("%s: txn_begin(read) dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + if ((status = mdb_cursor_open(txn, dict_lmdb->dbi, &dict_lmdb->cursor))) + msg_fatal("%s: cursor_open dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + } + + /* + * Database lookup. + */ + status = mdb_cursor_get(dict_lmdb->cursor, &mdb_key, &mdb_value, op); + if (status && status != MDB_NOTFOUND) + msg_fatal("%s: seeking dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + + if (status == MDB_NOTFOUND) { + + /* + * Caller must read to end, to ensure cursor gets closed. + */ + status = 1; + txn = mdb_cursor_txn(dict_lmdb->cursor); + mdb_cursor_close(dict_lmdb->cursor); + mdb_txn_abort(txn); + dict_lmdb->cursor = 0; + } else { + + /* + * Copy the key so that it is guaranteed null terminated. + */ + *key = SCOPY(dict_lmdb->key_buf, mdb_key.mv_data, mdb_key.mv_size); + + if (mdb_value.mv_data != 0 && mdb_value.mv_size > 0) { + + /* + * Copy the value so that it is guaranteed null terminated. + */ + *value = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size); + status = 0; + } + } + + return (status); +} + +/* dict_lmdb_lock - noop lock handler */ + +static int dict_lmdb_lock(DICT *dict, int unused_op) +{ + /* LMDB does its own concurrency control */ + return 0; +} + +/* dict_lmdb_close - disassociate from data base */ + +static void dict_lmdb_close(DICT *dict) +{ + DICT_LMDB *dict_lmdb = (DICT_LMDB *) dict; + + if (dict_lmdb->txn) { + int status = mdb_txn_commit(dict_lmdb->txn); + + if (status) + msg_fatal("%s: closing dictionary: %s", dict_lmdb->dict.name, mdb_strerror(status)); + dict_lmdb->cursor = NULL; + } + if (dict_lmdb->cursor) { + MDB_txn *txn = mdb_cursor_txn(dict_lmdb->cursor); + + mdb_cursor_close(dict_lmdb->cursor); + mdb_txn_abort(txn); + } + if (dict_lmdb->dict.stat_fd >= 0) + close(dict_lmdb->dict.stat_fd); + mdb_env_close(dict_lmdb->env); + if (dict_lmdb->key_buf) + vstring_free(dict_lmdb->key_buf); + if (dict_lmdb->val_buf) + vstring_free(dict_lmdb->val_buf); + if (dict->fold_buf) + vstring_free(dict->fold_buf); + dict_free(dict); +} + +/* dict_lmdb_open - open LMDB data base */ + +DICT *dict_lmdb_open(const char *path, int open_flags, int dict_flags) +{ + DICT_LMDB *dict_lmdb; + struct stat st; + MDB_env *env; + MDB_txn *txn; + MDB_dbi dbi; + char *mdb_path; + int env_flags, status; + + mdb_path = concatenate(path, ".lmdb", (char *) 0); + + env_flags = MDB_NOSUBDIR; + if (open_flags == O_RDONLY) + env_flags |= MDB_RDONLY; + + if ((status = mdb_env_create(&env))) + msg_fatal("env_create %s: %s", mdb_path, mdb_strerror(status)); + + if ((status = mdb_env_set_mapsize(env, dict_lmdb_map_size))) + msg_fatal("env_set_mapsize %s: %s", mdb_path, mdb_strerror(status)); + + if ((status = mdb_env_set_maxreaders(env, dict_lmdb_max_readers))) + msg_fatal("env_set_maxreaders %s: %s", mdb_path, mdb_strerror(status)); + + if ((status = mdb_env_open(env, mdb_path, env_flags, 0644))) + msg_fatal("env_open %s: %s", mdb_path, mdb_strerror(status)); + + if ((status = mdb_txn_begin(env, NULL, env_flags & MDB_RDONLY, &txn))) + msg_fatal("txn_begin %s: %s", mdb_path, mdb_strerror(status)); + + /* + * mdb_open requires a txn, but since the default DB always exists in an + * LMDB environment, we don't need to do anything else with the txn. + */ + if ((status = mdb_open(txn, NULL, 0, &dbi))) + msg_fatal("mdb_open %s: %s", mdb_path, mdb_strerror(status)); + + /* + * However, if O_TRUNC was specified, we need to do it now. Also with + * O_TRUNC we keep this write txn for as long as the database is open, + * since we'll probably be doing a bulk import immediately after. + */ + if (open_flags & O_TRUNC) { + if ((status = mdb_drop(txn, dbi, 0))) + msg_fatal("truncate %s: %s", mdb_path, mdb_strerror(status)); + } else { + mdb_txn_abort(txn); + txn = NULL; + } + + dict_lmdb = (DICT_LMDB *) dict_alloc(DICT_TYPE_LMDB, path, sizeof(*dict_lmdb)); + dict_lmdb->dict.lookup = dict_lmdb_lookup; + dict_lmdb->dict.update = dict_lmdb_update; + dict_lmdb->dict.delete = dict_lmdb_delete; + dict_lmdb->dict.sequence = dict_lmdb_sequence; + dict_lmdb->dict.close = dict_lmdb_close; + dict_lmdb->dict.lock = dict_lmdb_lock; + dict_lmdb->dict.stat_fd = open(mdb_path, O_RDONLY); + if (fstat(dict_lmdb->dict.stat_fd, &st) < 0) + msg_fatal("dict_lmdb_open: fstat: %m"); + dict_lmdb->dict.mtime = st.st_mtime; + dict_lmdb->dict.owner.uid = st.st_uid; + dict_lmdb->dict.owner.status = (st.st_uid != 0); + + /* + * Warn if the source file is newer than the indexed file, except when + * the source file changed only seconds ago. + */ + if ((dict_flags & DICT_FLAG_LOCK) != 0 + && stat(path, &st) == 0 + && st.st_mtime > dict_lmdb->dict.mtime + && st.st_mtime < time((time_t *) 0) - 100) + msg_warn("database %s is older than source file %s", mdb_path, path); + + close_on_exec(dict_lmdb->dict.stat_fd, CLOSE_ON_EXEC); + dict_lmdb->dict.flags = dict_flags | DICT_FLAG_FIXED; + if ((dict_flags & (DICT_FLAG_TRY0NULL | DICT_FLAG_TRY1NULL)) == 0) + dict_lmdb->dict.flags |= (DICT_FLAG_TRY0NULL | DICT_FLAG_TRY1NULL); + if (dict_flags & DICT_FLAG_FOLD_FIX) + dict_lmdb->dict.fold_buf = vstring_alloc(10); + dict_lmdb->env = env; + dict_lmdb->dbi = dbi; + + /* Save the write txn if we opened with O_TRUNC */ + dict_lmdb->txn = txn; + + dict_lmdb->cursor = 0; + dict_lmdb->key_buf = 0; + dict_lmdb->val_buf = 0; + + myfree(mdb_path); + + return (DICT_DEBUG (&dict_lmdb->dict)); +} + +#endif diff --git a/postfix/src/util/dict_lmdb.h b/postfix/src/util/dict_lmdb.h new file mode 100644 index 000000000..e44fcf6b3 --- /dev/null +++ b/postfix/src/util/dict_lmdb.h @@ -0,0 +1,41 @@ +#ifndef _DICT_LMDB_H_INCLUDED_ +#define _DICT_LMDB_H_INCLUDED_ + +/*++ +/* NAME +/* dict_lmdb 3h +/* SUMMARY +/* dictionary manager interface to OpenLDAP LMDB files +/* SYNOPSIS +/* #include +/* DESCRIPTION +/* .nf + + /* + * Utility library. + */ +#include + + /* + * External interface. + */ +#define DICT_TYPE_LMDB "lmdb" + +extern DICT *dict_lmdb_open(const char *, int, int); + + /* + * XXX Should be part of the DICT interface. + */ +extern size_t dict_lmdb_map_size; +extern unsigned int dict_lmdb_max_readers; + +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Howard Chu +/* Symas Corporation +/*--*/ + +#endif diff --git a/postfix/src/util/dict_open.c b/postfix/src/util/dict_open.c index 0f1778574..f0dca1db7 100644 --- a/postfix/src/util/dict_open.c +++ b/postfix/src/util/dict_open.c @@ -229,6 +229,7 @@ #include #include #include +#include #include #include #include @@ -271,6 +272,9 @@ static const DICT_OPEN_INFO dict_open_info[] = { DICT_TYPE_HASH, dict_hash_open, DICT_TYPE_BTREE, dict_btree_open, #endif +#if defined(HAS_LMDB) && defined(SNAPSHOT) + DICT_TYPE_LMDB, dict_lmdb_open, +#endif #ifdef HAS_NIS DICT_TYPE_NIS, dict_nis_open, #endif

Postfix extension Document Availability
Optional feature Document Availability
Berkeley DB database DB_README Postfix 1.0
LMDB database LMDB_README Postfix +2.11
LDAP database LDAP_README Postfix 1.0