From: Wietse Venema Date: Sun, 28 Dec 2014 05:00:00 +0000 (-0500) Subject: postfix-2.12-20150110-nonprod X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=36a173c9f9d2de9cab129524a963f5536859ebf0;p=thirdparty%2Fpostfix.git postfix-2.12-20150110-nonprod --- diff --git a/postfix/HISTORY b/postfix/HISTORY index 4b8cf432e..832c995bd 100644 --- a/postfix/HISTORY +++ b/postfix/HISTORY @@ -21209,3 +21209,113 @@ Apologies for any names omitted. either their result is a valid ASCII domain name or that it converts into a valid ASCII domain name. Files: util/midna.c, util/midna_test.in, util/midna_test.ref. + +20141230 + + Cleanup: s/midna/midna_domain/ for better specificity, + because we also need functions that act only on the domain + portion of an email address. Files: bounce/bounce_template.c, + global/midna_adomain.c, posttls-finger/posttls-finger.c, + smtp/smtp_addr.c, smtpd/smtpd_check.c, tls/tls_client.c, + util/midna_domain.[hc], util/valid_utf8_hostname.c. + + Infrastructure: function midna_adomain_to_utf8() (and + midna_adomain_to_ascii) to convert the domain portion of + an email address before table lookup. Files: + global/midna_adomain.[hc]. + +20141230-20140109 + + What is described here is the result of four iterations to + deal with malformed UTF-8 without massively contaminating + every Postfix program with new error-handling code paths, + in particular without triggering fatal errors that didn't + happen before. + + Infrastructure: function casefold() to support caseless + string comparison, primarily for table lookups. This function + supports two modes: case folding a la lowercase() for ASCII + byte values, and UTF-8 case folding. As recommended at + http://www.w3.org/International/wiki/Case_folding for + caseless string comparison, this uses the en_US locale to + avoid surprises. The implementatin handles + the entire RFC 3629 Unicode range (code points U+0000..U+10FFFF + including surrogates) and is chroot(2) safe. Files: casefold.c, stringops.h. + + Infrastructure: revised the midna_domain_to_ascii and + midna_domain_to_utf8 domain name conversion functions after + careful reading of the UTS #46 specification, and after + observing that ICU 4.8 library functions indeed implement + this spec, at least with default options. In particular, + midna_domain_to_utf8 takes an UTF-8 domain name and verifies + that its A-label form will pass the valid_hostname() test. + File: util/midna_domain.c. + + Infrastructure: handle UTF-8 errors in lookup table keys + or values without massively contaminating every Postfix + program with new error-handling code paths, in particular + without triggering fatal errors that didn't happen before. + The lookup/update/delete functions log a warning and ignore + a request with a bad key (it cannot exist); the update + functions ignore a request to store a bad value (it cannot + exist); and the lookup function reports a bad value as a + configuration error (it should not exist, but there it is). + Table iterators still report all (key, value) pairs in a + table. Files: util/dict.h, util/dict_open.c, util/dict_utf8.c, + global/mkmap_open.c. + + Note that with SMTPUTF8 turned on, each table-driven mechanism + (access, aliases, etc.) needs to make its own decision + whether UTF-8 syntax is required. We cannot blindly require + that everything has valid UTF-8 syntax. That would make + header/body_checks useless for content inspection, because + headers may be malformed and bodies may contain legitimate + binary content that isn't UTF-8. + + Note that with SMTPUTF8 turned off, Postfix must remain + 8-bit clean as it always has been. Table operations must + not complain that something violates UTF-8 syntax rules. + + UTF-8 sanitization in the Postfix SMTP server. With + smtputf8_enable=yes, SMTP commands with UTF-8 syntax errors + are rejected, table lookup results with invalid UTF-8 syntax + are handled as configuration errors, and UTF-8 syntax errors + in policy server replies result in execution of the policy + server's default action. + +20150102 + + Cleanup: propagate DICT_ERR_CONFIG through the proxymap + protocol. Files: global/dict_proxy.[hc], proxymap/proxymap.c. + +20150106 + + Robustness: don't segfault due to excessive recursion in + tok822_free_tree() after a faulty configuration runs into + the virtual_alias_recursion_limit. File: global/tok822_tree.c. + +20150109 + + Cleanup: the dict debug module now proxies dict flags. + File: util/dict_debug.c. + + With "smtputf8_enable = yes", the postmap and postalias + commands now enable UTF-8 by default (use "-u" to disable) + with one exception: UTF-8 remains disabled for header/body_checks + emulation (use "-U" to enable). Files: postmap/postmap.c, + postalias/postalias.c. + +20150110 + + Cleanup: the "inline" and "texthash" implementations now + reuse the "internal" database instead of reinventing the + wheel. Files: util/dict_inline.c, util/dict_thash.c. + + As a first step, with "smtputf8_enable = yes" all features + based on Postfix matchlists enable UTF-8 syntax checks and + casefolding support for table queries and results. That + includes mynetworks, mydestination, relay_domains, + virtual_alias_domains, and virtual_mailbox_domains. + + The next step is to turn on UTF-8 syntax checks and casefolding + support for access maps, address rewriting and routing. diff --git a/postfix/WISHLIST b/postfix/WISHLIST index 7621437f4..48bdf4958 100644 --- a/postfix/WISHLIST +++ b/postfix/WISHLIST @@ -8,6 +8,15 @@ Wish list: Things to do after the stable release: + Try to allow UTF-8 myhostname/mydomain, at least in bounce + template expansion. + + No enhanced status code when rejecting connection before + the HELO handshake is completed. + + Maybe don't whitelist a client that has maxed out its + per-MTA connection count limit. + Inline support for pcre:{/pattern/=action, ...} and ditto support for regexp: and cidr: tables. Factor out and reuse code that already exists in inline: and other tables. diff --git a/postfix/html/postalias.1.html b/postfix/html/postalias.1.html index 124781b05..3e67e0b76 100644 --- a/postfix/html/postalias.1.html +++ b/postfix/html/postalias.1.html @@ -10,7 +10,7 @@ POSTALIAS(1) POSTALIAS(1) postalias - Postfix alias database maintenance SYNOPSIS - postalias [-Nfinoprsvw] [-c config_dir] [-d key] [-q key] + postalias [-Nfinoprsuvw] [-c config_dir] [-d key] [-q key] [file_type:]file_name ... DESCRIPTION @@ -95,6 +95,10 @@ POSTALIAS(1) POSTALIAS(1) order. This feature is available in Postfix version 2.2 and later, and is not available for all database types. + -u Disable UTF-8 support. UTF-8 support is enabled by default when + "smtputf8_enable = yes". It requires that keys and values are + valid UTF-8 strings. + -v Enable verbose logging for debugging purposes. Multiple -v options make the software increasingly verbose. @@ -183,12 +187,16 @@ POSTALIAS(1) POSTALIAS(1) The default database type for use in newaliases(1), postalias(1) and postmap(1) commands. + smtputf8_enable (yes) + Enable experimental SMTPUTF8 support for the protocols described + in RFC 6531..6533. + syslog_facility (mail) The syslog facility of Postfix logging. syslog_name (see 'postconf -d' output) - The mail system name that is prepended to the process name in - syslog records, so that "smtpd" becomes, for example, "post- + The mail system name that is prepended to the process name in + syslog records, so that "smtpd" becomes, for example, "post- fix/smtpd". STANDARDS diff --git a/postfix/html/postconf.5.html b/postfix/html/postconf.5.html index cace24f43..919e372e1 100644 --- a/postfix/html/postconf.5.html +++ b/postfix/html/postconf.5.html @@ -9891,8 +9891,8 @@ SMTP servers that reject recipients after the DATA command. Use 
 /etc/postfix/transport:
-    smtp-domain_that_verifies_after_data    smtp-data-target:
-    lmtp-domain_that_verifies_after_data    lmtp-data-target:
+    smtp-domain-that-verifies-after-data    smtp-data-target:
+    lmtp-domain-that-verifies-after-data    lmtp-data-target:
diff --git a/postfix/html/postmap.1.html b/postfix/html/postmap.1.html index 55d1e9b2b..9822a2dd1 100644 --- a/postfix/html/postmap.1.html +++ b/postfix/html/postmap.1.html @@ -10,7 +10,7 @@ POSTMAP(1) POSTMAP(1) postmap - Postfix lookup table management SYNOPSIS - postmap [-Nbfhimnoprsvw] [-c config_dir] [-d key] [-q key] + postmap [-NbfhimnoprsuUvw] [-c config_dir] [-d key] [-q key] [file_type:]file_name ... DESCRIPTION @@ -66,6 +66,10 @@ POSTMAP(1) POSTMAP(1) style lookup keys for attachment MIME headers and for attached message/* headers. + NOTE: with "smtputf8_enable = yes", the -b option option dis- + ables UTF-8 syntax checks on query keys and lookup results. + Specify the -U option to force UTF-8 syntax checks anyway. + This feature is available in Postfix version 2.6 and later. -c config_dir @@ -99,6 +103,10 @@ POSTMAP(1) POSTMAP(1) also generates header-style lookup keys for attachment MIME headers and for attached message/* headers. + NOTE: with "smtputf8_enable = yes", the -b option option dis- + ables UTF-8 syntax checks on query keys and lookup results. + Specify the -U option to force UTF-8 syntax checks anyway. + This feature is available in Postfix version 2.6 and later. -i Incremental mode. Read entries from standard input and do not @@ -145,10 +153,17 @@ POSTMAP(1) POSTMAP(1) This feature is available in Postfix version 2.2 and later, and is not available for all database types. - -v Enable verbose logging for debugging purposes. Multiple -v + -u Disable UTF-8 support. UTF-8 support is enabled by default when + "smtputf8_enable = yes". It requires that keys and values are + valid UTF-8 strings. + + -U With "smtputf8_enable = yes", force UTF-8 syntax checks with the + -b and -h options. + + -v Enable verbose logging for debugging purposes. Multiple -v options make the software increasingly verbose. - -w When updating a table, do not complain about attempts to update + -w When updating a table, do not complain about attempts to update existing entries, and ignore those attempts. Arguments: @@ -160,32 +175,32 @@ POSTMAP(1) POSTMAP(1) The postmap(1) command can query any supported file type, but it can create only the following file types: - btree The output file is a btree file, named file_name.db. - This is available on systems with support for db data- + btree The output file is a btree file, named file_name.db. + This is available on systems with support for db data- bases. - cdb The output consists of one file, named file_name.cdb. - This is available on systems with support for cdb data- + cdb The output consists of one file, named file_name.cdb. + This is available on systems with support for cdb data- bases. dbm The output consists of two files, named file_name.pag and file_name.dir. This is available on systems with support for dbm databases. - hash The output file is a hashed file, named file_name.db. - This is available on systems with support for db data- + hash The output file is a hashed file, named file_name.db. + This is available on systems with support for db data- bases. - fail A table that reliably fails all requests. The lookup ta- - ble name is used for logging only. This table exists to + fail A table that reliably fails all requests. The lookup ta- + ble name is used for logging only. This table exists to simplify Postfix error tests. sdbm The output consists of two files, named file_name.pag and file_name.dir. This is available on systems with support for sdbm databases. - When no file_type is specified, the software uses the database - type specified via the default_database_type configuration + When no file_type is specified, the software uses the database + type specified via the default_database_type configuration parameter. file_name @@ -194,11 +209,11 @@ POSTMAP(1) POSTMAP(1) DIAGNOSTICS Problems are logged to the standard error stream and to syslogd(8). No - output means that no problems were detected. Duplicate entries are + output means that no problems were detected. Duplicate entries are skipped and are flagged with a warning. postmap(1) terminates with zero exit status in case of success (includ- - ing successful "postmap -q" lookup) and terminates with non-zero exit + ing successful "postmap -q" lookup) and terminates with non-zero exit status in case of failure. ENVIRONMENT @@ -209,12 +224,12 @@ POSTMAP(1) POSTMAP(1) Enable verbose logging for debugging purposes. CONFIGURATION PARAMETERS - The following main.cf parameters are especially relevant to this pro- - gram. The text below provides only a parameter summary. See post- + The following main.cf parameters are especially relevant to this pro- + gram. The text below provides only a parameter summary. See post- conf(5) for more details including examples. berkeley_db_create_buffer_size (16777216) - The per-table I/O buffer size for programs that create Berkeley + The per-table I/O buffer size for programs that create Berkeley DB hash or btree tables. berkeley_db_read_buffer_size (131072) @@ -222,13 +237,17 @@ POSTMAP(1) POSTMAP(1) hash or btree tables. config_directory (see 'postconf -d' output) - The default location of the Postfix main.cf and master.cf con- + The default location of the Postfix main.cf and master.cf con- figuration files. default_database_type (see 'postconf -d' output) The default database type for use in newaliases(1), postalias(1) and postmap(1) commands. + smtputf8_enable (yes) + Enable experimental SMTPUTF8 support for the protocols described + in RFC 6531..6533. + syslog_facility (mail) The syslog facility of Postfix logging. diff --git a/postfix/makedefs b/postfix/makedefs index d17621da7..4b16bc1d5 100644 --- a/postfix/makedefs +++ b/postfix/makedefs @@ -783,7 +783,7 @@ CCARGS="$CCARGS -DSNAPSHOT" # Non-production: needs thorough testing, or major changes are still # needed before the code stabilizes. -#CCARGS="$CCARGS -DNONPROD" +CCARGS="$CCARGS -DNONPROD" # Workaround: prepend Postfix include files before other include files. CCARGS="-I. -I../../include $CCARGS" diff --git a/postfix/man/man1/postalias.1 b/postfix/man/man1/postalias.1 index 9dda165f6..4a8b0f47e 100644 --- a/postfix/man/man1/postalias.1 +++ b/postfix/man/man1/postalias.1 @@ -9,7 +9,7 @@ Postfix alias database maintenance .na .nf .fi -\fBpostalias\fR [\fB-Nfinoprsvw\fR] [\fB-c \fIconfig_dir\fR] +\fBpostalias\fR [\fB-Nfinoprsuvw\fR] [\fB-c \fIconfig_dir\fR] [\fB-d \fIkey\fR] [\fB-q \fIkey\fR] [\fIfile_type\fR:]\fIfile_name\fR ... .SH DESCRIPTION @@ -99,6 +99,10 @@ printed in database order, which is not necessarily the same as the original input order. This feature is available in Postfix version 2.2 and later, and is not available for all database types. +.IP \fB-u\fR +Disable UTF-8 support. UTF-8 support is enabled by default +when "smtputf8_enable = yes". It requires that keys and +values are valid UTF-8 strings. .IP \fB-v\fR Enable verbose logging for debugging purposes. Multiple \fB-v\fR options make the software increasingly verbose. @@ -188,6 +192,9 @@ hash or btree tables. .IP "\fBdefault_database_type (see 'postconf -d' output)\fR" The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1) and \fBpostmap\fR(1) commands. +.IP "\fBsmtputf8_enable (yes)\fR" +Enable experimental SMTPUTF8 support for the protocols described +in RFC 6531..6533. .IP "\fBsyslog_facility (mail)\fR" The syslog facility of Postfix logging. .IP "\fBsyslog_name (see 'postconf -d' output)\fR" diff --git a/postfix/man/man1/postmap.1 b/postfix/man/man1/postmap.1 index cf6ae5240..6acadd528 100644 --- a/postfix/man/man1/postmap.1 +++ b/postfix/man/man1/postmap.1 @@ -9,7 +9,7 @@ Postfix lookup table management .na .nf .fi -\fBpostmap\fR [\fB-Nbfhimnoprsvw\fR] [\fB-c \fIconfig_dir\fR] +\fBpostmap\fR [\fB-NbfhimnoprsuUvw\fR] [\fB-c \fIconfig_dir\fR] [\fB-d \fIkey\fR] [\fB-q \fIkey\fR] [\fIfile_type\fR:]\fIfile_name\fR ... .SH DESCRIPTION @@ -81,6 +81,11 @@ parsing with \fB-m\fR. With this, the \fB-b\fR option generates no body-style lookup keys for attachment MIME headers and for attached message/* headers. .sp +NOTE: with "smtputf8_enable = yes", the \fB-b\fR option +option disables UTF-8 syntax checks on query keys and +lookup results. Specify the \fB-U\fR option to force UTF-8 +syntax checks anyway. +.sp This feature is available in Postfix version 2.6 and later. .IP "\fB-c \fIconfig_dir\fR" Read the \fBmain.cf\fR configuration file in the named directory @@ -114,6 +119,11 @@ parsing with \fB-m\fR. With this, the \fB-h\fR option also generates header-style lookup keys for attachment MIME headers and for attached message/* headers. .sp +NOTE: with "smtputf8_enable = yes", the \fB-b\fR option +option disables UTF-8 syntax checks on query keys and +lookup results. Specify the \fB-U\fR option to force UTF-8 +syntax checks anyway. +.sp This feature is available in Postfix version 2.6 and later. .IP \fB-i\fR Incremental mode. Read entries from standard input and do not @@ -161,6 +171,13 @@ as the original input order. .sp This feature is available in Postfix version 2.2 and later, and is not available for all database types. +.IP \fB-u\fR +Disable UTF-8 support. UTF-8 support is enabled by default +when "smtputf8_enable = yes". It requires that keys and +values are valid UTF-8 strings. +.IP \fB-U\fR +With "smtputf8_enable = yes", force UTF-8 syntax checks +with the \fB-b\fR and \fB-h\fR options. .IP \fB-v\fR Enable verbose logging for debugging purposes. Multiple \fB-v\fR options make the software increasingly verbose. @@ -245,6 +262,9 @@ configuration files. .IP "\fBdefault_database_type (see 'postconf -d' output)\fR" The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1) and \fBpostmap\fR(1) commands. +.IP "\fBsmtputf8_enable (yes)\fR" +Enable experimental SMTPUTF8 support for the protocols described +in RFC 6531..6533. .IP "\fBsyslog_facility (mail)\fR" The syslog facility of Postfix logging. .IP "\fBsyslog_name (see 'postconf -d' output)\fR" diff --git a/postfix/man/man5/postconf.5 b/postfix/man/man5/postconf.5 index f3cd3947d..4c34f1725 100644 --- a/postfix/man/man5/postconf.5 +++ b/postfix/man/man5/postconf.5 @@ -6113,8 +6113,8 @@ transport_maps to apply this feature selectively: .na .ft C /etc/postfix/transport: - smtp-domain_that_verifies_after_data smtp-data-target: - lmtp-domain_that_verifies_after_data lmtp-data-target: + smtp-domain-that-verifies-after-data smtp-data-target: + lmtp-domain-that-verifies-after-data lmtp-data-target: .fi .ad .ft R diff --git a/postfix/proto/postconf.proto b/postfix/proto/postconf.proto index d3783b48d..6fa78ff15 100644 --- a/postfix/proto/postconf.proto +++ b/postfix/proto/postconf.proto @@ -15437,8 +15437,8 @@ transport_maps to apply this feature selectively: 

 /etc/postfix/transport:
-    smtp-domain_that_verifies_after_data    smtp-data-target:
-    lmtp-domain_that_verifies_after_data    lmtp-data-target:
+    smtp-domain-that-verifies-after-data    smtp-data-target:
+    lmtp-domain-that-verifies-after-data    lmtp-data-target:
diff --git a/postfix/src/bounce/Makefile.in b/postfix/src/bounce/Makefile.in index 7eb5fc3d6..ef57d240b 100644 --- a/postfix/src/bounce/Makefile.in +++ b/postfix/src/bounce/Makefile.in @@ -326,7 +326,7 @@ bounce_template.o: ../../include/mac_parse.h bounce_template.o: ../../include/mail_conf.h bounce_template.o: ../../include/mail_params.h bounce_template.o: ../../include/mail_proto.h -bounce_template.o: ../../include/midna.h +bounce_template.o: ../../include/midna_domain.h bounce_template.o: ../../include/msg.h bounce_template.o: ../../include/mymalloc.h bounce_template.o: ../../include/nvtable.h diff --git a/postfix/src/bounce/bounce_template.c b/postfix/src/bounce/bounce_template.c index ca8c8c86f..39fef8fda 100644 --- a/postfix/src/bounce/bounce_template.c +++ b/postfix/src/bounce/bounce_template.c @@ -118,7 +118,7 @@ #include #include #ifndef NO_EAI -#include +#include #endif /* Global library. */ @@ -462,7 +462,7 @@ static const char *bounce_template_lookup(const char *key, int unused_mode, "non-ASCII input value: \"%s\"", tp->origin, key, asc_val); return (asc_val); - } else if ((utf8_val = midna_to_utf8(asc_val)) == 0) { + } else if ((utf8_val = midna_domain_to_utf8(asc_val)) == 0) { msg_warn("%s: conversion \"%s\" failed: " "input value: \"%s\"", tp->origin, key, asc_val); diff --git a/postfix/src/global/Makefile.in b/postfix/src/global/Makefile.in index 2f96f2521..763fb89bd 100644 --- a/postfix/src/global/Makefile.in +++ b/postfix/src/global/Makefile.in @@ -33,7 +33,7 @@ SRCS = abounce.c anvil_clnt.c been_here.c bounce.c bounce_log.c \ smtp_reply_footer.c safe_ultostr.c verify_sender_addr.c \ dict_memcache.c mail_version.c memcache_proto.c server_acl.c \ mkmap_fail.c haproxy_srvr.c dsn_filter.c dynamicmaps.c uxtext.c \ - smtputf8.c mail_conf_over.c mail_parm_split.c + smtputf8.c mail_conf_over.c mail_parm_split.c midna_adomain.c OBJS = abounce.o anvil_clnt.o been_here.o bounce.o bounce_log.o \ canon_addr.o cfg_parser.o cleanup_strerror.o cleanup_strflags.o \ clnt_stream.o conv_time.o db_common.o debug_peer.o debug_process.o \ @@ -68,40 +68,41 @@ OBJS = abounce.o anvil_clnt.o been_here.o bounce.o bounce_log.o \ smtp_reply_footer.o safe_ultostr.o verify_sender_addr.o \ dict_memcache.o mail_version.o memcache_proto.o server_acl.o \ mkmap_fail.o haproxy_srvr.o dsn_filter.o dynamicmaps.o uxtext.o \ - smtputf8.o attr_override.o mail_parm_split.o $(NON_PLUGIN_MAP_OBJ) -# MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf. -# When hard-linking these maps, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ), -# otherwise it sets the PLUGIN_* macros. -MAP_OBJ = dict_ldap.o dict_mysql.o dict_pgsql.o dict_sqlite.o mkmap_cdb.o \ - mkmap_lmdb.o mkmap_sdbm.o -HDRS = abounce.h anvil_clnt.h been_here.h bounce.h bounce_log.h \ - canon_addr.h cfg_parser.h cleanup_user.h clnt_stream.h config.h \ - conv_time.h db_common.h debug_peer.h debug_process.h defer.h \ - deliver_completed.h deliver_flock.h deliver_pass.h deliver_request.h \ - dict_ldap.h dict_mysql.h dict_pgsql.h dict_proxy.h dict_sqlite.h domain_list.h \ - dot_lockfile.h dot_lockfile_as.h dsb_scan.h dsn.h dsn_buf.h \ - dsn_mask.h dsn_print.h dsn_util.h ehlo_mask.h ext_prop.h \ - file_id.h flush_clnt.h header_opts.h header_token.h input_transp.h \ - int_filt.h is_header.h lex_822.h log_adhoc.h mail_addr.h \ - mail_addr_crunch.h mail_addr_find.h mail_addr_map.h mail_conf.h \ - mail_copy.h mail_date.h mail_dict.h mail_error.h mail_flush.h \ - mail_open_ok.h mail_params.h mail_proto.h mail_queue.h mail_run.h \ - mail_scan_dir.h mail_stream.h mail_task.h mail_version.h maps.h \ - mark_corrupt.h match_parent_style.h mbox_conf.h mbox_open.h \ - mime_state.h mkmap.h msg_stats.h mynetworks.h mypwd.h namadr_list.h \ - off_cvt.h opened.h own_inet_addr.h pipe_command.h post_mail.h \ - qmgr_user.h qmqp_proto.h quote_821_local.h quote_822_local.h \ - quote_flags.h rcpt_buf.h rcpt_print.h rec_attr_map.h rec_streamlf.h \ - rec_type.h recipient_list.h record.h resolve_clnt.h resolve_local.h \ - rewrite_clnt.h scache.h sent.h smtp_stream.h split_addr.h \ - string_list.h strip_addr.h sys_exits.h timed_ipc.h tok822.h \ - trace.h user_acl.h valid_mailhost_addr.h verify.h verify_clnt.h \ - verp_sender.h wildcard_inet_addr.h xtext.h delivered_hdr.h \ - fold_addr.h header_body_checks.h data_redirect.h match_service.h \ - addr_match_list.h smtp_reply_footer.h safe_ultostr.h \ - verify_sender_addr.h dict_memcache.h memcache_proto.h server_acl.h \ - haproxy_srvr.h dsn_filter.h dynamicmaps.h uxtext.h smtputf8.h \ - attr_override.h mail_parm_split.h + smtputf8.o attr_override.o mail_parm_split.o midna_adomain.o \ + $(NON_PLUGIN_MAP_OBJ) + # MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf. + # When hard-linking these maps, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ), + # otherwise it sets the PLUGIN_* macros. + MAP_OBJ = dict_ldap.o dict_mysql.o dict_pgsql.o dict_sqlite.o mkmap_cdb.o \ + mkmap_lmdb.o mkmap_sdbm.o + HDRS = abounce.h anvil_clnt.h been_here.h bounce.h bounce_log.h \ + canon_addr.h cfg_parser.h cleanup_user.h clnt_stream.h config.h \ + conv_time.h db_common.h debug_peer.h debug_process.h defer.h \ + deliver_completed.h deliver_flock.h deliver_pass.h deliver_request.h \ + dict_ldap.h dict_mysql.h dict_pgsql.h dict_proxy.h dict_sqlite.h domain_list.h \ + dot_lockfile.h dot_lockfile_as.h dsb_scan.h dsn.h dsn_buf.h \ + dsn_mask.h dsn_print.h dsn_util.h ehlo_mask.h ext_prop.h \ + file_id.h flush_clnt.h header_opts.h header_token.h input_transp.h \ + int_filt.h is_header.h lex_822.h log_adhoc.h mail_addr.h \ + mail_addr_crunch.h mail_addr_find.h mail_addr_map.h mail_conf.h \ + mail_copy.h mail_date.h mail_dict.h mail_error.h mail_flush.h \ + mail_open_ok.h mail_params.h mail_proto.h mail_queue.h mail_run.h \ + mail_scan_dir.h mail_stream.h mail_task.h mail_version.h maps.h \ + mark_corrupt.h match_parent_style.h mbox_conf.h mbox_open.h \ + mime_state.h mkmap.h msg_stats.h mynetworks.h mypwd.h namadr_list.h \ + off_cvt.h opened.h own_inet_addr.h pipe_command.h post_mail.h \ + qmgr_user.h qmqp_proto.h quote_821_local.h quote_822_local.h \ + quote_flags.h rcpt_buf.h rcpt_print.h rec_attr_map.h rec_streamlf.h \ + rec_type.h recipient_list.h record.h resolve_clnt.h resolve_local.h \ + rewrite_clnt.h scache.h sent.h smtp_stream.h split_addr.h \ + string_list.h strip_addr.h sys_exits.h timed_ipc.h tok822.h \ + trace.h user_acl.h valid_mailhost_addr.h verify.h verify_clnt.h \ + verp_sender.h wildcard_inet_addr.h xtext.h delivered_hdr.h \ + fold_addr.h header_body_checks.h data_redirect.h match_service.h \ + addr_match_list.h smtp_reply_footer.h safe_ultostr.h \ + verify_sender_addr.h dict_memcache.h memcache_proto.h server_acl.h \ + haproxy_srvr.h dsn_filter.h dynamicmaps.h uxtext.h smtputf8.h \ + attr_override.h mail_parm_split.h midna_adomain.h TESTSRC = rec2stream.c stream2rec.c recdump.c DEFS = -I. -I$(INC_DIR) -D$(SYSTYPE) CFLAGS = $(DEBUG) $(OPT) $(DEFS) @@ -1895,6 +1896,14 @@ memcache_proto.o: ../../include/vstring.h memcache_proto.o: ../../include/vstring_vstream.h memcache_proto.o: memcache_proto.c memcache_proto.o: memcache_proto.h +midna_adomain.o: ../../include/check_arg.h +midna_adomain.o: ../../include/midna_domain.h +midna_adomain.o: ../../include/stringops.h +midna_adomain.o: ../../include/sys_defs.h +midna_adomain.o: ../../include/vbuf.h +midna_adomain.o: ../../include/vstring.h +midna_adomain.o: midna_adomain.c +midna_adomain.o: midna_adomain.h mime_state.o: ../../include/check_arg.h mime_state.o: ../../include/msg.h mime_state.o: ../../include/mymalloc.h diff --git a/postfix/src/global/dict_ldap.c b/postfix/src/global/dict_ldap.c index aa8ad6b57..02db79894 100644 --- a/postfix/src/global/dict_ldap.c +++ b/postfix/src/global/dict_ldap.c @@ -1340,7 +1340,8 @@ static const char *dict_ldap_lookup(DICT *dict, const char *name) /* * Don't frustrate future attempts to make Postfix UTF-8 transparent. */ - if (!valid_utf8_string(name, strlen(name))) { + if (DICT_IS_ENABLE_UTF8(dict->flags) == 0 + && !valid_utf8_string(name, strlen(name))) { if (msg_verbose) msg_info("%s: %s: Skipping lookup of non-UTF-8 key '%s'", myname, dict_ldap->parser->name, name); @@ -1351,10 +1352,10 @@ static const char *dict_ldap_lookup(DICT *dict, const char *name) * Optionally fold the key. */ if (dict->flags & DICT_FLAG_FOLD_FIX) { - if (dict->fold_buf == 0) - dict->fold_buf = vstring_alloc(10); - vstring_strcpy(dict->fold_buf, name); - name = lowercase(vstring_str(dict->fold_buf)); + if (dict->fold_buf == 0) + dict->fold_buf = vstring_alloc(10); + vstring_strcpy(dict->fold_buf, name); + name = lowercase(vstring_str(dict->fold_buf)); } /* diff --git a/postfix/src/global/dict_proxy.c b/postfix/src/global/dict_proxy.c index 29410d785..0fafdb4bb 100644 --- a/postfix/src/global/dict_proxy.c +++ b/postfix/src/global/dict_proxy.c @@ -156,6 +156,9 @@ static int dict_proxy_sequence(DICT *dict, int function, case PROXY_STAT_RETRY: *key = *value = 0; DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR); + case PROXY_STAT_CONFIG: + *key = *value = 0; + DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR); default: msg_warn("%s sequence failed for table \"%s\" function %d: " "unexpected reply status %d", @@ -226,6 +229,8 @@ static const char *dict_proxy_lookup(DICT *dict, const char *key) DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, (char *) 0); case PROXY_STAT_RETRY: DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, (char *) 0); + case PROXY_STAT_CONFIG: + DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, (char *) 0); default: msg_warn("%s lookup failed for table \"%s\" key \"%s\": " "unexpected reply status %d", @@ -293,6 +298,8 @@ static int dict_proxy_update(DICT *dict, const char *key, const char *value) DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL); case PROXY_STAT_RETRY: DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR); + case PROXY_STAT_CONFIG: + DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR); default: msg_warn("%s update failed for table \"%s\" key \"%s\": " "unexpected reply status %d", @@ -360,6 +367,8 @@ static int dict_proxy_delete(DICT *dict, const char *key) DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL); case PROXY_STAT_RETRY: DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR); + case PROXY_STAT_CONFIG: + DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR); default: msg_warn("%s delete failed for table \"%s\" key \"%s\": " "unexpected reply status %d", diff --git a/postfix/src/global/dict_proxy.h b/postfix/src/global/dict_proxy.h index 80dd6e352..a5b79243a 100644 --- a/postfix/src/global/dict_proxy.h +++ b/postfix/src/global/dict_proxy.h @@ -37,6 +37,7 @@ extern DICT *dict_proxy_open(const char *, int, int); #define PROXY_STAT_RETRY 2 /* try lookup again later */ #define PROXY_STAT_BAD 3 /* invalid request parameter */ #define PROXY_STAT_DENY 4 /* table not approved for proxying */ +#define PROXY_STAT_CONFIG 5 /* DICT_ERR_CONFIG error */ /* LICENSE /* .ad diff --git a/postfix/src/global/dict_sqlite.c b/postfix/src/global/dict_sqlite.c index 3570f337c..92ef7d992 100644 --- a/postfix/src/global/dict_sqlite.c +++ b/postfix/src/global/dict_sqlite.c @@ -165,7 +165,8 @@ static const char *dict_sqlite_lookup(DICT *dict, const char *name) /* * Don't frustrate future attempts to make Postfix UTF-8 transparent. */ - if (!valid_utf8_string(name, strlen(name))) { + if (DICT_IS_ENABLE_UTF8(dict->flags) == 0 + && !valid_utf8_string(name, strlen(name))) { if (msg_verbose) msg_info("%s: %s: Skipping lookup of non-UTF-8 key '%s'", myname, dict_sqlite->parser->name, name); diff --git a/postfix/src/global/mail_params.c b/postfix/src/global/mail_params.c index 435bfd9bd..67b7a1130 100644 --- a/postfix/src/global/mail_params.c +++ b/postfix/src/global/mail_params.c @@ -842,7 +842,6 @@ void mail_params_init() dict_db_cache_size = var_db_read_buf; dict_lmdb_map_size = var_lmdb_map_size; inet_windowsize = var_inet_windowsize; - temp_utf8_kludge = var_smtputf8_enable; /* * Report run-time versus compile-time discrepancies. @@ -851,7 +850,9 @@ void mail_params_init() if (var_smtputf8_enable) msg_warn("%s is true, but EAI support is not compiled in", VAR_SMTPUTF8_ENABLE); + var_smtputf8_enable = 0; #endif + util_utf8_enable = var_smtputf8_enable; /* * Variables whose defaults are determined at runtime, after other diff --git a/postfix/src/global/midna_adomain.c b/postfix/src/global/midna_adomain.c new file mode 100644 index 000000000..81c98d453 --- /dev/null +++ b/postfix/src/global/midna_adomain.c @@ -0,0 +1,119 @@ +/*++ +/* NAME +/* midna_adomain 3 +/* SUMMARY +/* address domain part conversion +/* SYNOPSIS +/* #include +/* +/* char *midna_adomain_to_ascii( +/* VSTRING *dest, +/* const char *name) +/* +/* char *midna_adomain_to_utf8( +/* VSTRING *dest, +/* const char *name) +/* DESCRIPTION +/* The functions in this module transform the domain portion +/* of an email address between ASCII and UTF-8 form. Both +/* functions tolerate a missing domain, and both functions +/* return a copy of the input when the domain portion requires +/* no conversion. +/* +/* midna_adomain_to_ascii() converts an UTF-8 or ASCII domain +/* portion to ASCII. The result is a null pointer when +/* conversion fails. This function verifies that the resulting +/* domain passes valid_hostname(). +/* +/* midna_adomain_to_utf8() converts an UTF-8 or ASCII domain +/* name to UTF-8. The result is a null pointer when conversion +/* fails. This function verifies that the resulting domain, +/* after conversion to ASCII, passes valid_hostname(). +/* SEE ALSO +/* midna_domain(3), Postfix ASCII/UTF-8 domain name conversion +/* DIAGNOSTICS +/* Fatal errors: memory allocation problem. +/* Warnings: conversion error or result validation error. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + + /* + * System library. + */ +#include +#include + +#ifndef NO_EAI +#include + + /* + * Utility library. + */ +#include +#include +#include + + /* + * Global library. + */ +#include + +#define STR(x) vstring_str(x) + +/* midna_adomain_to_utf8 - convert address domain portion to UTF8 */ + +char *midna_adomain_to_utf8(VSTRING *dest, const char *src) +{ + const char *cp; + const char *domain_utf8; + + if ((cp = strrchr(src, '@')) == 0) { + vstring_strcpy(dest, src); + } else { + vstring_sprintf(dest, "%*s@", (int) (cp - src), src); + if (*(cp += 1)) { + if (allascii(cp) && strstr(cp, "--") == 0) { + vstring_strcat(dest, cp); + } else if ((domain_utf8 = midna_domain_to_utf8(cp)) == 0) { + return (0); + } else { + vstring_strcat(dest, domain_utf8); + } + } + } + return (STR(dest)); +} + +/* midna_adomain_to_ascii - convert address domain portion to ASCII */ + +char *midna_adomain_to_ascii(VSTRING *dest, const char *src) +{ + const char *cp; + const char *domain_ascii; + + if ((cp = strrchr(src, '@')) == 0) { + vstring_strcpy(dest, src); + } else { + vstring_sprintf(dest, "%*s@", (int) (cp - src), src); + if (*(cp += 1)) { + if (allascii(cp)) { + vstring_strcat(dest, cp); + } else if ((domain_ascii = midna_domain_to_ascii(cp + 1)) == 0) { + return (0); + } else { + vstring_strcat(dest, domain_ascii); + } + } + } + return (STR(dest)); +} + +#endif /* NO_IDNA */ diff --git a/postfix/src/global/midna_adomain.h b/postfix/src/global/midna_adomain.h new file mode 100644 index 000000000..14f02fefa --- /dev/null +++ b/postfix/src/global/midna_adomain.h @@ -0,0 +1,36 @@ +#ifndef _MIDNA_ADOMAIN_H_INCLUDED_ +#define _MIDNA_ADOMAIN_H_INCLUDED_ + +/*++ +/* NAME +/* midna_adomain 3h +/* SUMMARY +/* domain name conversion +/* SYNOPSIS +/* #include +/* DESCRIPTION +/* .nf + + /* + * Utility library. + */ +#include + + /* + * External interface. + */ +extern char *midna_adomain_to_utf8(VSTRING *, const char *); +extern char *midna_adomain_to_ascii(VSTRING *, const char *); + +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +#endif diff --git a/postfix/src/global/mkmap_open.c b/postfix/src/global/mkmap_open.c index 9374f8b45..10806495e 100644 --- a/postfix/src/global/mkmap_open.c +++ b/postfix/src/global/mkmap_open.c @@ -295,6 +295,13 @@ MKMAP *mkmap_open(const char *type, const char *path, if (mkmap->after_open) mkmap->after_open(mkmap); + /* + * Wrap the dictionary for UTF-8 syntax checks and casefolding. + */ + if ((mkmap->dict->flags & DICT_FLAG_UTF8_PROXY) == 0 + && DICT_IS_ENABLE_UTF8(dict_flags)) + mkmap->dict = dict_utf8_encapsulate(mkmap->dict); + /* * Resume signal delivery if multi-writer safe. */ diff --git a/postfix/src/global/tok822_tree.c b/postfix/src/global/tok822_tree.c index 16cec946a..a66cf1156 100644 --- a/postfix/src/global/tok822_tree.c +++ b/postfix/src/global/tok822_tree.c @@ -209,6 +209,7 @@ TOK822 *tok822_sub_append(TOK822 *t1, TOK822 *t2) return (t1->tail = tok822_append(t1->tail, t2)); } else { t1->head = t2; + t2->owner = t1; while (t2->next) (t2 = t2->next)->owner = t1; return (t1->tail = t2); @@ -227,6 +228,7 @@ TOK822 *tok822_sub_prepend(TOK822 *t1, TOK822 *t2) return (tp); } else { t1->head = t2; + t2->owner = t1; while (t2->next) (t2 = t2->next)->owner = t1; return (t1->tail = t2); @@ -259,11 +261,12 @@ TOK822 *tok822_sub_keep_after(TOK822 *t1, TOK822 *t2) TOK822 *tok822_free_tree(TOK822 *tp) { - if (tp) { - if (tp->next) - tok822_free_tree(tp->next); + TOK822 *next; + + for (/* void */; tp != 0; tp = next) { if (tp->head) tok822_free_tree(tp->head); + next = tp->next; tok822_free(tp); } return (0); diff --git a/postfix/src/postalias/postalias.c b/postfix/src/postalias/postalias.c index 633bdf770..8285fcef9 100644 --- a/postfix/src/postalias/postalias.c +++ b/postfix/src/postalias/postalias.c @@ -5,7 +5,7 @@ /* Postfix alias database maintenance /* SYNOPSIS /* .fi -/* \fBpostalias\fR [\fB-Nfinoprsvw\fR] [\fB-c \fIconfig_dir\fR] +/* \fBpostalias\fR [\fB-Nfinoprsuvw\fR] [\fB-c \fIconfig_dir\fR] /* [\fB-d \fIkey\fR] [\fB-q \fIkey\fR] /* [\fIfile_type\fR:]\fIfile_name\fR ... /* DESCRIPTION @@ -93,6 +93,10 @@ /* as the original input order. /* This feature is available in Postfix version 2.2 and later, /* and is not available for all database types. +/* .IP \fB-u\fR +/* Disable UTF-8 support. UTF-8 support is enabled by default +/* when "smtputf8_enable = yes". It requires that keys and +/* values are valid UTF-8 strings. /* .IP \fB-v\fR /* Enable verbose logging for debugging purposes. Multiple \fB-v\fR /* options make the software increasingly verbose. @@ -176,6 +180,9 @@ /* .IP "\fBdefault_database_type (see 'postconf -d' output)\fR" /* The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1) /* and \fBpostmap\fR(1) commands. +/* .IP "\fBsmtputf8_enable (yes)\fR" +/* Enable experimental SMTPUTF8 support for the protocols described +/* in RFC 6531..6533. /* .IP "\fBsyslog_facility (mail)\fR" /* The syslog facility of Postfix logging. /* .IP "\fBsyslog_name (see 'postconf -d' output)\fR" @@ -249,6 +256,7 @@ /* Application-specific. */ #define STR vstring_str +#define LEN VSTRING_LEN #define POSTALIAS_FLAG_AS_OWNER (1<<0) /* open dest as owner of source */ #define POSTALIAS_FLAG_SAVE_PERM (1<<1) /* copy access permission @@ -309,7 +317,6 @@ static void postalias(char *map_type, char *path_name, int postalias_flags, && (st.st_uid != geteuid() || st.st_gid != getegid())) set_eugid(st.st_uid, st.st_gid); - /* * Open the database, create it when it does not exist, truncate it when * it does exist, and lock out any spectators. @@ -338,6 +345,17 @@ static void postalias(char *map_type, char *path_name, int postalias_flags, last_line = 0; while (readllines(line_buffer, source_fp, &last_line, &lineno)) { + /* + * First some UTF-8 checks sans casefolding. + */ + if (DICT_IS_ENABLE_UTF8(dict_flags) + && !allascii(STR(line_buffer)) + && !valid_utf8_string(STR(line_buffer), LEN(line_buffer))) { + msg_warn("%s, line %d: non-UTF-8 input \"%s\"", + VSTREAM_PATH(source_fp), lineno, STR(line_buffer)); + continue; + } + /* * Tokenize the input, so that we do the right thing when a * quoted localpart contains special characters such as "@", ":" @@ -655,7 +673,7 @@ static void postalias_seq(const char *map_type, const char *map_name, static NORETURN usage(char *myname) { - msg_fatal("usage: %s [-Nfinoprsvw] [-c config_dir] [-d key] [-q key] [map_type:]file...", + msg_fatal("usage: %s [-Nfinoprsuvw] [-c config_dir] [-d key] [-q key] [map_type:]file...", myname); } @@ -670,7 +688,8 @@ int main(int argc, char **argv) struct stat st; int postalias_flags = POSTALIAS_FLAG_AS_OWNER | POSTALIAS_FLAG_SAVE_PERM; int open_flags = O_RDWR | O_CREAT | O_TRUNC; - int dict_flags = DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX; + int dict_flags = (DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX + | DICT_FLAG_UTF8_ENABLE); char *query = 0; char *delkey = 0; int sequence = 0; @@ -720,7 +739,7 @@ int main(int argc, char **argv) /* * Parse JCL. */ - while ((ch = GETOPT(argc, argv, "Nc:d:finopq:rsvw")) > 0) { + while ((ch = GETOPT(argc, argv, "Nc:d:finopq:rsuvw")) > 0) { switch (ch) { default: usage(argv[0]); @@ -768,6 +787,9 @@ int main(int argc, char **argv) msg_fatal("specify only one of -s or -q or -d"); sequence = 1; break; + case 'u': + dict_flags &= ~DICT_FLAG_UTF8_ENABLE; + break; case 'v': msg_verbose++; break; diff --git a/postfix/src/postmap/postmap.c b/postfix/src/postmap/postmap.c index af3e5a440..3730ffaf7 100644 --- a/postfix/src/postmap/postmap.c +++ b/postfix/src/postmap/postmap.c @@ -5,7 +5,7 @@ /* Postfix lookup table management /* SYNOPSIS /* .fi -/* \fBpostmap\fR [\fB-Nbfhimnoprsvw\fR] [\fB-c \fIconfig_dir\fR] +/* \fBpostmap\fR [\fB-NbfhimnoprsuUvw\fR] [\fB-c \fIconfig_dir\fR] /* [\fB-d \fIkey\fR] [\fB-q \fIkey\fR] /* [\fIfile_type\fR:]\fIfile_name\fR ... /* DESCRIPTION @@ -71,6 +71,11 @@ /* generates no body-style lookup keys for attachment MIME /* headers and for attached message/* headers. /* .sp +/* NOTE: with "smtputf8_enable = yes", the \fB-b\fR option +/* option disables UTF-8 syntax checks on query keys and +/* lookup results. Specify the \fB-U\fR option to force UTF-8 +/* syntax checks anyway. +/* .sp /* This feature is available in Postfix version 2.6 and later. /* .IP "\fB-c \fIconfig_dir\fR" /* Read the \fBmain.cf\fR configuration file in the named directory @@ -104,6 +109,11 @@ /* generates header-style lookup keys for attachment MIME /* headers and for attached message/* headers. /* .sp +/* NOTE: with "smtputf8_enable = yes", the \fB-b\fR option +/* option disables UTF-8 syntax checks on query keys and +/* lookup results. Specify the \fB-U\fR option to force UTF-8 +/* syntax checks anyway. +/* .sp /* This feature is available in Postfix version 2.6 and later. /* .IP \fB-i\fR /* Incremental mode. Read entries from standard input and do not @@ -151,6 +161,13 @@ /* .sp /* This feature is available in Postfix version 2.2 and later, /* and is not available for all database types. +/* .IP \fB-u\fR +/* Disable UTF-8 support. UTF-8 support is enabled by default +/* when "smtputf8_enable = yes". It requires that keys and +/* values are valid UTF-8 strings. +/* .IP \fB-U\fR +/* With "smtputf8_enable = yes", force UTF-8 syntax checks +/* with the \fB-b\fR and \fB-h\fR options. /* .IP \fB-v\fR /* Enable verbose logging for debugging purposes. Multiple \fB-v\fR /* options make the software increasingly verbose. @@ -229,6 +246,9 @@ /* .IP "\fBdefault_database_type (see 'postconf -d' output)\fR" /* The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1) /* and \fBpostmap\fR(1) commands. +/* .IP "\fBsmtputf8_enable (yes)\fR" +/* Enable experimental SMTPUTF8 support for the protocols described +/* in RFC 6531..6533. /* .IP "\fBsyslog_facility (mail)\fR" /* The syslog facility of Postfix logging. /* .IP "\fBsyslog_name (see 'postconf -d' output)\fR" @@ -401,6 +421,17 @@ static void postmap(char *map_type, char *path_name, int postmap_flags, last_line = 0; while (readllines(line_buffer, source_fp, &last_line, &lineno)) { + /* + * First some UTF-8 checks sans casefolding. + */ + if (DICT_IS_ENABLE_UTF8(dict_flags) + && !allascii(STR(line_buffer)) + && !valid_utf8_string(STR(line_buffer), LEN(line_buffer))) { + msg_warn("%s, line %d: non-UTF-8 input \"%s\"", + VSTREAM_PATH(source_fp), lineno, STR(line_buffer)); + continue; + } + /* * Split on the first whitespace character, then trim leading and * trailing whitespace from key and value. @@ -769,7 +800,7 @@ static void postmap_seq(const char *map_type, const char *map_name, static NORETURN usage(char *myname) { - msg_fatal("usage: %s [-Nfinoprsvw] [-c config_dir] [-d key] [-q key] [map_type:]file...", + msg_fatal("usage: %s [-NfinoprsuUvw] [-c config_dir] [-d key] [-q key] [map_type:]file...", myname); } @@ -784,11 +815,13 @@ int main(int argc, char **argv) struct stat st; int postmap_flags = POSTMAP_FLAG_AS_OWNER | POSTMAP_FLAG_SAVE_PERM; int open_flags = O_RDWR | O_CREAT | O_TRUNC; - int dict_flags = DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX; + int dict_flags = (DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX + | DICT_FLAG_UTF8_ENABLE); char *query = 0; char *delkey = 0; int sequence = 0; int found; + int force_utf8 = 0; /* * Fingerprint executables and core dumps. @@ -834,7 +867,7 @@ int main(int argc, char **argv) /* * Parse JCL. */ - while ((ch = GETOPT(argc, argv, "Nbc:d:fhimnopq:rsvw")) > 0) { + while ((ch = GETOPT(argc, argv, "Nbc:d:fhimnopq:rsuUvw")) > 0) { switch (ch) { default: usage(argv[0]); @@ -891,6 +924,12 @@ int main(int argc, char **argv) msg_fatal("specify only one of -s or -q or -d"); sequence = 1; break; + case 'u': + dict_flags &= ~DICT_FLAG_UTF8_ENABLE; + break; + case 'U': + force_utf8 = 1; + break; case 'v': msg_verbose++; break; @@ -911,7 +950,10 @@ int main(int argc, char **argv) && (postmap_flags & POSTMAP_FLAG_ANY_KEY) == (postmap_flags & POSTMAP_FLAG_MIME_KEY)) msg_warn("ignoring -m option without -b or -h"); - + if ((postmap_flags & (POSTMAP_FLAG_ANY_KEY & ~POSTMAP_FLAG_MIME_KEY)) + && force_utf8 == 0) + dict_flags &= ~DICT_FLAG_UTF8_MASK; + /* * Use the map type specified by the user, or fall back to a default * database type. diff --git a/postfix/src/posttls-finger/Makefile.in b/postfix/src/posttls-finger/Makefile.in index 02b4261ab..e693e4795 100644 --- a/postfix/src/posttls-finger/Makefile.in +++ b/postfix/src/posttls-finger/Makefile.in @@ -74,7 +74,7 @@ posttls-finger.o: ../../include/iostuff.h posttls-finger.o: ../../include/mail_conf.h posttls-finger.o: ../../include/mail_params.h posttls-finger.o: ../../include/mail_server.h -posttls-finger.o: ../../include/midna.h +posttls-finger.o: ../../include/midna_domain.h posttls-finger.o: ../../include/msg.h posttls-finger.o: ../../include/msg_vstream.h posttls-finger.o: ../../include/myaddrinfo.h diff --git a/postfix/src/posttls-finger/posttls-finger.c b/postfix/src/posttls-finger/posttls-finger.c index ac4bc8770..8be46e40b 100644 --- a/postfix/src/posttls-finger/posttls-finger.c +++ b/postfix/src/posttls-finger/posttls-finger.c @@ -323,7 +323,7 @@ #include #include #include -#include +#include #define STR(x) vstring_str(x) @@ -1103,7 +1103,7 @@ static DNS_RR *domain_addr(STATE *state, char *domain) * IDNA support. */ #ifndef NO_EAI - if (!allascii(domain) && (aname = midna_to_ascii(domain)) != 0) { + if (!allascii(domain) && (aname = midna_domain_to_ascii(domain)) != 0) { msg_info("%s asciified to %s", domain, aname); } else #endif @@ -1168,7 +1168,7 @@ static DNS_RR *host_addr(STATE *state, const char *host) * IDNA support. */ #ifndef NO_EAI - if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) { + if (!allascii(host) && (ahost = midna_domain_to_ascii(host)) != 0) { msg_info("%s asciified to %s", host, ahost); } else #endif diff --git a/postfix/src/proxymap/proxymap.c b/postfix/src/proxymap/proxymap.c index 416f523cb..262aeeb14 100644 --- a/postfix/src/proxymap/proxymap.c +++ b/postfix/src/proxymap/proxymap.c @@ -382,7 +382,8 @@ static void proxymap_sequence_service(VSTREAM *client_stream) reply_status = PROXY_STAT_NOKEY; reply_key = reply_value = ""; } else { - reply_status = PROXY_STAT_RETRY; + reply_status = (dict->error == DICT_ERR_RETRY ? + PROXY_STAT_RETRY : PROXY_STAT_CONFIG); reply_key = reply_value = ""; } } @@ -427,7 +428,8 @@ static void proxymap_lookup_service(VSTREAM *client_stream) reply_status = PROXY_STAT_NOKEY; reply_value = ""; } else { - reply_status = PROXY_STAT_RETRY; + reply_status = (dict->error == DICT_ERR_RETRY ? + PROXY_STAT_RETRY : PROXY_STAT_CONFIG); reply_value = ""; } @@ -482,7 +484,8 @@ static void proxymap_update_service(VSTREAM *client_stream) } else if (dict->error == 0) { reply_status = PROXY_STAT_NOKEY; } else { - reply_status = PROXY_STAT_RETRY; + reply_status = (dict->error == DICT_ERR_RETRY ? + PROXY_STAT_RETRY : PROXY_STAT_CONFIG); } } @@ -532,7 +535,8 @@ static void proxymap_delete_service(VSTREAM *client_stream) } else if (dict->error == 0) { reply_status = PROXY_STAT_NOKEY; } else { - reply_status = PROXY_STAT_RETRY; + reply_status = (dict->error == DICT_ERR_RETRY ? + PROXY_STAT_RETRY : PROXY_STAT_CONFIG); } } diff --git a/postfix/src/smtp/Makefile.in b/postfix/src/smtp/Makefile.in index bcbd2f2fa..7edc54ef7 100644 --- a/postfix/src/smtp/Makefile.in +++ b/postfix/src/smtp/Makefile.in @@ -154,7 +154,7 @@ smtp_addr.o: ../../include/inet_proto.h smtp_addr.o: ../../include/mail_params.h smtp_addr.o: ../../include/maps.h smtp_addr.o: ../../include/match_list.h -smtp_addr.o: ../../include/midna.h +smtp_addr.o: ../../include/midna_domain.h smtp_addr.o: ../../include/mime_state.h smtp_addr.o: ../../include/msg.h smtp_addr.o: ../../include/msg_stats.h @@ -380,12 +380,14 @@ smtp_proto.o: ../../include/mail_queue.h smtp_proto.o: ../../include/maps.h smtp_proto.o: ../../include/mark_corrupt.h smtp_proto.o: ../../include/match_list.h +smtp_proto.o: ../../include/match_parent_style.h smtp_proto.o: ../../include/mime_state.h smtp_proto.o: ../../include/msg.h smtp_proto.o: ../../include/msg_stats.h smtp_proto.o: ../../include/myaddrinfo.h smtp_proto.o: ../../include/myflock.h smtp_proto.o: ../../include/mymalloc.h +smtp_proto.o: ../../include/namadr_list.h smtp_proto.o: ../../include/name_code.h smtp_proto.o: ../../include/name_mask.h smtp_proto.o: ../../include/nvtable.h diff --git a/postfix/src/smtp/smtp_addr.c b/postfix/src/smtp/smtp_addr.c index 7ac4607f4..d02b25319 100644 --- a/postfix/src/smtp/smtp_addr.c +++ b/postfix/src/smtp/smtp_addr.c @@ -85,7 +85,7 @@ #include #include #include -#include +#include /* Global library. */ @@ -378,7 +378,7 @@ DNS_RR *smtp_domain_addr(const char *name, DNS_RR **mxrr, int misc_flags, * IDNA support. */ #ifndef NO_EAI - if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) { + if (!allascii(name) && (aname = midna_domain_to_ascii(name)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", name, aname); } else @@ -524,7 +524,7 @@ DNS_RR *smtp_host_addr(const char *host, int misc_flags, DSN_BUF *why) * IDNA support. */ #ifndef NO_EAI - if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) { + if (!allascii(host) && (ahost = midna_domain_to_ascii(host)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", host, ahost); } else diff --git a/postfix/src/smtpd/Makefile.in b/postfix/src/smtpd/Makefile.in index 13c6e6f09..4aee827e2 100644 --- a/postfix/src/smtpd/Makefile.in +++ b/postfix/src/smtpd/Makefile.in @@ -329,7 +329,7 @@ smtpd_check.o: ../../include/mail_stream.h smtpd_check.o: ../../include/maps.h smtpd_check.o: ../../include/match_list.h smtpd_check.o: ../../include/match_parent_style.h -smtpd_check.o: ../../include/midna.h +smtpd_check.o: ../../include/midna_domain.h smtpd_check.o: ../../include/milter.h smtpd_check.o: ../../include/msg.h smtpd_check.o: ../../include/msg_stats.h diff --git a/postfix/src/smtpd/smtpd.c b/postfix/src/smtpd/smtpd.c index 35aae24b6..d9312f9dd 100644 --- a/postfix/src/smtpd/smtpd.c +++ b/postfix/src/smtpd/smtpd.c @@ -3653,7 +3653,8 @@ static int etrn_cmd(SMTPD_STATE *state, int argc, SMTPD_TOKEN *argv) * As an extension to RFC 1985 we also allow an RFC 2821 address literal * enclosed in []. * - * XXX EAI: Convert to ASCII and use that form internally. + * XXX There does not appear to be an ETRN parameter to indicate that the + * domain name is UTF-8. */ if (!valid_hostname(argv[1].strval, DONT_GRIPE) && !valid_mailhost_literal(argv[1].strval, DONT_GRIPE)) { @@ -4948,6 +4949,14 @@ static void smtpd_proto(SMTPD_STATE *state) } watchdog_pat(); smtpd_chat_query(state); + /* Safety: protect internal interfaces against malformed UTF-8. */ + if (var_smtputf8_enable && valid_utf8_string(STR(state->buffer), + LEN(state->buffer)) == 0) { + state->error_mask |= MAIL_ERROR_PROTOCOL; + smtpd_chat_reply(state, "500 5.5.2 Error: bad UTF-8 syntax"); + state->error_count++; + continue; + } /* Move into smtpd_chat_query() and update session transcript. */ if (smtpd_cmd_filter != 0) { for (cp = STR(state->buffer); *cp && IS_SPACE_TAB(*cp); cp++) diff --git a/postfix/src/smtpd/smtpd_check.c b/postfix/src/smtpd/smtpd_check.c index fd114c0d1..ea77741d3 100644 --- a/postfix/src/smtpd/smtpd_check.c +++ b/postfix/src/smtpd/smtpd_check.c @@ -206,7 +206,7 @@ #include #include #include -#include +#include #include /* DNS library. */ @@ -1113,15 +1113,33 @@ static const char *check_mail_addr_find(SMTPD_STATE *state, char **ext) { const char *result; - + if ((result = mail_addr_find(maps, key, ext)) != 0 || maps->error == 0) return (result); if (maps->error == DICT_ERR_RETRY) + /* Warning is already logged. */ reject_dict_retry(state, reply_name); else reject_server_error(state); } +/* check_dict_get - reject with temporary failure if dict lookup fails */ + +static const char *check_dict_get(SMTPD_STATE *state, const char *table, + const char *reply_name, + DICT *dict, const char *key) +{ + const char *result; + + if ((result = dict_get(dict, key)) != 0 || dict->error == 0) + return (result); + if (dict->error == DICT_ERR_RETRY) { + msg_warn("%s: table lookup problem", table); + reject_dict_retry(state, reply_name); + } else + reject_server_error(state); +} + /* reject_unknown_reverse_name - fail if reverse client hostname is unknown */ static int reject_unknown_reverse_name(SMTPD_STATE *state) @@ -1417,7 +1435,7 @@ static int reject_unknown_mailhost(SMTPD_STATE *state, const char *name, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) { + if (!allascii(name) && (aname = midna_domain_to_ascii(name)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", name, aname); name = aname; @@ -1916,7 +1934,7 @@ static int permit_mx_backup(SMTPD_STATE *state, const char *recipient, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; @@ -2661,23 +2679,13 @@ static int check_access(SMTPD_STATE *state, const char *table, const char *name, if ((dict = dict_handle(table)) == 0) { msg_warn("%s: unexpected dictionary: %s", myname, table); - value = "451 4.3.5 Server configuration error"; - CHK_ACCESS_RETURN(check_table_result(state, table, value, name, - reply_name, reply_class, - def_acl), FOUND); + reject_server_error(state); } if (flags == 0 || (flags & dict->flags) != 0) { - if ((value = dict_get(dict, name)) != 0) + if ((value = check_dict_get(state, table, reply_name, dict, name)) != 0) CHK_ACCESS_RETURN(check_table_result(state, table, value, name, reply_name, reply_class, def_acl), FOUND); - if (dict->error != 0) { - msg_warn("%s: table lookup problem", table); - value = "451 4.3.5 Server configuration error"; - CHK_ACCESS_RETURN(check_table_result(state, table, value, name, - reply_name, reply_class, - def_acl), FOUND); - } } CHK_ACCESS_RETURN(SMTPD_CHECK_DUNNO, MISSED); } @@ -2711,24 +2719,15 @@ static int check_domain_access(SMTPD_STATE *state, const char *table, if ((dict = dict_handle(table)) == 0) { msg_warn("%s: unexpected dictionary: %s", myname, table); - value = "451 4.3.5 Server configuration error"; - CHK_DOMAIN_RETURN(check_table_result(state, table, value, - domain, reply_name, reply_class, - def_acl), FOUND); + reject_server_error(state); } for (name = domain; *name != 0; name = next) { if (flags == 0 || (flags & dict->flags) != 0) { - if ((value = dict_get(dict, name)) != 0) - CHK_DOMAIN_RETURN(check_table_result(state, table, value, - domain, reply_name, reply_class, - def_acl), FOUND); - if (dict->error != 0) { - msg_warn("%s: table lookup problem", table); - value = "451 4.3.5 Server configuration error"; + if ((value = check_dict_get(state, table, reply_name, + dict, name)) != 0) CHK_DOMAIN_RETURN(check_table_result(state, table, value, domain, reply_name, reply_class, def_acl), FOUND); - } } /* Don't apply subdomain magic to numerical hostnames. */ if (maybe_numerical @@ -2775,24 +2774,15 @@ static int check_addr_access(SMTPD_STATE *state, const char *table, if ((dict = dict_handle(table)) == 0) { msg_warn("%s: unexpected dictionary: %s", myname, table); - value = "451 4.3.5 Server configuration error"; - CHK_ADDR_RETURN(check_table_result(state, table, value, address, - reply_name, reply_class, - def_acl), FOUND); + reject_server_error(state); } do { if (flags == 0 || (flags & dict->flags) != 0) { - if ((value = dict_get(dict, addr)) != 0) + if ((value = check_dict_get(state, table, reply_name, + dict, addr)) != 0) CHK_ADDR_RETURN(check_table_result(state, table, value, address, reply_name, reply_class, def_acl), FOUND); - if (dict->error != 0) { - msg_warn("%s: table lookup problem", table); - value = "451 4.3.5 Server configuration error"; - CHK_ADDR_RETURN(check_table_result(state, table, value, address, - reply_name, reply_class, - def_acl), FOUND); - } } flags = PARTIAL; } while (split_at_right(addr, delim)); @@ -2914,7 +2904,7 @@ static int check_server_access(SMTPD_STATE *state, const char *table, * Fix 20140924: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; @@ -3634,7 +3624,7 @@ static const SMTPD_RBL_STATE *find_dnsxl_domain(SMTPD_STATE *state, * Fix 20140706: convert domain to ASCII. */ #ifndef NO_EAI - if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) { + if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, adomain); domain = adomain; @@ -3816,6 +3806,18 @@ static int reject_unauth_sender_login_mismatch(SMTPD_STATE *state, const char *s #endif +/* valid_utf8_action - validate UTF-8 policy server response */ + +static int valid_utf8_action(const char *server, const char *action) +{ + int retval; + + if ((retval = valid_utf8_string(action, strlen(action))) == 0) + msg_warn("malformed UTF-8 in policy server %s response: \"%s\"", + server, action); + return (retval); +} + /* check_policy_service - check delegated policy service */ static int check_policy_service(SMTPD_STATE *state, const char *server, @@ -3926,7 +3928,8 @@ static int check_policy_service(SMTPD_STATE *state, const char *server, ATTR_TYPE_END, ATTR_FLAG_MISSING, /* Reply attributes. */ RECV_ATTR_STR(MAIL_ATTR_ACTION, action), - ATTR_TYPE_END) != 1) { + ATTR_TYPE_END) != 1 + || (var_smtputf8_enable && valid_utf8_action(server, STR(action)) == 0)) { NOCLOBBER static int nesting_level = 0; jmp_buf savebuf; int status; diff --git a/postfix/src/tls/Makefile.in b/postfix/src/tls/Makefile.in index cddb4915b..1a4896053 100644 --- a/postfix/src/tls/Makefile.in +++ b/postfix/src/tls/Makefile.in @@ -140,7 +140,7 @@ tls_client.o: ../../include/dict.h tls_client.o: ../../include/dns.h tls_client.o: ../../include/iostuff.h tls_client.o: ../../include/mail_params.h -tls_client.o: ../../include/midna.h +tls_client.o: ../../include/midna_domain.h tls_client.o: ../../include/msg.h tls_client.o: ../../include/myaddrinfo.h tls_client.o: ../../include/myflock.h diff --git a/postfix/src/tls/tls_client.c b/postfix/src/tls/tls_client.c index f30f1e3ae..8211e2602 100644 --- a/postfix/src/tls/tls_client.c +++ b/postfix/src/tls/tls_client.c @@ -140,7 +140,7 @@ #include #include #include /* non-blocking */ -#include +#include /* Global library. */ @@ -535,7 +535,7 @@ static int match_servername(const char *certid, */ if (!allascii(certid)) return (0); - if (!allascii(nexthop) && (aname = midna_to_ascii(nexthop)) != 0) { + if (!allascii(nexthop) && (aname = midna_domain_to_ascii(nexthop)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", nexthop, aname); nexthop = aname; @@ -565,13 +565,19 @@ static int match_servername(const char *certid, #ifndef NO_EAI /* - * IDNA allows labels to be separated by any of the additional - * characters U+3002, U+FF0E, and U+FF61; that are Unicode - * variants. Their UTF-8 encodings are: E38082, EFBC8E and - * EFBDA1. + * Besides U+002E (full stop) IDNA2003 allows labels to be + * separated by any of the Unicode variants U+3002 (ideographic + * full stop), U+FF0E (fullwidth full stop), and U+FF61 + * (halfwidth ideographic full stop). Their respective UTF-8 + * encodings are: E38082, EFBC8E and EFBDA1. * - * It is not clear whether the IDNA to_ASCII conversion allows empty - * leading labels, so we handle these explicitly here. + * IDNA2008 does not permit (upper) case and other variant + * differences in U-labels. The midna_domain_to_ascii() function, + * based on UTS46, midna_domain_to_ascii() normalizes the + * differences away. + * + * The IDNA to_ASCII conversion does not allow empty leading labels, + * so we handle these explicitly here. */ else { unsigned char *cp = (unsigned char *) domain; @@ -586,7 +592,7 @@ static int match_servername(const char *certid, } } if (!allascii(domain) - && (aname = midna_to_ascii(domain)) != 0) { + && (aname = midna_domain_to_ascii(domain)) != 0) { if (msg_verbose) msg_info("%s asciified to %s", domain, aname); domain = aname; diff --git a/postfix/src/trivial-rewrite/resolve.c b/postfix/src/trivial-rewrite/resolve.c index 8ffb196ef..7d7b0b957 100644 --- a/postfix/src/trivial-rewrite/resolve.c +++ b/postfix/src/trivial-rewrite/resolve.c @@ -128,6 +128,7 @@ */ #define STR vstring_str +#define LEN VSTRING_LEN /* * Some of the lists that define the address domain classes. @@ -414,11 +415,14 @@ static void resolve_addr(RES_CONTEXT *rp, char *sender, char *addr, */ tok822_internalize(nextrcpt, tree, TOK822_STR_DEFL); rcpt_domain = strrchr(STR(nextrcpt), '@') + 1; - if (rcpt_domain == 0) + if (rcpt_domain == (char *) 1) msg_panic("no @ in address: \"%s\"", STR(nextrcpt)); if (*rcpt_domain == '[') { if (!valid_mailhost_literal(rcpt_domain, DONT_GRIPE)) *flags |= RESOLVE_FLAG_ERROR; + } else if (var_smtputf8_enable + && valid_utf8_string(STR(nextrcpt), LEN(nextrcpt)) == 0) { + *flags |= RESOLVE_FLAG_ERROR; } else if (!valid_utf8_hostname(var_smtputf8_enable, rcpt_domain, DONT_GRIPE)) { if (var_resolve_num_dom && valid_hostaddr(rcpt_domain, DONT_GRIPE)) { diff --git a/postfix/src/util/Makefile.in b/postfix/src/util/Makefile.in index b634d30ad..2090191bb 100644 --- a/postfix/src/util/Makefile.in +++ b/postfix/src/util/Makefile.in @@ -38,8 +38,8 @@ SRCS = alldig.c allprint.c argv.c argv_split.c attr_clnt.c attr_print0.c \ dict_fail.c msg_rate_delay.c dict_surrogate.c warn_stat.c \ dict_sockmap.c line_number.c recv_pass_attr.c pass_accept.c \ poll_fd.c timecmp.c slmdb.c dict_pipe.c dict_random.c \ - valid_utf8_hostname.c midna.c argv_splitq.c balpar.c dict_union.c \ - extpar.c dict_inline.c + valid_utf8_hostname.c midna_domain.c argv_splitq.c balpar.c dict_union.c \ + extpar.c dict_inline.c casefold.c dict_utf8.c OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ attr_print64.o attr_print_plain.o attr_scan0.o attr_scan64.o \ attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \ @@ -79,8 +79,8 @@ OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \ dict_fail.o msg_rate_delay.o dict_surrogate.o warn_stat.o \ dict_sockmap.o line_number.o recv_pass_attr.o pass_accept.o \ poll_fd.o timecmp.o $(NON_PLUGIN_MAP_OBJ) dict_pipe.o dict_random.o \ - valid_utf8_hostname.o midna.o argv_splitq.o balpar.o dict_union.o \ - extpar.o dict_inline.o + valid_utf8_hostname.o midna_domain.o argv_splitq.o balpar.o dict_union.o \ + extpar.o dict_inline.o casefold.o dict_utf8.o # MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf. # When hard-linking these, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ), # otherwise it sets the PLUGIN_* macros. @@ -109,7 +109,7 @@ HDRS = argv.h attr.h attr_clnt.h auto_clnt.h base64_code.h binhash.h \ edit_file.h dict_cache.h dict_thash.h ip_match.h nbbio.h base32_code.h \ dict_fail.h warn_stat.h dict_sockmap.h line_number.h timecmp.h \ slmdb.h compat_va_copy.h dict_pipe.h dict_random.h \ - valid_utf8_hostname.h midna.h dict_union.h dict_inline.h check_arg.h + valid_utf8_hostname.h midna_domain.h dict_union.h dict_inline.h check_arg.h TESTSRC = fifo_open.c fifo_rdwr_bug.c fifo_rdonly_bug.c select_bug.c \ stream_test.c dup2_pass_on_exec.c DEFS = -I. -D$(SYSTYPE) @@ -128,7 +128,7 @@ TESTPROG= dict_open dup2_pass_on_exec events exec_command fifo_open \ unix_recv_fd unix_send_fd stream_recv_fd stream_send_fd hex_code \ myaddrinfo myaddrinfo4 inet_proto sane_basename format_tv \ valid_utf8_string ip_match base32_code msg_rate_delay netstring \ - vstream timecmp dict_cache midna + vstream timecmp dict_cache midna_domain casefold PLUGIN_MAP_SO = $(LIB_PREFIX)pcre$(LIB_SUFFIX) LIB_DIR = ../../lib @@ -504,7 +504,12 @@ dict_cache: $(LIB) $(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS) mv junk $@.o -midna: $(LIB) +midna_domain: $(LIB) + mv $@.o junk + $(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS) + mv junk $@.o + +casefold: $(LIB) mv $@.o junk $(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS) mv junk $@.o @@ -515,7 +520,8 @@ tests: all valid_hostname_test mac_expand_test dict_test unescape_test \ dict_cidr_test attr_scan_plain_test htable_test hex_code_test \ myaddrinfo_test format_tv_test ip_match_test name_mask_tests \ base32_code_test dict_thash_test surrogate_test timecmp_test \ - dict_static_test dict_inline_test midna_test + dict_static_test dict_inline_test midna_domain_test casefold_test \ + dict_utf8_test root_tests: @@ -705,8 +711,8 @@ base32_code_test: base32_code $(SHLIB_ENV) ./base32_code dict_thash_test: ../postmap/postmap dict_thash.map - $(SHLIB_ENV) ../postmap/postmap -s texthash:dict_thash.map >dict_thash.tmp 2>&1 - sort dict_thash.tmp | diff -b dict_thash.map - + $(SHLIB_ENV) ../postmap/postmap -s texthash:dict_thash.map | sort >dict_thash.tmp 2>&1 + tr '[A-Z]' '[a-z]' dict_inline.tmp 2>&1 diff dict_inline.ref dict_inline.tmp rm -f dict_inline.tmp -midna_test: midna midna_test.in midna_test.ref - $(SHLIB_ENV) ./midna midna_test.tmp 2>&1 - diff midna_test.ref midna_test.tmp - rm -f midna_test.tmp +midna_domain_test: midna_domain midna_domain_test.in midna_domain_test.ref + $(SHLIB_ENV) ./midna_domain midna_domain_test.tmp 2>&1 + diff midna_domain_test.ref midna_domain_test.tmp + rm -f midna_domain_test.tmp + +casefold_test: casefold casefold_test.in casefold_test.ref + $(SHLIB_ENV) ./casefold casefold_test.tmp 2>&1 + diff casefold_test.ref casefold_test.tmp + rm -f casefold_test.tmp + +dict_utf8_test: dict_open dict_utf8_test.in dict_utf8_test.ref + $(SHLIB_ENV) sh dict_utf8_test.in >dict_utf8_test.tmp 2>&1 + diff dict_utf8_test.ref dict_utf8_test.tmp + rm -f dict_utf8_test.tmp depend: $(MAKES) (sed '1,/^# do not edit/!d' Makefile.in; \ @@ -945,6 +961,13 @@ binhash.o: binhash.h binhash.o: msg.h binhash.o: mymalloc.h binhash.o: sys_defs.h +casefold.o: casefold.c +casefold.o: check_arg.h +casefold.o: msg.h +casefold.o: stringops.h +casefold.o: sys_defs.h +casefold.o: vbuf.h +casefold.o: vstring.h chroot_uid.o: chroot_uid.c chroot_uid.o: chroot_uid.h chroot_uid.o: msg.h @@ -1132,6 +1155,7 @@ dict_ht.o: vstring.h dict_inline.o: argv.h dict_inline.o: check_arg.h dict_inline.o: dict.h +dict_inline.o: dict_ht.h dict_inline.o: dict_inline.c dict_inline.o: dict_inline.h dict_inline.o: htable.h @@ -1380,20 +1404,19 @@ dict_test.o: vstring_vstream.h dict_thash.o: argv.h dict_thash.o: check_arg.h dict_thash.o: dict.h +dict_thash.o: dict_ht.h dict_thash.o: dict_thash.c dict_thash.o: dict_thash.h dict_thash.o: htable.h dict_thash.o: iostuff.h dict_thash.o: msg.h dict_thash.o: myflock.h -dict_thash.o: mymalloc.h dict_thash.o: readlline.h dict_thash.o: stringops.h dict_thash.o: sys_defs.h dict_thash.o: vbuf.h dict_thash.o: vstream.h dict_thash.o: vstring.h -dict_thash.o: warn_stat.h dict_union.o: argv.h dict_union.o: check_arg.h dict_union.o: dict.h @@ -1421,6 +1444,18 @@ dict_unix.o: sys_defs.h dict_unix.o: vbuf.h dict_unix.o: vstream.h dict_unix.o: vstring.h +dict_utf8.o: argv.h +dict_utf8.o: check_arg.h +dict_utf8.o: dict.h +dict_utf8.o: dict_utf8.c +dict_utf8.o: msg.h +dict_utf8.o: myflock.h +dict_utf8.o: mymalloc.h +dict_utf8.o: stringops.h +dict_utf8.o: sys_defs.h +dict_utf8.o: vbuf.h +dict_utf8.o: vstream.h +dict_utf8.o: vstring.h dir_forest.o: check_arg.h dir_forest.o: dir_forest.c dir_forest.o: dir_forest.h @@ -1678,8 +1713,6 @@ load_file.o: vbuf.h load_file.o: vstream.h load_file.o: warn_stat.h load_lib.o: load_lib.c -load_lib.o: load_lib.h -load_lib.o: msg.h load_lib.o: sys_defs.h lowercase.o: check_arg.h lowercase.o: lowercase.c @@ -1755,17 +1788,17 @@ match_ops.o: sys_defs.h match_ops.o: vbuf.h match_ops.o: vstream.h match_ops.o: vstring.h -midna.o: check_arg.h -midna.o: ctable.h -midna.o: midna.c -midna.o: midna.h -midna.o: msg.h -midna.o: mymalloc.h -midna.o: stringops.h -midna.o: sys_defs.h -midna.o: valid_hostname.h -midna.o: vbuf.h -midna.o: vstring.h +midna_domain.o: check_arg.h +midna_domain.o: ctable.h +midna_domain.o: midna_domain.c +midna_domain.o: midna_domain.h +midna_domain.o: msg.h +midna_domain.o: mymalloc.h +midna_domain.o: stringops.h +midna_domain.o: sys_defs.h +midna_domain.o: valid_hostname.h +midna_domain.o: vbuf.h +midna_domain.o: vstring.h msg.o: msg.c msg.o: msg.h msg.o: msg_output.h @@ -2223,7 +2256,7 @@ valid_hostname.o: valid_hostname.h valid_hostname.o: vbuf.h valid_hostname.o: vstring.h valid_utf8_hostname.o: check_arg.h -valid_utf8_hostname.o: midna.h +valid_utf8_hostname.o: midna_domain.h valid_utf8_hostname.o: msg.h valid_utf8_hostname.o: mymalloc.h valid_utf8_hostname.o: stringops.h diff --git a/postfix/src/util/casefold.c b/postfix/src/util/casefold.c new file mode 100644 index 000000000..ed4b493a6 --- /dev/null +++ b/postfix/src/util/casefold.c @@ -0,0 +1,273 @@ +/*++ +/* NAME +/* casefold 3 +/* SUMMARY +/* casefold text for caseless comparison +/* SYNOPSIS +/* #include +/* +/* char *casefold( +/* VSTRING *src, +/* const char *src, +/* CONST_CHAR_STAR *err) +/* DESCRIPTION +/* casefold() converts text to a form that is suitable for +/* caseless comparison, rather than presentation to humans. +/* +/* When compiled without EAI support, casefold() implements +/* ASCII case folding, leaving non-ASCII byte values unchanged. +/* This mode has no error returns. +/* +/* When compiled with EAI support, casefold() implements UTF-8 +/* case folding using the en_US locale, as recommended when +/* the conversion result is not meant to be presented to humans. +/* When conversion fails the result is null, and the pointer +/* referenced by err is updated. +/* +/* With the ICU 4.8 library, there is no casefold error for +/* UTF-8 code points U+0000..U+10FFFF (including surrogate +/* range), not even when running inside an empty chroot jail. +/* +/* Arguments: +/* .IP src +/* Null-terminated input string. +/* .IP dest +/* Output buffer, null-terminated if the function completes +/* without reporting an error. +/* .IP err +/* Null pointer, or pointer to "const char *". for descriptive +/* text about errors. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + +/* System library. */ + +#include +#include +#include +#ifndef NO_EAI +#include +#include +#include +#endif + +/* Utility library. */ + +#include +#include + +#define STR(x) vstring_str(x) +#define LEN(x) VSTRING_LEN(x) + +/* casefold - casefold an UTF-8 string */ + +char *casefold(VSTRING *dest, const char *src, CONST_CHAR_STAR *err) +{ +#ifdef NO_EAI + + /* + * ASCII mode only + */ + vstring_strcpy(dest, src); + return (lowercase(STR(dest))); +#else + + /* + * Unicode mode. + */ + static UCaseMap *csm = 0; + UErrorCode error; + ssize_t space_needed; + int n; + + /* + * All-ASCII input. + */ + if (allascii(src)) { + vstring_strcpy(dest, src); + return (lowercase(STR(dest))); + } + + /* + * One-time initialization. With ICU 4.8 this works while chrooted. + */ + if (csm == 0) { + error = U_ZERO_ERROR; + csm = ucasemap_open("en_US", U_FOLD_CASE_DEFAULT, &error); + if (U_SUCCESS(error) == 0) + msg_fatal("ucasemap_open error: %s", u_errorName(error)); + } + + /* + * Fold the input, adjusting the buffer size if needed. Safety: don't + * loop forever. + */ + VSTRING_RESET(dest); + for (n = 0; n < 3; n++) { + error = U_ZERO_ERROR; + space_needed = + ucasemap_utf8FoldCase(csm, STR(dest), vstring_avail(dest), + src, strlen(src), &error); + if (error == U_BUFFER_OVERFLOW_ERROR) { + VSTRING_SPACE(dest, space_needed); + } else { + break; + } + } + + /* + * Report the result. With ICU 4.8, there are no casefolding errors for + * the entire RFC 3629 Unicode range (code points U+0000..U+10FFFF + * including surrogates). + */ + if (U_SUCCESS(error) == 0) { + if (err) + *err = u_errorName(error); + return (0); + } else { + /* Position the write pointer at the null terminator. */ + VSTRING_AT_OFFSET(dest, space_needed - 1); + return (STR(dest)); + } +#endif /* NO_EAI */ +} + +#ifdef TEST + +static void encode_utf8(VSTRING *buffer, int codepoint) +{ + const char myname[] = "encode_utf8"; + + VSTRING_RESET(buffer); + if (codepoint < 0x80) { + VSTRING_ADDCH(buffer, codepoint); + } else if (codepoint < 0x800) { + VSTRING_ADDCH(buffer, 0xc0 | (codepoint >> 6)); + VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f)); + } else if (codepoint < 0x10000) { + VSTRING_ADDCH(buffer, 0xe0 | (codepoint >> 12)); + VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 6) & 0x3f)); + VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f)); + } else if (codepoint <= 0x10FFFF) { + VSTRING_ADDCH(buffer, 0xf0 | (codepoint >> 18)); + VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 12) & 0x3f)); + VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 6) & 0x3f)); + VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f)); + } else { + msg_panic("%s: out-of-range codepoint U+%X", myname, codepoint); + } + VSTRING_TERMINATE(buffer); +} + +#include +#include +#include + +#include +#include +#include + +int main(int argc, char **argv) +{ + VSTRING *buffer = vstring_alloc(1); + VSTRING *dest = vstring_alloc(1); + char *bp; + char *conv_res; + const char *fold_err; + char *cmd; + int codepoint, first, last; + + if (setlocale(LC_ALL, "C") == 0) + msg_fatal("setlocale(LC_ALL, C) failed: %m"); + + msg_vstream_init(argv[0], VSTREAM_ERR); + + util_utf8_enable = 1; + + VSTRING_SPACE(buffer, 256); /* chroot pathname */ + + while (vstring_fgets_nonl(buffer, VSTREAM_IN)) { + bp = STR(buffer); + msg_info("> %s", bp); + cmd = mystrtok(&bp, CHARS_SPACE); + if (cmd == 0 || *cmd == '#') + continue; + while (ISSPACE(*bp)) + bp++; + + /* + * Null-terminated string. + */ + if (strcmp(cmd, "fold") == 0) { + if ((conv_res = casefold(dest, bp, &fold_err)) != 0) + msg_info("\"%s\" ->fold \"%s\"", bp, conv_res); + else + msg_warn("cannot casefold \"%s\": %s", bp, fold_err); + } + + /* + * Codepoint range. + */ + else if (strcmp(cmd, "range") == 0 + && sscanf(bp, "%i %i", &first, &last) == 2 + && first <= last) { + for (codepoint = first; codepoint <= last; codepoint++) { + if (codepoint >= 0xD800 && codepoint <= 0xDFFF) { + msg_warn("skipping surrogate range"); + codepoint = 0xDFFF; + } else { + encode_utf8(buffer, codepoint); + if (msg_verbose) + msg_info("U+%X -> %s", codepoint, STR(buffer)); + if (valid_utf8_string(STR(buffer), LEN(buffer)) == 0) + msg_fatal("bad utf-8 encoding for U+%X", codepoint); + if (casefold(dest, STR(buffer), &fold_err) == 0) + msg_warn("casefold error for U+%X: %s", + codepoint, fold_err); + } + } + msg_info("range completed: 0x%x..0x%x", first, last); + } + + /* + * Chroot directory. + */ + else if (strcmp(cmd, "chroot") == 0 + && sscanf(bp, "%255s", STR(buffer)) == 1) { + if (geteuid() == 0) { + if (chdir(STR(buffer)) < 0) + msg_fatal("chdir(%s): %m", STR(buffer)); + if (chroot(STR(buffer)) < 0) + msg_fatal("chroot(%s): %m", STR(buffer)); + msg_info("chroot %s completed", STR(buffer)); + } + } + + /* + * Verbose. + */ + else if (strcmp(cmd, "verbose") == 0 + && sscanf(bp, "%i", &msg_verbose) == 1) { + /* void */ ; + } + + /* + * Usage + */ + else { + msg_info("Usage: %s chroot | fold | range | verbose ", + argv[0]); + } + } + exit(0); +} + +#endif /* TEST */ diff --git a/postfix/src/util/casefold_test.in b/postfix/src/util/casefold_test.in new file mode 100644 index 000000000..f1a0098fd --- /dev/null +++ b/postfix/src/util/casefold_test.in @@ -0,0 +1,19 @@ +# Ignored when not running as root. +chroot /tmp +# Casefold U+0000 .. U+10FFFF excluding surrogates. +range 0x0 0xD7FF +range 0xD800 0xD800 +range 0xDFFF 0xDFFF +range 0xE000 0x10FFFF +# Demonstrate that range is not a noop. +verbose 1 +range 0xE000 0xE007 +verbose 0 +# Upper-case greek -> lower-case greek. +fold Δημοσθένους.example.com +# Upper-case ASCII -> lower-case ASCII. +fold HeLlO.ExAmPlE.CoM +# Folding does not change aliases for '.'. +fold x。example.com +fold x．example.com +fold x｡example.com diff --git a/postfix/src/util/casefold_test.ref b/postfix/src/util/casefold_test.ref new file mode 100644 index 000000000..18c6b6c54 --- /dev/null +++ b/postfix/src/util/casefold_test.ref @@ -0,0 +1,39 @@ +./casefold: > # Ignored when not running as root. +./casefold: > chroot /tmp +./casefold: > # Casefold U+0000 .. U+10FFFF excluding surrogates. +./casefold: > range 0x0 0xD7FF +./casefold: range completed: 0x0..0xd7ff +./casefold: > range 0xD800 0xD800 +./casefold: warning: skipping surrogate range +./casefold: range completed: 0xd800..0xd800 +./casefold: > range 0xDFFF 0xDFFF +./casefold: warning: skipping surrogate range +./casefold: range completed: 0xdfff..0xdfff +./casefold: > range 0xE000 0x10FFFF +./casefold: range completed: 0xe000..0x10ffff +./casefold: > # Demonstrate that range is not a noop. +./casefold: > verbose 1 +./casefold: > range 0xE000 0xE007 +./casefold: U+E000 ->  +./casefold: U+E001 ->  +./casefold: U+E002 ->  +./casefold: U+E003 ->  +./casefold: U+E004 ->  +./casefold: U+E005 ->  +./casefold: U+E006 ->  +./casefold: U+E007 ->  +./casefold: range completed: 0xe000..0xe007 +./casefold: > verbose 0 +./casefold: > # Upper-case greek -> lower-case greek. +./casefold: > fold Δημοσθένους.example.com +./casefold: "Δημοσθένους.example.com" ->fold "δημοσθένουσ.example.com" +./casefold: > # Upper-case ASCII -> lower-case ASCII. +./casefold: > fold HeLlO.ExAmPlE.CoM +./casefold: "HeLlO.ExAmPlE.CoM" ->fold "hello.example.com" +./casefold: > # Folding does not change aliases for '.'. +./casefold: > fold x。example.com +./casefold: "x。example.com" ->fold "x。example.com" +./casefold: > fold x．example.com +./casefold: "x．example.com" ->fold "x．example.com" +./casefold: > fold x｡example.com +./casefold: "x｡example.com" ->fold "x｡example.com" diff --git a/postfix/src/util/dict.c b/postfix/src/util/dict.c index df583179e..0ec4a3d75 100644 --- a/postfix/src/util/dict.c +++ b/postfix/src/util/dict.c @@ -637,6 +637,8 @@ static const NAME_MASK dict_mask[] = { "open_lock", DICT_FLAG_OPEN_LOCK, /* permanent lock upon open */ "bulk_update", DICT_FLAG_BULK_UPDATE, /* bulk update if supported */ "multi_writer", DICT_FLAG_MULTI_WRITER, /* multi-writer safe */ + "utf8_enable", DICT_FLAG_UTF8_ENABLE, /* enable UTF-8 checks/fold */ + "utf8_proxy", DICT_FLAG_UTF8_PROXY, /* UTF-8 proxy is present */ 0, }; diff --git a/postfix/src/util/dict.h b/postfix/src/util/dict.h index 104996ae5..e48b81c45 100644 --- a/postfix/src/util/dict.h +++ b/postfix/src/util/dict.h @@ -92,6 +92,7 @@ typedef struct DICT { VSTRING *fold_buf; /* key folding buffer */ DICT_OWNER owner; /* provenance */ int error; /* last operation only */ + ssize_t size; /* size of this thing */ DICT_JMP_BUF *jbuf; /* exception handling */ } DICT; @@ -126,6 +127,10 @@ extern DICT *dict_debug(DICT *); #define DICT_FLAG_OPEN_LOCK (1<<16) /* perm lock if not multi-writer safe */ #define DICT_FLAG_BULK_UPDATE (1<<17) /* optimize for bulk updates */ #define DICT_FLAG_MULTI_WRITER (1<<18) /* multi-writer safe map */ +#define DICT_FLAG_UTF8_ENABLE (1<<19) /* enable UTF-8 checks */ +#define DICT_FLAG_UTF8_PROXY (1<<20) /* UTF-8 proxy layer is present */ + +#define DICT_FLAG_UTF8_MASK (DICT_FLAG_UTF8_ENABLE) /* IMPORTANT: Update the dict_mask[] table when the above changes */ @@ -157,9 +162,17 @@ extern DICT *dict_debug(DICT *); #define DICT_FLAG_RQST_MASK (DICT_FLAG_FOLD_ANY | DICT_FLAG_LOCK | \ DICT_FLAG_DUP_REPLACE | DICT_FLAG_DUP_WARN | \ DICT_FLAG_DUP_IGNORE | DICT_FLAG_SYNC_UPDATE | \ - DICT_FLAG_PARANOID) + DICT_FLAG_PARANOID | DICT_FLAG_UTF8_MASK) #define DICT_FLAG_INST_MASK ~(DICT_FLAG_IMPL_MASK | DICT_FLAG_RQST_MASK) + /* + * Feature tests. + */ +extern int util_utf8_enable; + +#define DICT_IS_ENABLE_UTF8(flags) \ + (util_utf8_enable && (flags & DICT_FLAG_UTF8_MASK)) + /* * dict->error values. Errors must be negative; smtpd_check depends on this. */ @@ -233,6 +246,14 @@ extern int dict_changed(void); extern const char *dict_changed_name(void); extern const char *dict_flags_str(int); extern int dict_flags_mask(const char *); +extern void dict_type_override(DICT *, const char *); + + /* + * Check and convert UTF-8 keys and values. + */ +extern DICT *dict_utf8_encapsulate(DICT *); +extern char *dict_utf8_check_fold(DICT *, const char *, CONST_CHAR_STAR *); +extern int dict_utf8_check(const char *, CONST_CHAR_STAR *); /* * Driver for interactive or scripted tests. @@ -262,13 +283,13 @@ extern DICT *PRINTFLIKE(5, 6) dict_surrogate(const char *, const char *, int, in * systems have bugs in their implementation. */ #ifdef NO_SIGSETJMP -#define dict_setjmp(stream) setjmp((stream)->jbuf[0]) -#define dict_longjmp(stream, val) longjmp((stream)->jbuf[0], (val)) +#define dict_setjmp(dict) setjmp((dict)->jbuf[0]) +#define dict_longjmp(dict, val) longjmp((dict)->jbuf[0], (val)) #else -#define dict_setjmp(stream) sigsetjmp((stream)->jbuf[0], 1) -#define dict_longjmp(stream, val) siglongjmp((stream)->jbuf[0], (val)) +#define dict_setjmp(dict) sigsetjmp((dict)->jbuf[0], 1) +#define dict_longjmp(dict, val) siglongjmp((dict)->jbuf[0], (val)) #endif -#define dict_isjmp(stream) ((stream)->jbuf != 0) +#define dict_isjmp(dict) ((dict)->jbuf != 0) /* * Temporary API. If exception handling proves to be useful, diff --git a/postfix/src/util/dict_alloc.c b/postfix/src/util/dict_alloc.c index f370569a8..094e2193a 100644 --- a/postfix/src/util/dict_alloc.c +++ b/postfix/src/util/dict_alloc.c @@ -153,6 +153,7 @@ DICT *dict_alloc(const char *dict_type, const char *dict_name, ssize_t size) dict->owner.status = DICT_OWNER_UNKNOWN; dict->owner.uid = INT_MAX; dict->error = DICT_ERR_NONE; + dict->size = size; dict->jbuf = 0; return dict; } diff --git a/postfix/src/util/dict_cdb.c b/postfix/src/util/dict_cdb.c index dbd4bd9ad..85e49a4ca 100644 --- a/postfix/src/util/dict_cdb.c +++ b/postfix/src/util/dict_cdb.c @@ -195,7 +195,7 @@ static DICT *dict_cdbq_open(const char *path, int dict_flags) if ((fd = open(cdb_path, O_RDONLY)) < 0) DICT_CDBQ_OPEN_RETURN(dict_surrogate(DICT_TYPE_CDB, path, - O_RDONLY, dict_flags, + O_RDONLY, dict_flags, "open database %s: %m", cdb_path)); dict_cdbq = (DICT_CDBQ *) dict_alloc(DICT_TYPE_CDB, diff --git a/postfix/src/util/dict_debug.c b/postfix/src/util/dict_debug.c index 3d9a44325..46634d40c 100644 --- a/postfix/src/util/dict_debug.c +++ b/postfix/src/util/dict_debug.c @@ -59,7 +59,9 @@ static const char *dict_debug_lookup(DICT *dict, const char *key) DICT *real_dict = dict_debug->real_dict; const char *result; + real_dict->flags = dict->flags; result = dict_get(real_dict, key); + dict->flags = real_dict->flags; msg_info("%s:%s lookup: \"%s\" = \"%s\"", dict->type, dict->name, key, result ? result : real_dict->error ? "error" : "not_found"); DICT_ERR_VAL_RETURN(dict, real_dict->error, result); @@ -73,7 +75,9 @@ static int dict_debug_update(DICT *dict, const char *key, const char *value) DICT *real_dict = dict_debug->real_dict; int result; + real_dict->flags = dict->flags; result = dict_put(real_dict, key, value); + dict->flags = real_dict->flags; msg_info("%s:%s update: \"%s\" = \"%s\": %s", dict->type, dict->name, key, value, result == 0 ? "success" : real_dict->error ? "error" : "failed"); @@ -88,7 +92,9 @@ static int dict_debug_delete(DICT *dict, const char *key) DICT *real_dict = dict_debug->real_dict; int result; + real_dict->flags = dict->flags; result = dict_del(real_dict, key); + dict->flags = real_dict->flags; msg_info("%s:%s delete: \"%s\": %s", dict->type, dict->name, key, result == 0 ? "success" : real_dict->error ? "error" : "failed"); @@ -104,7 +110,9 @@ static int dict_debug_sequence(DICT *dict, int function, DICT *real_dict = dict_debug->real_dict; int result; + real_dict->flags = dict->flags; result = dict_seq(real_dict, function, key, value); + dict->flags = real_dict->flags; if (result == 0) msg_info("%s:%s sequence: \"%s\" = \"%s\"", dict->type, dict->name, *key, *value); diff --git a/postfix/src/util/dict_inline.c b/postfix/src/util/dict_inline.c index af646007c..7e2231171 100644 --- a/postfix/src/util/dict_inline.c +++ b/postfix/src/util/dict_inline.c @@ -31,114 +31,30 @@ /* System library. */ #include +#include /* Utility library. */ #include #include -#include #include #include +#include #include /* Application-specific. */ -typedef struct { - DICT dict; /* generic members */ - HTABLE *table; /* lookup table */ - HTABLE_INFO **info; /* for iterator */ - HTABLE_INFO **cursor; /* ditto */ -} DICT_INLINE; - -/* dict_inline_lookup - search inline table */ - -static const char *dict_inline_lookup(DICT *dict, const char *name) -{ - DICT_INLINE *dict_inline = (DICT_INLINE *) dict; - const char *result = 0; - - /* - * Optionally fold the key. - */ - if (dict->flags & DICT_FLAG_FOLD_FIX) { - if (dict->fold_buf == 0) - dict->fold_buf = vstring_alloc(10); - vstring_strcpy(dict->fold_buf, name); - name = lowercase(vstring_str(dict->fold_buf)); - } - - /* - * Look up the value. - */ - result = htable_find(dict_inline->table, name); - - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, result); -} - -/* dict_inline_sequence - traverse the dictionary */ - -static int dict_inline_sequence(DICT *dict, int function, - const char **key, const char **value) -{ - const char *myname = "dict_inline_sequence"; - DICT_INLINE *dict_inline = (DICT_INLINE *) dict; - - /* - * Determine and execute the seek function. - */ - switch (function) { - case DICT_SEQ_FUN_FIRST: - if (dict_inline->info == 0) - dict_inline->info = htable_list(dict_inline->table); - dict_inline->cursor = dict_inline->info; - break; - case DICT_SEQ_FUN_NEXT: - if (dict_inline->cursor[0]) - dict_inline->cursor += 1; - break; - default: - msg_panic("%s: invalid function: %d", myname, function); - } - - /* - * Return the entry under the cursor. - */ - if (dict_inline->cursor[0]) { - *key = dict_inline->cursor[0]->key; - *value = dict_inline->cursor[0]->value; - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_SUCCESS); - } else { - *key = 0; - *value = 0; - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL); - } -} - -/* dict_inline_close - disassociate from inline table */ - -static void dict_inline_close(DICT *dict) -{ - DICT_INLINE *dict_inline = (DICT_INLINE *) dict; - - htable_free(dict_inline->table, myfree); - if (dict_inline->info) - myfree((void *) dict_inline->info); - if (dict->fold_buf) - vstring_free(dict->fold_buf); - dict_free(dict); -} - /* dict_inline_open - open inline table */ DICT *dict_inline_open(const char *name, int open_flags, int dict_flags) { - DICT_INLINE *dict_inline; + DICT *dict; char *cp, *saved_name = 0; size_t len; - HTABLE *table = 0; char *nameval, *vname, *value; const char *err = 0; char *xperr = 0; + int count = 0; /* * Clarity first. Let the optimizer worry about redundant code. @@ -149,8 +65,6 @@ DICT *dict_inline_open(const char *name, int open_flags, int dict_flags) myfree(saved_name); \ if (xperr != 0) \ myfree(xperr); \ - if (table != 0) \ - htable_free(table, myfree); \ return (__d); \ } while (0) @@ -163,6 +77,19 @@ DICT *dict_inline_open(const char *name, int open_flags, int dict_flags) "%s:%s map requires O_RDONLY access mode", DICT_TYPE_INLINE, name)); + /* + * UTF-8 syntax check. + */ + if (DICT_IS_ENABLE_UTF8(dict_flags) + && allascii(name) == 0 + && valid_utf8_string(name, strlen(name)) == 0) + DICT_INLINE_RETURN(dict_surrogate(DICT_TYPE_INLINE, name, + open_flags, dict_flags, + "bad UTF-8 syntax: \"%s:%s\"; " + "need \"%s:{name=value...}\"", + DICT_TYPE_INLINE, name, + DICT_TYPE_INLINE)); + /* * Parse the table into its constituent name=value pairs. */ @@ -175,15 +102,23 @@ DICT *dict_inline_open(const char *name, int open_flags, int dict_flags) DICT_TYPE_INLINE, name, DICT_TYPE_INLINE)); - table = htable_create(5); + /* + * Reuse the "internal" dictionary type. + */ + dict = dict_open3(DICT_TYPE_HT, name, open_flags, dict_flags); + dict_type_override(dict, DICT_TYPE_INLINE); while ((nameval = mystrtokq(&cp, CHARS_COMMA_SP, CHARS_BRACE)) != 0) { if ((nameval[0] != CHARS_BRACE[0] || (err = xperr = extpar(&nameval, CHARS_BRACE, EXTPAR_FLAG_STRIP)) == 0) && (err = split_nameval(nameval, &vname, &value)) != 0) break; - (void) htable_enter(table, vname, mystrdup(value)); + + /* No duplicate checks. See comments in dict_thash.c. */ + dict->update(dict, vname, value); + count += 1; } - if (err != 0 || table->used == 0) + if (err != 0 || count == 0) { + dict->close(dict); DICT_INLINE_RETURN(dict_surrogate(DICT_TYPE_INLINE, name, open_flags, dict_flags, "%s: \"%s:%s\"; " @@ -191,21 +126,8 @@ DICT *dict_inline_open(const char *name, int open_flags, int dict_flags) err != 0 ? err : "empty table", DICT_TYPE_INLINE, name, DICT_TYPE_INLINE)); + } + dict->owner.status = DICT_OWNER_TRUSTED; - /* - * Bundle up the result. - */ - dict_inline = (DICT_INLINE *) - dict_alloc(DICT_TYPE_INLINE, name, sizeof(*dict_inline)); - dict_inline->dict.lookup = dict_inline_lookup; - dict_inline->dict.sequence = dict_inline_sequence; - dict_inline->dict.close = dict_inline_close; - dict_inline->dict.flags = dict_flags | DICT_FLAG_FIXED; - dict_inline->dict.owner.status = DICT_OWNER_TRUSTED; - if (dict_flags & DICT_FLAG_FOLD_FIX) - dict_inline->dict.fold_buf = vstring_alloc(10); - dict_inline->info = 0; - dict_inline->table = table; - table = 0; - DICT_INLINE_RETURN(DICT_DEBUG (&dict_inline->dict)); + DICT_INLINE_RETURN(DICT_DEBUG (dict)); } diff --git a/postfix/src/util/dict_inline.ref b/postfix/src/util/dict_inline.ref index 771ca9f6a..38332d526 100644 --- a/postfix/src/util/dict_inline.ref +++ b/postfix/src/util/dict_inline.ref @@ -10,7 +10,7 @@ owner=trusted (uid=2147483647) owner=trusted (uid=2147483647) owner=trusted (uid=2147483647) > get foo -foo=xx +foo=XX > get bar bar=lotsa stuff > get baz diff --git a/postfix/src/util/dict_lmdb.c b/postfix/src/util/dict_lmdb.c index 3cbea12f9..50f5c7090 100644 --- a/postfix/src/util/dict_lmdb.c +++ b/postfix/src/util/dict_lmdb.c @@ -235,7 +235,6 @@ static int dict_lmdb_update(DICT *dict, const char *name, const char *value) name = lowercase(vstring_str(dict->fold_buf)); } mdb_key.mv_data = (void *) name; - mdb_value.mv_data = (void *) value; mdb_key.mv_size = strlen(name); mdb_value.mv_size = strlen(value); @@ -435,6 +434,8 @@ static int dict_lmdb_sequence(DICT *dict, int function, if (mdb_value.mv_data != 0 && mdb_value.mv_size > 0) *value = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data, mdb_value.mv_size); + else + *value = ""; /* XXX */ break; /* diff --git a/postfix/src/util/dict_open.c b/postfix/src/util/dict_open.c index 0880b95cd..4b11986d6 100644 --- a/postfix/src/util/dict_open.c +++ b/postfix/src/util/dict_open.c @@ -66,6 +66,10 @@ /* int dict_longjmp(dict, val) /* DICT *dict; /* int val; +/* +/* void dict_type_override(dict, type) +/* DICT *dict; +/* const char *type; /* DESCRIPTION /* This module implements a low-level interface to multiple /* physical dictionary types. @@ -119,11 +123,14 @@ /* With databases whose lookup fields are fixed-case strings, /* fold the search string to lower case before accessing the /* database. This includes hash:, cdb:, dbm:. nis:, ldap:, -/* *sql. +/* *sql. WARNING: case folding is supported only for ASCII or +/* valid UTF-8. /* .IP DICT_FLAG_FOLD_MUL /* With databases where one lookup field can match both upper /* and lower case, fold the search key to lower case before -/* accessing the database. This includes regexp: and pcre: +/* accessing the database. This includes regexp: and pcre:. +/* WARNING: case folding is supported only for ASCII or valid +/* UTF-8. /* .IP DICT_FLAG_FOLD_ANY /* Short-hand for (DICT_FLAG_FOLD_FIX | DICT_FLAG_FOLD_MUL). /* .IP DICT_FLAG_SYNC_UPDATE @@ -149,6 +156,12 @@ /* and must trap exceptions from the database client with dict_setjmp(). /* .IP DICT_FLAG_DEBUG /* Enable additional logging. +/* .IP DICT_FLAG_UTF8_ENABLE +/* With util_utf8_enable != 0, require that lookup/update/delete +/* keys and values are valid UTF-8. Skip a lookup/update/delete +/* request with a non-UTF-8 key, skip an update request with +/* a non-UTF-8 value, and fail a lookup request with a non-UTF-8 +/* value. /* .PP /* Specify DICT_FLAG_NONE for no special processing. /* @@ -179,6 +192,10 @@ /* dict_open3() takes separate arguments for dictionary type and /* name, but otherwise performs the same functions as dict_open(). /* +/* The dict_get(), dict_put(), dict_del(), and dict_seq() +/* macros evaluate their first argument multiple times. +/* These names should have been in uppercase. +/* /* dict_get() retrieves the value stored in the named dictionary /* under the given key. A null pointer means the value was not found. /* As with dict_lookup(), the result is owned by the lookup table @@ -226,6 +243,10 @@ /* NB: non-local jumps such as dict_longjmp() are not safe for /* jumping out of any routine that manipulates DICT data. /* longjmp() like calls are best avoided in signal handlers. +/* +/* dict_type_override() changes the symbolic dictionary type. +/* This is used by dictionaries whose internals are based on +/* some other dictionary type. /* DIAGNOSTICS /* Fatal error: open error, unsupported dictionary type, attempt to /* update non-writable dictionary. @@ -457,6 +478,10 @@ DICT *dict_open3(const char *dict_type, const char *dict_name, msg_fatal("%s:%s: unable to get exclusive lock: %m", dict_type, dict_name); } + /* Last step: insert proxy for UTF-8 syntax checks and casefolding. */ + if ((dict->flags & DICT_FLAG_UTF8_PROXY) == 0 + && DICT_IS_ENABLE_UTF8(dict_flags)) + dict = dict_utf8_encapsulate(dict); return (dict); } @@ -532,6 +557,14 @@ DICT_MAPNAMES_EXTEND_FN dict_mapnames_extend(DICT_MAPNAMES_EXTEND_FN new_cb) return (old_cb); } +/* dict_type_override - disguise a dictionary type */ + +void dict_type_override(DICT *dict, const char *type) +{ + myfree(dict->type); + dict->type = mystrdup(type); +} + #ifdef TEST /* diff --git a/postfix/src/util/dict_test.c b/postfix/src/util/dict_test.c index 0fb0133bf..836798592 100644 --- a/postfix/src/util/dict_test.c +++ b/postfix/src/util/dict_test.c @@ -76,9 +76,11 @@ void dict_test(int argc, char **argv) dict_flags |= DICT_FLAG_LOCK; if ((dict_flags & (DICT_FLAG_DUP_WARN | DICT_FLAG_DUP_IGNORE)) == 0) dict_flags |= DICT_FLAG_DUP_REPLACE; + dict_flags |= DICT_FLAG_UTF8_ENABLE; vstream_fflush(VSTREAM_OUT); dict_name = argv[optind]; dict_allow_surrogate = 1; + util_utf8_enable = 1; dict = dict_open(dict_name, open_flags, dict_flags); dict_register(dict_name, dict); vstream_printf("owner=%s (uid=%ld)\n", @@ -125,8 +127,6 @@ void dict_test(int argc, char **argv) if (dict_put(dict, key, value) != 0) vstream_printf("%s: %s\n", key, dict->error ? "error" : "not updated"); - else - vstream_printf("%s=%s\n", key, value); } else if (strcmp(cmd, "first") == 0 && !key && !value) { if (dict_seq(dict, DICT_SEQ_FUN_FIRST, &key, &value) == 0) vstream_printf("%s=%s\n", key, value); diff --git a/postfix/src/util/dict_test.ref b/postfix/src/util/dict_test.ref index 0872fb09a..54e91f88d 100644 --- a/postfix/src/util/dict_test.ref +++ b/postfix/src/util/dict_test.ref @@ -12,7 +12,6 @@ foo=fooval > del foo foo: deleted > put baz bazval -baz=bazval > get baz baz=bazval > del baz diff --git a/postfix/src/util/dict_thash.c b/postfix/src/util/dict_thash.c index aaef6613f..c638bb6e3 100644 --- a/postfix/src/util/dict_thash.c +++ b/postfix/src/util/dict_thash.c @@ -41,121 +41,33 @@ /* Utility library. */ #include -#include -#include #include #include #include #include #include +#include #include -#include /* Application-specific. */ -typedef struct { - DICT dict; /* generic members */ - HTABLE *table; /* in-memory hash */ - HTABLE_INFO **info; /* for iterator */ - HTABLE_INFO **cursor; /* ditto */ -} DICT_THASH; - #define STR vstring_str - -/* dict_thash_lookup - find database entry */ - -static const char *dict_thash_lookup(DICT *dict, const char *name) -{ - DICT_THASH *dict_thash = (DICT_THASH *) dict; - const char *result = 0; - - /* - * Optionally fold the key. - */ - if (dict->flags & DICT_FLAG_FOLD_FIX) { - if (dict->fold_buf == 0) - dict->fold_buf = vstring_alloc(10); - vstring_strcpy(dict->fold_buf, name); - name = lowercase(vstring_str(dict->fold_buf)); - } - - /* - * Look up the value. - */ - result = htable_find(dict_thash->table, name); - - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, result); -} - -/* dict_thash_sequence - traverse the dictionary */ - -static int dict_thash_sequence(DICT *dict, int function, - const char **key, const char **value) -{ - const char *myname = "dict_thash_sequence"; - DICT_THASH *dict_thash = (DICT_THASH *) dict; - - /* - * Determine and execute the seek function. - */ - switch (function) { - case DICT_SEQ_FUN_FIRST: - if (dict_thash->info == 0) - dict_thash->info = htable_list(dict_thash->table); - dict_thash->cursor = dict_thash->info; - break; - case DICT_SEQ_FUN_NEXT: - if (dict_thash->cursor[0]) - dict_thash->cursor += 1; - break; - default: - msg_panic("%s: invalid function: %d", myname, function); - } - - /* - * Return the entry under the cursor. - */ - if (dict_thash->cursor[0]) { - *key = dict_thash->cursor[0]->key; - *value = dict_thash->cursor[0]->value; - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_SUCCESS); - } else { - *key = 0; - *value = 0; - DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL); - } -} - -/* dict_thash_close - disassociate from data base */ - -static void dict_thash_close(DICT *dict) -{ - DICT_THASH *dict_thash = (DICT_THASH *) dict; - - htable_free(dict_thash->table, myfree); - if (dict_thash->info) - myfree((void *) dict_thash->info); - if (dict->fold_buf) - vstring_free(dict->fold_buf); - dict_free(dict); -} +#define LEN VSTRING_LEN /* dict_thash_open - open flat text data base */ DICT *dict_thash_open(const char *path, int open_flags, int dict_flags) { - DICT_THASH *dict_thash; - VSTREAM *fp = 0; + DICT *dict; + VSTREAM *fp = 0; /* DICT_THASH_OPEN_RETURN() */ struct stat st; time_t before; time_t after; - VSTRING *line_buffer = 0; + VSTRING *line_buffer = 0; /* DICT_THASH_OPEN_RETURN() */ int lineno; int last_line; char *key; char *value; - HTABLE *table; - HTABLE_INFO *ht; /* * Let the optimizer worry about eliminating redundant code. @@ -187,13 +99,31 @@ DICT *dict_thash_open(const char *path, int open_flags, int dict_flags) DICT_THASH_OPEN_RETURN(dict_surrogate(DICT_TYPE_THASH, path, open_flags, dict_flags, "open database %s: %m", path)); - } + } + + /* + * Reuse the "internal" dictionary type. + */ + dict = dict_open3(DICT_TYPE_HT, path, open_flags, dict_flags); + dict_type_override(dict, DICT_TYPE_THASH); + if (line_buffer == 0) line_buffer = vstring_alloc(100); last_line = 0; - table = htable_create(13); while (readllines(line_buffer, fp, &last_line, &lineno)) { + /* + * First some UTF-8 checks sans casefolding. + */ + if (DICT_IS_ENABLE_UTF8(dict_flags) + && allascii(STR(line_buffer)) == 0 + && valid_utf8_string(STR(line_buffer), LEN(line_buffer)) == 0) { + msg_warn("%s, line %d: non-UTF-8 input \"%s\"" + " -- ignoring this line", + VSTREAM_PATH(fp), lineno, STR(line_buffer)); + continue; + } + /* * Split on the first whitespace character, then trim leading and * trailing whitespace from key and value. @@ -220,31 +150,31 @@ DICT *dict_thash_open(const char *path, int open_flags, int dict_flags) msg_warn("%s, line %d: record is in \"key: value\" format;" " is this an alias file?", path, lineno); - /* - * Optionally fold the key. - */ - if (dict_flags & DICT_FLAG_FOLD_FIX) - lowercase(key); - /* * Store the value under the key. Handle duplicates - * appropriately. + * appropriately. XXX Move this into dict_ht, but 1) that map + * ignores duplicates by default and we would have to check that + * we won't break existing code that depends on such benavior; 2) + * by inlining the checks here we can degrade gracefully instead + * of terminating with a fatal error. See comment in dict_inline.c. */ - if ((ht = htable_locate(table, key)) != 0) { + if (dict->lookup(dict, key) != 0) { if (dict_flags & DICT_FLAG_DUP_IGNORE) { /* void */ ; } else if (dict_flags & DICT_FLAG_DUP_REPLACE) { - myfree(ht->value); - ht->value = mystrdup(value); + dict->update(dict, key, value); } else if (dict_flags & DICT_FLAG_DUP_WARN) { msg_warn("%s, line %d: duplicate entry: \"%s\"", path, lineno, key); } else { - msg_fatal("%s, line %d: duplicate entry: \"%s\"", - path, lineno, key); + dict->close(dict); + DICT_THASH_OPEN_RETURN(dict_surrogate(DICT_TYPE_THASH, path, + open_flags, dict_flags, + "%s, line %d: duplicate entry: \"%s\"", + path, lineno, key)); } } else { - htable_enter(table, key, mystrdup(value)); + dict->update(dict, key, value); } } @@ -263,27 +193,14 @@ DICT *dict_thash_open(const char *path, int open_flags, int dict_flags) /* * Yes, it is hot. Discard the result and read the file again. */ - htable_free(table, myfree); + dict->close(dict); if (msg_verbose > 1) msg_info("pausing to let file %s cool down", path); doze(300000); } - /* - * Create the in-memory table. - */ - dict_thash = (DICT_THASH *) - dict_alloc(DICT_TYPE_THASH, path, sizeof(*dict_thash)); - dict_thash->dict.lookup = dict_thash_lookup; - dict_thash->dict.sequence = dict_thash_sequence; - dict_thash->dict.close = dict_thash_close; - dict_thash->dict.flags = dict_flags | DICT_FLAG_DUP_WARN | DICT_FLAG_FIXED; - if (dict_flags & DICT_FLAG_FOLD_FIX) - dict_thash->dict.fold_buf = vstring_alloc(10); - dict_thash->info = 0; - dict_thash->table = table; - dict_thash->dict.owner.uid = st.st_uid; - dict_thash->dict.owner.status = (st.st_uid != 0); + dict->owner.uid = st.st_uid; + dict->owner.status = (st.st_uid != 0); - DICT_THASH_OPEN_RETURN(DICT_DEBUG (&dict_thash->dict)); + DICT_THASH_OPEN_RETURN(DICT_DEBUG (dict)); } diff --git a/postfix/src/util/dict_thash.map b/postfix/src/util/dict_thash.map index 67e75784f..95b54fd82 100644 --- a/postfix/src/util/dict_thash.map +++ b/postfix/src/util/dict_thash.map @@ -12,3 +12,4 @@ attr_scan0.c 15454 attr_scan64.c 17256 attr_scan_plain.c 16924 auto_clnt.c 9819 +ABCDEF 012345 diff --git a/postfix/src/util/dict_utf8.c b/postfix/src/util/dict_utf8.c new file mode 100644 index 000000000..b07c6e095 --- /dev/null +++ b/postfix/src/util/dict_utf8.c @@ -0,0 +1,318 @@ +/*++ +/* NAME +/* dict_utf8 3 +/* SUMMARY +/* dictionary UTF-8 helpers +/* SYNOPSIS +/* #include +/* +/* DICT *dict_utf8_encapsulate( +/* DICT *dict) +/* +/* char *dict_utf8_check_fold( +/* DICT *dict, +/* const char *string, +/* CONST_CHAR_STAR *err, +/* int fold_flag) +/* +/* int dict_utf8_check( +/* const char *string, +/* CONST_CHAR_STAR *err) +/* DESCRIPTION +/* dict_utf8_encapsulate() wraps a dictionary's lookup/update/delete +/* methods with code that enforces UTF-8 checks on keys and +/* values, and that logs a warning when incorrect UTF-8 is +/* encountered. The original dictionary handle becomes invalid. +/* +/* The wrapper code enforces a policy that maximizes application +/* robustness. Attempts to store non-UTF-8 keys or values are +/* skipped while reporting success, attempts to look up or +/* delete non-UTF-8 keys are skipped while reporting success, +/* and attempts to look up a non-UTF-8 value are flagged while +/* reporting a configuration error. +/* +/* The dict_utf8_check* functions may be invoked to perform +/* UTF-8 validity checks when util_utf8_enable is non-zero and +/* DICT_FLAG_UTF8_ENABLE is set. Otherwise both functions +/* always report success. +/* +/* dict_utf8_check_fold() optionally folds a string, and checks +/* it for UTF-8 validity. The result is the possibly-folded +/* string, or a null pointer in case of error. +/* +/* dict_utf8_check() checks a string for UTF-8 validity. The +/* result is zero in case of error. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + + /* + * System library. + */ +#include +#include + + /* + * Utility library. + */ +#include +#include +#include +#include +#include + + /* + * The goal is to maximize robustness: bad UTF-8 should not appear in keys, + * because those are derived from controlled inputs, and values should be + * printable before they are stored. But if we failed to check something + * then it should not result in fatal errors and thus open up the system for + * a denial-of-service attack. + * + * Proposed over-all policy: skip attempts to store invalid UTF-8 lookup keys + * or values. Rationale: some storage may not permit malformed UTF-8. This + * maximizes program robustness. If we get an invalid lookup result, report + * a configuration error. + * + * LOOKUP + * + * If the key is invalid, log a warning and skip the request. Rationale: the + * item cannot exist. + * + * If the lookup result is invalid, log a warning and return a configuration + * error. + * + * UPDATE + * + * If the key is invalid, then log a warning and skip the request. Rationale: + * the item cannot exist. + * + * If the value is invalid, log a warning and skip the request. Rationale: + * storage may not permit malformed UTF-8. This maximizes program + * robustness. + * + * DELETE + * + * If the key is invalid, then skip the request. Rationale: the item cannot + * exist. + */ + +/* dict_utf8_check_fold - casefold or validate string */ + +char *dict_utf8_check_fold(DICT *dict, const char *string, + CONST_CHAR_STAR *err) +{ + int fold_flag = (dict->flags & DICT_FLAG_FOLD_ANY); + + /* + * Casefold and implicitly validate UTF-8. + */ + if (fold_flag != 0 && (fold_flag == (dict->flags & DICT_FLAG_FIXED) ? + DICT_FLAG_FOLD_FIX : DICT_FLAG_FOLD_MUL)) { + if (dict->fold_buf == 0) + dict->fold_buf = vstring_alloc(10); + return (casefold(dict->fold_buf, string, err)); + } + + /* + * Validate UTF-8 without casefolding. + */ + if (!allascii(string) && valid_utf8_string(string, strlen(string)) == 0) { + if (err) + *err = "malformed UTF-8 or invalid codepoint"; + return (0); + } + return ((char *) string); +} + +/* dict_utf8_check validate UTF-8 string */ + +int dict_utf8_check(const char *string, CONST_CHAR_STAR *err) +{ + if (!allascii(string) && valid_utf8_string(string, strlen(string)) == 0) { + if (err) + *err = "malformed UTF-8 or invalid codepoint"; + return (0); + } + return (1); +} + +/* dict_utf8_lookup - UTF-8 lookup method wrapper */ + +static const char *dict_utf8_lookup(DICT *self, const char *key) +{ + DICT *dict; + const char *utf8_err; + const char *fold_res; + const char *value; + + /* + * Validate and optionally fold the key, and if invalid skip the request. + */ + if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) { + msg_warn("%s:%s: non-UTF-8 key \"%s\": %s", + self->type, self->name, key, utf8_err); + self->error = DICT_ERR_NONE; + return (0); + } + + /* + * Proxy the request. + */ + dict = (void *) self - self->size; + dict->flags = self->flags; + value = dict->lookup(dict, fold_res); + self->flags = dict->flags; + self->error = dict->error; + + /* + * Validate the result, and if invalid fail the request. + */ + if (value != 0 && dict_utf8_check(value, &utf8_err) == 0) { + msg_warn("%s:%s: key \"%s\": non-UTF-8 value \"%s\": %s", + self->type, self->name, key, value, utf8_err); + self->error = DICT_ERR_CONFIG; + return (0); + } else { + return (value); + } +} + +/* dict_utf8_update - UTF-8 update method wrapper */ + +static int dict_utf8_update(DICT *self, const char *key, const char *value) +{ + DICT *dict; + const char *utf8_err; + const char *fold_res; + int status; + + /* + * Validate or fold the key, and if invalid skip the request. + */ + if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) { + msg_warn("%s:%s: non-UTF-8 key \"%s\": %s", + self->type, self->name, key, utf8_err); + self->error = DICT_ERR_NONE; + return (DICT_STAT_SUCCESS); + } + + /* + * Validate the value, and if invalid skip the request. + */ + else if (dict_utf8_check(value, &utf8_err) == 0) { + msg_warn("%s:%s: key \"%s\": non-UTF-8 value \"%s\": %s", + self->type, self->name, key, value, utf8_err); + self->error = DICT_ERR_NONE; + return (DICT_STAT_SUCCESS); + } + + /* + * Proxy the request. + */ + else { + dict = (void *) self - self->size; + dict->flags = self->flags; + status = dict->update(dict, fold_res, value); + self->flags = dict->flags; + self->error = dict->error; + return (status); + } +} + +/* dict_utf8_delete - UTF-8 delete method wrapper */ + +static int dict_utf8_delete(DICT *self, const char *key) +{ + DICT *dict; + const char *utf8_err; + const char *fold_res; + int status; + + /* + * Validate and optionally fold the key, and if invalid skip the request. + */ + if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) { + msg_warn("%s:%s: non-UTF-8 key \"%s\": %s", + self->type, self->name, key, utf8_err); + self->error = DICT_ERR_NONE; + return (DICT_STAT_SUCCESS); + } + + /* + * Proxy the request. + */ + else { + dict = (void *) self - self->size; + dict->flags = self->flags; + status = dict->delete(dict, fold_res); + self->flags = dict->flags; + self->error = dict->error; + return (status); + } +} + +/* dict_utf8_close - dummy */ + +static void dict_utf8_close(DICT *self) +{ + DICT *dict; + + /* + * Destroy the dict object that we are appended to, and thereby destroy + * ourselves. + */ + dict = (void *) self - self->size; + dict->close(dict); +} + +/* dict_utf8_encapsulate - wrap a legacy dict object for UTF-8 processing */ + +DICT *dict_utf8_encapsulate(DICT *dict) +{ + DICT *self; + + /* + * Sanity check. + */ + if (dict->flags & DICT_FLAG_UTF8_PROXY) + msg_panic("dict_utf8_encapsulate: %s:%s is already encapsulated", + dict->type, dict->name); + + /* + * Append ourselves to the dict object, so that dict_close(dict) will do + * the right thing. dict->size is based on the actual size of the dict + * object's subclass, so we don't have to worry about alignment problems. + * + * XXX Add dict_flags argument to dict_alloc() so that it can allocate the + * right memory amount, and we can avoid having to resize an object. + */ + dict = myrealloc(dict, dict->size + sizeof(*self)); + self = (void *) dict + dict->size; + *self = *dict; + + /* + * Interpose on the lookup/update/delete/close methods. In particular we + * do not interpose on the iterator. Invalid keys are not stored, and we + * want to be able to delete an invalid value. + */ + self->lookup = dict_utf8_lookup; + self->update = dict_utf8_update; + self->delete = dict_utf8_delete; + self->close = dict_utf8_close; + + /* + * Finally, disable casefolding in the dict object. It now happens in the + * lookup/update/delete wrappers. + */ + dict->flags &= ~DICT_FLAG_FOLD_ANY; + self->flags |= DICT_FLAG_UTF8_PROXY; + + return (self); +} diff --git a/postfix/src/util/dict_utf8_test.in b/postfix/src/util/dict_utf8_test.in new file mode 100644 index 000000000..9f7743a4e --- /dev/null +++ b/postfix/src/util/dict_utf8_test.in @@ -0,0 +1,12 @@ +#!/bin/sh + +awk 'BEGIN { + print "flags" + print "verbose" + printf "get foo\n" + printf "put %c%c%c xxx\n", 128, 128, 128 + printf "get %c%c%c\n", 128, 128, 128 + printf "put xxx %c%c%c\n", 128, 128, 128 + printf "get xxx\n" + exit +}' | ./dict_open internal:whatever write utf8_enable diff --git a/postfix/src/util/dict_utf8_test.ref b/postfix/src/util/dict_utf8_test.ref new file mode 100644 index 000000000..2a13513d8 --- /dev/null +++ b/postfix/src/util/dict_utf8_test.ref @@ -0,0 +1,15 @@ +owner=trusted (uid=2147483647) +> flags +dict flags fixed|lock|replace|utf8_enable|utf8_proxy +> verbose +> get foo +foo: not found +> put €€€ xxx +./dict_open: warning: internal:whatever: non-UTF-8 key "???": malformed UTF-8 or invalid codepoint +> get €€€ +./dict_open: warning: internal:whatever: non-UTF-8 key "???": malformed UTF-8 or invalid codepoint +€€€: not found +> put xxx €€€ +./dict_open: warning: internal:whatever: key "xxx": non-UTF-8 value "???": malformed UTF-8 or invalid codepoint +> get xxx +xxx: not found diff --git a/postfix/src/util/host_port.c b/postfix/src/util/host_port.c index defa3b455..c4e86166d 100644 --- a/postfix/src/util/host_port.c +++ b/postfix/src/util/host_port.c @@ -91,7 +91,7 @@ #include #include -#include /* XXX temp_utf8_kludge */ +#include /* XXX util_utf8_enable */ #include /* Global library. */ @@ -158,7 +158,7 @@ const char *host_port(char *buf, char **host, char *def_host, * network addresses instead of requiring proper [ipaddress] forms. */ if (*host != def_host - && !valid_utf8_hostname(temp_utf8_kludge, *host, DONT_GRIPE) + && !valid_utf8_hostname(util_utf8_enable, *host, DONT_GRIPE) && !valid_hostaddr(*host, DONT_GRIPE)) return ("valid hostname or network address required"); if (*port != def_service && ISDIGIT(**port) && !alldig(*port)) diff --git a/postfix/src/util/match_list.c b/postfix/src/util/match_list.c index 7bf88bd58..714d474a5 100644 --- a/postfix/src/util/match_list.c +++ b/postfix/src/util/match_list.c @@ -107,7 +107,8 @@ static ARGV *match_list_parse(ARGV *list, char *string, int init_match) int match; #define OPEN_FLAGS O_RDONLY -#define DICT_FLAGS (DICT_FLAG_LOCK | DICT_FLAG_FOLD_FIX) +#define DICT_FLAGS (DICT_FLAG_LOCK | DICT_FLAG_FOLD_FIX \ + | DICT_FLAG_UTF8_ENABLE) #define STR(x) vstring_str(x) /* diff --git a/postfix/src/util/midna.c b/postfix/src/util/midna.c deleted file mode 100644 index 0f4118ee1..000000000 --- a/postfix/src/util/midna.c +++ /dev/null @@ -1,335 +0,0 @@ -/*++ -/* NAME -/* midna 3 -/* SUMMARY -/* Postfix domain name conversion -/* SYNOPSIS -/* #include -/* -/* int midna_cache_size; -/* -/* const char *midna_to_ascii( -/* const char *name) -/* -/* const char *midna_to_utf8( -/* const char *name) -/* -/* const char *midna_suffix_to_ascii( -/* const char *name) -/* -/* const char *midna_suffix_to_utf8( -/* const char *name) -/* DESCRIPTION -/* The functions in this module transform domain names from -/* or to IDNA form. The result is cached to avoid repeated -/* conversion of the same name. -/* -/* midna_to_ascii() converts an UTF-8 or ASCII domain name to -/* ASCII. The result is a null pointer in case of error. This -/* function verifies that the result is a valid ASCII domainname. -/* -/* midna_to_utf8() converts an UTF-8 or ASCII domain name to -/* UTF-8. The result is a null pointer in case of error. This -/* function verifies that the result converts to a valid ASCII -/* domainname. -/* -/* midna_suffix_to_ascii() and midna_suffix_to_utf8() take a -/* name that starts with '.' and otherwise perform the same -/* operations as midna_to_ascii() and midna_to_utf8(). -/* -/* midna_cache_size specifies the size of the conversion result -/* cache. This value is used only once, upon the first lookup -/* request. -/* SEE ALSO -/* msg(3) diagnostics interface -/* DIAGNOSTICS -/* Fatal errors: memory allocation problem. -/* Warnings: conversion error or result validation error. -/* LICENSE -/* .ad -/* .fi -/* The Secure Mailer license must be distributed with this software. -/* AUTHOR(S) -/* Arnt Gulbrandsen -/* -/* Wietse Venema -/* IBM T.J. Watson Research -/* P.O. Box 704 -/* Yorktown Heights, NY 10598, USA -/*--*/ - - /* - * System library. - */ -#include -#include -#include - -#ifndef NO_EAI -#include - - /* - * Utility library. - */ -#include -#include -#include -#include -#include -#include - - /* - * Application-specific. - */ -#define DEF_MIDNA_CACHE_SIZE 256 - -int midna_cache_size = DEF_MIDNA_CACHE_SIZE; -static VSTRING *midna_buf; /* x.suffix */ - -#define STR(x) vstring_str(x) - -/* midna_to_ascii_create - convert domain to ASCII */ - -static void *midna_to_ascii_create(const char *name, void *unused_context) -{ - static const char myname[] = "midna_to_ascii_create"; - char buf[1024]; /* XXX */ - UErrorCode error = U_ZERO_ERROR; - UIDNAInfo info = UIDNA_INFO_INITIALIZER; - UIDNA *idna; - int anl; - - /* - * Paranoia: do not expose uidna_*() to unfiltered network data. - */ - if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { - msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", - myname, name, "malformed UTF-8"); - return (0); - } - - /* - * Perform the requested conversion. - */ - idna = uidna_openUTS46(UIDNA_DEFAULT, &error); - anl = uidna_nameToASCII_UTF8(idna, - name, strlen(name), - buf, sizeof(buf), - &info, - &error); - uidna_close(idna); - - /* - * Paranoia: verify that the result is a valid ASCII domain name. A quick - * check shows that the UTS46 implementation will reject labels that - * start or end in '-' or that are over-long, but let's play safe here. - */ - if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { - buf[anl] = 0; /* XXX */ - if (!valid_hostname(buf, DONT_GRIPE)) { - msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", - myname, name, "malformed ASCII label(s)"); - return (0); - } - return (mystrndup(buf, anl)); - } else { - msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s", - myname, name, u_errorName(error)); - return (0); - } -} - -/* midna_to_utf8_create - convert domain to UTF8 */ - -static void *midna_to_utf8_create(const char *name, void *unused_context) -{ - static const char myname[] = "midna_to_utf8_create"; - char buf[1024]; /* XXX */ - UErrorCode error = U_ZERO_ERROR; - UIDNAInfo info = UIDNA_INFO_INITIALIZER; - UIDNA *idna; - int anl; - - /* - * Paranoia: do not expose uidna_*() to unfiltered network data. - */ - if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { - msg_warn("%s: Problem translating domain \"%s\" to UTF-8 form: %s", - myname, name, "malformed UTF-8"); - return (0); - } - - /* - * Perform the requested conversion. - */ - idna = uidna_openUTS46(UIDNA_DEFAULT, &error); - anl = uidna_nameToUnicodeUTF8(idna, - name, strlen(name), - buf, sizeof(buf), - &info, - &error); - uidna_close(idna); - - /* - * Paranoia: UTS46 toUTF8 will accept and produce a name that does not - * convert to a valid ASCII domain name. So we enforce sanity here. - */ - if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { - buf[anl] = 0; /* XXX */ - if (midna_to_ascii(buf) == 0) - return (0); - return (mystrndup(buf, anl)); - } else { - msg_warn("%s: Problem translating domain \"%s\" to UTF8 form: %s", - myname, name, u_errorName(error)); - return (0); - } -} - -/* midna_cache_free - cache element destructor */ - -static void midna_cache_free(void *value, void *unused_context) -{ - if (value) - myfree(value); -} - -/* midna_to_ascii - convert name to ASCII */ - -const char *midna_to_ascii(const char *name) -{ - static CTABLE *midna_to_ascii_cache = 0; - - if (midna_to_ascii_cache == 0) - midna_to_ascii_cache = ctable_create(midna_cache_size, - midna_to_ascii_create, - midna_cache_free, - (void *) 0); - return (ctable_locate(midna_to_ascii_cache, name)); -} - -/* midna_to_utf8 - convert name to UTF8 */ - -const char *midna_to_utf8(const char *name) -{ - static CTABLE *midna_to_utf8_cache = 0; - - if (midna_to_utf8_cache == 0) - midna_to_utf8_cache = ctable_create(midna_cache_size, - midna_to_utf8_create, - midna_cache_free, - (void *) 0); - return (ctable_locate(midna_to_utf8_cache, name)); -} - -/* midna_suffix_to_ascii - convert .name to ASCII */ - -const char *midna_suffix_to_ascii(const char *suffix) -{ - const char *cache_res; - - /* - * If prepending x to .name causes the result to become too long, then - * the suffix is bad. - */ - if (midna_buf == 0) - midna_buf = vstring_alloc(100); - vstring_sprintf(midna_buf, "x%s", suffix); - if ((cache_res = midna_to_ascii(STR(midna_buf))) == 0) - return (0); - else - return (cache_res + 1); -} - -/* midna_suffix_to_utf8 - convert .name to UTF8 */ - -const char *midna_suffix_to_utf8(const char *name) -{ - const char *cache_res; - - /* - * If prepending x to .name causes the result to become too long, then - * the suffix is bad. - */ - if (midna_buf == 0) - midna_buf = vstring_alloc(100); - vstring_sprintf(midna_buf, "x%s", name); - if ((cache_res = midna_to_utf8(STR(midna_buf))) == 0) - return (0); - else - return (cache_res + 1); -} - -#ifdef TEST - - /* - * Test program - reads names from stdin, reports invalid names to stderr. - */ -#include -#include - -#include /* XXX temp_utf8_kludge */ -#include -#include -#include -#include - -int main(int argc, char **argv) -{ - VSTRING *buffer = vstring_alloc(1); - const char *bp; - const char *ascii; - const char *utf8; - - if (setlocale(LC_ALL, "C") == 0) - msg_fatal("setlocale(LC_ALL, C) failed: %m"); - - msg_vstream_init(argv[0], VSTREAM_ERR); - msg_verbose = 1; - temp_utf8_kludge = 1; - - while (vstring_fgets_nonl(buffer, VSTREAM_IN)) { - bp = STR(buffer); - msg_info("> %s", bp); - while (ISSPACE(*bp)) - bp++; - if (*bp == '#' || *bp == 0) - continue; - if (!allascii(bp)) { - utf8 = midna_to_utf8(bp); - if (utf8 != 0) - msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8); - ascii = midna_to_ascii(bp); - if (ascii != 0) { - msg_info("\"%s\" ->ascii \"%s\"", bp, ascii); - utf8 = midna_to_utf8(ascii); - if (utf8 != 0) { - msg_info("\"%s\" ->ascii \"%s\" ->utf8 \"%s\"", - bp, ascii, utf8); - if (strcmp(utf8, bp) != 0) - msg_warn("\"%s\" != \"%s\"", bp, utf8); - } - } - } else { - ascii = midna_to_ascii(bp); - if (ascii != 0) - msg_info("\"%s\" ->ascii \"%s\"", bp, ascii); - utf8 = midna_to_utf8(bp); - if (utf8 != 0) { - msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8); - ascii = midna_to_ascii(utf8); - if (ascii != 0) { - msg_info("\"%s\" ->utf8 \"%s\" ->ascii \"%s\"", - bp, utf8, ascii); - if (strcmp(ascii, bp) != 0) - msg_warn("\"%s\" != \"%s\"", bp, ascii); - } - } - } - } - exit(0); -} - -#endif /* TEST */ - -#endif /* NO_EAI */ diff --git a/postfix/src/util/midna_domain.c b/postfix/src/util/midna_domain.c new file mode 100644 index 000000000..1563cac0e --- /dev/null +++ b/postfix/src/util/midna_domain.c @@ -0,0 +1,345 @@ +/*++ +/* NAME +/* midna_domain 3 +/* SUMMARY +/* ASCII/UTF-8 domain name conversion +/* SYNOPSIS +/* #include +/* +/* int midna_domain_cache_size; +/* +/* const char *midna_domain_to_ascii( +/* const char *name) +/* +/* const char *midna_domain_to_utf8( +/* const char *name) +/* +/* const char *midna_domain_suffix_to_ascii( +/* const char *name) +/* +/* const char *midna_domain_suffix_to_utf8( +/* const char *name) +/* DESCRIPTION +/* The functions in this module transform domain names from/to +/* ASCII and UTF-8 form. The result is cached to avoid repeated +/* conversion. +/* +/* This module builds on the ICU library implementation of the +/* UTS #46 specification, using default ICU library options +/* because those are likely best tested: with transitional +/* processing, with case mapping, with normalization, with +/* limited IDNA2003 compatibility, without STD3 ASCII rules. +/* +/* midna_domain_to_ascii() converts an UTF-8 or ASCII domain +/* name to ASCII. The result is a null pointer in case of +/* error. This function verifies that the result passes +/* valid_hostname(). +/* +/* midna_domain_to_utf8() converts an UTF-8 or ASCII domain +/* name to UTF-8. The result is a null pointer in case of +/* error. This function verifies that the result, after +/* conversion to ASCII, passes valid_hostname(). +/* +/* midna_domain_suffix_to_ascii() and midna_domain_suffix_to_utf8() +/* take a name that starts with '.' and otherwise perform the +/* same operations as midna_domain_to_ascii() and +/* midna_domain_to_utf8(). +/* +/* midna_domain_cache_size specifies the size of the conversion +/* result cache. This value is used only once, upon the first +/* lookup +/* request. +/* SEE ALSO +/* http://unicode.org/reports/tr46/ Unicode IDNA Compatibility processing +/* msg(3) diagnostics interface +/* DIAGNOSTICS +/* Fatal errors: memory allocation problem. +/* Warnings: conversion error or result validation error. +/* LICENSE +/* .ad +/* .fi +/* The Secure Mailer license must be distributed with this software. +/* AUTHOR(S) +/* Arnt Gulbrandsen +/* +/* Wietse Venema +/* IBM T.J. Watson Research +/* P.O. Box 704 +/* Yorktown Heights, NY 10598, USA +/*--*/ + + /* + * System library. + */ +#include +#include +#include + +#ifndef NO_EAI +#include + + /* + * Utility library. + */ +#include +#include +#include +#include +#include +#include + + /* + * Application-specific. + */ +#define DEF_MIDNA_CACHE_SIZE 256 + +int midna_domain_cache_size = DEF_MIDNA_CACHE_SIZE; +static VSTRING *midna_domain_buf; /* x.suffix */ + +#define STR(x) vstring_str(x) + +/* midna_domain_to_ascii_create - convert domain to ASCII */ + +static void *midna_domain_to_ascii_create(const char *name, void *unused_context) +{ + static const char myname[] = "midna_domain_to_ascii_create"; + char buf[1024]; /* XXX */ + UErrorCode error = U_ZERO_ERROR; + UIDNAInfo info = UIDNA_INFO_INITIALIZER; + UIDNA *idna; + int anl; + + /* + * Paranoia: do not expose uidna_*() to unfiltered network data. + */ + if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { + msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s", + myname, name, "malformed UTF-8"); + return (0); + } + + /* + * Perform the requested conversion. + */ + idna = uidna_openUTS46(UIDNA_DEFAULT, &error);/* XXX check error */ + anl = uidna_nameToASCII_UTF8(idna, + name, strlen(name), + buf, sizeof(buf) - 1, + &info, + &error); + uidna_close(idna); + + /* + * Paranoia: verify that the result passes valid_hostname(). A quick + * check shows that UTS46 ToASCII by default rejects inputs with labels + * that start or end in '-', with names or labels that are over-long, or + * "fake" A-labels, as required by UTS 46 section 4.1, but we rely on + * valid_hostname() on the output side just to be sure. + */ + if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { + buf[anl] = 0; /* XXX */ + if (!valid_hostname(buf, DONT_GRIPE)) { + msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s", + myname, name, "malformed ASCII label(s)"); + return (0); + } + return (mystrndup(buf, anl)); + } else { + msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s", + myname, name, u_errorName(info.errors)); + return (0); + } +} + +/* midna_domain_to_utf8_create - convert domain to UTF8 */ + +static void *midna_domain_to_utf8_create(const char *name, void *unused_context) +{ + static const char myname[] = "midna_domain_to_utf8_create"; + char buf[1024]; /* XXX */ + UErrorCode error = U_ZERO_ERROR; + UIDNAInfo info = UIDNA_INFO_INITIALIZER; + UIDNA *idna; + int anl; + + /* + * Paranoia: do not expose uidna_*() to unfiltered network data. + */ + if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) { + msg_warn("%s: Problem translating domain \"%.100s\" to UTF-8 form: %s", + myname, name, "malformed UTF-8"); + return (0); + } + + /* + * Perform the requested conversion. + */ + idna = uidna_openUTS46(UIDNA_DEFAULT, &error);/* XXX check error */ + anl = uidna_nameToUnicodeUTF8(idna, + name, strlen(name), + buf, sizeof(buf) - 1, + &info, + &error); + uidna_close(idna); + + /* + * Paranoia: UTS46 toUTF8 by default accepts and produces an over-long + * name or a name that contains an over-long NR-LDH label (and perhaps + * other invalid forms that are not covered in UTS 46, section 4.1). We + * rely on midna_domain_to_ascii() to validate the output. + */ + if (U_SUCCESS(error) && info.errors == 0 && anl > 0) { + buf[anl] = 0; /* XXX */ + if (midna_domain_to_ascii(buf) == 0) + return (0); + return (mystrndup(buf, anl)); + } else { + msg_warn("%s: Problem translating domain \"%.100s\" to UTF8 form: %s", + myname, name, u_errorName(info.errors)); + return (0); + } +} + +/* midna_domain_cache_free - cache element destructor */ + +static void midna_domain_cache_free(void *value, void *unused_context) +{ + if (value) + myfree(value); +} + +/* midna_domain_to_ascii - convert name to ASCII */ + +const char *midna_domain_to_ascii(const char *name) +{ + static CTABLE *midna_domain_to_ascii_cache = 0; + + if (midna_domain_to_ascii_cache == 0) + midna_domain_to_ascii_cache = ctable_create(midna_domain_cache_size, + midna_domain_to_ascii_create, + midna_domain_cache_free, + (void *) 0); + return (ctable_locate(midna_domain_to_ascii_cache, name)); +} + +/* midna_domain_to_utf8 - convert name to UTF8 */ + +const char *midna_domain_to_utf8(const char *name) +{ + static CTABLE *midna_domain_to_utf8_cache = 0; + + if (midna_domain_to_utf8_cache == 0) + midna_domain_to_utf8_cache = ctable_create(midna_domain_cache_size, + midna_domain_to_utf8_create, + midna_domain_cache_free, + (void *) 0); + return (ctable_locate(midna_domain_to_utf8_cache, name)); +} + +/* midna_domain_suffix_to_ascii - convert .name to ASCII */ + +const char *midna_domain_suffix_to_ascii(const char *suffix) +{ + const char *cache_res; + + /* + * If prepending x to .name causes the result to become too long, then + * the suffix is bad. + */ + if (midna_domain_buf == 0) + midna_domain_buf = vstring_alloc(100); + vstring_sprintf(midna_domain_buf, "x%s", suffix); + if ((cache_res = midna_domain_to_ascii(STR(midna_domain_buf))) == 0) + return (0); + else + return (cache_res + 1); +} + +/* midna_domain_suffix_to_utf8 - convert .name to UTF8 */ + +const char *midna_domain_suffix_to_utf8(const char *name) +{ + const char *cache_res; + + /* + * If prepending x to .name causes the result to become too long, then + * the suffix is bad. + */ + if (midna_domain_buf == 0) + midna_domain_buf = vstring_alloc(100); + vstring_sprintf(midna_domain_buf, "x%s", name); + if ((cache_res = midna_domain_to_utf8(STR(midna_domain_buf))) == 0) + return (0); + else + return (cache_res + 1); +} + +#ifdef TEST + + /* + * Test program - reads names from stdin, reports invalid names to stderr. + */ +#include +#include + +#include /* XXX util_utf8_enable */ +#include +#include +#include +#include + +int main(int argc, char **argv) +{ + VSTRING *buffer = vstring_alloc(1); + const char *bp; + const char *ascii; + const char *utf8; + + if (setlocale(LC_ALL, "C") == 0) + msg_fatal("setlocale(LC_ALL, C) failed: %m"); + + msg_vstream_init(argv[0], VSTREAM_ERR); + /* msg_verbose = 1; */ + util_utf8_enable = 1; + + while (vstring_fgets_nonl(buffer, VSTREAM_IN)) { + bp = STR(buffer); + msg_info("> %s", bp); + while (ISSPACE(*bp)) + bp++; + if (*bp == '#' || *bp == 0) + continue; + msg_info("unconditional conversions:"); + utf8 = midna_domain_to_utf8(bp); + msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8 ? utf8 : "(error)"); + ascii = midna_domain_to_ascii(bp); + msg_info("\"%s\" ->ascii \"%s\"", bp, ascii ? ascii : "(error)"); + msg_info("conditional conversions:"); + if (!allascii(bp)) { + if (ascii != 0) { + utf8 = midna_domain_to_utf8(ascii); + msg_info("\"%s\" ->ascii \"%s\" ->utf8 \"%s\"", + bp, ascii, utf8 ? utf8 : "(error)"); + if (utf8 != 0) { + if (strcmp(utf8, bp) != 0) + msg_warn("\"%s\" != \"%s\"", bp, utf8); + } + } + } else { + if (utf8 != 0) { + ascii = midna_domain_to_ascii(utf8); + msg_info("\"%s\" ->utf8 \"%s\" ->ascii \"%s\"", + bp, utf8, ascii ? ascii : "(error)"); + if (ascii != 0) { + if (strcmp(ascii, bp) != 0) + msg_warn("\"%s\" != \"%s\"", bp, ascii); + } + } + } + } + exit(0); +} + +#endif /* TEST */ + +#endif /* NO_EAI */ diff --git a/postfix/src/util/midna.h b/postfix/src/util/midna_domain.h similarity index 54% rename from postfix/src/util/midna.h rename to postfix/src/util/midna_domain.h index 599097832..29cfc8c84 100644 --- a/postfix/src/util/midna.h +++ b/postfix/src/util/midna_domain.h @@ -3,21 +3,21 @@ /*++ /* NAME -/* mail_idna 3h +/* midna_domain 3h /* SUMMARY -/* domain name conversion +/* ASCII/UTF-8 domain name conversion /* SYNOPSIS -/* #include +/* #include /* DESCRIPTION /* .nf /* * External interface. */ -extern const char *midna_to_ascii(const char *); -extern const char *midna_to_utf8(const char *); -extern const char *midna_suffix_to_ascii(const char *); -extern const char *midna_suffix_to_utf8(const char *); +extern const char *midna_domain_to_ascii(const char *); +extern const char *midna_domain_to_utf8(const char *); +extern const char *midna_domain_suffix_to_ascii(const char *); +extern const char *midna_domain_suffix_to_utf8(const char *); /* LICENSE /* .ad diff --git a/postfix/src/util/midna_domain_test.in b/postfix/src/util/midna_domain_test.in new file mode 100644 index 000000000..55491e41f --- /dev/null +++ b/postfix/src/util/midna_domain_test.in @@ -0,0 +1,21 @@ +# Upper-case greek -> lower-case greek. +Δημοσθένους.example.com +# Upper-case ASCII -> lower-case ASCII. +Hello.example.com +# Invalid LDH label('-' at begin or end). +bad-.example.com +-bad.example.com +# Invalid LDH (label > 63 bytes). +abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com +# Valid LDH label (label <= 63 bytes). +abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com +# Invalid name (length > 255 bytes). +abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com +# Aliases for '.' -> '.'. +x。example.com +x．example.com +x｡example.com +# Good a-label. +xn--mumble.example.com +# Bad a-label. +xn--123456.example.com diff --git a/postfix/src/util/midna_domain_test.ref b/postfix/src/util/midna_domain_test.ref new file mode 100644 index 000000000..17e4fcc68 --- /dev/null +++ b/postfix/src/util/midna_domain_test.ref @@ -0,0 +1,89 @@ +./midna_domain: > # Upper-case greek -> lower-case greek. +./midna_domain: > Δημοσθένους.example.com +./midna_domain: unconditional conversions: +./midna_domain: "Δημοσθένους.example.com" ->utf8 "δημοσθένουσ.example.com" +./midna_domain: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" +./midna_domain: conditional conversions: +./midna_domain: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" ->utf8 "δημοσθένουσ.example.com" +./midna_domain: warning: "Δημοσθένους.example.com" != "δημοσθένουσ.example.com" +./midna_domain: > # Upper-case ASCII -> lower-case ASCII. +./midna_domain: > Hello.example.com +./midna_domain: unconditional conversions: +./midna_domain: "Hello.example.com" ->utf8 "hello.example.com" +./midna_domain: "Hello.example.com" ->ascii "hello.example.com" +./midna_domain: conditional conversions: +./midna_domain: "Hello.example.com" ->utf8 "hello.example.com" ->ascii "hello.example.com" +./midna_domain: warning: "Hello.example.com" != "hello.example.com" +./midna_domain: > # Invalid LDH label('-' at begin or end). +./midna_domain: > bad-.example.com +./midna_domain: unconditional conversions: +./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "bad-.example.com" to UTF8 form: U_UNSUPPORTED_ERROR +./midna_domain: "bad-.example.com" ->utf8 "(error)" +./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "bad-.example.com" to ASCII form: U_UNSUPPORTED_ERROR +./midna_domain: "bad-.example.com" ->ascii "(error)" +./midna_domain: conditional conversions: +./midna_domain: > -bad.example.com +./midna_domain: unconditional conversions: +./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "-bad.example.com" to UTF8 form: U_INDEX_OUTOFBOUNDS_ERROR +./midna_domain: "-bad.example.com" ->utf8 "(error)" +./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "-bad.example.com" to ASCII form: U_INDEX_OUTOFBOUNDS_ERROR +./midna_domain: "-bad.example.com" ->ascii "(error)" +./midna_domain: conditional conversions: +./midna_domain: > # Invalid LDH (label > 63 bytes). +./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com +./midna_domain: unconditional conversions: +./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" to ASCII form: U_MISSING_RESOURCE_ERROR +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" ->utf8 "(error)" +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" ->ascii "(error)" +./midna_domain: conditional conversions: +./midna_domain: > # Valid LDH label (label <= 63 bytes). +./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com +./midna_domain: unconditional conversions: +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->utf8 "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->ascii "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" +./midna_domain: conditional conversions: +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->utf8 "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->ascii "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" +./midna_domain: > # Invalid name (length > 255 bytes). +./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com +./midna_domain: unconditional conversions: +./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcde" to ASCII form: U_FILE_ACCESS_ERROR +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com" ->utf8 "(error)" +./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com" ->ascii "(error)" +./midna_domain: conditional conversions: +./midna_domain: > # Aliases for '.' -> '.'. +./midna_domain: > x。example.com +./midna_domain: unconditional conversions: +./midna_domain: "x。example.com" ->utf8 "x.example.com" +./midna_domain: "x。example.com" ->ascii "x.example.com" +./midna_domain: conditional conversions: +./midna_domain: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna_domain: warning: "x。example.com" != "x.example.com" +./midna_domain: > x．example.com +./midna_domain: unconditional conversions: +./midna_domain: "x．example.com" ->utf8 "x.example.com" +./midna_domain: "x．example.com" ->ascii "x.example.com" +./midna_domain: conditional conversions: +./midna_domain: "x．example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna_domain: warning: "x．example.com" != "x.example.com" +./midna_domain: > x｡example.com +./midna_domain: unconditional conversions: +./midna_domain: "x｡example.com" ->utf8 "x.example.com" +./midna_domain: "x｡example.com" ->ascii "x.example.com" +./midna_domain: conditional conversions: +./midna_domain: "x｡example.com" ->ascii "x.example.com" ->utf8 "x.example.com" +./midna_domain: warning: "x｡example.com" != "x.example.com" +./midna_domain: > # Good a-label. +./midna_domain: > xn--mumble.example.com +./midna_domain: unconditional conversions: +./midna_domain: "xn--mumble.example.com" ->utf8 "㲹㲺㲵㲴.example.com" +./midna_domain: "xn--mumble.example.com" ->ascii "xn--mumble.example.com" +./midna_domain: conditional conversions: +./midna_domain: "xn--mumble.example.com" ->utf8 "㲹㲺㲵㲴.example.com" ->ascii "xn--mumble.example.com" +./midna_domain: > # Bad a-label. +./midna_domain: > xn--123456.example.com +./midna_domain: unconditional conversions: +./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "xn--123456.example.com" to UTF8 form: [BOGUS UErrorCode] +./midna_domain: "xn--123456.example.com" ->utf8 "(error)" +./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "xn--123456.example.com" to ASCII form: [BOGUS UErrorCode] +./midna_domain: "xn--123456.example.com" ->ascii "(error)" +./midna_domain: conditional conversions: diff --git a/postfix/src/util/midna_test.in b/postfix/src/util/midna_test.in deleted file mode 100644 index b309be965..000000000 --- a/postfix/src/util/midna_test.in +++ /dev/null @@ -1,14 +0,0 @@ -# Upper-case greek -> lower-case greek. -Δημοσθένους.example.com -# Upper-case ASCII -> lower-case ASCII. -Hello.example.com -# Invalid domain name ('-' at begin or end). --bad-.example.com -# Invalid domain name (label > 62 bytes). -abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com -# Valid domain name. -abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -# Aliases for '.' -> '.'. -x。example.com -x．example.com -x｡example.com diff --git a/postfix/src/util/midna_test.ref b/postfix/src/util/midna_test.ref deleted file mode 100644 index 1df1eb61e..000000000 --- a/postfix/src/util/midna_test.ref +++ /dev/null @@ -1,71 +0,0 @@ -./midna: > # Upper-case greek -> lower-case greek. -./midna: > Δημοσθένους.example.com -./midna: ctable_locate: install entry key δημοσθένουσ.example.com -./midna: ctable_locate: install entry key Δημοσθένους.example.com -./midna: "Δημοσθένους.example.com" ->utf8 "δημοσθένουσ.example.com" -./midna: ctable_locate: install entry key Δημοσθένους.example.com -./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" -./midna: ctable_locate: move existing entry key δημοσθένουσ.example.com -./midna: ctable_locate: install entry key xn--ixanjetild6aev.example.com -./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" ->utf8 "δημοσθένουσ.example.com" -./midna: warning: "Δημοσθένους.example.com" != "δημοσθένουσ.example.com" -./midna: > # Upper-case ASCII -> lower-case ASCII. -./midna: > Hello.example.com -./midna: ctable_locate: install entry key Hello.example.com -./midna: "Hello.example.com" ->ascii "hello.example.com" -./midna: ctable_locate: install entry key hello.example.com -./midna: ctable_locate: install entry key Hello.example.com -./midna: "Hello.example.com" ->utf8 "hello.example.com" -./midna: ctable_locate: leave existing entry key hello.example.com -./midna: "Hello.example.com" ->utf8 "hello.example.com" ->ascii "hello.example.com" -./midna: warning: "Hello.example.com" != "hello.example.com" -./midna: > # Invalid domain name ('-' at begin or end). -./midna: > -bad-.example.com -./midna: warning: midna_to_ascii_create: Problem translating domain "-bad-.example.com" to ASCII form: U_ZERO_ERROR -./midna: ctable_locate: install entry key -bad-.example.com -./midna: warning: midna_to_utf8_create: Problem translating domain "-bad-.example.com" to UTF8 form: U_ZERO_ERROR -./midna: ctable_locate: install entry key -bad-.example.com -./midna: > # Invalid domain name (label > 62 bytes). -./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com -./midna: warning: midna_to_ascii_create: Problem translating domain "abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com" to ASCII form: U_ZERO_ERROR -./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com -./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com -./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com -./midna: > # Valid domain name. -./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" -./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" -./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com -./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" -./midna: > # Aliases for '.' -> '.'. -./midna: > x。example.com -./midna: ctable_locate: install entry key x.example.com -./midna: ctable_locate: install entry key x。example.com -./midna: "x。example.com" ->utf8 "x.example.com" -./midna: ctable_locate: install entry key x。example.com -./midna: "x。example.com" ->ascii "x.example.com" -./midna: ctable_locate: move existing entry key x.example.com -./midna: ctable_locate: install entry key x.example.com -./midna: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com" -./midna: warning: "x。example.com" != "x.example.com" -./midna: > x．example.com -./midna: ctable_locate: leave existing entry key x.example.com -./midna: ctable_locate: install entry key x．example.com -./midna: "x．example.com" ->utf8 "x.example.com" -./midna: ctable_locate: install entry key x．example.com -./midna: "x．example.com" ->ascii "x.example.com" -./midna: ctable_locate: move existing entry key x.example.com -./midna: "x．example.com" ->ascii "x.example.com" ->utf8 "x.example.com" -./midna: warning: "x．example.com" != "x.example.com" -./midna: > x｡example.com -./midna: ctable_locate: move existing entry key x.example.com -./midna: ctable_locate: install entry key x｡example.com -./midna: "x｡example.com" ->utf8 "x.example.com" -./midna: ctable_locate: install entry key x｡example.com -./midna: "x｡example.com" ->ascii "x.example.com" -./midna: ctable_locate: move existing entry key x.example.com -./midna: "x｡example.com" ->ascii "x.example.com" ->utf8 "x.example.com" -./midna: warning: "x｡example.com" != "x.example.com" diff --git a/postfix/src/util/printable.c b/postfix/src/util/printable.c index 1ba0731c7..ff2ac63b3 100644 --- a/postfix/src/util/printable.c +++ b/postfix/src/util/printable.c @@ -6,7 +6,7 @@ /* SYNOPSIS /* #include /* -/* int temp_utf8_kludge; +/* int util_utf8_enable; /* /* char *printable(buffer, replacement) /* char *buffer; @@ -15,7 +15,7 @@ /* printable() replaces non-printable characters /* in its input with the given replacement. /* -/* temp_utf8_kludge controls whether UTF8 is considered printable. +/* util_utf8_enable controls whether UTF8 is considered printable. /* By default, non-ASCII text is replaced. /* /* Arguments: @@ -44,7 +44,7 @@ #include "stringops.h" -int temp_utf8_kludge = 0; +int util_utf8_enable = 0; char *printable(char *string, int replacement) { @@ -59,7 +59,7 @@ char *printable(char *string, int replacement) while ((ch = *cp) != 0) { if (ISASCII(ch) && ISPRINT(ch)) { /* ok */ - } else if (temp_utf8_kludge && ch >= 194 && ch <= 254 + } else if (util_utf8_enable && ch >= 194 && ch <= 254 && cp[1] >= 128 && cp[1] < 192) { /* UTF8; skip the rest of the bytes in the character. */ while (cp[1] >= 128 && cp[1] < 192) diff --git a/postfix/src/util/stringops.h b/postfix/src/util/stringops.h index 3d4d3b869..cac9eca55 100644 --- a/postfix/src/util/stringops.h +++ b/postfix/src/util/stringops.h @@ -19,10 +19,11 @@ /* * External interface. */ -extern int temp_utf8_kludge; +extern int util_utf8_enable; extern char *printable(char *, int); extern char *neuter(char *, const char *, int); extern char *lowercase(char *); +extern char *casefold(VSTRING *, const char *, CONST_CHAR_STAR *); extern char *uppercase(char *); extern char *skipblanks(const char *); extern char *trimblanks(char *, ssize_t); diff --git a/postfix/src/util/valid_utf8_hostname.c b/postfix/src/util/valid_utf8_hostname.c index f42e21f53..3d6922aa9 100644 --- a/postfix/src/util/valid_utf8_hostname.c +++ b/postfix/src/util/valid_utf8_hostname.c @@ -8,8 +8,8 @@ /* /* int valid_utf8_hostname( /* int enable_utf8, -/* const char *domain, -/* int gripe) +/* const char *domain, +/* int gripe) /* DESCRIPTION /* valid_utf8_hostname() is a wrapper around valid_hostname(). /* If EAI support is compiled in, and enable_utf8 is true, the @@ -43,7 +43,7 @@ #include #include #include -#include +#include #include /* valid_utf8_hostname - validate internationalized domain name */ @@ -51,8 +51,6 @@ int valid_utf8_hostname(int enable_utf8, const char *name, int gripe) { static const char myname[] = "valid_utf8_hostname"; - const char *aname; - int ret; /* * Trivial cases first. @@ -64,23 +62,26 @@ int valid_utf8_hostname(int enable_utf8, const char *name, int gripe) } /* - * Convert domain name to ASCII form. + * Convert non-ASCII domain name to ASCII and validate the result per + * STD3. midna_domain_to_ascii() applies valid_hostname() to the result. + * Propagate the gripe parameter for better diagnostics (note that + * midna_domain_to_ascii() logs a problem only when the result is not + * cached). */ #ifndef NO_EAI if (enable_utf8 && !allascii(name)) { - if ((aname = midna_to_ascii(name)) == 0) { + if (midna_domain_to_ascii(name) == 0) { if (gripe) msg_warn("%s: malformed UTF-8 domain name", myname); return (0); + } else { + return (1); } - } else + } #endif - aname = name; /* - * Validate the name per STD3 (if the IDNA routines didn't already). + * Validate ASCII name per STD3. */ - ret = valid_hostname(aname, gripe); - - return (ret); + return (valid_hostname(name, gripe)); }