either their result is a valid ASCII domain name or that
it converts into a valid ASCII domain name. Files:
util/midna.c, util/midna_test.in, util/midna_test.ref.
+
+20141230
+
+ Cleanup: s/midna/midna_domain/ for better specificity,
+ because we also need functions that act only on the domain
+ portion of an email address. Files: bounce/bounce_template.c,
+ global/midna_adomain.c, posttls-finger/posttls-finger.c,
+ smtp/smtp_addr.c, smtpd/smtpd_check.c, tls/tls_client.c,
+ util/midna_domain.[hc], util/valid_utf8_hostname.c.
+
+ Infrastructure: function midna_adomain_to_utf8() (and
+ midna_adomain_to_ascii) to convert the domain portion of
+ an email address before table lookup. Files:
+ global/midna_adomain.[hc].
+
+20141230-20140109
+
+ What is described here is the result of four iterations to
+ deal with malformed UTF-8 without massively contaminating
+ every Postfix program with new error-handling code paths,
+ in particular without triggering fatal errors that didn't
+ happen before.
+
+ Infrastructure: function casefold() to support caseless
+ string comparison, primarily for table lookups. This function
+ supports two modes: case folding a la lowercase() for ASCII
+ byte values, and UTF-8 case folding. As recommended at
+ http://www.w3.org/International/wiki/Case_folding for
+ caseless string comparison, this uses the en_US locale to
+ avoid surprises. The implementatin handles
+ the entire RFC 3629 Unicode range (code points U+0000..U+10FFFF
+ including surrogates) and is chroot(2) safe. Files: casefold.c, stringops.h.
+
+ Infrastructure: revised the midna_domain_to_ascii and
+ midna_domain_to_utf8 domain name conversion functions after
+ careful reading of the UTS #46 specification, and after
+ observing that ICU 4.8 library functions indeed implement
+ this spec, at least with default options. In particular,
+ midna_domain_to_utf8 takes an UTF-8 domain name and verifies
+ that its A-label form will pass the valid_hostname() test.
+ File: util/midna_domain.c.
+
+ Infrastructure: handle UTF-8 errors in lookup table keys
+ or values without massively contaminating every Postfix
+ program with new error-handling code paths, in particular
+ without triggering fatal errors that didn't happen before.
+ The lookup/update/delete functions log a warning and ignore
+ a request with a bad key (it cannot exist); the update
+ functions ignore a request to store a bad value (it cannot
+ exist); and the lookup function reports a bad value as a
+ configuration error (it should not exist, but there it is).
+ Table iterators still report all (key, value) pairs in a
+ table. Files: util/dict.h, util/dict_open.c, util/dict_utf8.c,
+ global/mkmap_open.c.
+
+ Note that with SMTPUTF8 turned on, each table-driven mechanism
+ (access, aliases, etc.) needs to make its own decision
+ whether UTF-8 syntax is required. We cannot blindly require
+ that everything has valid UTF-8 syntax. That would make
+ header/body_checks useless for content inspection, because
+ headers may be malformed and bodies may contain legitimate
+ binary content that isn't UTF-8.
+
+ Note that with SMTPUTF8 turned off, Postfix must remain
+ 8-bit clean as it always has been. Table operations must
+ not complain that something violates UTF-8 syntax rules.
+
+ UTF-8 sanitization in the Postfix SMTP server. With
+ smtputf8_enable=yes, SMTP commands with UTF-8 syntax errors
+ are rejected, table lookup results with invalid UTF-8 syntax
+ are handled as configuration errors, and UTF-8 syntax errors
+ in policy server replies result in execution of the policy
+ server's default action.
+
+20150102
+
+ Cleanup: propagate DICT_ERR_CONFIG through the proxymap
+ protocol. Files: global/dict_proxy.[hc], proxymap/proxymap.c.
+
+20150106
+
+ Robustness: don't segfault due to excessive recursion in
+ tok822_free_tree() after a faulty configuration runs into
+ the virtual_alias_recursion_limit. File: global/tok822_tree.c.
+
+20150109
+
+ Cleanup: the dict debug module now proxies dict flags.
+ File: util/dict_debug.c.
+
+ With "smtputf8_enable = yes", the postmap and postalias
+ commands now enable UTF-8 by default (use "-u" to disable)
+ with one exception: UTF-8 remains disabled for header/body_checks
+ emulation (use "-U" to enable). Files: postmap/postmap.c,
+ postalias/postalias.c.
+
+20150110
+
+ Cleanup: the "inline" and "texthash" implementations now
+ reuse the "internal" database instead of reinventing the
+ wheel. Files: util/dict_inline.c, util/dict_thash.c.
+
+ As a first step, with "smtputf8_enable = yes" all features
+ based on Postfix matchlists enable UTF-8 syntax checks and
+ casefolding support for table queries and results. That
+ includes mynetworks, mydestination, relay_domains,
+ virtual_alias_domains, and virtual_mailbox_domains.
+
+ The next step is to turn on UTF-8 syntax checks and casefolding
+ support for access maps, address rewriting and routing.
Things to do after the stable release:
+ Try to allow UTF-8 myhostname/mydomain, at least in bounce
+ template expansion.
+
+ No enhanced status code when rejecting connection before
+ the HELO handshake is completed.
+
+ Maybe don't whitelist a client that has maxed out its
+ per-MTA connection count limit.
+
Inline support for pcre:{/pattern/=action, ...} and ditto
support for regexp: and cidr: tables. Factor out and reuse
code that already exists in inline: and other tables.
postalias - Postfix alias database maintenance
<b>SYNOPSIS</b>
- <b>postalias</b> [<b>-Nfinoprsvw</b>] [<b>-c</b> <i>config</i><b>_</b><i>dir</i>] [<b>-d</b> <i>key</i>] [<b>-q</b> <i>key</i>]
+ <b>postalias</b> [<b>-Nfinoprsuvw</b>] [<b>-c</b> <i>config</i><b>_</b><i>dir</i>] [<b>-d</b> <i>key</i>] [<b>-q</b> <i>key</i>]
[<i>file</i><b>_</b><i>type</i>:]<i>file</i><b>_</b><i>name</i> ...
<b>DESCRIPTION</b>
order. This feature is available in Postfix version 2.2 and
later, and is not available for all database types.
+ <b>-u</b> Disable UTF-8 support. UTF-8 support is enabled by default when
+ "<a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> = yes". It requires that keys and values are
+ valid UTF-8 strings.
+
<b>-v</b> Enable verbose logging for debugging purposes. Multiple <b>-v</b>
options make the software increasingly verbose.
The default database type for use in <a href="newaliases.1.html"><b>newaliases</b>(1)</a>, <a href="postalias.1.html"><b>postalias</b>(1)</a>
and <a href="postmap.1.html"><b>postmap</b>(1)</a> commands.
+ <b><a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> (yes)</b>
+ Enable experimental SMTPUTF8 support for the protocols described
+ in <a href="http://tools.ietf.org/html/rfc6531">RFC 6531</a>..6533.
+
<b><a href="postconf.5.html#syslog_facility">syslog_facility</a> (mail)</b>
The syslog facility of Postfix logging.
<b><a href="postconf.5.html#syslog_name">syslog_name</a> (see 'postconf -d' output)</b>
- The mail system name that is prepended to the process name in
- syslog records, so that "smtpd" becomes, for example, "post-
+ The mail system name that is prepended to the process name in
+ syslog records, so that "smtpd" becomes, for example, "post-
fix/smtpd".
<b>STANDARDS</b>
<blockquote>
<pre>
/etc/postfix/transport:
- smtp-domain_that_verifies_after_data smtp-data-target:
- lmtp-domain_that_verifies_after_data lmtp-data-target:
+ smtp-domain-that-verifies-after-data smtp-data-target:
+ lmtp-domain-that-verifies-after-data lmtp-data-target:
</pre>
</blockquote>
postmap - Postfix lookup table management
<b>SYNOPSIS</b>
- <b>postmap</b> [<b>-Nbfhimnoprsvw</b>] [<b>-c</b> <i>config</i><b>_</b><i>dir</i>] [<b>-d</b> <i>key</i>] [<b>-q</b> <i>key</i>]
+ <b>postmap</b> [<b>-NbfhimnoprsuUvw</b>] [<b>-c</b> <i>config</i><b>_</b><i>dir</i>] [<b>-d</b> <i>key</i>] [<b>-q</b> <i>key</i>]
[<i>file</i><b>_</b><i>type</i>:]<i>file</i><b>_</b><i>name</i> ...
<b>DESCRIPTION</b>
style lookup keys for attachment MIME headers and for attached
message/* headers.
+ NOTE: with "<a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> = yes", the <b>-b</b> option option dis-
+ ables UTF-8 syntax checks on query keys and lookup results.
+ Specify the <b>-U</b> option to force UTF-8 syntax checks anyway.
+
This feature is available in Postfix version 2.6 and later.
<b>-c</b> <i>config</i><b>_</b><i>dir</i>
also generates header-style lookup keys for attachment MIME
headers and for attached message/* headers.
+ NOTE: with "<a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> = yes", the <b>-b</b> option option dis-
+ ables UTF-8 syntax checks on query keys and lookup results.
+ Specify the <b>-U</b> option to force UTF-8 syntax checks anyway.
+
This feature is available in Postfix version 2.6 and later.
<b>-i</b> Incremental mode. Read entries from standard input and do not
This feature is available in Postfix version 2.2 and later, and
is not available for all database types.
- <b>-v</b> Enable verbose logging for debugging purposes. Multiple <b>-v</b>
+ <b>-u</b> Disable UTF-8 support. UTF-8 support is enabled by default when
+ "<a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> = yes". It requires that keys and values are
+ valid UTF-8 strings.
+
+ <b>-U</b> With "<a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> = yes", force UTF-8 syntax checks with the
+ <b>-b</b> and <b>-h</b> options.
+
+ <b>-v</b> Enable verbose logging for debugging purposes. Multiple <b>-v</b>
options make the software increasingly verbose.
- <b>-w</b> When updating a table, do not complain about attempts to update
+ <b>-w</b> When updating a table, do not complain about attempts to update
existing entries, and ignore those attempts.
Arguments:
The <a href="postmap.1.html"><b>postmap</b>(1)</a> command can query any supported file type, but it
can create only the following file types:
- <b>btree</b> The output file is a btree file, named <i>file</i><b>_</b><i>name</i><b>.db</b>.
- This is available on systems with support for <b>db</b> data-
+ <b>btree</b> The output file is a btree file, named <i>file</i><b>_</b><i>name</i><b>.db</b>.
+ This is available on systems with support for <b>db</b> data-
bases.
- <b>cdb</b> The output consists of one file, named <i>file</i><b>_</b><i>name</i><b>.cdb</b>.
- This is available on systems with support for <b>cdb</b> data-
+ <b>cdb</b> The output consists of one file, named <i>file</i><b>_</b><i>name</i><b>.cdb</b>.
+ This is available on systems with support for <b>cdb</b> data-
bases.
<b>dbm</b> The output consists of two files, named <i>file</i><b>_</b><i>name</i><b>.pag</b> and
<i>file</i><b>_</b><i>name</i><b>.dir</b>. This is available on systems with support
for <b>dbm</b> databases.
- <b>hash</b> The output file is a hashed file, named <i>file</i><b>_</b><i>name</i><b>.db</b>.
- This is available on systems with support for <b>db</b> data-
+ <b>hash</b> The output file is a hashed file, named <i>file</i><b>_</b><i>name</i><b>.db</b>.
+ This is available on systems with support for <b>db</b> data-
bases.
- <b>fail</b> A table that reliably fails all requests. The lookup ta-
- ble name is used for logging only. This table exists to
+ <b>fail</b> A table that reliably fails all requests. The lookup ta-
+ ble name is used for logging only. This table exists to
simplify Postfix error tests.
<b>sdbm</b> The output consists of two files, named <i>file</i><b>_</b><i>name</i><b>.pag</b> and
<i>file</i><b>_</b><i>name</i><b>.dir</b>. This is available on systems with support
for <b>sdbm</b> databases.
- When no <i>file</i><b>_</b><i>type</i> is specified, the software uses the database
- type specified via the <b><a href="postconf.5.html#default_database_type">default_database_type</a></b> configuration
+ When no <i>file</i><b>_</b><i>type</i> is specified, the software uses the database
+ type specified via the <b><a href="postconf.5.html#default_database_type">default_database_type</a></b> configuration
parameter.
<i>file</i><b>_</b><i>name</i>
<b>DIAGNOSTICS</b>
Problems are logged to the standard error stream and to <b>syslogd</b>(8). No
- output means that no problems were detected. Duplicate entries are
+ output means that no problems were detected. Duplicate entries are
skipped and are flagged with a warning.
<a href="postmap.1.html"><b>postmap</b>(1)</a> terminates with zero exit status in case of success (includ-
- ing successful "<b>postmap -q</b>" lookup) and terminates with non-zero exit
+ ing successful "<b>postmap -q</b>" lookup) and terminates with non-zero exit
status in case of failure.
<b>ENVIRONMENT</b>
Enable verbose logging for debugging purposes.
<b>CONFIGURATION PARAMETERS</b>
- The following <a href="postconf.5.html"><b>main.cf</b></a> parameters are especially relevant to this pro-
- gram.
The text below provides only a parameter summary. See <a href="postconf.5.html"><b>post-</b></a>
+ The following <a href="postconf.5.html"><b>main.cf</b></a> parameters are especially relevant to this pro-
+ gram.
The text below provides only a parameter summary. See <a href="postconf.5.html"><b>post-</b></a>
<a href="postconf.5.html"><b>conf</b>(5)</a> for more details including examples.
<b><a href="postconf.5.html#berkeley_db_create_buffer_size">berkeley_db_create_buffer_size</a> (16777216)</b>
- The per-table I/O buffer size for programs that create Berkeley
+ The per-table I/O buffer size for programs that create Berkeley
DB hash or btree tables.
<b><a href="postconf.5.html#berkeley_db_read_buffer_size">berkeley_db_read_buffer_size</a> (131072)</b>
hash or btree tables.
<b><a href="postconf.5.html#config_directory">config_directory</a> (see 'postconf -d' output)</b>
- The default location of the Postfix <a href="postconf.5.html">main.cf</a> and <a href="master.5.html">master.cf</a> con-
+ The default location of the Postfix <a href="postconf.5.html">main.cf</a> and <a href="master.5.html">master.cf</a> con-
figuration files.
<b><a href="postconf.5.html#default_database_type">default_database_type</a> (see 'postconf -d' output)</b>
The default database type for use in <a href="newaliases.1.html"><b>newaliases</b>(1)</a>, <a href="postalias.1.html"><b>postalias</b>(1)</a>
and <a href="postmap.1.html"><b>postmap</b>(1)</a> commands.
+ <b><a href="postconf.5.html#smtputf8_enable">smtputf8_enable</a> (yes)</b>
+ Enable experimental SMTPUTF8 support for the protocols described
+ in <a href="http://tools.ietf.org/html/rfc6531">RFC 6531</a>..6533.
+
<b><a href="postconf.5.html#syslog_facility">syslog_facility</a> (mail)</b>
The syslog facility of Postfix logging.
# Non-production: needs thorough testing, or major changes are still
# needed before the code stabilizes.
-#CCARGS="$CCARGS -DNONPROD"
+CCARGS="$CCARGS -DNONPROD"
# Workaround: prepend Postfix include files before other include files.
CCARGS="-I. -I../../include $CCARGS"
.na
.nf
.fi
-\fBpostalias\fR [\fB-Nfinoprsvw\fR] [\fB-c \fIconfig_dir\fR]
+\fBpostalias\fR [\fB-Nfinoprsuvw\fR] [\fB-c \fIconfig_dir\fR]
[\fB-d \fIkey\fR] [\fB-q \fIkey\fR]
[\fIfile_type\fR:]\fIfile_name\fR ...
.SH DESCRIPTION
as the original input order.
This feature is available in Postfix version 2.2 and later,
and is not available for all database types.
+.IP \fB-u\fR
+Disable UTF-8 support. UTF-8 support is enabled by default
+when "smtputf8_enable = yes". It requires that keys and
+values are valid UTF-8 strings.
.IP \fB-v\fR
Enable verbose logging for debugging purposes. Multiple \fB-v\fR
options make the software increasingly verbose.
.IP "\fBdefault_database_type (see 'postconf -d' output)\fR"
The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1)
and \fBpostmap\fR(1) commands.
+.IP "\fBsmtputf8_enable (yes)\fR"
+Enable experimental SMTPUTF8 support for the protocols described
+in RFC 6531..6533.
.IP "\fBsyslog_facility (mail)\fR"
The syslog facility of Postfix logging.
.IP "\fBsyslog_name (see 'postconf -d' output)\fR"
.na
.nf
.fi
-\fBpostmap\fR [\fB-Nbfhimnoprsvw\fR] [\fB-c \fIconfig_dir\fR]
+\fBpostmap\fR [\fB-NbfhimnoprsuUvw\fR] [\fB-c \fIconfig_dir\fR]
[\fB-d \fIkey\fR] [\fB-q \fIkey\fR]
[\fIfile_type\fR:]\fIfile_name\fR ...
.SH DESCRIPTION
generates no body-style lookup keys for attachment MIME
headers and for attached message/* headers.
.sp
+NOTE: with "smtputf8_enable = yes", the \fB-b\fR option
+option disables UTF-8 syntax checks on query keys and
+lookup results. Specify the \fB-U\fR option to force UTF-8
+syntax checks anyway.
+.sp
This feature is available in Postfix version 2.6 and later.
.IP "\fB-c \fIconfig_dir\fR"
Read the \fBmain.cf\fR configuration file in the named directory
generates header-style lookup keys for attachment MIME
headers and for attached message/* headers.
.sp
+NOTE: with "smtputf8_enable = yes", the \fB-b\fR option
+option disables UTF-8 syntax checks on query keys and
+lookup results. Specify the \fB-U\fR option to force UTF-8
+syntax checks anyway.
+.sp
This feature is available in Postfix version 2.6 and later.
.IP \fB-i\fR
Incremental mode. Read entries from standard input and do not
.sp
This feature is available in Postfix version 2.2 and later,
and is not available for all database types.
+.IP \fB-u\fR
+Disable UTF-8 support. UTF-8 support is enabled by default
+when "smtputf8_enable = yes". It requires that keys and
+values are valid UTF-8 strings.
+.IP \fB-U\fR
+With "smtputf8_enable = yes", force UTF-8 syntax checks
+with the \fB-b\fR and \fB-h\fR options.
.IP \fB-v\fR
Enable verbose logging for debugging purposes. Multiple \fB-v\fR
options make the software increasingly verbose.
.IP "\fBdefault_database_type (see 'postconf -d' output)\fR"
The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1)
and \fBpostmap\fR(1) commands.
+.IP "\fBsmtputf8_enable (yes)\fR"
+Enable experimental SMTPUTF8 support for the protocols described
+in RFC 6531..6533.
.IP "\fBsyslog_facility (mail)\fR"
The syslog facility of Postfix logging.
.IP "\fBsyslog_name (see 'postconf -d' output)\fR"
.na
.ft C
/etc/postfix/transport:
- smtp-domain_that_verifies_after_data smtp-data-target:
- lmtp-domain_that_verifies_after_data lmtp-data-target:
+ smtp-domain-that-verifies-after-data smtp-data-target:
+ lmtp-domain-that-verifies-after-data lmtp-data-target:
.fi
.ad
.ft R
<blockquote>
<pre>
/etc/postfix/transport:
- smtp-domain_that_verifies_after_data smtp-data-target:
- lmtp-domain_that_verifies_after_data lmtp-data-target:
+ smtp-domain-that-verifies-after-data smtp-data-target:
+ lmtp-domain-that-verifies-after-data lmtp-data-target:
</pre>
</blockquote>
bounce_template.o: ../../include/mail_conf.h
bounce_template.o: ../../include/mail_params.h
bounce_template.o: ../../include/mail_proto.h
-bounce_template.o: ../../include/midna.h
+bounce_template.o: ../../include/midna_domain.h
bounce_template.o: ../../include/msg.h
bounce_template.o: ../../include/mymalloc.h
bounce_template.o: ../../include/nvtable.h
#include <stringops.h>
#include <mymalloc.h>
#ifndef NO_EAI
-#include <midna.h>
+#include <midna_domain.h>
#endif
/* Global library. */
"non-ASCII input value: \"%s\"",
tp->origin, key, asc_val);
return (asc_val);
- } else if ((utf8_val = midna_to_utf8(asc_val)) == 0) {
+ } else if ((utf8_val = midna_domain_to_utf8(asc_val)) == 0) {
msg_warn("%s: conversion \"%s\" failed: "
"input value: \"%s\"",
tp->origin, key, asc_val);
smtp_reply_footer.c safe_ultostr.c verify_sender_addr.c \
dict_memcache.c mail_version.c memcache_proto.c server_acl.c \
mkmap_fail.c haproxy_srvr.c dsn_filter.c dynamicmaps.c uxtext.c \
- smtputf8.c mail_conf_over.c mail_parm_split.c
+ smtputf8.c mail_conf_over.c mail_parm_split.c midna_adomain.c
OBJS = abounce.o anvil_clnt.o been_here.o bounce.o bounce_log.o \
canon_addr.o cfg_parser.o cleanup_strerror.o cleanup_strflags.o \
clnt_stream.o conv_time.o db_common.o debug_peer.o debug_process.o \
smtp_reply_footer.o safe_ultostr.o verify_sender_addr.o \
dict_memcache.o mail_version.o memcache_proto.o server_acl.o \
mkmap_fail.o haproxy_srvr.o dsn_filter.o dynamicmaps.o uxtext.o \
- smtputf8.o attr_override.o mail_parm_split.o $(NON_PLUGIN_MAP_OBJ)
-# MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf.
-# When hard-linking these maps, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ),
-# otherwise it sets the PLUGIN_* macros.
-MAP_OBJ = dict_ldap.o dict_mysql.o dict_pgsql.o dict_sqlite.o mkmap_cdb.o \
- mkmap_lmdb.o mkmap_sdbm.o
-HDRS = abounce.h anvil_clnt.h been_here.h bounce.h bounce_log.h \
- canon_addr.h cfg_parser.h cleanup_user.h clnt_stream.h config.h \
- conv_time.h db_common.h debug_peer.h debug_process.h defer.h \
- deliver_completed.h deliver_flock.h deliver_pass.h deliver_request.h \
- dict_ldap.h dict_mysql.h dict_pgsql.h dict_proxy.h dict_sqlite.h domain_list.h \
- dot_lockfile.h dot_lockfile_as.h dsb_scan.h dsn.h dsn_buf.h \
- dsn_mask.h dsn_print.h dsn_util.h ehlo_mask.h ext_prop.h \
- file_id.h flush_clnt.h header_opts.h header_token.h input_transp.h \
- int_filt.h is_header.h lex_822.h log_adhoc.h mail_addr.h \
- mail_addr_crunch.h mail_addr_find.h mail_addr_map.h mail_conf.h \
- mail_copy.h mail_date.h mail_dict.h mail_error.h mail_flush.h \
- mail_open_ok.h mail_params.h mail_proto.h mail_queue.h mail_run.h \
- mail_scan_dir.h mail_stream.h mail_task.h mail_version.h maps.h \
- mark_corrupt.h match_parent_style.h mbox_conf.h mbox_open.h \
- mime_state.h mkmap.h msg_stats.h mynetworks.h mypwd.h namadr_list.h \
- off_cvt.h opened.h own_inet_addr.h pipe_command.h post_mail.h \
- qmgr_user.h qmqp_proto.h quote_821_local.h quote_822_local.h \
- quote_flags.h rcpt_buf.h rcpt_print.h rec_attr_map.h rec_streamlf.h \
- rec_type.h recipient_list.h record.h resolve_clnt.h resolve_local.h \
- rewrite_clnt.h scache.h sent.h smtp_stream.h split_addr.h \
- string_list.h strip_addr.h sys_exits.h timed_ipc.h tok822.h \
- trace.h user_acl.h valid_mailhost_addr.h verify.h verify_clnt.h \
- verp_sender.h wildcard_inet_addr.h xtext.h delivered_hdr.h \
- fold_addr.h header_body_checks.h data_redirect.h match_service.h \
- addr_match_list.h smtp_reply_footer.h safe_ultostr.h \
- verify_sender_addr.h dict_memcache.h memcache_proto.h server_acl.h \
- haproxy_srvr.h dsn_filter.h dynamicmaps.h uxtext.h smtputf8.h \
- attr_override.h mail_parm_split.h
+ smtputf8.o attr_override.o mail_parm_split.o midna_adomain.o \
+ $(NON_PLUGIN_MAP_OBJ)
+ # MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf.
+ # When hard-linking these maps, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ),
+ # otherwise it sets the PLUGIN_* macros.
+ MAP_OBJ = dict_ldap.o dict_mysql.o dict_pgsql.o dict_sqlite.o mkmap_cdb.o \
+ mkmap_lmdb.o mkmap_sdbm.o
+ HDRS = abounce.h anvil_clnt.h been_here.h bounce.h bounce_log.h \
+ canon_addr.h cfg_parser.h cleanup_user.h clnt_stream.h config.h \
+ conv_time.h db_common.h debug_peer.h debug_process.h defer.h \
+ deliver_completed.h deliver_flock.h deliver_pass.h deliver_request.h \
+ dict_ldap.h dict_mysql.h dict_pgsql.h dict_proxy.h dict_sqlite.h domain_list.h \
+ dot_lockfile.h dot_lockfile_as.h dsb_scan.h dsn.h dsn_buf.h \
+ dsn_mask.h dsn_print.h dsn_util.h ehlo_mask.h ext_prop.h \
+ file_id.h flush_clnt.h header_opts.h header_token.h input_transp.h \
+ int_filt.h is_header.h lex_822.h log_adhoc.h mail_addr.h \
+ mail_addr_crunch.h mail_addr_find.h mail_addr_map.h mail_conf.h \
+ mail_copy.h mail_date.h mail_dict.h mail_error.h mail_flush.h \
+ mail_open_ok.h mail_params.h mail_proto.h mail_queue.h mail_run.h \
+ mail_scan_dir.h mail_stream.h mail_task.h mail_version.h maps.h \
+ mark_corrupt.h match_parent_style.h mbox_conf.h mbox_open.h \
+ mime_state.h mkmap.h msg_stats.h mynetworks.h mypwd.h namadr_list.h \
+ off_cvt.h opened.h own_inet_addr.h pipe_command.h post_mail.h \
+ qmgr_user.h qmqp_proto.h quote_821_local.h quote_822_local.h \
+ quote_flags.h rcpt_buf.h rcpt_print.h rec_attr_map.h rec_streamlf.h \
+ rec_type.h recipient_list.h record.h resolve_clnt.h resolve_local.h \
+ rewrite_clnt.h scache.h sent.h smtp_stream.h split_addr.h \
+ string_list.h strip_addr.h sys_exits.h timed_ipc.h tok822.h \
+ trace.h user_acl.h valid_mailhost_addr.h verify.h verify_clnt.h \
+ verp_sender.h wildcard_inet_addr.h xtext.h delivered_hdr.h \
+ fold_addr.h header_body_checks.h data_redirect.h match_service.h \
+ addr_match_list.h smtp_reply_footer.h safe_ultostr.h \
+ verify_sender_addr.h dict_memcache.h memcache_proto.h server_acl.h \
+ haproxy_srvr.h dsn_filter.h dynamicmaps.h uxtext.h smtputf8.h \
+ attr_override.h mail_parm_split.h midna_adomain.h
TESTSRC = rec2stream.c stream2rec.c recdump.c
DEFS = -I. -I$(INC_DIR) -D$(SYSTYPE)
CFLAGS = $(DEBUG) $(OPT) $(DEFS)
memcache_proto.o: ../../include/vstring_vstream.h
memcache_proto.o: memcache_proto.c
memcache_proto.o: memcache_proto.h
+midna_adomain.o: ../../include/check_arg.h
+midna_adomain.o: ../../include/midna_domain.h
+midna_adomain.o: ../../include/stringops.h
+midna_adomain.o: ../../include/sys_defs.h
+midna_adomain.o: ../../include/vbuf.h
+midna_adomain.o: ../../include/vstring.h
+midna_adomain.o: midna_adomain.c
+midna_adomain.o: midna_adomain.h
mime_state.o: ../../include/check_arg.h
mime_state.o: ../../include/msg.h
mime_state.o: ../../include/mymalloc.h
/*
* Don't frustrate future attempts to make Postfix UTF-8 transparent.
*/
- if (!valid_utf8_string(name, strlen(name))) {
+ if (DICT_IS_ENABLE_UTF8(dict->flags) == 0
+ && !valid_utf8_string(name, strlen(name))) {
if (msg_verbose)
msg_info("%s: %s: Skipping lookup of non-UTF-8 key '%s'",
myname, dict_ldap->parser->name, name);
* Optionally fold the key.
*/
if (dict->flags & DICT_FLAG_FOLD_FIX) {
- if (dict->fold_buf == 0)
- dict->fold_buf = vstring_alloc(10);
- vstring_strcpy(dict->fold_buf, name);
- name = lowercase(vstring_str(dict->fold_buf));
+ if (dict->fold_buf == 0)
+ dict->fold_buf = vstring_alloc(10);
+ vstring_strcpy(dict->fold_buf, name);
+ name = lowercase(vstring_str(dict->fold_buf));
}
/*
case PROXY_STAT_RETRY:
*key = *value = 0;
DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR);
+ case PROXY_STAT_CONFIG:
+ *key = *value = 0;
+ DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR);
default:
msg_warn("%s sequence failed for table \"%s\" function %d: "
"unexpected reply status %d",
DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, (char *) 0);
case PROXY_STAT_RETRY:
DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, (char *) 0);
+ case PROXY_STAT_CONFIG:
+ DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, (char *) 0);
default:
msg_warn("%s lookup failed for table \"%s\" key \"%s\": "
"unexpected reply status %d",
DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL);
case PROXY_STAT_RETRY:
DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR);
+ case PROXY_STAT_CONFIG:
+ DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR);
default:
msg_warn("%s update failed for table \"%s\" key \"%s\": "
"unexpected reply status %d",
DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL);
case PROXY_STAT_RETRY:
DICT_ERR_VAL_RETURN(dict, DICT_ERR_RETRY, DICT_STAT_ERROR);
+ case PROXY_STAT_CONFIG:
+ DICT_ERR_VAL_RETURN(dict, DICT_ERR_CONFIG, DICT_STAT_ERROR);
default:
msg_warn("%s delete failed for table \"%s\" key \"%s\": "
"unexpected reply status %d",
#define PROXY_STAT_RETRY 2 /* try lookup again later */
#define PROXY_STAT_BAD 3 /* invalid request parameter */
#define PROXY_STAT_DENY 4 /* table not approved for proxying */
+#define PROXY_STAT_CONFIG 5 /* DICT_ERR_CONFIG error */
/* LICENSE
/* .ad
/*
* Don't frustrate future attempts to make Postfix UTF-8 transparent.
*/
- if (!valid_utf8_string(name, strlen(name))) {
+ if (DICT_IS_ENABLE_UTF8(dict->flags) == 0
+ && !valid_utf8_string(name, strlen(name))) {
if (msg_verbose)
msg_info("%s: %s: Skipping lookup of non-UTF-8 key '%s'",
myname, dict_sqlite->parser->name, name);
dict_db_cache_size = var_db_read_buf;
dict_lmdb_map_size = var_lmdb_map_size;
inet_windowsize = var_inet_windowsize;
- temp_utf8_kludge = var_smtputf8_enable;
/*
* Report run-time versus compile-time discrepancies.
if (var_smtputf8_enable)
msg_warn("%s is true, but EAI support is not compiled in",
VAR_SMTPUTF8_ENABLE);
+ var_smtputf8_enable = 0;
#endif
+ util_utf8_enable = var_smtputf8_enable;
/*
* Variables whose defaults are determined at runtime, after other
--- /dev/null
+/*++
+/* NAME
+/* midna_adomain 3
+/* SUMMARY
+/* address domain part conversion
+/* SYNOPSIS
+/* #include <midna_adomain.h>
+/*
+/* char *midna_adomain_to_ascii(
+/* VSTRING *dest,
+/* const char *name)
+/*
+/* char *midna_adomain_to_utf8(
+/* VSTRING *dest,
+/* const char *name)
+/* DESCRIPTION
+/* The functions in this module transform the domain portion
+/* of an email address between ASCII and UTF-8 form. Both
+/* functions tolerate a missing domain, and both functions
+/* return a copy of the input when the domain portion requires
+/* no conversion.
+/*
+/* midna_adomain_to_ascii() converts an UTF-8 or ASCII domain
+/* portion to ASCII. The result is a null pointer when
+/* conversion fails. This function verifies that the resulting
+/* domain passes valid_hostname().
+/*
+/* midna_adomain_to_utf8() converts an UTF-8 or ASCII domain
+/* name to UTF-8. The result is a null pointer when conversion
+/* fails. This function verifies that the resulting domain,
+/* after conversion to ASCII, passes valid_hostname().
+/* SEE ALSO
+/* midna_domain(3), Postfix ASCII/UTF-8 domain name conversion
+/* DIAGNOSTICS
+/* Fatal errors: memory allocation problem.
+/* Warnings: conversion error or result validation error.
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+ /*
+ * System library.
+ */
+#include <sys_defs.h>
+#include <string.h>
+
+#ifndef NO_EAI
+#include <unicode/uidna.h>
+
+ /*
+ * Utility library.
+ */
+#include <vstring.h>
+#include <stringops.h>
+#include <midna_domain.h>
+
+ /*
+ * Global library.
+ */
+#include <midna_adomain.h>
+
+#define STR(x) vstring_str(x)
+
+/* midna_adomain_to_utf8 - convert address domain portion to UTF8 */
+
+char *midna_adomain_to_utf8(VSTRING *dest, const char *src)
+{
+ const char *cp;
+ const char *domain_utf8;
+
+ if ((cp = strrchr(src, '@')) == 0) {
+ vstring_strcpy(dest, src);
+ } else {
+ vstring_sprintf(dest, "%*s@", (int) (cp - src), src);
+ if (*(cp += 1)) {
+ if (allascii(cp) && strstr(cp, "--") == 0) {
+ vstring_strcat(dest, cp);
+ } else if ((domain_utf8 = midna_domain_to_utf8(cp)) == 0) {
+ return (0);
+ } else {
+ vstring_strcat(dest, domain_utf8);
+ }
+ }
+ }
+ return (STR(dest));
+}
+
+/* midna_adomain_to_ascii - convert address domain portion to ASCII */
+
+char *midna_adomain_to_ascii(VSTRING *dest, const char *src)
+{
+ const char *cp;
+ const char *domain_ascii;
+
+ if ((cp = strrchr(src, '@')) == 0) {
+ vstring_strcpy(dest, src);
+ } else {
+ vstring_sprintf(dest, "%*s@", (int) (cp - src), src);
+ if (*(cp += 1)) {
+ if (allascii(cp)) {
+ vstring_strcat(dest, cp);
+ } else if ((domain_ascii = midna_domain_to_ascii(cp + 1)) == 0) {
+ return (0);
+ } else {
+ vstring_strcat(dest, domain_ascii);
+ }
+ }
+ }
+ return (STR(dest));
+}
+
+#endif /* NO_IDNA */
--- /dev/null
+#ifndef _MIDNA_ADOMAIN_H_INCLUDED_
+#define _MIDNA_ADOMAIN_H_INCLUDED_
+
+/*++
+/* NAME
+/* midna_adomain 3h
+/* SUMMARY
+/* domain name conversion
+/* SYNOPSIS
+/* #include <midna_adomain.h>
+/* DESCRIPTION
+/* .nf
+
+ /*
+ * Utility library.
+ */
+#include <vstring.h>
+
+ /*
+ * External interface.
+ */
+extern char *midna_adomain_to_utf8(VSTRING *, const char *);
+extern char *midna_adomain_to_ascii(VSTRING *, const char *);
+
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+#endif
if (mkmap->after_open)
mkmap->after_open(mkmap);
+ /*
+ * Wrap the dictionary for UTF-8 syntax checks and casefolding.
+ */
+ if ((mkmap->dict->flags & DICT_FLAG_UTF8_PROXY) == 0
+ && DICT_IS_ENABLE_UTF8(dict_flags))
+ mkmap->dict = dict_utf8_encapsulate(mkmap->dict);
+
/*
* Resume signal delivery if multi-writer safe.
*/
return (t1->tail = tok822_append(t1->tail, t2));
} else {
t1->head = t2;
+ t2->owner = t1;
while (t2->next)
(t2 = t2->next)->owner = t1;
return (t1->tail = t2);
return (tp);
} else {
t1->head = t2;
+ t2->owner = t1;
while (t2->next)
(t2 = t2->next)->owner = t1;
return (t1->tail = t2);
TOK822 *tok822_free_tree(TOK822 *tp)
{
- if (tp) {
- if (tp->next)
- tok822_free_tree(tp->next);
+ TOK822 *next;
+
+ for (/* void */; tp != 0; tp = next) {
if (tp->head)
tok822_free_tree(tp->head);
+ next = tp->next;
tok822_free(tp);
}
return (0);
/* Postfix alias database maintenance
/* SYNOPSIS
/* .fi
-/* \fBpostalias\fR [\fB-Nfinoprsvw\fR] [\fB-c \fIconfig_dir\fR]
+/* \fBpostalias\fR [\fB-Nfinoprsuvw\fR] [\fB-c \fIconfig_dir\fR]
/* [\fB-d \fIkey\fR] [\fB-q \fIkey\fR]
/* [\fIfile_type\fR:]\fIfile_name\fR ...
/* DESCRIPTION
/* as the original input order.
/* This feature is available in Postfix version 2.2 and later,
/* and is not available for all database types.
+/* .IP \fB-u\fR
+/* Disable UTF-8 support. UTF-8 support is enabled by default
+/* when "smtputf8_enable = yes". It requires that keys and
+/* values are valid UTF-8 strings.
/* .IP \fB-v\fR
/* Enable verbose logging for debugging purposes. Multiple \fB-v\fR
/* options make the software increasingly verbose.
/* .IP "\fBdefault_database_type (see 'postconf -d' output)\fR"
/* The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1)
/* and \fBpostmap\fR(1) commands.
+/* .IP "\fBsmtputf8_enable (yes)\fR"
+/* Enable experimental SMTPUTF8 support for the protocols described
+/* in RFC 6531..6533.
/* .IP "\fBsyslog_facility (mail)\fR"
/* The syslog facility of Postfix logging.
/* .IP "\fBsyslog_name (see 'postconf -d' output)\fR"
/* Application-specific. */
#define STR vstring_str
+#define LEN VSTRING_LEN
#define POSTALIAS_FLAG_AS_OWNER (1<<0) /* open dest as owner of source */
#define POSTALIAS_FLAG_SAVE_PERM (1<<1) /* copy access permission
&& (st.st_uid != geteuid() || st.st_gid != getegid()))
set_eugid(st.st_uid, st.st_gid);
-
/*
* Open the database, create it when it does not exist, truncate it when
* it does exist, and lock out any spectators.
last_line = 0;
while (readllines(line_buffer, source_fp, &last_line, &lineno)) {
+ /*
+ * First some UTF-8 checks sans casefolding.
+ */
+ if (DICT_IS_ENABLE_UTF8(dict_flags)
+ && !allascii(STR(line_buffer))
+ && !valid_utf8_string(STR(line_buffer), LEN(line_buffer))) {
+ msg_warn("%s, line %d: non-UTF-8 input \"%s\"",
+ VSTREAM_PATH(source_fp), lineno, STR(line_buffer));
+ continue;
+ }
+
/*
* Tokenize the input, so that we do the right thing when a
* quoted localpart contains special characters such as "@", ":"
static NORETURN usage(char *myname)
{
- msg_fatal("usage: %s [-Nfinoprsvw] [-c config_dir] [-d key] [-q key] [map_type:]file...",
+ msg_fatal("usage: %s [-Nfinoprsuvw] [-c config_dir] [-d key] [-q key] [map_type:]file...",
myname);
}
struct stat st;
int postalias_flags = POSTALIAS_FLAG_AS_OWNER | POSTALIAS_FLAG_SAVE_PERM;
int open_flags = O_RDWR | O_CREAT | O_TRUNC;
- int dict_flags = DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX;
+ int dict_flags = (DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX
+ | DICT_FLAG_UTF8_ENABLE);
char *query = 0;
char *delkey = 0;
int sequence = 0;
/*
* Parse JCL.
*/
- while ((ch = GETOPT(argc, argv, "Nc:d:finopq:rsvw")) > 0) {
+ while ((ch = GETOPT(argc, argv, "Nc:d:finopq:rsuvw")) > 0) {
switch (ch) {
default:
usage(argv[0]);
msg_fatal("specify only one of -s or -q or -d");
sequence = 1;
break;
+ case 'u':
+ dict_flags &= ~DICT_FLAG_UTF8_ENABLE;
+ break;
case 'v':
msg_verbose++;
break;
/* Postfix lookup table management
/* SYNOPSIS
/* .fi
-/* \fBpostmap\fR [\fB-Nbfhimnoprsvw\fR] [\fB-c \fIconfig_dir\fR]
+/* \fBpostmap\fR [\fB-NbfhimnoprsuUvw\fR] [\fB-c \fIconfig_dir\fR]
/* [\fB-d \fIkey\fR] [\fB-q \fIkey\fR]
/* [\fIfile_type\fR:]\fIfile_name\fR ...
/* DESCRIPTION
/* generates no body-style lookup keys for attachment MIME
/* headers and for attached message/* headers.
/* .sp
+/* NOTE: with "smtputf8_enable = yes", the \fB-b\fR option
+/* option disables UTF-8 syntax checks on query keys and
+/* lookup results. Specify the \fB-U\fR option to force UTF-8
+/* syntax checks anyway.
+/* .sp
/* This feature is available in Postfix version 2.6 and later.
/* .IP "\fB-c \fIconfig_dir\fR"
/* Read the \fBmain.cf\fR configuration file in the named directory
/* generates header-style lookup keys for attachment MIME
/* headers and for attached message/* headers.
/* .sp
+/* NOTE: with "smtputf8_enable = yes", the \fB-b\fR option
+/* option disables UTF-8 syntax checks on query keys and
+/* lookup results. Specify the \fB-U\fR option to force UTF-8
+/* syntax checks anyway.
+/* .sp
/* This feature is available in Postfix version 2.6 and later.
/* .IP \fB-i\fR
/* Incremental mode. Read entries from standard input and do not
/* .sp
/* This feature is available in Postfix version 2.2 and later,
/* and is not available for all database types.
+/* .IP \fB-u\fR
+/* Disable UTF-8 support. UTF-8 support is enabled by default
+/* when "smtputf8_enable = yes". It requires that keys and
+/* values are valid UTF-8 strings.
+/* .IP \fB-U\fR
+/* With "smtputf8_enable = yes", force UTF-8 syntax checks
+/* with the \fB-b\fR and \fB-h\fR options.
/* .IP \fB-v\fR
/* Enable verbose logging for debugging purposes. Multiple \fB-v\fR
/* options make the software increasingly verbose.
/* .IP "\fBdefault_database_type (see 'postconf -d' output)\fR"
/* The default database type for use in \fBnewaliases\fR(1), \fBpostalias\fR(1)
/* and \fBpostmap\fR(1) commands.
+/* .IP "\fBsmtputf8_enable (yes)\fR"
+/* Enable experimental SMTPUTF8 support for the protocols described
+/* in RFC 6531..6533.
/* .IP "\fBsyslog_facility (mail)\fR"
/* The syslog facility of Postfix logging.
/* .IP "\fBsyslog_name (see 'postconf -d' output)\fR"
last_line = 0;
while (readllines(line_buffer, source_fp, &last_line, &lineno)) {
+ /*
+ * First some UTF-8 checks sans casefolding.
+ */
+ if (DICT_IS_ENABLE_UTF8(dict_flags)
+ && !allascii(STR(line_buffer))
+ && !valid_utf8_string(STR(line_buffer), LEN(line_buffer))) {
+ msg_warn("%s, line %d: non-UTF-8 input \"%s\"",
+ VSTREAM_PATH(source_fp), lineno, STR(line_buffer));
+ continue;
+ }
+
/*
* Split on the first whitespace character, then trim leading and
* trailing whitespace from key and value.
static NORETURN usage(char *myname)
{
- msg_fatal("usage: %s [-Nfinoprsvw] [-c config_dir] [-d key] [-q key] [map_type:]file...",
+ msg_fatal("usage: %s [-NfinoprsuUvw] [-c config_dir] [-d key] [-q key] [map_type:]file...",
myname);
}
struct stat st;
int postmap_flags = POSTMAP_FLAG_AS_OWNER | POSTMAP_FLAG_SAVE_PERM;
int open_flags = O_RDWR | O_CREAT | O_TRUNC;
- int dict_flags = DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX;
+ int dict_flags = (DICT_FLAG_DUP_WARN | DICT_FLAG_FOLD_FIX
+ | DICT_FLAG_UTF8_ENABLE);
char *query = 0;
char *delkey = 0;
int sequence = 0;
int found;
+ int force_utf8 = 0;
/*
* Fingerprint executables and core dumps.
/*
* Parse JCL.
*/
- while ((ch = GETOPT(argc, argv, "Nbc:d:fhimnopq:rsvw")) > 0) {
+ while ((ch = GETOPT(argc, argv, "Nbc:d:fhimnopq:rsuUvw")) > 0) {
switch (ch) {
default:
usage(argv[0]);
msg_fatal("specify only one of -s or -q or -d");
sequence = 1;
break;
+ case 'u':
+ dict_flags &= ~DICT_FLAG_UTF8_ENABLE;
+ break;
+ case 'U':
+ force_utf8 = 1;
+ break;
case 'v':
msg_verbose++;
break;
&& (postmap_flags & POSTMAP_FLAG_ANY_KEY)
== (postmap_flags & POSTMAP_FLAG_MIME_KEY))
msg_warn("ignoring -m option without -b or -h");
-
+ if ((postmap_flags & (POSTMAP_FLAG_ANY_KEY & ~POSTMAP_FLAG_MIME_KEY))
+ && force_utf8 == 0)
+ dict_flags &= ~DICT_FLAG_UTF8_MASK;
+
/*
* Use the map type specified by the user, or fall back to a default
* database type.
posttls-finger.o: ../../include/mail_conf.h
posttls-finger.o: ../../include/mail_params.h
posttls-finger.o: ../../include/mail_server.h
-posttls-finger.o: ../../include/midna.h
+posttls-finger.o: ../../include/midna_domain.h
posttls-finger.o: ../../include/msg.h
posttls-finger.o: ../../include/msg_vstream.h
posttls-finger.o: ../../include/myaddrinfo.h
#include <sane_connect.h>
#include <myaddrinfo.h>
#include <sock_addr.h>
-#include <midna.h>
+#include <midna_domain.h>
#define STR(x) vstring_str(x)
* IDNA support.
*/
#ifndef NO_EAI
- if (!allascii(domain) && (aname = midna_to_ascii(domain)) != 0) {
+ if (!allascii(domain) && (aname = midna_domain_to_ascii(domain)) != 0) {
msg_info("%s asciified to %s", domain, aname);
} else
#endif
* IDNA support.
*/
#ifndef NO_EAI
- if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) {
+ if (!allascii(host) && (ahost = midna_domain_to_ascii(host)) != 0) {
msg_info("%s asciified to %s", host, ahost);
} else
#endif
reply_status = PROXY_STAT_NOKEY;
reply_key = reply_value = "";
} else {
- reply_status = PROXY_STAT_RETRY;
+ reply_status = (dict->error == DICT_ERR_RETRY ?
+ PROXY_STAT_RETRY : PROXY_STAT_CONFIG);
reply_key = reply_value = "";
}
}
reply_status = PROXY_STAT_NOKEY;
reply_value = "";
} else {
- reply_status = PROXY_STAT_RETRY;
+ reply_status = (dict->error == DICT_ERR_RETRY ?
+ PROXY_STAT_RETRY : PROXY_STAT_CONFIG);
reply_value = "";
}
} else if (dict->error == 0) {
reply_status = PROXY_STAT_NOKEY;
} else {
- reply_status = PROXY_STAT_RETRY;
+ reply_status = (dict->error == DICT_ERR_RETRY ?
+ PROXY_STAT_RETRY : PROXY_STAT_CONFIG);
}
}
} else if (dict->error == 0) {
reply_status = PROXY_STAT_NOKEY;
} else {
- reply_status = PROXY_STAT_RETRY;
+ reply_status = (dict->error == DICT_ERR_RETRY ?
+ PROXY_STAT_RETRY : PROXY_STAT_CONFIG);
}
}
smtp_addr.o: ../../include/mail_params.h
smtp_addr.o: ../../include/maps.h
smtp_addr.o: ../../include/match_list.h
-smtp_addr.o: ../../include/midna.h
+smtp_addr.o: ../../include/midna_domain.h
smtp_addr.o: ../../include/mime_state.h
smtp_addr.o: ../../include/msg.h
smtp_addr.o: ../../include/msg_stats.h
smtp_proto.o: ../../include/maps.h
smtp_proto.o: ../../include/mark_corrupt.h
smtp_proto.o: ../../include/match_list.h
+smtp_proto.o: ../../include/match_parent_style.h
smtp_proto.o: ../../include/mime_state.h
smtp_proto.o: ../../include/msg.h
smtp_proto.o: ../../include/msg_stats.h
smtp_proto.o: ../../include/myaddrinfo.h
smtp_proto.o: ../../include/myflock.h
smtp_proto.o: ../../include/mymalloc.h
+smtp_proto.o: ../../include/namadr_list.h
smtp_proto.o: ../../include/name_code.h
smtp_proto.o: ../../include/name_mask.h
smtp_proto.o: ../../include/nvtable.h
#include <stringops.h>
#include <myaddrinfo.h>
#include <inet_proto.h>
-#include <midna.h>
+#include <midna_domain.h>
/* Global library. */
* IDNA support.
*/
#ifndef NO_EAI
- if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) {
+ if (!allascii(name) && (aname = midna_domain_to_ascii(name)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", name, aname);
} else
* IDNA support.
*/
#ifndef NO_EAI
- if (!allascii(host) && (ahost = midna_to_ascii(host)) != 0) {
+ if (!allascii(host) && (ahost = midna_domain_to_ascii(host)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", host, ahost);
} else
smtpd_check.o: ../../include/maps.h
smtpd_check.o: ../../include/match_list.h
smtpd_check.o: ../../include/match_parent_style.h
-smtpd_check.o: ../../include/midna.h
+smtpd_check.o: ../../include/midna_domain.h
smtpd_check.o: ../../include/milter.h
smtpd_check.o: ../../include/msg.h
smtpd_check.o: ../../include/msg_stats.h
* As an extension to RFC 1985 we also allow an RFC 2821 address literal
* enclosed in [].
*
- * XXX EAI: Convert to ASCII and use that form internally.
+ * XXX There does not appear to be an ETRN parameter to indicate that the
+ * domain name is UTF-8.
*/
if (!valid_hostname(argv[1].strval, DONT_GRIPE)
&& !valid_mailhost_literal(argv[1].strval, DONT_GRIPE)) {
}
watchdog_pat();
smtpd_chat_query(state);
+ /* Safety: protect internal interfaces against malformed UTF-8. */
+ if (var_smtputf8_enable && valid_utf8_string(STR(state->buffer),
+ LEN(state->buffer)) == 0) {
+ state->error_mask |= MAIL_ERROR_PROTOCOL;
+ smtpd_chat_reply(state, "500 5.5.2 Error: bad UTF-8 syntax");
+ state->error_count++;
+ continue;
+ }
/* Move into smtpd_chat_query() and update session transcript. */
if (smtpd_cmd_filter != 0) {
for (cp = STR(state->buffer); *cp && IS_SPACE_TAB(*cp); cp++)
#include <inet_proto.h>
#include <ip_match.h>
#include <valid_utf8_hostname.h>
-#include <midna.h>
+#include <midna_domain.h>
#include <mynetworks.h>
/* DNS library. */
char **ext)
{
const char *result;
-
+
if ((result = mail_addr_find(maps, key, ext)) != 0 || maps->error == 0)
return (result);
if (maps->error == DICT_ERR_RETRY)
+ /* Warning is already logged. */
reject_dict_retry(state, reply_name);
else
reject_server_error(state);
}
+/* check_dict_get - reject with temporary failure if dict lookup fails */
+
+static const char *check_dict_get(SMTPD_STATE *state, const char *table,
+ const char *reply_name,
+ DICT *dict, const char *key)
+{
+ const char *result;
+
+ if ((result = dict_get(dict, key)) != 0 || dict->error == 0)
+ return (result);
+ if (dict->error == DICT_ERR_RETRY) {
+ msg_warn("%s: table lookup problem", table);
+ reject_dict_retry(state, reply_name);
+ } else
+ reject_server_error(state);
+}
+
/* reject_unknown_reverse_name - fail if reverse client hostname is unknown */
static int reject_unknown_reverse_name(SMTPD_STATE *state)
* Fix 20140924: convert domain to ASCII.
*/
#ifndef NO_EAI
- if (!allascii(name) && (aname = midna_to_ascii(name)) != 0) {
+ if (!allascii(name) && (aname = midna_domain_to_ascii(name)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", name, aname);
name = aname;
* Fix 20140924: convert domain to ASCII.
*/
#ifndef NO_EAI
- if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) {
+ if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", domain, adomain);
domain = adomain;
if ((dict = dict_handle(table)) == 0) {
msg_warn("%s: unexpected dictionary: %s", myname, table);
- value = "451 4.3.5 Server configuration error";
- CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
- reply_name, reply_class,
- def_acl), FOUND);
+ reject_server_error(state);
}
if (flags == 0 || (flags & dict->flags) != 0) {
- if ((value = dict_get(dict, name)) != 0)
+ if ((value = check_dict_get(state, table, reply_name, dict, name)) != 0)
CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
reply_name, reply_class,
def_acl), FOUND);
- if (dict->error != 0) {
- msg_warn("%s: table lookup problem", table);
- value = "451 4.3.5 Server configuration error";
- CHK_ACCESS_RETURN(check_table_result(state, table, value, name,
- reply_name, reply_class,
- def_acl), FOUND);
- }
}
CHK_ACCESS_RETURN(SMTPD_CHECK_DUNNO, MISSED);
}
if ((dict = dict_handle(table)) == 0) {
msg_warn("%s: unexpected dictionary: %s", myname, table);
- value = "451 4.3.5 Server configuration error";
- CHK_DOMAIN_RETURN(check_table_result(state, table, value,
- domain, reply_name, reply_class,
- def_acl), FOUND);
+ reject_server_error(state);
}
for (name = domain; *name != 0; name = next) {
if (flags == 0 || (flags & dict->flags) != 0) {
- if ((value = dict_get(dict, name)) != 0)
- CHK_DOMAIN_RETURN(check_table_result(state, table, value,
- domain, reply_name, reply_class,
- def_acl), FOUND);
- if (dict->error != 0) {
- msg_warn("%s: table lookup problem", table);
- value = "451 4.3.5 Server configuration error";
+ if ((value = check_dict_get(state, table, reply_name,
+ dict, name)) != 0)
CHK_DOMAIN_RETURN(check_table_result(state, table, value,
domain, reply_name, reply_class,
def_acl), FOUND);
- }
}
/* Don't apply subdomain magic to numerical hostnames. */
if (maybe_numerical
if ((dict = dict_handle(table)) == 0) {
msg_warn("%s: unexpected dictionary: %s", myname, table);
- value = "451 4.3.5 Server configuration error";
- CHK_ADDR_RETURN(check_table_result(state, table, value, address,
- reply_name, reply_class,
- def_acl), FOUND);
+ reject_server_error(state);
}
do {
if (flags == 0 || (flags & dict->flags) != 0) {
- if ((value = dict_get(dict, addr)) != 0)
+ if ((value = check_dict_get(state, table, reply_name,
+ dict, addr)) != 0)
CHK_ADDR_RETURN(check_table_result(state, table, value, address,
reply_name, reply_class,
def_acl), FOUND);
- if (dict->error != 0) {
- msg_warn("%s: table lookup problem", table);
- value = "451 4.3.5 Server configuration error";
- CHK_ADDR_RETURN(check_table_result(state, table, value, address,
- reply_name, reply_class,
- def_acl), FOUND);
- }
}
flags = PARTIAL;
} while (split_at_right(addr, delim));
* Fix 20140924: convert domain to ASCII.
*/
#ifndef NO_EAI
- if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) {
+ if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", domain, adomain);
domain = adomain;
* Fix 20140706: convert domain to ASCII.
*/
#ifndef NO_EAI
- if (!allascii(domain) && (adomain = midna_to_ascii(domain)) != 0) {
+ if (!allascii(domain) && (adomain = midna_domain_to_ascii(domain)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", domain, adomain);
domain = adomain;
#endif
+/* valid_utf8_action - validate UTF-8 policy server response */
+
+static int valid_utf8_action(const char *server, const char *action)
+{
+ int retval;
+
+ if ((retval = valid_utf8_string(action, strlen(action))) == 0)
+ msg_warn("malformed UTF-8 in policy server %s response: \"%s\"",
+ server, action);
+ return (retval);
+}
+
/* check_policy_service - check delegated policy service */
static int check_policy_service(SMTPD_STATE *state, const char *server,
ATTR_TYPE_END,
ATTR_FLAG_MISSING, /* Reply attributes. */
RECV_ATTR_STR(MAIL_ATTR_ACTION, action),
- ATTR_TYPE_END) != 1) {
+ ATTR_TYPE_END) != 1
+ || (var_smtputf8_enable && valid_utf8_action(server, STR(action)) == 0)) {
NOCLOBBER static int nesting_level = 0;
jmp_buf savebuf;
int status;
tls_client.o: ../../include/dns.h
tls_client.o: ../../include/iostuff.h
tls_client.o: ../../include/mail_params.h
-tls_client.o: ../../include/midna.h
+tls_client.o: ../../include/midna_domain.h
tls_client.o: ../../include/msg.h
tls_client.o: ../../include/myaddrinfo.h
tls_client.o: ../../include/myflock.h
#include <stringops.h>
#include <msg.h>
#include <iostuff.h> /* non-blocking */
-#include <midna.h>
+#include <midna_domain.h>
/* Global library. */
*/
if (!allascii(certid))
return (0);
- if (!allascii(nexthop) && (aname = midna_to_ascii(nexthop)) != 0) {
+ if (!allascii(nexthop) && (aname = midna_domain_to_ascii(nexthop)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", nexthop, aname);
nexthop = aname;
#ifndef NO_EAI
/*
- * IDNA allows labels to be separated by any of the additional
- * characters U+3002, U+FF0E, and U+FF61; that are Unicode
- * variants. Their UTF-8 encodings are: E38082, EFBC8E and
- * EFBDA1.
+ * Besides U+002E (full stop) IDNA2003 allows labels to be
+ * separated by any of the Unicode variants U+3002 (ideographic
+ * full stop), U+FF0E (fullwidth full stop), and U+FF61
+ * (halfwidth ideographic full stop). Their respective UTF-8
+ * encodings are: E38082, EFBC8E and EFBDA1.
*
- * It is not clear whether the IDNA to_ASCII conversion allows empty
- * leading labels, so we handle these explicitly here.
+ * IDNA2008 does not permit (upper) case and other variant
+ * differences in U-labels. The midna_domain_to_ascii() function,
+ * based on UTS46, midna_domain_to_ascii() normalizes the
+ * differences away.
+ *
+ * The IDNA to_ASCII conversion does not allow empty leading labels,
+ * so we handle these explicitly here.
*/
else {
unsigned char *cp = (unsigned char *) domain;
}
}
if (!allascii(domain)
- && (aname = midna_to_ascii(domain)) != 0) {
+ && (aname = midna_domain_to_ascii(domain)) != 0) {
if (msg_verbose)
msg_info("%s asciified to %s", domain, aname);
domain = aname;
*/
#define STR vstring_str
+#define LEN VSTRING_LEN
/*
* Some of the lists that define the address domain classes.
*/
tok822_internalize(nextrcpt, tree, TOK822_STR_DEFL);
rcpt_domain = strrchr(STR(nextrcpt), '@') + 1;
- if (rcpt_domain == 0)
+ if (rcpt_domain == (char *) 1)
msg_panic("no @ in address: \"%s\"", STR(nextrcpt));
if (*rcpt_domain == '[') {
if (!valid_mailhost_literal(rcpt_domain, DONT_GRIPE))
*flags |= RESOLVE_FLAG_ERROR;
+ } else if (var_smtputf8_enable
+ && valid_utf8_string(STR(nextrcpt), LEN(nextrcpt)) == 0) {
+ *flags |= RESOLVE_FLAG_ERROR;
} else if (!valid_utf8_hostname(var_smtputf8_enable, rcpt_domain,
DONT_GRIPE)) {
if (var_resolve_num_dom && valid_hostaddr(rcpt_domain, DONT_GRIPE)) {
dict_fail.c msg_rate_delay.c dict_surrogate.c warn_stat.c \
dict_sockmap.c line_number.c recv_pass_attr.c pass_accept.c \
poll_fd.c timecmp.c slmdb.c dict_pipe.c dict_random.c \
- valid_utf8_hostname.c midna.c argv_splitq.c balpar.c dict_union.c \
- extpar.c dict_inline.c
+ valid_utf8_hostname.c midna_domain.c argv_splitq.c balpar.c dict_union.c \
+ extpar.c dict_inline.c casefold.c dict_utf8.c
OBJS = alldig.o allprint.o argv.o argv_split.o attr_clnt.o attr_print0.o \
attr_print64.o attr_print_plain.o attr_scan0.o attr_scan64.o \
attr_scan_plain.o auto_clnt.o base64_code.o basename.o binhash.o \
dict_fail.o msg_rate_delay.o dict_surrogate.o warn_stat.o \
dict_sockmap.o line_number.o recv_pass_attr.o pass_accept.o \
poll_fd.o timecmp.o $(NON_PLUGIN_MAP_OBJ) dict_pipe.o dict_random.o \
- valid_utf8_hostname.o midna.o argv_splitq.o balpar.o dict_union.o \
- extpar.o dict_inline.o
+ valid_utf8_hostname.o midna_domain.o argv_splitq.o balpar.o dict_union.o \
+ extpar.o dict_inline.o casefold.o dict_utf8.o
# MAP_OBJ is for maps that may be dynamically loaded with dynamicmaps.cf.
# When hard-linking these, makedefs sets NON_PLUGIN_MAP_OBJ=$(MAP_OBJ),
# otherwise it sets the PLUGIN_* macros.
edit_file.h dict_cache.h dict_thash.h ip_match.h nbbio.h base32_code.h \
dict_fail.h warn_stat.h dict_sockmap.h line_number.h timecmp.h \
slmdb.h compat_va_copy.h dict_pipe.h dict_random.h \
- valid_utf8_hostname.h midna.h dict_union.h dict_inline.h check_arg.h
+ valid_utf8_hostname.h midna_domain.h dict_union.h dict_inline.h check_arg.h
TESTSRC = fifo_open.c fifo_rdwr_bug.c fifo_rdonly_bug.c select_bug.c \
stream_test.c dup2_pass_on_exec.c
DEFS = -I. -D$(SYSTYPE)
unix_recv_fd unix_send_fd stream_recv_fd stream_send_fd hex_code \
myaddrinfo myaddrinfo4 inet_proto sane_basename format_tv \
valid_utf8_string ip_match base32_code msg_rate_delay netstring \
- vstream timecmp dict_cache midna
+ vstream timecmp dict_cache midna_domain casefold
PLUGIN_MAP_SO = $(LIB_PREFIX)pcre$(LIB_SUFFIX)
LIB_DIR = ../../lib
$(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS)
mv junk $@.o
-midna: $(LIB)
+midna_domain: $(LIB)
+ mv $@.o junk
+ $(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS)
+ mv junk $@.o
+
+casefold: $(LIB)
mv $@.o junk
$(CC) $(CFLAGS) -DTEST -o $@ $@.c $(LIB) $(SYSLIBS)
mv junk $@.o
dict_cidr_test attr_scan_plain_test htable_test hex_code_test \
myaddrinfo_test format_tv_test ip_match_test name_mask_tests \
base32_code_test dict_thash_test surrogate_test timecmp_test \
- dict_static_test dict_inline_test midna_test
+ dict_static_test dict_inline_test midna_domain_test casefold_test \
+ dict_utf8_test
root_tests:
$(SHLIB_ENV) ./base32_code
dict_thash_test: ../postmap/postmap dict_thash.map
- $(SHLIB_ENV) ../postmap/postmap -s texthash:dict_thash.map >dict_thash.tmp 2>&1
- sort dict_thash.tmp | diff -b dict_thash.map -
+ $(SHLIB_ENV) ../postmap/postmap -s texthash:dict_thash.map | sort >dict_thash.tmp 2>&1
+ tr '[A-Z]' '[a-z]' <dict_thash.map | sort | diff -b dict_thash.tmp -
rm -f dict_thash.tmp
surrogate_test: dict_open surrogate.ref
$(SHLIB_ENV) ./dict_open inline:'{ foo=xx x' read </dev/null; \
$(SHLIB_ENV) ./dict_open inline:'{ foo=xx {x=y}x}' read </dev/null; \
(echo get foo; echo get bar; echo get baz) | $(SHLIB_ENV) \
- ./dict_open inline:'{ foo=xx, { bar = lotsa stuff }}' read; \
+ ./dict_open inline:'{ foo=XX, { bAr = lotsa stuff }}' read fold_fix; \
) >dict_inline.tmp 2>&1
diff dict_inline.ref dict_inline.tmp
rm -f dict_inline.tmp
-midna_test: midna midna_test.in midna_test.ref
- $(SHLIB_ENV) ./midna <midna_test.in >midna_test.tmp 2>&1
- diff midna_test.ref midna_test.tmp
- rm -f midna_test.tmp
+midna_domain_test: midna_domain midna_domain_test.in midna_domain_test.ref
+ $(SHLIB_ENV) ./midna_domain <midna_domain_test.in >midna_domain_test.tmp 2>&1
+ diff midna_domain_test.ref midna_domain_test.tmp
+ rm -f midna_domain_test.tmp
+
+casefold_test: casefold casefold_test.in casefold_test.ref
+ $(SHLIB_ENV) ./casefold <casefold_test.in >casefold_test.tmp 2>&1
+ diff casefold_test.ref casefold_test.tmp
+ rm -f casefold_test.tmp
+
+dict_utf8_test: dict_open dict_utf8_test.in dict_utf8_test.ref
+ $(SHLIB_ENV) sh dict_utf8_test.in >dict_utf8_test.tmp 2>&1
+ diff dict_utf8_test.ref dict_utf8_test.tmp
+ rm -f dict_utf8_test.tmp
depend: $(MAKES)
(sed '1,/^# do not edit/!d' Makefile.in; \
binhash.o: msg.h
binhash.o: mymalloc.h
binhash.o: sys_defs.h
+casefold.o: casefold.c
+casefold.o: check_arg.h
+casefold.o: msg.h
+casefold.o: stringops.h
+casefold.o: sys_defs.h
+casefold.o: vbuf.h
+casefold.o: vstring.h
chroot_uid.o: chroot_uid.c
chroot_uid.o: chroot_uid.h
chroot_uid.o: msg.h
dict_inline.o: argv.h
dict_inline.o: check_arg.h
dict_inline.o: dict.h
+dict_inline.o: dict_ht.h
dict_inline.o: dict_inline.c
dict_inline.o: dict_inline.h
dict_inline.o: htable.h
dict_thash.o: argv.h
dict_thash.o: check_arg.h
dict_thash.o: dict.h
+dict_thash.o: dict_ht.h
dict_thash.o: dict_thash.c
dict_thash.o: dict_thash.h
dict_thash.o: htable.h
dict_thash.o: iostuff.h
dict_thash.o: msg.h
dict_thash.o: myflock.h
-dict_thash.o: mymalloc.h
dict_thash.o: readlline.h
dict_thash.o: stringops.h
dict_thash.o: sys_defs.h
dict_thash.o: vbuf.h
dict_thash.o: vstream.h
dict_thash.o: vstring.h
-dict_thash.o: warn_stat.h
dict_union.o: argv.h
dict_union.o: check_arg.h
dict_union.o: dict.h
dict_unix.o: vbuf.h
dict_unix.o: vstream.h
dict_unix.o: vstring.h
+dict_utf8.o: argv.h
+dict_utf8.o: check_arg.h
+dict_utf8.o: dict.h
+dict_utf8.o: dict_utf8.c
+dict_utf8.o: msg.h
+dict_utf8.o: myflock.h
+dict_utf8.o: mymalloc.h
+dict_utf8.o: stringops.h
+dict_utf8.o: sys_defs.h
+dict_utf8.o: vbuf.h
+dict_utf8.o: vstream.h
+dict_utf8.o: vstring.h
dir_forest.o: check_arg.h
dir_forest.o: dir_forest.c
dir_forest.o: dir_forest.h
load_file.o: vstream.h
load_file.o: warn_stat.h
load_lib.o: load_lib.c
-load_lib.o: load_lib.h
-load_lib.o: msg.h
load_lib.o: sys_defs.h
lowercase.o: check_arg.h
lowercase.o: lowercase.c
match_ops.o: vbuf.h
match_ops.o: vstream.h
match_ops.o: vstring.h
-midna.o: check_arg.h
-midna.o: ctable.h
-midna.o: midna.c
-midna.o: midna.h
-midna.o: msg.h
-midna.o: mymalloc.h
-midna.o: stringops.h
-midna.o: sys_defs.h
-midna.o: valid_hostname.h
-midna.o: vbuf.h
-midna.o: vstring.h
+midna_domain.o: check_arg.h
+midna_domain.o: ctable.h
+midna_domain.o: midna_domain.c
+midna_domain.o: midna_domain.h
+midna_domain.o: msg.h
+midna_domain.o: mymalloc.h
+midna_domain.o: stringops.h
+midna_domain.o: sys_defs.h
+midna_domain.o: valid_hostname.h
+midna_domain.o: vbuf.h
+midna_domain.o: vstring.h
msg.o: msg.c
msg.o: msg.h
msg.o: msg_output.h
valid_hostname.o: vbuf.h
valid_hostname.o: vstring.h
valid_utf8_hostname.o: check_arg.h
-valid_utf8_hostname.o: midna.h
+valid_utf8_hostname.o: midna_domain.h
valid_utf8_hostname.o: msg.h
valid_utf8_hostname.o: mymalloc.h
valid_utf8_hostname.o: stringops.h
--- /dev/null
+/*++
+/* NAME
+/* casefold 3
+/* SUMMARY
+/* casefold text for caseless comparison
+/* SYNOPSIS
+/* #include <stringops.h>
+/*
+/* char *casefold(
+/* VSTRING *src,
+/* const char *src,
+/* CONST_CHAR_STAR *err)
+/* DESCRIPTION
+/* casefold() converts text to a form that is suitable for
+/* caseless comparison, rather than presentation to humans.
+/*
+/* When compiled without EAI support, casefold() implements
+/* ASCII case folding, leaving non-ASCII byte values unchanged.
+/* This mode has no error returns.
+/*
+/* When compiled with EAI support, casefold() implements UTF-8
+/* case folding using the en_US locale, as recommended when
+/* the conversion result is not meant to be presented to humans.
+/* When conversion fails the result is null, and the pointer
+/* referenced by err is updated.
+/*
+/* With the ICU 4.8 library, there is no casefold error for
+/* UTF-8 code points U+0000..U+10FFFF (including surrogate
+/* range), not even when running inside an empty chroot jail.
+/*
+/* Arguments:
+/* .IP src
+/* Null-terminated input string.
+/* .IP dest
+/* Output buffer, null-terminated if the function completes
+/* without reporting an error.
+/* .IP err
+/* Null pointer, or pointer to "const char *". for descriptive
+/* text about errors.
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+/* System library. */
+
+#include <sys_defs.h>
+#include <string.h>
+#include <ctype.h>
+#ifndef NO_EAI
+#include <unicode/ucasemap.h>
+#include <unicode/ustring.h>
+#include <unicode/uchar.h>
+#endif
+
+/* Utility library. */
+
+#include <msg.h>
+#include <stringops.h>
+
+#define STR(x) vstring_str(x)
+#define LEN(x) VSTRING_LEN(x)
+
+/* casefold - casefold an UTF-8 string */
+
+char *casefold(VSTRING *dest, const char *src, CONST_CHAR_STAR *err)
+{
+#ifdef NO_EAI
+
+ /*
+ * ASCII mode only
+ */
+ vstring_strcpy(dest, src);
+ return (lowercase(STR(dest)));
+#else
+
+ /*
+ * Unicode mode.
+ */
+ static UCaseMap *csm = 0;
+ UErrorCode error;
+ ssize_t space_needed;
+ int n;
+
+ /*
+ * All-ASCII input.
+ */
+ if (allascii(src)) {
+ vstring_strcpy(dest, src);
+ return (lowercase(STR(dest)));
+ }
+
+ /*
+ * One-time initialization. With ICU 4.8 this works while chrooted.
+ */
+ if (csm == 0) {
+ error = U_ZERO_ERROR;
+ csm = ucasemap_open("en_US", U_FOLD_CASE_DEFAULT, &error);
+ if (U_SUCCESS(error) == 0)
+ msg_fatal("ucasemap_open error: %s", u_errorName(error));
+ }
+
+ /*
+ * Fold the input, adjusting the buffer size if needed. Safety: don't
+ * loop forever.
+ */
+ VSTRING_RESET(dest);
+ for (n = 0; n < 3; n++) {
+ error = U_ZERO_ERROR;
+ space_needed =
+ ucasemap_utf8FoldCase(csm, STR(dest), vstring_avail(dest),
+ src, strlen(src), &error);
+ if (error == U_BUFFER_OVERFLOW_ERROR) {
+ VSTRING_SPACE(dest, space_needed);
+ } else {
+ break;
+ }
+ }
+
+ /*
+ * Report the result. With ICU 4.8, there are no casefolding errors for
+ * the entire RFC 3629 Unicode range (code points U+0000..U+10FFFF
+ * including surrogates).
+ */
+ if (U_SUCCESS(error) == 0) {
+ if (err)
+ *err = u_errorName(error);
+ return (0);
+ } else {
+ /* Position the write pointer at the null terminator. */
+ VSTRING_AT_OFFSET(dest, space_needed - 1);
+ return (STR(dest));
+ }
+#endif /* NO_EAI */
+}
+
+#ifdef TEST
+
+static void encode_utf8(VSTRING *buffer, int codepoint)
+{
+ const char myname[] = "encode_utf8";
+
+ VSTRING_RESET(buffer);
+ if (codepoint < 0x80) {
+ VSTRING_ADDCH(buffer, codepoint);
+ } else if (codepoint < 0x800) {
+ VSTRING_ADDCH(buffer, 0xc0 | (codepoint >> 6));
+ VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f));
+ } else if (codepoint < 0x10000) {
+ VSTRING_ADDCH(buffer, 0xe0 | (codepoint >> 12));
+ VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 6) & 0x3f));
+ VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f));
+ } else if (codepoint <= 0x10FFFF) {
+ VSTRING_ADDCH(buffer, 0xf0 | (codepoint >> 18));
+ VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 12) & 0x3f));
+ VSTRING_ADDCH(buffer, 0x80 | ((codepoint >> 6) & 0x3f));
+ VSTRING_ADDCH(buffer, 0x80 | (codepoint & 0x3f));
+ } else {
+ msg_panic("%s: out-of-range codepoint U+%X", myname, codepoint);
+ }
+ VSTRING_TERMINATE(buffer);
+}
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <locale.h>
+
+#include <vstream.h>
+#include <vstring_vstream.h>
+#include <msg_vstream.h>
+
+int main(int argc, char **argv)
+{
+ VSTRING *buffer = vstring_alloc(1);
+ VSTRING *dest = vstring_alloc(1);
+ char *bp;
+ char *conv_res;
+ const char *fold_err;
+ char *cmd;
+ int codepoint, first, last;
+
+ if (setlocale(LC_ALL, "C") == 0)
+ msg_fatal("setlocale(LC_ALL, C) failed: %m");
+
+ msg_vstream_init(argv[0], VSTREAM_ERR);
+
+ util_utf8_enable = 1;
+
+ VSTRING_SPACE(buffer, 256); /* chroot pathname */
+
+ while (vstring_fgets_nonl(buffer, VSTREAM_IN)) {
+ bp = STR(buffer);
+ msg_info("> %s", bp);
+ cmd = mystrtok(&bp, CHARS_SPACE);
+ if (cmd == 0 || *cmd == '#')
+ continue;
+ while (ISSPACE(*bp))
+ bp++;
+
+ /*
+ * Null-terminated string.
+ */
+ if (strcmp(cmd, "fold") == 0) {
+ if ((conv_res = casefold(dest, bp, &fold_err)) != 0)
+ msg_info("\"%s\" ->fold \"%s\"", bp, conv_res);
+ else
+ msg_warn("cannot casefold \"%s\": %s", bp, fold_err);
+ }
+
+ /*
+ * Codepoint range.
+ */
+ else if (strcmp(cmd, "range") == 0
+ && sscanf(bp, "%i %i", &first, &last) == 2
+ && first <= last) {
+ for (codepoint = first; codepoint <= last; codepoint++) {
+ if (codepoint >= 0xD800 && codepoint <= 0xDFFF) {
+ msg_warn("skipping surrogate range");
+ codepoint = 0xDFFF;
+ } else {
+ encode_utf8(buffer, codepoint);
+ if (msg_verbose)
+ msg_info("U+%X -> %s", codepoint, STR(buffer));
+ if (valid_utf8_string(STR(buffer), LEN(buffer)) == 0)
+ msg_fatal("bad utf-8 encoding for U+%X", codepoint);
+ if (casefold(dest, STR(buffer), &fold_err) == 0)
+ msg_warn("casefold error for U+%X: %s",
+ codepoint, fold_err);
+ }
+ }
+ msg_info("range completed: 0x%x..0x%x", first, last);
+ }
+
+ /*
+ * Chroot directory.
+ */
+ else if (strcmp(cmd, "chroot") == 0
+ && sscanf(bp, "%255s", STR(buffer)) == 1) {
+ if (geteuid() == 0) {
+ if (chdir(STR(buffer)) < 0)
+ msg_fatal("chdir(%s): %m", STR(buffer));
+ if (chroot(STR(buffer)) < 0)
+ msg_fatal("chroot(%s): %m", STR(buffer));
+ msg_info("chroot %s completed", STR(buffer));
+ }
+ }
+
+ /*
+ * Verbose.
+ */
+ else if (strcmp(cmd, "verbose") == 0
+ && sscanf(bp, "%i", &msg_verbose) == 1) {
+ /* void */ ;
+ }
+
+ /*
+ * Usage
+ */
+ else {
+ msg_info("Usage: %s chroot <path> | fold <text> | range <first> <last> | verbose <int>",
+ argv[0]);
+ }
+ }
+ exit(0);
+}
+
+#endif /* TEST */
--- /dev/null
+# Ignored when not running as root.
+chroot /tmp
+# Casefold U+0000 .. U+10FFFF excluding surrogates.
+range 0x0 0xD7FF
+range 0xD800 0xD800
+range 0xDFFF 0xDFFF
+range 0xE000 0x10FFFF
+# Demonstrate that range is not a noop.
+verbose 1
+range 0xE000 0xE007
+verbose 0
+# Upper-case greek -> lower-case greek.
+fold Δημοσθένους.example.com
+# Upper-case ASCII -> lower-case ASCII.
+fold HeLlO.ExAmPlE.CoM
+# Folding does not change aliases for '.'.
+fold x。example.com
+fold x．example.com
+fold x｡example.com
--- /dev/null
+./casefold: > # Ignored when not running as root.
+./casefold: > chroot /tmp
+./casefold: > # Casefold U+0000 .. U+10FFFF excluding surrogates.
+./casefold: > range 0x0 0xD7FF
+./casefold: range completed: 0x0..0xd7ff
+./casefold: > range 0xD800 0xD800
+./casefold: warning: skipping surrogate range
+./casefold: range completed: 0xd800..0xd800
+./casefold: > range 0xDFFF 0xDFFF
+./casefold: warning: skipping surrogate range
+./casefold: range completed: 0xdfff..0xdfff
+./casefold: > range 0xE000 0x10FFFF
+./casefold: range completed: 0xe000..0x10ffff
+./casefold: > # Demonstrate that range is not a noop.
+./casefold: > verbose 1
+./casefold: > range 0xE000 0xE007
+./casefold: U+E000 ->
+./casefold: U+E001 ->
+./casefold: U+E002 ->
+./casefold: U+E003 ->
+./casefold: U+E004 ->
+./casefold: U+E005 ->
+./casefold: U+E006 ->
+./casefold: U+E007 ->
+./casefold: range completed: 0xe000..0xe007
+./casefold: > verbose 0
+./casefold: > # Upper-case greek -> lower-case greek.
+./casefold: > fold Δημοσθένους.example.com
+./casefold: "Δημοσθένους.example.com" ->fold "δημοσθένουσ.example.com"
+./casefold: > # Upper-case ASCII -> lower-case ASCII.
+./casefold: > fold HeLlO.ExAmPlE.CoM
+./casefold: "HeLlO.ExAmPlE.CoM" ->fold "hello.example.com"
+./casefold: > # Folding does not change aliases for '.'.
+./casefold: > fold x。example.com
+./casefold: "x。example.com" ->fold "x。example.com"
+./casefold: > fold x．example.com
+./casefold: "x．example.com" ->fold "x．example.com"
+./casefold: > fold x｡example.com
+./casefold: "x｡example.com" ->fold "x｡example.com"
"open_lock", DICT_FLAG_OPEN_LOCK, /* permanent lock upon open */
"bulk_update", DICT_FLAG_BULK_UPDATE, /* bulk update if supported */
"multi_writer", DICT_FLAG_MULTI_WRITER, /* multi-writer safe */
+ "utf8_enable", DICT_FLAG_UTF8_ENABLE, /* enable UTF-8 checks/fold */
+ "utf8_proxy", DICT_FLAG_UTF8_PROXY, /* UTF-8 proxy is present */
0,
};
VSTRING *fold_buf; /* key folding buffer */
DICT_OWNER owner; /* provenance */
int error; /* last operation only */
+ ssize_t size; /* size of this thing */
DICT_JMP_BUF *jbuf; /* exception handling */
} DICT;
#define DICT_FLAG_OPEN_LOCK (1<<16) /* perm lock if not multi-writer safe */
#define DICT_FLAG_BULK_UPDATE (1<<17) /* optimize for bulk updates */
#define DICT_FLAG_MULTI_WRITER (1<<18) /* multi-writer safe map */
+#define DICT_FLAG_UTF8_ENABLE (1<<19) /* enable UTF-8 checks */
+#define DICT_FLAG_UTF8_PROXY (1<<20) /* UTF-8 proxy layer is present */
+
+#define DICT_FLAG_UTF8_MASK (DICT_FLAG_UTF8_ENABLE)
/* IMPORTANT: Update the dict_mask[] table when the above changes */
#define DICT_FLAG_RQST_MASK (DICT_FLAG_FOLD_ANY | DICT_FLAG_LOCK | \
DICT_FLAG_DUP_REPLACE | DICT_FLAG_DUP_WARN | \
DICT_FLAG_DUP_IGNORE | DICT_FLAG_SYNC_UPDATE | \
- DICT_FLAG_PARANOID)
+ DICT_FLAG_PARANOID | DICT_FLAG_UTF8_MASK)
#define DICT_FLAG_INST_MASK ~(DICT_FLAG_IMPL_MASK | DICT_FLAG_RQST_MASK)
+ /*
+ * Feature tests.
+ */
+extern int util_utf8_enable;
+
+#define DICT_IS_ENABLE_UTF8(flags) \
+ (util_utf8_enable && (flags & DICT_FLAG_UTF8_MASK))
+
/*
* dict->error values. Errors must be negative; smtpd_check depends on this.
*/
extern const char *dict_changed_name(void);
extern const char *dict_flags_str(int);
extern int dict_flags_mask(const char *);
+extern void dict_type_override(DICT *, const char *);
+
+ /*
+ * Check and convert UTF-8 keys and values.
+ */
+extern DICT *dict_utf8_encapsulate(DICT *);
+extern char *dict_utf8_check_fold(DICT *, const char *, CONST_CHAR_STAR *);
+extern int dict_utf8_check(const char *, CONST_CHAR_STAR *);
/*
* Driver for interactive or scripted tests.
* systems have bugs in their implementation.
*/
#ifdef NO_SIGSETJMP
-#define dict_setjmp(stream) setjmp((stream)->jbuf[0])
-#define dict_longjmp(stream, val) longjmp((stream)->jbuf[0], (val))
+#define dict_setjmp(dict) setjmp((dict)->jbuf[0])
+#define dict_longjmp(dict, val) longjmp((dict)->jbuf[0], (val))
#else
-#define dict_setjmp(stream) sigsetjmp((stream)->jbuf[0], 1)
-#define dict_longjmp(stream, val) siglongjmp((stream)->jbuf[0], (val))
+#define dict_setjmp(dict) sigsetjmp((dict)->jbuf[0], 1)
+#define dict_longjmp(dict, val) siglongjmp((dict)->jbuf[0], (val))
#endif
-#define dict_isjmp(stream) ((stream)->jbuf != 0)
+#define dict_isjmp(dict) ((dict)->jbuf != 0)
/*
* Temporary API. If exception handling proves to be useful,
dict->owner.status = DICT_OWNER_UNKNOWN;
dict->owner.uid = INT_MAX;
dict->error = DICT_ERR_NONE;
+ dict->size = size;
dict->jbuf = 0;
return dict;
}
if ((fd = open(cdb_path, O_RDONLY)) < 0)
DICT_CDBQ_OPEN_RETURN(dict_surrogate(DICT_TYPE_CDB, path,
- O_RDONLY, dict_flags,
+ O_RDONLY, dict_flags,
"open database %s: %m", cdb_path));
dict_cdbq = (DICT_CDBQ *) dict_alloc(DICT_TYPE_CDB,
DICT *real_dict = dict_debug->real_dict;
const char *result;
+ real_dict->flags = dict->flags;
result = dict_get(real_dict, key);
+ dict->flags = real_dict->flags;
msg_info("%s:%s lookup: \"%s\" = \"%s\"", dict->type, dict->name, key,
result ? result : real_dict->error ? "error" : "not_found");
DICT_ERR_VAL_RETURN(dict, real_dict->error, result);
DICT *real_dict = dict_debug->real_dict;
int result;
+ real_dict->flags = dict->flags;
result = dict_put(real_dict, key, value);
+ dict->flags = real_dict->flags;
msg_info("%s:%s update: \"%s\" = \"%s\": %s", dict->type, dict->name,
key, value, result == 0 ? "success" : real_dict->error ?
"error" : "failed");
DICT *real_dict = dict_debug->real_dict;
int result;
+ real_dict->flags = dict->flags;
result = dict_del(real_dict, key);
+ dict->flags = real_dict->flags;
msg_info("%s:%s delete: \"%s\": %s", dict->type, dict->name, key,
result == 0 ? "success" : real_dict->error ?
"error" : "failed");
DICT *real_dict = dict_debug->real_dict;
int result;
+ real_dict->flags = dict->flags;
result = dict_seq(real_dict, function, key, value);
+ dict->flags = real_dict->flags;
if (result == 0)
msg_info("%s:%s sequence: \"%s\" = \"%s\"", dict->type, dict->name,
*key, *value);
/* System library. */
#include <sys_defs.h>
+#include <string.h>
/* Utility library. */
#include <msg.h>
#include <mymalloc.h>
-#include <htable.h>
#include <stringops.h>
#include <dict.h>
+#include <dict_ht.h>
#include <dict_inline.h>
/* Application-specific. */
-typedef struct {
- DICT dict; /* generic members */
- HTABLE *table; /* lookup table */
- HTABLE_INFO **info; /* for iterator */
- HTABLE_INFO **cursor; /* ditto */
-} DICT_INLINE;
-
-/* dict_inline_lookup - search inline table */
-
-static const char *dict_inline_lookup(DICT *dict, const char *name)
-{
- DICT_INLINE *dict_inline = (DICT_INLINE *) dict;
- const char *result = 0;
-
- /*
- * Optionally fold the key.
- */
- if (dict->flags & DICT_FLAG_FOLD_FIX) {
- if (dict->fold_buf == 0)
- dict->fold_buf = vstring_alloc(10);
- vstring_strcpy(dict->fold_buf, name);
- name = lowercase(vstring_str(dict->fold_buf));
- }
-
- /*
- * Look up the value.
- */
- result = htable_find(dict_inline->table, name);
-
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, result);
-}
-
-/* dict_inline_sequence - traverse the dictionary */
-
-static int dict_inline_sequence(DICT *dict, int function,
- const char **key, const char **value)
-{
- const char *myname = "dict_inline_sequence";
- DICT_INLINE *dict_inline = (DICT_INLINE *) dict;
-
- /*
- * Determine and execute the seek function.
- */
- switch (function) {
- case DICT_SEQ_FUN_FIRST:
- if (dict_inline->info == 0)
- dict_inline->info = htable_list(dict_inline->table);
- dict_inline->cursor = dict_inline->info;
- break;
- case DICT_SEQ_FUN_NEXT:
- if (dict_inline->cursor[0])
- dict_inline->cursor += 1;
- break;
- default:
- msg_panic("%s: invalid function: %d", myname, function);
- }
-
- /*
- * Return the entry under the cursor.
- */
- if (dict_inline->cursor[0]) {
- *key = dict_inline->cursor[0]->key;
- *value = dict_inline->cursor[0]->value;
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_SUCCESS);
- } else {
- *key = 0;
- *value = 0;
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL);
- }
-}
-
-/* dict_inline_close - disassociate from inline table */
-
-static void dict_inline_close(DICT *dict)
-{
- DICT_INLINE *dict_inline = (DICT_INLINE *) dict;
-
- htable_free(dict_inline->table, myfree);
- if (dict_inline->info)
- myfree((void *) dict_inline->info);
- if (dict->fold_buf)
- vstring_free(dict->fold_buf);
- dict_free(dict);
-}
-
/* dict_inline_open - open inline table */
DICT *dict_inline_open(const char *name, int open_flags, int dict_flags)
{
- DICT_INLINE *dict_inline;
+ DICT *dict;
char *cp, *saved_name = 0;
size_t len;
- HTABLE *table = 0;
char *nameval, *vname, *value;
const char *err = 0;
char *xperr = 0;
+ int count = 0;
/*
* Clarity first. Let the optimizer worry about redundant code.
myfree(saved_name); \
if (xperr != 0) \
myfree(xperr); \
- if (table != 0) \
- htable_free(table, myfree); \
return (__d); \
} while (0)
"%s:%s map requires O_RDONLY access mode",
DICT_TYPE_INLINE, name));
+ /*
+ * UTF-8 syntax check.
+ */
+ if (DICT_IS_ENABLE_UTF8(dict_flags)
+ && allascii(name) == 0
+ && valid_utf8_string(name, strlen(name)) == 0)
+ DICT_INLINE_RETURN(dict_surrogate(DICT_TYPE_INLINE, name,
+ open_flags, dict_flags,
+ "bad UTF-8 syntax: \"%s:%s\"; "
+ "need \"%s:{name=value...}\"",
+ DICT_TYPE_INLINE, name,
+ DICT_TYPE_INLINE));
+
/*
* Parse the table into its constituent name=value pairs.
*/
DICT_TYPE_INLINE, name,
DICT_TYPE_INLINE));
- table = htable_create(5);
+ /*
+ * Reuse the "internal" dictionary type.
+ */
+ dict = dict_open3(DICT_TYPE_HT, name, open_flags, dict_flags);
+ dict_type_override(dict, DICT_TYPE_INLINE);
while ((nameval = mystrtokq(&cp, CHARS_COMMA_SP, CHARS_BRACE)) != 0) {
if ((nameval[0] != CHARS_BRACE[0]
|| (err = xperr = extpar(&nameval, CHARS_BRACE, EXTPAR_FLAG_STRIP)) == 0)
&& (err = split_nameval(nameval, &vname, &value)) != 0)
break;
- (void) htable_enter(table, vname, mystrdup(value));
+
+ /* No duplicate checks. See comments in dict_thash.c. */
+ dict->update(dict, vname, value);
+ count += 1;
}
- if (err != 0 || table->used == 0)
+ if (err != 0 || count == 0) {
+ dict->close(dict);
DICT_INLINE_RETURN(dict_surrogate(DICT_TYPE_INLINE, name,
open_flags, dict_flags,
"%s: \"%s:%s\"; "
err != 0 ? err : "empty table",
DICT_TYPE_INLINE, name,
DICT_TYPE_INLINE));
+ }
+ dict->owner.status = DICT_OWNER_TRUSTED;
- /*
- * Bundle up the result.
- */
- dict_inline = (DICT_INLINE *)
- dict_alloc(DICT_TYPE_INLINE, name, sizeof(*dict_inline));
- dict_inline->dict.lookup = dict_inline_lookup;
- dict_inline->dict.sequence = dict_inline_sequence;
- dict_inline->dict.close = dict_inline_close;
- dict_inline->dict.flags = dict_flags | DICT_FLAG_FIXED;
- dict_inline->dict.owner.status = DICT_OWNER_TRUSTED;
- if (dict_flags & DICT_FLAG_FOLD_FIX)
- dict_inline->dict.fold_buf = vstring_alloc(10);
- dict_inline->info = 0;
- dict_inline->table = table;
- table = 0;
- DICT_INLINE_RETURN(DICT_DEBUG (&dict_inline->dict));
+ DICT_INLINE_RETURN(DICT_DEBUG (dict));
}
owner=trusted (uid=2147483647)
owner=trusted (uid=2147483647)
> get foo
-foo=xx
+foo=XX
> get bar
bar=lotsa stuff
> get baz
name = lowercase(vstring_str(dict->fold_buf));
}
mdb_key.mv_data = (void *) name;
-
mdb_value.mv_data = (void *) value;
mdb_key.mv_size = strlen(name);
mdb_value.mv_size = strlen(value);
if (mdb_value.mv_data != 0 && mdb_value.mv_size > 0)
*value = SCOPY(dict_lmdb->val_buf, mdb_value.mv_data,
mdb_value.mv_size);
+ else
+ *value = ""; /* XXX */
break;
/*
/* int dict_longjmp(dict, val)
/* DICT *dict;
/* int val;
+/*
+/* void dict_type_override(dict, type)
+/* DICT *dict;
+/* const char *type;
/* DESCRIPTION
/* This module implements a low-level interface to multiple
/* physical dictionary types.
/* With databases whose lookup fields are fixed-case strings,
/* fold the search string to lower case before accessing the
/* database. This includes hash:, cdb:, dbm:. nis:, ldap:,
-/* *sql.
+/* *sql. WARNING: case folding is supported only for ASCII or
+/* valid UTF-8.
/* .IP DICT_FLAG_FOLD_MUL
/* With databases where one lookup field can match both upper
/* and lower case, fold the search key to lower case before
-/* accessing the database. This includes regexp: and pcre:
+/* accessing the database. This includes regexp: and pcre:.
+/* WARNING: case folding is supported only for ASCII or valid
+/* UTF-8.
/* .IP DICT_FLAG_FOLD_ANY
/* Short-hand for (DICT_FLAG_FOLD_FIX | DICT_FLAG_FOLD_MUL).
/* .IP DICT_FLAG_SYNC_UPDATE
/* and must trap exceptions from the database client with dict_setjmp().
/* .IP DICT_FLAG_DEBUG
/* Enable additional logging.
+/* .IP DICT_FLAG_UTF8_ENABLE
+/* With util_utf8_enable != 0, require that lookup/update/delete
+/* keys and values are valid UTF-8. Skip a lookup/update/delete
+/* request with a non-UTF-8 key, skip an update request with
+/* a non-UTF-8 value, and fail a lookup request with a non-UTF-8
+/* value.
/* .PP
/* Specify DICT_FLAG_NONE for no special processing.
/*
/* dict_open3() takes separate arguments for dictionary type and
/* name, but otherwise performs the same functions as dict_open().
/*
+/* The dict_get(), dict_put(), dict_del(), and dict_seq()
+/* macros evaluate their first argument multiple times.
+/* These names should have been in uppercase.
+/*
/* dict_get() retrieves the value stored in the named dictionary
/* under the given key. A null pointer means the value was not found.
/* As with dict_lookup(), the result is owned by the lookup table
/* NB: non-local jumps such as dict_longjmp() are not safe for
/* jumping out of any routine that manipulates DICT data.
/* longjmp() like calls are best avoided in signal handlers.
+/*
+/* dict_type_override() changes the symbolic dictionary type.
+/* This is used by dictionaries whose internals are based on
+/* some other dictionary type.
/* DIAGNOSTICS
/* Fatal error: open error, unsupported dictionary type, attempt to
/* update non-writable dictionary.
msg_fatal("%s:%s: unable to get exclusive lock: %m",
dict_type, dict_name);
}
+ /* Last step: insert proxy for UTF-8 syntax checks and casefolding. */
+ if ((dict->flags & DICT_FLAG_UTF8_PROXY) == 0
+ && DICT_IS_ENABLE_UTF8(dict_flags))
+ dict = dict_utf8_encapsulate(dict);
return (dict);
}
return (old_cb);
}
+/* dict_type_override - disguise a dictionary type */
+
+void dict_type_override(DICT *dict, const char *type)
+{
+ myfree(dict->type);
+ dict->type = mystrdup(type);
+}
+
#ifdef TEST
/*
dict_flags |= DICT_FLAG_LOCK;
if ((dict_flags & (DICT_FLAG_DUP_WARN | DICT_FLAG_DUP_IGNORE)) == 0)
dict_flags |= DICT_FLAG_DUP_REPLACE;
+ dict_flags |= DICT_FLAG_UTF8_ENABLE;
vstream_fflush(VSTREAM_OUT);
dict_name = argv[optind];
dict_allow_surrogate = 1;
+ util_utf8_enable = 1;
dict = dict_open(dict_name, open_flags, dict_flags);
dict_register(dict_name, dict);
vstream_printf("owner=%s (uid=%ld)\n",
if (dict_put(dict, key, value) != 0)
vstream_printf("%s: %s\n", key, dict->error ?
"error" : "not updated");
- else
- vstream_printf("%s=%s\n", key, value);
} else if (strcmp(cmd, "first") == 0 && !key && !value) {
if (dict_seq(dict, DICT_SEQ_FUN_FIRST, &key, &value) == 0)
vstream_printf("%s=%s\n", key, value);
> del foo
foo: deleted
> put baz bazval
-baz=bazval
> get baz
baz=bazval
> del baz
/* Utility library. */
#include <msg.h>
-#include <mymalloc.h>
-#include <htable.h>
#include <iostuff.h>
#include <vstring.h>
#include <stringops.h>
#include <readlline.h>
#include <dict.h>
+#include <dict_ht.h>
#include <dict_thash.h>
-#include <warn_stat.h>
/* Application-specific. */
-typedef struct {
- DICT dict; /* generic members */
- HTABLE *table; /* in-memory hash */
- HTABLE_INFO **info; /* for iterator */
- HTABLE_INFO **cursor; /* ditto */
-} DICT_THASH;
-
#define STR vstring_str
-
-/* dict_thash_lookup - find database entry */
-
-static const char *dict_thash_lookup(DICT *dict, const char *name)
-{
- DICT_THASH *dict_thash = (DICT_THASH *) dict;
- const char *result = 0;
-
- /*
- * Optionally fold the key.
- */
- if (dict->flags & DICT_FLAG_FOLD_FIX) {
- if (dict->fold_buf == 0)
- dict->fold_buf = vstring_alloc(10);
- vstring_strcpy(dict->fold_buf, name);
- name = lowercase(vstring_str(dict->fold_buf));
- }
-
- /*
- * Look up the value.
- */
- result = htable_find(dict_thash->table, name);
-
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, result);
-}
-
-/* dict_thash_sequence - traverse the dictionary */
-
-static int dict_thash_sequence(DICT *dict, int function,
- const char **key, const char **value)
-{
- const char *myname = "dict_thash_sequence";
- DICT_THASH *dict_thash = (DICT_THASH *) dict;
-
- /*
- * Determine and execute the seek function.
- */
- switch (function) {
- case DICT_SEQ_FUN_FIRST:
- if (dict_thash->info == 0)
- dict_thash->info = htable_list(dict_thash->table);
- dict_thash->cursor = dict_thash->info;
- break;
- case DICT_SEQ_FUN_NEXT:
- if (dict_thash->cursor[0])
- dict_thash->cursor += 1;
- break;
- default:
- msg_panic("%s: invalid function: %d", myname, function);
- }
-
- /*
- * Return the entry under the cursor.
- */
- if (dict_thash->cursor[0]) {
- *key = dict_thash->cursor[0]->key;
- *value = dict_thash->cursor[0]->value;
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_SUCCESS);
- } else {
- *key = 0;
- *value = 0;
- DICT_ERR_VAL_RETURN(dict, DICT_ERR_NONE, DICT_STAT_FAIL);
- }
-}
-
-/* dict_thash_close - disassociate from data base */
-
-static void dict_thash_close(DICT *dict)
-{
- DICT_THASH *dict_thash = (DICT_THASH *) dict;
-
- htable_free(dict_thash->table, myfree);
- if (dict_thash->info)
- myfree((void *) dict_thash->info);
- if (dict->fold_buf)
- vstring_free(dict->fold_buf);
- dict_free(dict);
-}
+#define LEN VSTRING_LEN
/* dict_thash_open - open flat text data base */
DICT *dict_thash_open(const char *path, int open_flags, int dict_flags)
{
- DICT_THASH *dict_thash;
- VSTREAM *fp = 0;
+ DICT *dict;
+ VSTREAM *fp = 0; /* DICT_THASH_OPEN_RETURN() */
struct stat st;
time_t before;
time_t after;
- VSTRING *line_buffer = 0;
+ VSTRING *line_buffer = 0; /* DICT_THASH_OPEN_RETURN() */
int lineno;
int last_line;
char *key;
char *value;
- HTABLE *table;
- HTABLE_INFO *ht;
/*
* Let the optimizer worry about eliminating redundant code.
DICT_THASH_OPEN_RETURN(dict_surrogate(DICT_TYPE_THASH, path,
open_flags, dict_flags,
"open database %s: %m", path));
- }
+ }
+
+ /*
+ * Reuse the "internal" dictionary type.
+ */
+ dict = dict_open3(DICT_TYPE_HT, path, open_flags, dict_flags);
+ dict_type_override(dict, DICT_TYPE_THASH);
+
if (line_buffer == 0)
line_buffer = vstring_alloc(100);
last_line = 0;
- table = htable_create(13);
while (readllines(line_buffer, fp, &last_line, &lineno)) {
+ /*
+ * First some UTF-8 checks sans casefolding.
+ */
+ if (DICT_IS_ENABLE_UTF8(dict_flags)
+ && allascii(STR(line_buffer)) == 0
+ && valid_utf8_string(STR(line_buffer), LEN(line_buffer)) == 0) {
+ msg_warn("%s, line %d: non-UTF-8 input \"%s\""
+ " -- ignoring this line",
+ VSTREAM_PATH(fp), lineno, STR(line_buffer));
+ continue;
+ }
+
/*
* Split on the first whitespace character, then trim leading and
* trailing whitespace from key and value.
msg_warn("%s, line %d: record is in \"key: value\" format;"
" is this an alias file?", path, lineno);
- /*
- * Optionally fold the key.
- */
- if (dict_flags & DICT_FLAG_FOLD_FIX)
- lowercase(key);
-
/*
* Store the value under the key. Handle duplicates
- * appropriately.
+ * appropriately. XXX Move this into dict_ht, but 1) that map
+ * ignores duplicates by default and we would have to check that
+ * we won't break existing code that depends on such benavior; 2)
+ * by inlining the checks here we can degrade gracefully instead
+ * of terminating with a fatal error. See comment in dict_inline.c.
*/
- if ((ht = htable_locate(table, key)) != 0) {
+ if (dict->lookup(dict, key) != 0) {
if (dict_flags & DICT_FLAG_DUP_IGNORE) {
/* void */ ;
} else if (dict_flags & DICT_FLAG_DUP_REPLACE) {
- myfree(ht->value);
- ht->value = mystrdup(value);
+ dict->update(dict, key, value);
} else if (dict_flags & DICT_FLAG_DUP_WARN) {
msg_warn("%s, line %d: duplicate entry: \"%s\"",
path, lineno, key);
} else {
- msg_fatal("%s, line %d: duplicate entry: \"%s\"",
- path, lineno, key);
+ dict->close(dict);
+ DICT_THASH_OPEN_RETURN(dict_surrogate(DICT_TYPE_THASH, path,
+ open_flags, dict_flags,
+ "%s, line %d: duplicate entry: \"%s\"",
+ path, lineno, key));
}
} else {
- htable_enter(table, key, mystrdup(value));
+ dict->update(dict, key, value);
}
}
/*
* Yes, it is hot. Discard the result and read the file again.
*/
- htable_free(table, myfree);
+ dict->close(dict);
if (msg_verbose > 1)
msg_info("pausing to let file %s cool down", path);
doze(300000);
}
- /*
- * Create the in-memory table.
- */
- dict_thash = (DICT_THASH *)
- dict_alloc(DICT_TYPE_THASH, path, sizeof(*dict_thash));
- dict_thash->dict.lookup = dict_thash_lookup;
- dict_thash->dict.sequence = dict_thash_sequence;
- dict_thash->dict.close = dict_thash_close;
- dict_thash->dict.flags = dict_flags | DICT_FLAG_DUP_WARN | DICT_FLAG_FIXED;
- if (dict_flags & DICT_FLAG_FOLD_FIX)
- dict_thash->dict.fold_buf = vstring_alloc(10);
- dict_thash->info = 0;
- dict_thash->table = table;
- dict_thash->dict.owner.uid = st.st_uid;
- dict_thash->dict.owner.status = (st.st_uid != 0);
+ dict->owner.uid = st.st_uid;
+ dict->owner.status = (st.st_uid != 0);
- DICT_THASH_OPEN_RETURN(DICT_DEBUG (&dict_thash->dict));
+ DICT_THASH_OPEN_RETURN(DICT_DEBUG (dict));
}
attr_scan64.c 17256
attr_scan_plain.c 16924
auto_clnt.c 9819
+ABCDEF 012345
--- /dev/null
+/*++
+/* NAME
+/* dict_utf8 3
+/* SUMMARY
+/* dictionary UTF-8 helpers
+/* SYNOPSIS
+/* #include <dict.h>
+/*
+/* DICT *dict_utf8_encapsulate(
+/* DICT *dict)
+/*
+/* char *dict_utf8_check_fold(
+/* DICT *dict,
+/* const char *string,
+/* CONST_CHAR_STAR *err,
+/* int fold_flag)
+/*
+/* int dict_utf8_check(
+/* const char *string,
+/* CONST_CHAR_STAR *err)
+/* DESCRIPTION
+/* dict_utf8_encapsulate() wraps a dictionary's lookup/update/delete
+/* methods with code that enforces UTF-8 checks on keys and
+/* values, and that logs a warning when incorrect UTF-8 is
+/* encountered. The original dictionary handle becomes invalid.
+/*
+/* The wrapper code enforces a policy that maximizes application
+/* robustness. Attempts to store non-UTF-8 keys or values are
+/* skipped while reporting success, attempts to look up or
+/* delete non-UTF-8 keys are skipped while reporting success,
+/* and attempts to look up a non-UTF-8 value are flagged while
+/* reporting a configuration error.
+/*
+/* The dict_utf8_check* functions may be invoked to perform
+/* UTF-8 validity checks when util_utf8_enable is non-zero and
+/* DICT_FLAG_UTF8_ENABLE is set. Otherwise both functions
+/* always report success.
+/*
+/* dict_utf8_check_fold() optionally folds a string, and checks
+/* it for UTF-8 validity. The result is the possibly-folded
+/* string, or a null pointer in case of error.
+/*
+/* dict_utf8_check() checks a string for UTF-8 validity. The
+/* result is zero in case of error.
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+ /*
+ * System library.
+ */
+#include <sys_defs.h>
+#include <string.h>
+
+ /*
+ * Utility library.
+ */
+#include <msg.h>
+#include <stringops.h>
+#include <dict.h>
+#include <mymalloc.h>
+#include <msg.h>
+
+ /*
+ * The goal is to maximize robustness: bad UTF-8 should not appear in keys,
+ * because those are derived from controlled inputs, and values should be
+ * printable before they are stored. But if we failed to check something
+ * then it should not result in fatal errors and thus open up the system for
+ * a denial-of-service attack.
+ *
+ * Proposed over-all policy: skip attempts to store invalid UTF-8 lookup keys
+ * or values. Rationale: some storage may not permit malformed UTF-8. This
+ * maximizes program robustness. If we get an invalid lookup result, report
+ * a configuration error.
+ *
+ * LOOKUP
+ *
+ * If the key is invalid, log a warning and skip the request. Rationale: the
+ * item cannot exist.
+ *
+ * If the lookup result is invalid, log a warning and return a configuration
+ * error.
+ *
+ * UPDATE
+ *
+ * If the key is invalid, then log a warning and skip the request. Rationale:
+ * the item cannot exist.
+ *
+ * If the value is invalid, log a warning and skip the request. Rationale:
+ * storage may not permit malformed UTF-8. This maximizes program
+ * robustness.
+ *
+ * DELETE
+ *
+ * If the key is invalid, then skip the request. Rationale: the item cannot
+ * exist.
+ */
+
+/* dict_utf8_check_fold - casefold or validate string */
+
+char *dict_utf8_check_fold(DICT *dict, const char *string,
+ CONST_CHAR_STAR *err)
+{
+ int fold_flag = (dict->flags & DICT_FLAG_FOLD_ANY);
+
+ /*
+ * Casefold and implicitly validate UTF-8.
+ */
+ if (fold_flag != 0 && (fold_flag == (dict->flags & DICT_FLAG_FIXED) ?
+ DICT_FLAG_FOLD_FIX : DICT_FLAG_FOLD_MUL)) {
+ if (dict->fold_buf == 0)
+ dict->fold_buf = vstring_alloc(10);
+ return (casefold(dict->fold_buf, string, err));
+ }
+
+ /*
+ * Validate UTF-8 without casefolding.
+ */
+ if (!allascii(string) && valid_utf8_string(string, strlen(string)) == 0) {
+ if (err)
+ *err = "malformed UTF-8 or invalid codepoint";
+ return (0);
+ }
+ return ((char *) string);
+}
+
+/* dict_utf8_check validate UTF-8 string */
+
+int dict_utf8_check(const char *string, CONST_CHAR_STAR *err)
+{
+ if (!allascii(string) && valid_utf8_string(string, strlen(string)) == 0) {
+ if (err)
+ *err = "malformed UTF-8 or invalid codepoint";
+ return (0);
+ }
+ return (1);
+}
+
+/* dict_utf8_lookup - UTF-8 lookup method wrapper */
+
+static const char *dict_utf8_lookup(DICT *self, const char *key)
+{
+ DICT *dict;
+ const char *utf8_err;
+ const char *fold_res;
+ const char *value;
+
+ /*
+ * Validate and optionally fold the key, and if invalid skip the request.
+ */
+ if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) {
+ msg_warn("%s:%s: non-UTF-8 key \"%s\": %s",
+ self->type, self->name, key, utf8_err);
+ self->error = DICT_ERR_NONE;
+ return (0);
+ }
+
+ /*
+ * Proxy the request.
+ */
+ dict = (void *) self - self->size;
+ dict->flags = self->flags;
+ value = dict->lookup(dict, fold_res);
+ self->flags = dict->flags;
+ self->error = dict->error;
+
+ /*
+ * Validate the result, and if invalid fail the request.
+ */
+ if (value != 0 && dict_utf8_check(value, &utf8_err) == 0) {
+ msg_warn("%s:%s: key \"%s\": non-UTF-8 value \"%s\": %s",
+ self->type, self->name, key, value, utf8_err);
+ self->error = DICT_ERR_CONFIG;
+ return (0);
+ } else {
+ return (value);
+ }
+}
+
+/* dict_utf8_update - UTF-8 update method wrapper */
+
+static int dict_utf8_update(DICT *self, const char *key, const char *value)
+{
+ DICT *dict;
+ const char *utf8_err;
+ const char *fold_res;
+ int status;
+
+ /*
+ * Validate or fold the key, and if invalid skip the request.
+ */
+ if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) {
+ msg_warn("%s:%s: non-UTF-8 key \"%s\": %s",
+ self->type, self->name, key, utf8_err);
+ self->error = DICT_ERR_NONE;
+ return (DICT_STAT_SUCCESS);
+ }
+
+ /*
+ * Validate the value, and if invalid skip the request.
+ */
+ else if (dict_utf8_check(value, &utf8_err) == 0) {
+ msg_warn("%s:%s: key \"%s\": non-UTF-8 value \"%s\": %s",
+ self->type, self->name, key, value, utf8_err);
+ self->error = DICT_ERR_NONE;
+ return (DICT_STAT_SUCCESS);
+ }
+
+ /*
+ * Proxy the request.
+ */
+ else {
+ dict = (void *) self - self->size;
+ dict->flags = self->flags;
+ status = dict->update(dict, fold_res, value);
+ self->flags = dict->flags;
+ self->error = dict->error;
+ return (status);
+ }
+}
+
+/* dict_utf8_delete - UTF-8 delete method wrapper */
+
+static int dict_utf8_delete(DICT *self, const char *key)
+{
+ DICT *dict;
+ const char *utf8_err;
+ const char *fold_res;
+ int status;
+
+ /*
+ * Validate and optionally fold the key, and if invalid skip the request.
+ */
+ if ((fold_res = dict_utf8_check_fold(self, key, &utf8_err)) == 0) {
+ msg_warn("%s:%s: non-UTF-8 key \"%s\": %s",
+ self->type, self->name, key, utf8_err);
+ self->error = DICT_ERR_NONE;
+ return (DICT_STAT_SUCCESS);
+ }
+
+ /*
+ * Proxy the request.
+ */
+ else {
+ dict = (void *) self - self->size;
+ dict->flags = self->flags;
+ status = dict->delete(dict, fold_res);
+ self->flags = dict->flags;
+ self->error = dict->error;
+ return (status);
+ }
+}
+
+/* dict_utf8_close - dummy */
+
+static void dict_utf8_close(DICT *self)
+{
+ DICT *dict;
+
+ /*
+ * Destroy the dict object that we are appended to, and thereby destroy
+ * ourselves.
+ */
+ dict = (void *) self - self->size;
+ dict->close(dict);
+}
+
+/* dict_utf8_encapsulate - wrap a legacy dict object for UTF-8 processing */
+
+DICT *dict_utf8_encapsulate(DICT *dict)
+{
+ DICT *self;
+
+ /*
+ * Sanity check.
+ */
+ if (dict->flags & DICT_FLAG_UTF8_PROXY)
+ msg_panic("dict_utf8_encapsulate: %s:%s is already encapsulated",
+ dict->type, dict->name);
+
+ /*
+ * Append ourselves to the dict object, so that dict_close(dict) will do
+ * the right thing. dict->size is based on the actual size of the dict
+ * object's subclass, so we don't have to worry about alignment problems.
+ *
+ * XXX Add dict_flags argument to dict_alloc() so that it can allocate the
+ * right memory amount, and we can avoid having to resize an object.
+ */
+ dict = myrealloc(dict, dict->size + sizeof(*self));
+ self = (void *) dict + dict->size;
+ *self = *dict;
+
+ /*
+ * Interpose on the lookup/update/delete/close methods. In particular we
+ * do not interpose on the iterator. Invalid keys are not stored, and we
+ * want to be able to delete an invalid value.
+ */
+ self->lookup = dict_utf8_lookup;
+ self->update = dict_utf8_update;
+ self->delete = dict_utf8_delete;
+ self->close = dict_utf8_close;
+
+ /*
+ * Finally, disable casefolding in the dict object. It now happens in the
+ * lookup/update/delete wrappers.
+ */
+ dict->flags &= ~DICT_FLAG_FOLD_ANY;
+ self->flags |= DICT_FLAG_UTF8_PROXY;
+
+ return (self);
+}
--- /dev/null
+#!/bin/sh
+
+awk 'BEGIN {
+ print "flags"
+ print "verbose"
+ printf "get foo\n"
+ printf "put %c%c%c xxx\n", 128, 128, 128
+ printf "get %c%c%c\n", 128, 128, 128
+ printf "put xxx %c%c%c\n", 128, 128, 128
+ printf "get xxx\n"
+ exit
+}' | ./dict_open internal:whatever write utf8_enable
--- /dev/null
+owner=trusted (uid=2147483647)
+> flags
+dict flags fixed|lock|replace|utf8_enable|utf8_proxy
+> verbose
+> get foo
+foo: not found
+> put \80\80\80 xxx
+./dict_open: warning: internal:whatever: non-UTF-8 key "???": malformed UTF-8 or invalid codepoint
+> get \80\80\80
+./dict_open: warning: internal:whatever: non-UTF-8 key "???": malformed UTF-8 or invalid codepoint
+\80\80\80: not found
+> put xxx \80\80\80
+./dict_open: warning: internal:whatever: key "xxx": non-UTF-8 value "???": malformed UTF-8 or invalid codepoint
+> get xxx
+xxx: not found
#include <msg.h>
#include <split_at.h>
-#include <stringops.h> /* XXX temp_utf8_kludge */
+#include <stringops.h> /* XXX util_utf8_enable */
#include <valid_utf8_hostname.h>
/* Global library. */
* network addresses instead of requiring proper [ipaddress] forms.
*/
if (*host != def_host
- && !valid_utf8_hostname(temp_utf8_kludge, *host, DONT_GRIPE)
+ && !valid_utf8_hostname(util_utf8_enable, *host, DONT_GRIPE)
&& !valid_hostaddr(*host, DONT_GRIPE))
return ("valid hostname or network address required");
if (*port != def_service && ISDIGIT(**port) && !alldig(*port))
int match;
#define OPEN_FLAGS O_RDONLY
-#define DICT_FLAGS (DICT_FLAG_LOCK | DICT_FLAG_FOLD_FIX)
+#define DICT_FLAGS (DICT_FLAG_LOCK | DICT_FLAG_FOLD_FIX \
+ | DICT_FLAG_UTF8_ENABLE)
#define STR(x) vstring_str(x)
/*
+++ /dev/null
-/*++
-/* NAME
-/* midna 3
-/* SUMMARY
-/* Postfix domain name conversion
-/* SYNOPSIS
-/* #include <midna.h>
-/*
-/* int midna_cache_size;
-/*
-/* const char *midna_to_ascii(
-/* const char *name)
-/*
-/* const char *midna_to_utf8(
-/* const char *name)
-/*
-/* const char *midna_suffix_to_ascii(
-/* const char *name)
-/*
-/* const char *midna_suffix_to_utf8(
-/* const char *name)
-/* DESCRIPTION
-/* The functions in this module transform domain names from
-/* or to IDNA form. The result is cached to avoid repeated
-/* conversion of the same name.
-/*
-/* midna_to_ascii() converts an UTF-8 or ASCII domain name to
-/* ASCII. The result is a null pointer in case of error. This
-/* function verifies that the result is a valid ASCII domainname.
-/*
-/* midna_to_utf8() converts an UTF-8 or ASCII domain name to
-/* UTF-8. The result is a null pointer in case of error. This
-/* function verifies that the result converts to a valid ASCII
-/* domainname.
-/*
-/* midna_suffix_to_ascii() and midna_suffix_to_utf8() take a
-/* name that starts with '.' and otherwise perform the same
-/* operations as midna_to_ascii() and midna_to_utf8().
-/*
-/* midna_cache_size specifies the size of the conversion result
-/* cache. This value is used only once, upon the first lookup
-/* request.
-/* SEE ALSO
-/* msg(3) diagnostics interface
-/* DIAGNOSTICS
-/* Fatal errors: memory allocation problem.
-/* Warnings: conversion error or result validation error.
-/* LICENSE
-/* .ad
-/* .fi
-/* The Secure Mailer license must be distributed with this software.
-/* AUTHOR(S)
-/* Arnt Gulbrandsen
-/*
-/* Wietse Venema
-/* IBM T.J. Watson Research
-/* P.O. Box 704
-/* Yorktown Heights, NY 10598, USA
-/*--*/
-
- /*
- * System library.
- */
-#include <sys_defs.h>
-#include <string.h>
-#include <ctype.h>
-
-#ifndef NO_EAI
-#include <unicode/uidna.h>
-
- /*
- * Utility library.
- */
-#include <mymalloc.h>
-#include <msg.h>
-#include <ctable.h>
-#include <stringops.h>
-#include <valid_hostname.h>
-#include <midna.h>
-
- /*
- * Application-specific.
- */
-#define DEF_MIDNA_CACHE_SIZE 256
-
-int midna_cache_size = DEF_MIDNA_CACHE_SIZE;
-static VSTRING *midna_buf; /* x.suffix */
-
-#define STR(x) vstring_str(x)
-
-/* midna_to_ascii_create - convert domain to ASCII */
-
-static void *midna_to_ascii_create(const char *name, void *unused_context)
-{
- static const char myname[] = "midna_to_ascii_create";
- char buf[1024]; /* XXX */
- UErrorCode error = U_ZERO_ERROR;
- UIDNAInfo info = UIDNA_INFO_INITIALIZER;
- UIDNA *idna;
- int anl;
-
- /*
- * Paranoia: do not expose uidna_*() to unfiltered network data.
- */
- if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) {
- msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s",
- myname, name, "malformed UTF-8");
- return (0);
- }
-
- /*
- * Perform the requested conversion.
- */
- idna = uidna_openUTS46(UIDNA_DEFAULT, &error);
- anl = uidna_nameToASCII_UTF8(idna,
- name, strlen(name),
- buf, sizeof(buf),
- &info,
- &error);
- uidna_close(idna);
-
- /*
- * Paranoia: verify that the result is a valid ASCII domain name. A quick
- * check shows that the UTS46 implementation will reject labels that
- * start or end in '-' or that are over-long, but let's play safe here.
- */
- if (U_SUCCESS(error) && info.errors == 0 && anl > 0) {
- buf[anl] = 0; /* XXX */
- if (!valid_hostname(buf, DONT_GRIPE)) {
- msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s",
- myname, name, "malformed ASCII label(s)");
- return (0);
- }
- return (mystrndup(buf, anl));
- } else {
- msg_warn("%s: Problem translating domain \"%s\" to ASCII form: %s",
- myname, name, u_errorName(error));
- return (0);
- }
-}
-
-/* midna_to_utf8_create - convert domain to UTF8 */
-
-static void *midna_to_utf8_create(const char *name, void *unused_context)
-{
- static const char myname[] = "midna_to_utf8_create";
- char buf[1024]; /* XXX */
- UErrorCode error = U_ZERO_ERROR;
- UIDNAInfo info = UIDNA_INFO_INITIALIZER;
- UIDNA *idna;
- int anl;
-
- /*
- * Paranoia: do not expose uidna_*() to unfiltered network data.
- */
- if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) {
- msg_warn("%s: Problem translating domain \"%s\" to UTF-8 form: %s",
- myname, name, "malformed UTF-8");
- return (0);
- }
-
- /*
- * Perform the requested conversion.
- */
- idna = uidna_openUTS46(UIDNA_DEFAULT, &error);
- anl = uidna_nameToUnicodeUTF8(idna,
- name, strlen(name),
- buf, sizeof(buf),
- &info,
- &error);
- uidna_close(idna);
-
- /*
- * Paranoia: UTS46 toUTF8 will accept and produce a name that does not
- * convert to a valid ASCII domain name. So we enforce sanity here.
- */
- if (U_SUCCESS(error) && info.errors == 0 && anl > 0) {
- buf[anl] = 0; /* XXX */
- if (midna_to_ascii(buf) == 0)
- return (0);
- return (mystrndup(buf, anl));
- } else {
- msg_warn("%s: Problem translating domain \"%s\" to UTF8 form: %s",
- myname, name, u_errorName(error));
- return (0);
- }
-}
-
-/* midna_cache_free - cache element destructor */
-
-static void midna_cache_free(void *value, void *unused_context)
-{
- if (value)
- myfree(value);
-}
-
-/* midna_to_ascii - convert name to ASCII */
-
-const char *midna_to_ascii(const char *name)
-{
- static CTABLE *midna_to_ascii_cache = 0;
-
- if (midna_to_ascii_cache == 0)
- midna_to_ascii_cache = ctable_create(midna_cache_size,
- midna_to_ascii_create,
- midna_cache_free,
- (void *) 0);
- return (ctable_locate(midna_to_ascii_cache, name));
-}
-
-/* midna_to_utf8 - convert name to UTF8 */
-
-const char *midna_to_utf8(const char *name)
-{
- static CTABLE *midna_to_utf8_cache = 0;
-
- if (midna_to_utf8_cache == 0)
- midna_to_utf8_cache = ctable_create(midna_cache_size,
- midna_to_utf8_create,
- midna_cache_free,
- (void *) 0);
- return (ctable_locate(midna_to_utf8_cache, name));
-}
-
-/* midna_suffix_to_ascii - convert .name to ASCII */
-
-const char *midna_suffix_to_ascii(const char *suffix)
-{
- const char *cache_res;
-
- /*
- * If prepending x to .name causes the result to become too long, then
- * the suffix is bad.
- */
- if (midna_buf == 0)
- midna_buf = vstring_alloc(100);
- vstring_sprintf(midna_buf, "x%s", suffix);
- if ((cache_res = midna_to_ascii(STR(midna_buf))) == 0)
- return (0);
- else
- return (cache_res + 1);
-}
-
-/* midna_suffix_to_utf8 - convert .name to UTF8 */
-
-const char *midna_suffix_to_utf8(const char *name)
-{
- const char *cache_res;
-
- /*
- * If prepending x to .name causes the result to become too long, then
- * the suffix is bad.
- */
- if (midna_buf == 0)
- midna_buf = vstring_alloc(100);
- vstring_sprintf(midna_buf, "x%s", name);
- if ((cache_res = midna_to_utf8(STR(midna_buf))) == 0)
- return (0);
- else
- return (cache_res + 1);
-}
-
-#ifdef TEST
-
- /*
- * Test program - reads names from stdin, reports invalid names to stderr.
- */
-#include <stdlib.h>
-#include <locale.h>
-
-#include <stringops.h> /* XXX temp_utf8_kludge */
-#include <vstring.h>
-#include <vstream.h>
-#include <vstring_vstream.h>
-#include <msg_vstream.h>
-
-int main(int argc, char **argv)
-{
- VSTRING *buffer = vstring_alloc(1);
- const char *bp;
- const char *ascii;
- const char *utf8;
-
- if (setlocale(LC_ALL, "C") == 0)
- msg_fatal("setlocale(LC_ALL, C) failed: %m");
-
- msg_vstream_init(argv[0], VSTREAM_ERR);
- msg_verbose = 1;
- temp_utf8_kludge = 1;
-
- while (vstring_fgets_nonl(buffer, VSTREAM_IN)) {
- bp = STR(buffer);
- msg_info("> %s", bp);
- while (ISSPACE(*bp))
- bp++;
- if (*bp == '#' || *bp == 0)
- continue;
- if (!allascii(bp)) {
- utf8 = midna_to_utf8(bp);
- if (utf8 != 0)
- msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8);
- ascii = midna_to_ascii(bp);
- if (ascii != 0) {
- msg_info("\"%s\" ->ascii \"%s\"", bp, ascii);
- utf8 = midna_to_utf8(ascii);
- if (utf8 != 0) {
- msg_info("\"%s\" ->ascii \"%s\" ->utf8 \"%s\"",
- bp, ascii, utf8);
- if (strcmp(utf8, bp) != 0)
- msg_warn("\"%s\" != \"%s\"", bp, utf8);
- }
- }
- } else {
- ascii = midna_to_ascii(bp);
- if (ascii != 0)
- msg_info("\"%s\" ->ascii \"%s\"", bp, ascii);
- utf8 = midna_to_utf8(bp);
- if (utf8 != 0) {
- msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8);
- ascii = midna_to_ascii(utf8);
- if (ascii != 0) {
- msg_info("\"%s\" ->utf8 \"%s\" ->ascii \"%s\"",
- bp, utf8, ascii);
- if (strcmp(ascii, bp) != 0)
- msg_warn("\"%s\" != \"%s\"", bp, ascii);
- }
- }
- }
- }
- exit(0);
-}
-
-#endif /* TEST */
-
-#endif /* NO_EAI */
--- /dev/null
+/*++
+/* NAME
+/* midna_domain 3
+/* SUMMARY
+/* ASCII/UTF-8 domain name conversion
+/* SYNOPSIS
+/* #include <midna_domain.h>
+/*
+/* int midna_domain_cache_size;
+/*
+/* const char *midna_domain_to_ascii(
+/* const char *name)
+/*
+/* const char *midna_domain_to_utf8(
+/* const char *name)
+/*
+/* const char *midna_domain_suffix_to_ascii(
+/* const char *name)
+/*
+/* const char *midna_domain_suffix_to_utf8(
+/* const char *name)
+/* DESCRIPTION
+/* The functions in this module transform domain names from/to
+/* ASCII and UTF-8 form. The result is cached to avoid repeated
+/* conversion.
+/*
+/* This module builds on the ICU library implementation of the
+/* UTS #46 specification, using default ICU library options
+/* because those are likely best tested: with transitional
+/* processing, with case mapping, with normalization, with
+/* limited IDNA2003 compatibility, without STD3 ASCII rules.
+/*
+/* midna_domain_to_ascii() converts an UTF-8 or ASCII domain
+/* name to ASCII. The result is a null pointer in case of
+/* error. This function verifies that the result passes
+/* valid_hostname().
+/*
+/* midna_domain_to_utf8() converts an UTF-8 or ASCII domain
+/* name to UTF-8. The result is a null pointer in case of
+/* error. This function verifies that the result, after
+/* conversion to ASCII, passes valid_hostname().
+/*
+/* midna_domain_suffix_to_ascii() and midna_domain_suffix_to_utf8()
+/* take a name that starts with '.' and otherwise perform the
+/* same operations as midna_domain_to_ascii() and
+/* midna_domain_to_utf8().
+/*
+/* midna_domain_cache_size specifies the size of the conversion
+/* result cache. This value is used only once, upon the first
+/* lookup
+/* request.
+/* SEE ALSO
+/* http://unicode.org/reports/tr46/ Unicode IDNA Compatibility processing
+/* msg(3) diagnostics interface
+/* DIAGNOSTICS
+/* Fatal errors: memory allocation problem.
+/* Warnings: conversion error or result validation error.
+/* LICENSE
+/* .ad
+/* .fi
+/* The Secure Mailer license must be distributed with this software.
+/* AUTHOR(S)
+/* Arnt Gulbrandsen
+/*
+/* Wietse Venema
+/* IBM T.J. Watson Research
+/* P.O. Box 704
+/* Yorktown Heights, NY 10598, USA
+/*--*/
+
+ /*
+ * System library.
+ */
+#include <sys_defs.h>
+#include <string.h>
+#include <ctype.h>
+
+#ifndef NO_EAI
+#include <unicode/uidna.h>
+
+ /*
+ * Utility library.
+ */
+#include <mymalloc.h>
+#include <msg.h>
+#include <ctable.h>
+#include <stringops.h>
+#include <valid_hostname.h>
+#include <midna_domain.h>
+
+ /*
+ * Application-specific.
+ */
+#define DEF_MIDNA_CACHE_SIZE 256
+
+int midna_domain_cache_size = DEF_MIDNA_CACHE_SIZE;
+static VSTRING *midna_domain_buf; /* x.suffix */
+
+#define STR(x) vstring_str(x)
+
+/* midna_domain_to_ascii_create - convert domain to ASCII */
+
+static void *midna_domain_to_ascii_create(const char *name, void *unused_context)
+{
+ static const char myname[] = "midna_domain_to_ascii_create";
+ char buf[1024]; /* XXX */
+ UErrorCode error = U_ZERO_ERROR;
+ UIDNAInfo info = UIDNA_INFO_INITIALIZER;
+ UIDNA *idna;
+ int anl;
+
+ /*
+ * Paranoia: do not expose uidna_*() to unfiltered network data.
+ */
+ if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) {
+ msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s",
+ myname, name, "malformed UTF-8");
+ return (0);
+ }
+
+ /*
+ * Perform the requested conversion.
+ */
+ idna = uidna_openUTS46(UIDNA_DEFAULT, &error);/* XXX check error */
+ anl = uidna_nameToASCII_UTF8(idna,
+ name, strlen(name),
+ buf, sizeof(buf) - 1,
+ &info,
+ &error);
+ uidna_close(idna);
+
+ /*
+ * Paranoia: verify that the result passes valid_hostname(). A quick
+ * check shows that UTS46 ToASCII by default rejects inputs with labels
+ * that start or end in '-', with names or labels that are over-long, or
+ * "fake" A-labels, as required by UTS 46 section 4.1, but we rely on
+ * valid_hostname() on the output side just to be sure.
+ */
+ if (U_SUCCESS(error) && info.errors == 0 && anl > 0) {
+ buf[anl] = 0; /* XXX */
+ if (!valid_hostname(buf, DONT_GRIPE)) {
+ msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s",
+ myname, name, "malformed ASCII label(s)");
+ return (0);
+ }
+ return (mystrndup(buf, anl));
+ } else {
+ msg_warn("%s: Problem translating domain \"%.100s\" to ASCII form: %s",
+ myname, name, u_errorName(info.errors));
+ return (0);
+ }
+}
+
+/* midna_domain_to_utf8_create - convert domain to UTF8 */
+
+static void *midna_domain_to_utf8_create(const char *name, void *unused_context)
+{
+ static const char myname[] = "midna_domain_to_utf8_create";
+ char buf[1024]; /* XXX */
+ UErrorCode error = U_ZERO_ERROR;
+ UIDNAInfo info = UIDNA_INFO_INITIALIZER;
+ UIDNA *idna;
+ int anl;
+
+ /*
+ * Paranoia: do not expose uidna_*() to unfiltered network data.
+ */
+ if (allascii(name) == 0 && valid_utf8_string(name, strlen(name)) == 0) {
+ msg_warn("%s: Problem translating domain \"%.100s\" to UTF-8 form: %s",
+ myname, name, "malformed UTF-8");
+ return (0);
+ }
+
+ /*
+ * Perform the requested conversion.
+ */
+ idna = uidna_openUTS46(UIDNA_DEFAULT, &error);/* XXX check error */
+ anl = uidna_nameToUnicodeUTF8(idna,
+ name, strlen(name),
+ buf, sizeof(buf) - 1,
+ &info,
+ &error);
+ uidna_close(idna);
+
+ /*
+ * Paranoia: UTS46 toUTF8 by default accepts and produces an over-long
+ * name or a name that contains an over-long NR-LDH label (and perhaps
+ * other invalid forms that are not covered in UTS 46, section 4.1). We
+ * rely on midna_domain_to_ascii() to validate the output.
+ */
+ if (U_SUCCESS(error) && info.errors == 0 && anl > 0) {
+ buf[anl] = 0; /* XXX */
+ if (midna_domain_to_ascii(buf) == 0)
+ return (0);
+ return (mystrndup(buf, anl));
+ } else {
+ msg_warn("%s: Problem translating domain \"%.100s\" to UTF8 form: %s",
+ myname, name, u_errorName(info.errors));
+ return (0);
+ }
+}
+
+/* midna_domain_cache_free - cache element destructor */
+
+static void midna_domain_cache_free(void *value, void *unused_context)
+{
+ if (value)
+ myfree(value);
+}
+
+/* midna_domain_to_ascii - convert name to ASCII */
+
+const char *midna_domain_to_ascii(const char *name)
+{
+ static CTABLE *midna_domain_to_ascii_cache = 0;
+
+ if (midna_domain_to_ascii_cache == 0)
+ midna_domain_to_ascii_cache = ctable_create(midna_domain_cache_size,
+ midna_domain_to_ascii_create,
+ midna_domain_cache_free,
+ (void *) 0);
+ return (ctable_locate(midna_domain_to_ascii_cache, name));
+}
+
+/* midna_domain_to_utf8 - convert name to UTF8 */
+
+const char *midna_domain_to_utf8(const char *name)
+{
+ static CTABLE *midna_domain_to_utf8_cache = 0;
+
+ if (midna_domain_to_utf8_cache == 0)
+ midna_domain_to_utf8_cache = ctable_create(midna_domain_cache_size,
+ midna_domain_to_utf8_create,
+ midna_domain_cache_free,
+ (void *) 0);
+ return (ctable_locate(midna_domain_to_utf8_cache, name));
+}
+
+/* midna_domain_suffix_to_ascii - convert .name to ASCII */
+
+const char *midna_domain_suffix_to_ascii(const char *suffix)
+{
+ const char *cache_res;
+
+ /*
+ * If prepending x to .name causes the result to become too long, then
+ * the suffix is bad.
+ */
+ if (midna_domain_buf == 0)
+ midna_domain_buf = vstring_alloc(100);
+ vstring_sprintf(midna_domain_buf, "x%s", suffix);
+ if ((cache_res = midna_domain_to_ascii(STR(midna_domain_buf))) == 0)
+ return (0);
+ else
+ return (cache_res + 1);
+}
+
+/* midna_domain_suffix_to_utf8 - convert .name to UTF8 */
+
+const char *midna_domain_suffix_to_utf8(const char *name)
+{
+ const char *cache_res;
+
+ /*
+ * If prepending x to .name causes the result to become too long, then
+ * the suffix is bad.
+ */
+ if (midna_domain_buf == 0)
+ midna_domain_buf = vstring_alloc(100);
+ vstring_sprintf(midna_domain_buf, "x%s", name);
+ if ((cache_res = midna_domain_to_utf8(STR(midna_domain_buf))) == 0)
+ return (0);
+ else
+ return (cache_res + 1);
+}
+
+#ifdef TEST
+
+ /*
+ * Test program - reads names from stdin, reports invalid names to stderr.
+ */
+#include <stdlib.h>
+#include <locale.h>
+
+#include <stringops.h> /* XXX util_utf8_enable */
+#include <vstring.h>
+#include <vstream.h>
+#include <vstring_vstream.h>
+#include <msg_vstream.h>
+
+int main(int argc, char **argv)
+{
+ VSTRING *buffer = vstring_alloc(1);
+ const char *bp;
+ const char *ascii;
+ const char *utf8;
+
+ if (setlocale(LC_ALL, "C") == 0)
+ msg_fatal("setlocale(LC_ALL, C) failed: %m");
+
+ msg_vstream_init(argv[0], VSTREAM_ERR);
+ /* msg_verbose = 1; */
+ util_utf8_enable = 1;
+
+ while (vstring_fgets_nonl(buffer, VSTREAM_IN)) {
+ bp = STR(buffer);
+ msg_info("> %s", bp);
+ while (ISSPACE(*bp))
+ bp++;
+ if (*bp == '#' || *bp == 0)
+ continue;
+ msg_info("unconditional conversions:");
+ utf8 = midna_domain_to_utf8(bp);
+ msg_info("\"%s\" ->utf8 \"%s\"", bp, utf8 ? utf8 : "(error)");
+ ascii = midna_domain_to_ascii(bp);
+ msg_info("\"%s\" ->ascii \"%s\"", bp, ascii ? ascii : "(error)");
+ msg_info("conditional conversions:");
+ if (!allascii(bp)) {
+ if (ascii != 0) {
+ utf8 = midna_domain_to_utf8(ascii);
+ msg_info("\"%s\" ->ascii \"%s\" ->utf8 \"%s\"",
+ bp, ascii, utf8 ? utf8 : "(error)");
+ if (utf8 != 0) {
+ if (strcmp(utf8, bp) != 0)
+ msg_warn("\"%s\" != \"%s\"", bp, utf8);
+ }
+ }
+ } else {
+ if (utf8 != 0) {
+ ascii = midna_domain_to_ascii(utf8);
+ msg_info("\"%s\" ->utf8 \"%s\" ->ascii \"%s\"",
+ bp, utf8, ascii ? ascii : "(error)");
+ if (ascii != 0) {
+ if (strcmp(ascii, bp) != 0)
+ msg_warn("\"%s\" != \"%s\"", bp, ascii);
+ }
+ }
+ }
+ }
+ exit(0);
+}
+
+#endif /* TEST */
+
+#endif /* NO_EAI */
/*++
/* NAME
-/* mail_idna 3h
+/* midna_domain 3h
/* SUMMARY
-/* domain name conversion
+/* ASCII/UTF-8 domain name conversion
/* SYNOPSIS
-/* #include <mail_idna.h>
+/* #include <midna_domain.h>
/* DESCRIPTION
/* .nf
/*
* External interface.
*/
-extern const char *midna_to_ascii(const char *);
-extern const char *midna_to_utf8(const char *);
-extern const char *midna_suffix_to_ascii(const char *);
-extern const char *midna_suffix_to_utf8(const char *);
+extern const char *midna_domain_to_ascii(const char *);
+extern const char *midna_domain_to_utf8(const char *);
+extern const char *midna_domain_suffix_to_ascii(const char *);
+extern const char *midna_domain_suffix_to_utf8(const char *);
/* LICENSE
/* .ad
--- /dev/null
+# Upper-case greek -> lower-case greek.
+Δημοσθένους.example.com
+# Upper-case ASCII -> lower-case ASCII.
+Hello.example.com
+# Invalid LDH label('-' at begin or end).
+bad-.example.com
+-bad.example.com
+# Invalid LDH (label > 63 bytes).
+abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com
+# Valid LDH label (label <= 63 bytes).
+abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com
+# Invalid name (length > 255 bytes).
+abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com
+# Aliases for '.' -> '.'.
+x。example.com
+x．example.com
+x｡example.com
+# Good a-label.
+xn--mumble.example.com
+# Bad a-label.
+xn--123456.example.com
--- /dev/null
+./midna_domain: > # Upper-case greek -> lower-case greek.
+./midna_domain: > Δημοσθένους.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "Δημοσθένους.example.com" ->utf8 "δημοσθένουσ.example.com"
+./midna_domain: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" ->utf8 "δημοσθένουσ.example.com"
+./midna_domain: warning: "Δημοσθένους.example.com" != "δημοσθένουσ.example.com"
+./midna_domain: > # Upper-case ASCII -> lower-case ASCII.
+./midna_domain: > Hello.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "Hello.example.com" ->utf8 "hello.example.com"
+./midna_domain: "Hello.example.com" ->ascii "hello.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "Hello.example.com" ->utf8 "hello.example.com" ->ascii "hello.example.com"
+./midna_domain: warning: "Hello.example.com" != "hello.example.com"
+./midna_domain: > # Invalid LDH label('-' at begin or end).
+./midna_domain: > bad-.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "bad-.example.com" to UTF8 form: U_UNSUPPORTED_ERROR
+./midna_domain: "bad-.example.com" ->utf8 "(error)"
+./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "bad-.example.com" to ASCII form: U_UNSUPPORTED_ERROR
+./midna_domain: "bad-.example.com" ->ascii "(error)"
+./midna_domain: conditional conversions:
+./midna_domain: > -bad.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "-bad.example.com" to UTF8 form: U_INDEX_OUTOFBOUNDS_ERROR
+./midna_domain: "-bad.example.com" ->utf8 "(error)"
+./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "-bad.example.com" to ASCII form: U_INDEX_OUTOFBOUNDS_ERROR
+./midna_domain: "-bad.example.com" ->ascii "(error)"
+./midna_domain: conditional conversions:
+./midna_domain: > # Invalid LDH (label > 63 bytes).
+./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" to ASCII form: U_MISSING_RESOURCE_ERROR
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" ->utf8 "(error)"
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789.example.com" ->ascii "(error)"
+./midna_domain: conditional conversions:
+./midna_domain: > # Valid LDH label (label <= 63 bytes).
+./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->utf8 "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com"
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->ascii "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->utf8 "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com" ->ascii "abcdef0123456789abcdef0123456789abcdef0123456789abcdef012345678.example.com"
+./midna_domain: > # Invalid name (length > 255 bytes).
+./midna_domain: > abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcde" to ASCII form: U_FILE_ACCESS_ERROR
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com" ->utf8 "(error)"
+./midna_domain: "abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234567.example.com" ->ascii "(error)"
+./midna_domain: conditional conversions:
+./midna_domain: > # Aliases for '.' -> '.'.
+./midna_domain: > x。example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "x。example.com" ->utf8 "x.example.com"
+./midna_domain: "x。example.com" ->ascii "x.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
+./midna_domain: warning: "x。example.com" != "x.example.com"
+./midna_domain: > x．example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "x．example.com" ->utf8 "x.example.com"
+./midna_domain: "x．example.com" ->ascii "x.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "x．example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
+./midna_domain: warning: "x．example.com" != "x.example.com"
+./midna_domain: > x｡example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "x｡example.com" ->utf8 "x.example.com"
+./midna_domain: "x｡example.com" ->ascii "x.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "x｡example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
+./midna_domain: warning: "x｡example.com" != "x.example.com"
+./midna_domain: > # Good a-label.
+./midna_domain: > xn--mumble.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: "xn--mumble.example.com" ->utf8 "㲹㲺㲵㲴.example.com"
+./midna_domain: "xn--mumble.example.com" ->ascii "xn--mumble.example.com"
+./midna_domain: conditional conversions:
+./midna_domain: "xn--mumble.example.com" ->utf8 "㲹㲺㲵㲴.example.com" ->ascii "xn--mumble.example.com"
+./midna_domain: > # Bad a-label.
+./midna_domain: > xn--123456.example.com
+./midna_domain: unconditional conversions:
+./midna_domain: warning: midna_domain_to_utf8_create: Problem translating domain "xn--123456.example.com" to UTF8 form: [BOGUS UErrorCode]
+./midna_domain: "xn--123456.example.com" ->utf8 "(error)"
+./midna_domain: warning: midna_domain_to_ascii_create: Problem translating domain "xn--123456.example.com" to ASCII form: [BOGUS UErrorCode]
+./midna_domain: "xn--123456.example.com" ->ascii "(error)"
+./midna_domain: conditional conversions:
+++ /dev/null
-# Upper-case greek -> lower-case greek.
-Δημοσθένους.example.com
-# Upper-case ASCII -> lower-case ASCII.
-Hello.example.com
-# Invalid domain name ('-' at begin or end).
--bad-.example.com
-# Invalid domain name (label > 62 bytes).
-abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com
-# Valid domain name.
-abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-# Aliases for '.' -> '.'.
-x。example.com
-x．example.com
-x｡example.com
+++ /dev/null
-./midna: > # Upper-case greek -> lower-case greek.
-./midna: > Δημοσθένους.example.com
-./midna: ctable_locate: install entry key δημοσθένουσ.example.com
-./midna: ctable_locate: install entry key Δημοσθένους.example.com
-./midna: "Δημοσθένους.example.com" ->utf8 "δημοσθένουσ.example.com"
-./midna: ctable_locate: install entry key Δημοσθένους.example.com
-./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com"
-./midna: ctable_locate: move existing entry key δημοσθένουσ.example.com
-./midna: ctable_locate: install entry key xn--ixanjetild6aev.example.com
-./midna: "Δημοσθένους.example.com" ->ascii "xn--ixanjetild6aev.example.com" ->utf8 "δημοσθένουσ.example.com"
-./midna: warning: "Δημοσθένους.example.com" != "δημοσθένουσ.example.com"
-./midna: > # Upper-case ASCII -> lower-case ASCII.
-./midna: > Hello.example.com
-./midna: ctable_locate: install entry key Hello.example.com
-./midna: "Hello.example.com" ->ascii "hello.example.com"
-./midna: ctable_locate: install entry key hello.example.com
-./midna: ctable_locate: install entry key Hello.example.com
-./midna: "Hello.example.com" ->utf8 "hello.example.com"
-./midna: ctable_locate: leave existing entry key hello.example.com
-./midna: "Hello.example.com" ->utf8 "hello.example.com" ->ascii "hello.example.com"
-./midna: warning: "Hello.example.com" != "hello.example.com"
-./midna: > # Invalid domain name ('-' at begin or end).
-./midna: > -bad-.example.com
-./midna: warning: midna_to_ascii_create: Problem translating domain "-bad-.example.com" to ASCII form: U_ZERO_ERROR
-./midna: ctable_locate: install entry key -bad-.example.com
-./midna: warning: midna_to_utf8_create: Problem translating domain "-bad-.example.com" to UTF8 form: U_ZERO_ERROR
-./midna: ctable_locate: install entry key -bad-.example.com
-./midna: > # Invalid domain name (label > 62 bytes).
-./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com
-./midna: warning: midna_to_ascii_create: Problem translating domain "abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com" to ASCII form: U_ZERO_ERROR
-./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com
-./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com
-./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef0123456789.example.com
-./midna: > # Valid domain name.
-./midna: > abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com"
-./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-./midna: ctable_locate: install entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com"
-./midna: ctable_locate: leave existing entry key abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com
-./midna: "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->utf8 "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com" ->ascii "abcdef01234567890abcdef01234567890abcdef01234567890abcdef012345.example.com"
-./midna: > # Aliases for '.' -> '.'.
-./midna: > x。example.com
-./midna: ctable_locate: install entry key x.example.com
-./midna: ctable_locate: install entry key x。example.com
-./midna: "x。example.com" ->utf8 "x.example.com"
-./midna: ctable_locate: install entry key x。example.com
-./midna: "x。example.com" ->ascii "x.example.com"
-./midna: ctable_locate: move existing entry key x.example.com
-./midna: ctable_locate: install entry key x.example.com
-./midna: "x。example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
-./midna: warning: "x。example.com" != "x.example.com"
-./midna: > x．example.com
-./midna: ctable_locate: leave existing entry key x.example.com
-./midna: ctable_locate: install entry key x．example.com
-./midna: "x．example.com" ->utf8 "x.example.com"
-./midna: ctable_locate: install entry key x．example.com
-./midna: "x．example.com" ->ascii "x.example.com"
-./midna: ctable_locate: move existing entry key x.example.com
-./midna: "x．example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
-./midna: warning: "x．example.com" != "x.example.com"
-./midna: > x｡example.com
-./midna: ctable_locate: move existing entry key x.example.com
-./midna: ctable_locate: install entry key x｡example.com
-./midna: "x｡example.com" ->utf8 "x.example.com"
-./midna: ctable_locate: install entry key x｡example.com
-./midna: "x｡example.com" ->ascii "x.example.com"
-./midna: ctable_locate: move existing entry key x.example.com
-./midna: "x｡example.com" ->ascii "x.example.com" ->utf8 "x.example.com"
-./midna: warning: "x｡example.com" != "x.example.com"
/* SYNOPSIS
/* #include <stringops.h>
/*
-/* int temp_utf8_kludge;
+/* int util_utf8_enable;
/*
/* char *printable(buffer, replacement)
/* char *buffer;
/* printable() replaces non-printable characters
/* in its input with the given replacement.
/*
-/* temp_utf8_kludge controls whether UTF8 is considered printable.
+/* util_utf8_enable controls whether UTF8 is considered printable.
/* By default, non-ASCII text is replaced.
/*
/* Arguments:
#include "stringops.h"
-int temp_utf8_kludge = 0;
+int util_utf8_enable = 0;
char *printable(char *string, int replacement)
{
while ((ch = *cp) != 0) {
if (ISASCII(ch) && ISPRINT(ch)) {
/* ok */
- } else if (temp_utf8_kludge && ch >= 194 && ch <= 254
+ } else if (util_utf8_enable && ch >= 194 && ch <= 254
&& cp[1] >= 128 && cp[1] < 192) {
/* UTF8; skip the rest of the bytes in the character. */
while (cp[1] >= 128 && cp[1] < 192)
/*
* External interface.
*/
-extern int temp_utf8_kludge;
+extern int util_utf8_enable;
extern char *printable(char *, int);
extern char *neuter(char *, const char *, int);
extern char *lowercase(char *);
+extern char *casefold(VSTRING *, const char *, CONST_CHAR_STAR *);
extern char *uppercase(char *);
extern char *skipblanks(const char *);
extern char *trimblanks(char *, ssize_t);
/*
/* int valid_utf8_hostname(
/* int enable_utf8,
-/* const char *domain,
-/* int gripe)
+/* const char *domain,
+/* int gripe)
/* DESCRIPTION
/* valid_utf8_hostname() is a wrapper around valid_hostname().
/* If EAI support is compiled in, and enable_utf8 is true, the
#include <mymalloc.h>
#include <stringops.h>
#include <valid_hostname.h>
-#include <midna.h>
+#include <midna_domain.h>
#include <valid_utf8_hostname.h>
/* valid_utf8_hostname - validate internationalized domain name */
int valid_utf8_hostname(int enable_utf8, const char *name, int gripe)
{
static const char myname[] = "valid_utf8_hostname";
- const char *aname;
- int ret;
/*
* Trivial cases first.
}
/*
- * Convert domain name to ASCII form.
+ * Convert non-ASCII domain name to ASCII and validate the result per
+ * STD3. midna_domain_to_ascii() applies valid_hostname() to the result.
+ * Propagate the gripe parameter for better diagnostics (note that
+ * midna_domain_to_ascii() logs a problem only when the result is not
+ * cached).
*/
#ifndef NO_EAI
if (enable_utf8 && !allascii(name)) {
- if ((aname = midna_to_ascii(name)) == 0) {
+ if (midna_domain_to_ascii(name) == 0) {
if (gripe)
msg_warn("%s: malformed UTF-8 domain name", myname);
return (0);
+ } else {
+ return (1);
}
- } else
+ }
#endif
- aname = name;
/*
- * Validate the name per STD3 (if the IDNA routines didn't already).
+ * Validate ASCII name per STD3.
*/
- ret = valid_hostname(aname, gripe);
-
- return (ret);
+ return (valid_hostname(name, gripe));
}
projects / thirdparty / postfix.git / commitdiff
