--- /dev/null
+
+
+
+
+
+
+Network Working Group P. Faltstrom
+Request for Comments: 3490 Cisco
+Category: Standards Track P. Hoffman
+ IMC & VPNC
+ A. Costello
+ UC Berkeley
+ March 2003
+
+
+ Internationalizing Domain Names in Applications (IDNA)
+
+Status of this Memo
+
+ This document specifies an Internet standards track protocol for the
+ Internet community, and requests discussion and suggestions for
+ improvements. Please refer to the current edition of the "Internet
+ Official Protocol Standards" (STD 1) for the standardization state
+ and status of this protocol. Distribution of this memo is unlimited.
+
+Copyright Notice
+
+ Copyright (C) The Internet Society (2003). All Rights Reserved.
+
+Abstract
+
+ Until now, there has been no standard method for domain names to use
+ characters outside the ASCII repertoire. This document defines
+ internationalized domain names (IDNs) and a mechanism called
+ Internationalizing Domain Names in Applications (IDNA) for handling
+ them in a standard fashion. IDNs use characters drawn from a large
+ repertoire (Unicode), but IDNA allows the non-ASCII characters to be
+ represented using only the ASCII characters already allowed in so-
+ called host names today. This backward-compatible representation is
+ required in existing protocols like DNS, so that IDNs can be
+ introduced with no changes to the existing infrastructure. IDNA is
+ only meant for processing domain names, not free text.
+
+Table of Contents
+
+ 1. Introduction.................................................. 2
+ 1.1 Problem Statement......................................... 3
+ 1.2 Limitations of IDNA....................................... 3
+ 1.3 Brief overview for application developers................. 4
+ 2. Terminology................................................... 5
+ 3. Requirements and applicability................................ 7
+ 3.1 Requirements.............................................. 7
+ 3.2 Applicability............................................. 8
+ 3.2.1. DNS resource records................................ 8
+
+
+
+Faltstrom, et al. Standards Track [Page 1]
+\f
+RFC 3490 IDNA March 2003
+
+
+ 3.2.2. Non-domain-name data types stored in domain names... 9
+ 4. Conversion operations......................................... 9
+ 4.1 ToASCII................................................... 10
+ 4.2 ToUnicode................................................. 11
+ 5. ACE prefix.................................................... 12
+ 6. Implications for typical applications using DNS............... 13
+ 6.1 Entry and display in applications......................... 14
+ 6.2 Applications and resolver libraries....................... 15
+ 6.3 DNS servers............................................... 15
+ 6.4 Avoiding exposing users to the raw ACE encoding........... 16
+ 6.5 DNSSEC authentication of IDN domain names................ 16
+ 7. Name server considerations.................................... 17
+ 8. Root server considerations.................................... 17
+ 9. References.................................................... 18
+ 9.1 Normative References...................................... 18
+ 9.2 Informative References.................................... 18
+ 10. Security Considerations...................................... 19
+ 11. IANA Considerations.......................................... 20
+ 12. Authors' Addresses........................................... 21
+ 13. Full Copyright Statement..................................... 22
+
+1. Introduction
+
+ IDNA works by allowing applications to use certain ASCII name labels
+ (beginning with a special prefix) to represent non-ASCII name labels.
+ Lower-layer protocols need not be aware of this; therefore IDNA does
+ not depend on changes to any infrastructure. In particular, IDNA
+ does not depend on any changes to DNS servers, resolvers, or protocol
+ elements, because the ASCII name service provided by the existing DNS
+ is entirely sufficient for IDNA.
+
+ This document does not require any applications to conform to IDNA,
+ but applications can elect to use IDNA in order to support IDN while
+ maintaining interoperability with existing infrastructure. If an
+ application wants to use non-ASCII characters in domain names, IDNA
+ is the only currently-defined option. Adding IDNA support to an
+ existing application entails changes to the application only, and
+ leaves room for flexibility in the user interface.
+
+ A great deal of the discussion of IDN solutions has focused on
+ transition issues and how IDN will work in a world where not all of
+ the components have been updated. Proposals that were not chosen by
+ the IDN Working Group would depend on user applications, resolvers,
+ and DNS servers being updated in order for a user to use an
+ internationalized domain name. Rather than rely on widespread
+ updating of all components, IDNA depends on updates to user
+ applications only; no changes are needed to the DNS protocol or any
+ DNS servers or the resolvers on user's computers.
+
+
+
+Faltstrom, et al. Standards Track [Page 2]
+\f
+RFC 3490 IDNA March 2003
+
+
+1.1 Problem Statement
+
+ The IDNA specification solves the problem of extending the repertoire
+ of characters that can be used in domain names to include the Unicode
+ repertoire (with some restrictions).
+
+ IDNA does not extend the service offered by DNS to the applications.
+ Instead, the applications (and, by implication, the users) continue
+ to see an exact-match lookup service. Either there is a single
+ exactly-matching name or there is no match. This model has served
+ the existing applications well, but it requires, with or without
+ internationalized domain names, that users know the exact spelling of
+ the domain names that the users type into applications such as web
+ browsers and mail user agents. The introduction of the larger
+ repertoire of characters potentially makes the set of misspellings
+ larger, especially given that in some cases the same appearance, for
+ example on a business card, might visually match several Unicode code
+ points or several sequences of code points.
+
+ IDNA allows the graceful introduction of IDNs not only by avoiding
+ upgrades to existing infrastructure (such as DNS servers and mail
+ transport agents), but also by allowing some rudimentary use of IDNs
+ in applications by using the ASCII representation of the non-ASCII
+ name labels. While such names are very user-unfriendly to read and
+ type, and hence are not suitable for user input, they allow (for
+ instance) replying to email and clicking on URLs even though the
+ domain name displayed is incomprehensible to the user. In order to
+ allow user-friendly input and output of the IDNs, the applications
+ need to be modified to conform to this specification.
+
+ IDNA uses the Unicode character repertoire, which avoids the
+ significant delays that would be inherent in waiting for a different
+ and specific character set be defined for IDN purposes by some other
+ standards developing organization.
+
+1.2 Limitations of IDNA
+
+ The IDNA protocol does not solve all linguistic issues with users
+ inputting names in different scripts. Many important language-based
+ and script-based mappings are not covered in IDNA and need to be
+ handled outside the protocol. For example, names that are entered in
+ a mix of traditional and simplified Chinese characters will not be
+ mapped to a single canonical name. Another example is Scandinavian
+ names that are entered with U+00F6 (LATIN SMALL LETTER O WITH
+ DIAERESIS) will not be mapped to U+00F8 (LATIN SMALL LETTER O WITH
+ STROKE).
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 3]
+\f
+RFC 3490 IDNA March 2003
+
+
+ An example of an important issue that is not considered in detail in
+ IDNA is how to provide a high probability that a user who is entering
+ a domain name based on visual information (such as from a business
+ card or billboard) or aural information (such as from a telephone or
+ radio) would correctly enter the IDN. Similar issues exist for ASCII
+ domain names, for example the possible visual confusion between the
+ letter 'O' and the digit zero, but the introduction of the larger
+ repertoire of characters creates more opportunities of similar
+ looking and similar sounding names. Note that this is a complex
+ issue relating to languages, input methods on computers, and so on.
+ Furthermore, the kind of matching and searching necessary for a high
+ probability of success would not fit the role of the DNS and its
+ exact matching function.
+
+1.3 Brief overview for application developers
+
+ Applications can use IDNA to support internationalized domain names
+ anywhere that ASCII domain names are already supported, including DNS
+ master files and resolver interfaces. (Applications can also define
+ protocols and interfaces that support IDNs directly using non-ASCII
+ representations. IDNA does not prescribe any particular
+ representation for new protocols, but it still defines which names
+ are valid and how they are compared.)
+
+ The IDNA protocol is contained completely within applications. It is
+ not a client-server or peer-to-peer protocol: everything is done
+ inside the application itself. When used with a DNS resolver
+ library, IDNA is inserted as a "shim" between the application and the
+ resolver library. When used for writing names into a DNS zone, IDNA
+ is used just before the name is committed to the zone.
+
+ There are two operations described in section 4 of this document:
+
+ - The ToASCII operation is used before sending an IDN to something
+ that expects ASCII names (such as a resolver) or writing an IDN
+ into a place that expects ASCII names (such as a DNS master file).
+
+ - The ToUnicode operation is used when displaying names to users,
+ for example names obtained from a DNS zone.
+
+ It is important to note that the ToASCII operation can fail. If it
+ fails when processing a domain name, that domain name cannot be used
+ as an internationalized domain name and the application has to have
+ some method of dealing with this failure.
+
+ IDNA requires that implementations process input strings with
+ Nameprep [NAMEPREP], which is a profile of Stringprep [STRINGPREP],
+ and then with Punycode [PUNYCODE]. Implementations of IDNA MUST
+
+
+
+Faltstrom, et al. Standards Track [Page 4]
+\f
+RFC 3490 IDNA March 2003
+
+
+ fully implement Nameprep and Punycode; neither Nameprep nor Punycode
+ are optional.
+
+2. Terminology
+
+ The key words "MUST", "SHALL", "REQUIRED", "SHOULD", "RECOMMENDED",
+ and "MAY" in this document are to be interpreted as described in BCP
+ 14, RFC 2119 [RFC2119].
+
+ A code point is an integer value associated with a character in a
+ coded character set.
+
+ Unicode [UNICODE] is a coded character set containing tens of
+ thousands of characters. A single Unicode code point is denoted by
+ "U+" followed by four to six hexadecimal digits, while a range of
+ Unicode code points is denoted by two hexadecimal numbers separated
+ by "..", with no prefixes.
+
+ ASCII means US-ASCII [USASCII], a coded character set containing 128
+ characters associated with code points in the range 0..7F. Unicode
+ is an extension of ASCII: it includes all the ASCII characters and
+ associates them with the same code points.
+
+ The term "LDH code points" is defined in this document to mean the
+ code points associated with ASCII letters, digits, and the hyphen-
+ minus; that is, U+002D, 30..39, 41..5A, and 61..7A. "LDH" is an
+ abbreviation for "letters, digits, hyphen".
+
+ [STD13] talks about "domain names" and "host names", but many people
+ use the terms interchangeably. Further, because [STD13] was not
+ terribly clear, many people who are sure they know the exact
+ definitions of each of these terms disagree on the definitions. In
+ this document the term "domain name" is used in general. This
+ document explicitly cites [STD3] whenever referring to the host name
+ syntax restrictions defined therein.
+
+ A label is an individual part of a domain name. Labels are usually
+ shown separated by dots; for example, the domain name
+ "www.example.com" is composed of three labels: "www", "example", and
+ "com". (The zero-length root label described in [STD13], which can
+ be explicit as in "www.example.com." or implicit as in
+ "www.example.com", is not considered a label in this specification.)
+ IDNA extends the set of usable characters in labels that are text.
+ For the rest of this document, the term "label" is shorthand for
+ "text label", and "every label" means "every text label".
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 5]
+\f
+RFC 3490 IDNA March 2003
+
+
+ An "internationalized label" is a label to which the ToASCII
+ operation (see section 4) can be applied without failing (with the
+ UseSTD3ASCIIRules flag unset). This implies that every ASCII label
+ that satisfies the [STD13] length restriction is an internationalized
+ label. Therefore the term "internationalized label" is a
+ generalization, embracing both old ASCII labels and new non-ASCII
+ labels. Although most Unicode characters can appear in
+ internationalized labels, ToASCII will fail for some input strings,
+ and such strings are not valid internationalized labels.
+
+ An "internationalized domain name" (IDN) is a domain name in which
+ every label is an internationalized label. This implies that every
+ ASCII domain name is an IDN (which implies that it is possible for a
+ name to be an IDN without it containing any non-ASCII characters).
+ This document does not attempt to define an "internationalized host
+ name". Just as has been the case with ASCII names, some DNS zone
+ administrators may impose restrictions, beyond those imposed by DNS
+ or IDNA, on the characters or strings that may be registered as
+ labels in their zones. Such restrictions have no impact on the
+ syntax or semantics of DNS protocol messages; a query for a name that
+ matches no records will yield the same response regardless of the
+ reason why it is not in the zone. Clients issuing queries or
+ interpreting responses cannot be assumed to have any knowledge of
+ zone-specific restrictions or conventions.
+
+ In IDNA, equivalence of labels is defined in terms of the ToASCII
+ operation, which constructs an ASCII form for a given label, whether
+ or not the label was already an ASCII label. Labels are defined to
+ be equivalent if and only if their ASCII forms produced by ToASCII
+ match using a case-insensitive ASCII comparison. ASCII labels
+ already have a notion of equivalence: upper case and lower case are
+ considered equivalent. The IDNA notion of equivalence is an
+ extension of that older notion. Equivalent labels in IDNA are
+ treated as alternate forms of the same label, just as "foo" and "Foo"
+ are treated as alternate forms of the same label.
+
+ To allow internationalized labels to be handled by existing
+ applications, IDNA uses an "ACE label" (ACE stands for ASCII
+ Compatible Encoding). An ACE label is an internationalized label
+ that can be rendered in ASCII and is equivalent to an
+ internationalized label that cannot be rendered in ASCII. Given any
+ internationalized label that cannot be rendered in ASCII, the ToASCII
+ operation will convert it to an equivalent ACE label (whereas an
+ ASCII label will be left unaltered by ToASCII). ACE labels are
+ unsuitable for display to users. The ToUnicode operation will
+ convert any label to an equivalent non-ACE label. In fact, an ACE
+ label is formally defined to be any label that the ToUnicode
+ operation would alter (whereas non-ACE labels are left unaltered by
+
+
+
+Faltstrom, et al. Standards Track [Page 6]
+\f
+RFC 3490 IDNA March 2003
+
+
+ ToUnicode). Every ACE label begins with the ACE prefix specified in
+ section 5. The ToASCII and ToUnicode operations are specified in
+ section 4.
+
+ The "ACE prefix" is defined in this document to be a string of ASCII
+ characters that appears at the beginning of every ACE label. It is
+ specified in section 5.
+
+ A "domain name slot" is defined in this document to be a protocol
+ element or a function argument or a return value (and so on)
+ explicitly designated for carrying a domain name. Examples of domain
+ name slots include: the QNAME field of a DNS query; the name argument
+ of the gethostbyname() library function; the part of an email address
+ following the at-sign (@) in the From: field of an email message
+ header; and the host portion of the URI in the src attribute of an
+ HTML <IMG> tag. General text that just happens to contain a domain
+ name is not a domain name slot; for example, a domain name appearing
+ in the plain text body of an email message is not occupying a domain
+ name slot.
+
+ An "IDN-aware domain name slot" is defined in this document to be a
+ domain name slot explicitly designated for carrying an
+ internationalized domain name as defined in this document. The
+ designation may be static (for example, in the specification of the
+ protocol or interface) or dynamic (for example, as a result of
+ negotiation in an interactive session).
+
+ An "IDN-unaware domain name slot" is defined in this document to be
+ any domain name slot that is not an IDN-aware domain name slot.
+ Obviously, this includes any domain name slot whose specification
+ predates IDNA.
+
+3. Requirements and applicability
+
+3.1 Requirements
+
+ IDNA conformance means adherence to the following four requirements:
+
+ 1) Whenever dots are used as label separators, the following
+ characters MUST be recognized as dots: U+002E (full stop), U+3002
+ (ideographic full stop), U+FF0E (fullwidth full stop), U+FF61
+ (halfwidth ideographic full stop).
+
+ 2) Whenever a domain name is put into an IDN-unaware domain name slot
+ (see section 2), it MUST contain only ASCII characters. Given an
+ internationalized domain name (IDN), an equivalent domain name
+ satisfying this requirement can be obtained by applying the
+
+
+
+
+Faltstrom, et al. Standards Track [Page 7]
+\f
+RFC 3490 IDNA March 2003
+
+
+ ToASCII operation (see section 4) to each label and, if dots are
+ used as label separators, changing all the label separators to
+ U+002E.
+
+ 3) ACE labels obtained from domain name slots SHOULD be hidden from
+ users when it is known that the environment can handle the non-ACE
+ form, except when the ACE form is explicitly requested. When it
+ is not known whether or not the environment can handle the non-ACE
+ form, the application MAY use the non-ACE form (which might fail,
+ such as by not being displayed properly), or it MAY use the ACE
+ form (which will look unintelligle to the user). Given an
+ internationalized domain name, an equivalent domain name
+ containing no ACE labels can be obtained by applying the ToUnicode
+ operation (see section 4) to each label. When requirements 2 and
+ 3 both apply, requirement 2 takes precedence.
+
+ 4) Whenever two labels are compared, they MUST be considered to match
+ if and only if they are equivalent, that is, their ASCII forms
+ (obtained by applying ToASCII) match using a case-insensitive
+ ASCII comparison. Whenever two names are compared, they MUST be
+ considered to match if and only if their corresponding labels
+ match, regardless of whether the names use the same forms of label
+ separators.
+
+3.2 Applicability
+
+ IDNA is applicable to all domain names in all domain name slots
+ except where it is explicitly excluded.
+
+ This implies that IDNA is applicable to many protocols that predate
+ IDNA. Note that IDNs occupying domain name slots in those protocols
+ MUST be in ASCII form (see section 3.1, requirement 2).
+
+3.2.1. DNS resource records
+
+ IDNA does not apply to domain names in the NAME and RDATA fields of
+ DNS resource records whose CLASS is not IN. This exclusion applies
+ to every non-IN class, present and future, except where future
+ standards override this exclusion by explicitly inviting the use of
+ IDNA.
+
+ There are currently no other exclusions on the applicability of IDNA
+ to DNS resource records; it depends entirely on the CLASS, and not on
+ the TYPE. This will remain true, even as new types are defined,
+ unless there is a compelling reason for a new type to complicate
+ matters by imposing type-specific rules.
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 8]
+\f
+RFC 3490 IDNA March 2003
+
+
+3.2.2. Non-domain-name data types stored in domain names
+
+ Although IDNA enables the representation of non-ASCII characters in
+ domain names, that does not imply that IDNA enables the
+ representation of non-ASCII characters in other data types that are
+ stored in domain names. For example, an email address local part is
+ sometimes stored in a domain label (hostmaster@example.com would be
+ represented as hostmaster.example.com in the RDATA field of an SOA
+ record). IDNA does not update the existing email standards, which
+ allow only ASCII characters in local parts. Therefore, unless the
+ email standards are revised to invite the use of IDNA for local
+ parts, a domain label that holds the local part of an email address
+ SHOULD NOT begin with the ACE prefix, and even if it does, it is to
+ be interpreted literally as a local part that happens to begin with
+ the ACE prefix.
+
+4. Conversion operations
+
+ An application converts a domain name put into an IDN-unaware slot or
+ displayed to a user. This section specifies the steps to perform in
+ the conversion, and the ToASCII and ToUnicode operations.
+
+ The input to ToASCII or ToUnicode is a single label that is a
+ sequence of Unicode code points (remember that all ASCII code points
+ are also Unicode code points). If a domain name is represented using
+ a character set other than Unicode or US-ASCII, it will first need to
+ be transcoded to Unicode.
+
+ Starting from a whole domain name, the steps that an application
+ takes to do the conversions are:
+
+ 1) Decide whether the domain name is a "stored string" or a "query
+ string" as described in [STRINGPREP]. If this conversion follows
+ the "queries" rule from [STRINGPREP], set the flag called
+ "AllowUnassigned".
+
+ 2) Split the domain name into individual labels as described in
+ section 3.1. The labels do not include the separator.
+
+ 3) For each label, decide whether or not to enforce the restrictions
+ on ASCII characters in host names [STD3]. (Applications already
+ faced this choice before the introduction of IDNA, and can
+ continue to make the decision the same way they always have; IDNA
+ makes no new recommendations regarding this choice.) If the
+ restrictions are to be enforced, set the flag called
+ "UseSTD3ASCIIRules" for that label.
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 9]
+\f
+RFC 3490 IDNA March 2003
+
+
+ 4) Process each label with either the ToASCII or the ToUnicode
+ operation as appropriate. Typically, you use the ToASCII
+ operation if you are about to put the name into an IDN-unaware
+ slot, and you use the ToUnicode operation if you are displaying
+ the name to a user; section 3.1 gives greater detail on the
+ applicable requirements.
+
+ 5) If ToASCII was applied in step 4 and dots are used as label
+ separators, change all the label separators to U+002E (full stop).
+
+ The following two subsections define the ToASCII and ToUnicode
+ operations that are used in step 4.
+
+ This description of the protocol uses specific procedure names, names
+ of flags, and so on, in order to facilitate the specification of the
+ protocol. These names, as well as the actual steps of the
+ procedures, are not required of an implementation. In fact, any
+ implementation which has the same external behavior as specified in
+ this document conforms to this specification.
+
+4.1 ToASCII
+
+ The ToASCII operation takes a sequence of Unicode code points that
+ make up one label and transforms it into a sequence of code points in
+ the ASCII range (0..7F). If ToASCII succeeds, the original sequence
+ and the resulting sequence are equivalent labels.
+
+ It is important to note that the ToASCII operation can fail. ToASCII
+ fails if any step of it fails. If any step of the ToASCII operation
+ fails on any label in a domain name, that domain name MUST NOT be
+ used as an internationalized domain name. The method for dealing
+ with this failure is application-specific.
+
+ The inputs to ToASCII are a sequence of code points, the
+ AllowUnassigned flag, and the UseSTD3ASCIIRules flag. The output of
+ ToASCII is either a sequence of ASCII code points or a failure
+ condition.
+
+ ToASCII never alters a sequence of code points that are all in the
+ ASCII range to begin with (although it could fail). Applying the
+ ToASCII operation multiple times has exactly the same effect as
+ applying it just once.
+
+ ToASCII consists of the following steps:
+
+ 1. If the sequence contains any code points outside the ASCII range
+ (0..7F) then proceed to step 2, otherwise skip to step 3.
+
+
+
+
+Faltstrom, et al. Standards Track [Page 10]
+\f
+RFC 3490 IDNA March 2003
+
+
+ 2. Perform the steps specified in [NAMEPREP] and fail if there is an
+ error. The AllowUnassigned flag is used in [NAMEPREP].
+
+ 3. If the UseSTD3ASCIIRules flag is set, then perform these checks:
+
+ (a) Verify the absence of non-LDH ASCII code points; that is, the
+ absence of 0..2C, 2E..2F, 3A..40, 5B..60, and 7B..7F.
+
+ (b) Verify the absence of leading and trailing hyphen-minus; that
+ is, the absence of U+002D at the beginning and end of the
+ sequence.
+
+ 4. If the sequence contains any code points outside the ASCII range
+ (0..7F) then proceed to step 5, otherwise skip to step 8.
+
+ 5. Verify that the sequence does NOT begin with the ACE prefix.
+
+ 6. Encode the sequence using the encoding algorithm in [PUNYCODE] and
+ fail if there is an error.
+
+ 7. Prepend the ACE prefix.
+
+ 8. Verify that the number of code points is in the range 1 to 63
+ inclusive.
+
+4.2 ToUnicode
+
+ The ToUnicode operation takes a sequence of Unicode code points that
+ make up one label and returns a sequence of Unicode code points. If
+ the input sequence is a label in ACE form, then the result is an
+ equivalent internationalized label that is not in ACE form, otherwise
+ the original sequence is returned unaltered.
+
+ ToUnicode never fails. If any step fails, then the original input
+ sequence is returned immediately in that step.
+
+ The ToUnicode output never contains more code points than its input.
+ Note that the number of octets needed to represent a sequence of code
+ points depends on the particular character encoding used.
+
+ The inputs to ToUnicode are a sequence of code points, the
+ AllowUnassigned flag, and the UseSTD3ASCIIRules flag. The output of
+ ToUnicode is always a sequence of Unicode code points.
+
+ 1. If all code points in the sequence are in the ASCII range (0..7F)
+ then skip to step 3.
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 11]
+\f
+RFC 3490 IDNA March 2003
+
+
+ 2. Perform the steps specified in [NAMEPREP] and fail if there is an
+ error. (If step 3 of ToASCII is also performed here, it will not
+ affect the overall behavior of ToUnicode, but it is not
+ necessary.) The AllowUnassigned flag is used in [NAMEPREP].
+
+ 3. Verify that the sequence begins with the ACE prefix, and save a
+ copy of the sequence.
+
+ 4. Remove the ACE prefix.
+
+ 5. Decode the sequence using the decoding algorithm in [PUNYCODE] and
+ fail if there is an error. Save a copy of the result of this
+ step.
+
+ 6. Apply ToASCII.
+
+ 7. Verify that the result of step 6 matches the saved copy from step
+ 3, using a case-insensitive ASCII comparison.
+
+ 8. Return the saved copy from step 5.
+
+5. ACE prefix
+
+ The ACE prefix, used in the conversion operations (section 4), is two
+ alphanumeric ASCII characters followed by two hyphen-minuses. It
+ cannot be any of the prefixes already used in earlier documents,
+ which includes the following: "bl--", "bq--", "dq--", "lq--", "mq--",
+ "ra--", "wq--" and "zq--". The ToASCII and ToUnicode operations MUST
+ recognize the ACE prefix in a case-insensitive manner.
+
+ The ACE prefix for IDNA is "xn--" or any capitalization thereof.
+
+ This means that an ACE label might be "xn--de-jg4avhby1noc0d", where
+ "de-jg4avhby1noc0d" is the part of the ACE label that is generated by
+ the encoding steps in [PUNYCODE].
+
+ While all ACE labels begin with the ACE prefix, not all labels
+ beginning with the ACE prefix are necessarily ACE labels. Non-ACE
+ labels that begin with the ACE prefix will confuse users and SHOULD
+ NOT be allowed in DNS zones.
+
+
+
+
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 12]
+\f
+RFC 3490 IDNA March 2003
+
+
+6. Implications for typical applications using DNS
+
+ In IDNA, applications perform the processing needed to input
+ internationalized domain names from users, display internationalized
+ domain names to users, and process the inputs and outputs from DNS
+ and other protocols that carry domain names.
+
+ The components and interfaces between them can be represented
+ pictorially as:
+
+ +------+
+ | User |
+ +------+
+ ^
+ | Input and display: local interface methods
+ | (pen, keyboard, glowing phosphorus, ...)
+ +-------------------|-------------------------------+
+ | v |
+ | +-----------------------------+ |
+ | | Application | |
+ | | (ToASCII and ToUnicode | |
+ | | operations may be | |
+ | | called here) | |
+ | +-----------------------------+ |
+ | ^ ^ | End system
+ | | | |
+ | Call to resolver: | | Application-specific |
+ | ACE | | protocol: |
+ | v | ACE unless the |
+ | +----------+ | protocol is updated |
+ | | Resolver | | to handle other |
+ | +----------+ | encodings |
+ | ^ | |
+ +-----------------|----------|----------------------+
+ DNS protocol: | |
+ ACE | |
+ v v
+ +-------------+ +---------------------+
+ | DNS servers | | Application servers |
+ +-------------+ +---------------------+
+
+ The box labeled "Application" is where the application splits a
+ domain name into labels, sets the appropriate flags, and performs the
+ ToASCII and ToUnicode operations. This is described in section 4.
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 13]
+\f
+RFC 3490 IDNA March 2003
+
+
+6.1 Entry and display in applications
+
+ Applications can accept domain names using any character set or sets
+ desired by the application developer, and can display domain names in
+ any charset. That is, the IDNA protocol does not affect the
+ interface between users and applications.
+
+ An IDNA-aware application can accept and display internationalized
+ domain names in two formats: the internationalized character set(s)
+ supported by the application, and as an ACE label. ACE labels that
+ are displayed or input MUST always include the ACE prefix.
+ Applications MAY allow input and display of ACE labels, but are not
+ encouraged to do so except as an interface for special purposes,
+ possibly for debugging, or to cope with display limitations as
+ described in section 6.4.. ACE encoding is opaque and ugly, and
+ should thus only be exposed to users who absolutely need it. Because
+ name labels encoded as ACE name labels can be rendered either as the
+ encoded ASCII characters or the proper decoded characters, the
+ application MAY have an option for the user to select the preferred
+ method of display; if it does, rendering the ACE SHOULD NOT be the
+ default.
+
+ Domain names are often stored and transported in many places. For
+ example, they are part of documents such as mail messages and web
+ pages. They are transported in many parts of many protocols, such as
+ both the control commands and the RFC 2822 body parts of SMTP, and
+ the headers and the body content in HTTP. It is important to
+ remember that domain names appear both in domain name slots and in
+ the content that is passed over protocols.
+
+ In protocols and document formats that define how to handle
+ specification or negotiation of charsets, labels can be encoded in
+ any charset allowed by the protocol or document format. If a
+ protocol or document format only allows one charset, the labels MUST
+ be given in that charset.
+
+ In any place where a protocol or document format allows transmission
+ of the characters in internationalized labels, internationalized
+ labels SHOULD be transmitted using whatever character encoding and
+ escape mechanism that the protocol or document format uses at that
+ place.
+
+ All protocols that use domain name slots already have the capacity
+ for handling domain names in the ASCII charset. Thus, ACE labels
+ (internationalized labels that have been processed with the ToASCII
+ operation) can inherently be handled by those protocols.
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 14]
+\f
+RFC 3490 IDNA March 2003
+
+
+6.2 Applications and resolver libraries
+
+ Applications normally use functions in the operating system when they
+ resolve DNS queries. Those functions in the operating system are
+ often called "the resolver library", and the applications communicate
+ with the resolver libraries through a programming interface (API).
+
+ Because these resolver libraries today expect only domain names in
+ ASCII, applications MUST prepare labels that are passed to the
+ resolver library using the ToASCII operation. Labels received from
+ the resolver library contain only ASCII characters; internationalized
+ labels that cannot be represented directly in ASCII use the ACE form.
+ ACE labels always include the ACE prefix.
+
+ An operating system might have a set of libraries for performing the
+ ToASCII operation. The input to such a library might be in one or
+ more charsets that are used in applications (UTF-8 and UTF-16 are
+ likely candidates for almost any operating system, and script-
+ specific charsets are likely for localized operating systems).
+
+ IDNA-aware applications MUST be able to work with both non-
+ internationalized labels (those that conform to [STD13] and [STD3])
+ and internationalized labels.
+
+ It is expected that new versions of the resolver libraries in the
+ future will be able to accept domain names in other charsets than
+ ASCII, and application developers might one day pass not only domain
+ names in Unicode, but also in local script to a new API for the
+ resolver libraries in the operating system. Thus the ToASCII and
+ ToUnicode operations might be performed inside these new versions of
+ the resolver libraries.
+
+ Domain names passed to resolvers or put into the question section of
+ DNS requests follow the rules for "queries" from [STRINGPREP].
+
+6.3 DNS servers
+
+ Domain names stored in zones follow the rules for "stored strings"
+ from [STRINGPREP].
+
+ For internationalized labels that cannot be represented directly in
+ ASCII, DNS servers MUST use the ACE form produced by the ToASCII
+ operation. All IDNs served by DNS servers MUST contain only ASCII
+ characters.
+
+ If a signaling system which makes negotiation possible between old
+ and new DNS clients and servers is standardized in the future, the
+ encoding of the query in the DNS protocol itself can be changed from
+
+
+
+Faltstrom, et al. Standards Track [Page 15]
+\f
+RFC 3490 IDNA March 2003
+
+
+ ACE to something else, such as UTF-8. The question whether or not
+ this should be used is, however, a separate problem and is not
+ discussed in this memo.
+
+6.4 Avoiding exposing users to the raw ACE encoding
+
+ Any application that might show the user a domain name obtained from
+ a domain name slot, such as from gethostbyaddr or part of a mail
+ header, will need to be updated if it is to prevent users from seeing
+ the ACE.
+
+ If an application decodes an ACE name using ToUnicode but cannot show
+ all of the characters in the decoded name, such as if the name
+ contains characters that the output system cannot display, the
+ application SHOULD show the name in ACE format (which always includes
+ the ACE prefix) instead of displaying the name with the replacement
+ character (U+FFFD). This is to make it easier for the user to
+ transfer the name correctly to other programs. Programs that by
+ default show the ACE form when they cannot show all the characters in
+ a name label SHOULD also have a mechanism to show the name that is
+ produced by the ToUnicode operation with as many characters as
+ possible and replacement characters in the positions where characters
+ cannot be displayed.
+
+ The ToUnicode operation does not alter labels that are not valid ACE
+ labels, even if they begin with the ACE prefix. After ToUnicode has
+ been applied, if a label still begins with the ACE prefix, then it is
+ not a valid ACE label, and is not equivalent to any of the
+ intermediate Unicode strings constructed by ToUnicode.
+
+6.5 DNSSEC authentication of IDN domain names
+
+ DNS Security [RFC2535] is a method for supplying cryptographic
+ verification information along with DNS messages. Public Key
+ Cryptography is used in conjunction with digital signatures to
+ provide a means for a requester of domain information to authenticate
+ the source of the data. This ensures that it can be traced back to a
+ trusted source, either directly, or via a chain of trust linking the
+ source of the information to the top of the DNS hierarchy.
+
+ IDNA specifies that all internationalized domain names served by DNS
+ servers that cannot be represented directly in ASCII must use the ACE
+ form produced by the ToASCII operation. This operation must be
+ performed prior to a zone being signed by the private key for that
+ zone. Because of this ordering, it is important to recognize that
+ DNSSEC authenticates the ASCII domain name, not the Unicode form or
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 16]
+\f
+RFC 3490 IDNA March 2003
+
+
+ the mapping between the Unicode form and the ASCII form. In the
+ presence of DNSSEC, this is the name that MUST be signed in the zone
+ and MUST be validated against.
+
+ One consequence of this for sites deploying IDNA in the presence of
+ DNSSEC is that any special purpose proxies or forwarders used to
+ transform user input into IDNs must be earlier in the resolution flow
+ than DNSSEC authenticating nameservers for DNSSEC to work.
+
+7. Name server considerations
+
+ Existing DNS servers do not know the IDNA rules for handling non-
+ ASCII forms of IDNs, and therefore need to be shielded from them.
+ All existing channels through which names can enter a DNS server
+ database (for example, master files [STD13] and DNS update messages
+ [RFC2136]) are IDN-unaware because they predate IDNA, and therefore
+ requirement 2 of section 3.1 of this document provides the needed
+ shielding, by ensuring that internationalized domain names entering
+ DNS server databases through such channels have already been
+ converted to their equivalent ASCII forms.
+
+ It is imperative that there be only one ASCII encoding for a
+ particular domain name. Because of the design of the ToASCII and
+ ToUnicode operations, there are no ACE labels that decode to ASCII
+ labels, and therefore name servers cannot contain multiple ASCII
+ encodings of the same domain name.
+
+ [RFC2181] explicitly allows domain labels to contain octets beyond
+ the ASCII range (0..7F), and this document does not change that.
+ Note, however, that there is no defined interpretation of octets
+ 80..FF as characters. If labels containing these octets are returned
+ to applications, unpredictable behavior could result. The ASCII form
+ defined by ToASCII is the only standard representation for
+ internationalized labels in the current DNS protocol.
+
+8. Root server considerations
+
+ IDNs are likely to be somewhat longer than current domain names, so
+ the bandwidth needed by the root servers is likely to go up by a
+ small amount. Also, queries and responses for IDNs will probably be
+ somewhat longer than typical queries today, so more queries and
+ responses may be forced to go to TCP instead of UDP.
+
+
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 17]
+\f
+RFC 3490 IDNA March 2003
+
+
+9. References
+
+9.1 Normative References
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+ [STRINGPREP] Hoffman, P. and M. Blanchet, "Preparation of
+ Internationalized Strings ("stringprep")", RFC 3454,
+ December 2002.
+
+ [NAMEPREP] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
+ Profile for Internationalized Domain Names (IDN)", RFC
+ 3491, March 2003.
+
+ [PUNYCODE] Costello, A., "Punycode: A Bootstring encoding of
+ Unicode for use with Internationalized Domain Names in
+ Applications (IDNA)", RFC 3492, March 2003.
+
+ [STD3] Braden, R., "Requirements for Internet Hosts --
+ Communication Layers", STD 3, RFC 1122, and
+ "Requirements for Internet Hosts -- Application and
+ Support", STD 3, RFC 1123, October 1989.
+
+ [STD13] Mockapetris, P., "Domain names - concepts and
+ facilities", STD 13, RFC 1034 and "Domain names -
+ implementation and specification", STD 13, RFC 1035,
+ November 1987.
+
+9.2 Informative References
+
+ [RFC2535] Eastlake, D., "Domain Name System Security Extensions",
+ RFC 2535, March 1999.
+
+ [RFC2181] Elz, R. and R. Bush, "Clarifications to the DNS
+ Specification", RFC 2181, July 1997.
+
+ [UAX9] Unicode Standard Annex #9, The Bidirectional Algorithm,
+ <http://www.unicode.org/unicode/reports/tr9/>.
+
+ [UNICODE] The Unicode Consortium. The Unicode Standard, Version
+ 3.2.0 is defined by The Unicode Standard, Version 3.0
+ (Reading, MA, Addison-Wesley, 2000. ISBN 0-201-61633-5),
+ as amended by the Unicode Standard Annex #27: Unicode
+ 3.1 (http://www.unicode.org/reports/tr27/) and by the
+ Unicode Standard Annex #28: Unicode 3.2
+ (http://www.unicode.org/reports/tr28/).
+
+
+
+
+Faltstrom, et al. Standards Track [Page 18]
+\f
+RFC 3490 IDNA March 2003
+
+
+ [USASCII] Cerf, V., "ASCII format for Network Interchange", RFC
+ 20, October 1969.
+
+10. Security Considerations
+
+ Security on the Internet partly relies on the DNS. Thus, any change
+ to the characteristics of the DNS can change the security of much of
+ the Internet.
+
+ This memo describes an algorithm which encodes characters that are
+ not valid according to STD3 and STD13 into octet values that are
+ valid. No security issues such as string length increases or new
+ allowed values are introduced by the encoding process or the use of
+ these encoded values, apart from those introduced by the ACE encoding
+ itself.
+
+ Domain names are used by users to identify and connect to Internet
+ servers. The security of the Internet is compromised if a user
+ entering a single internationalized name is connected to different
+ servers based on different interpretations of the internationalized
+ domain name.
+
+ When systems use local character sets other than ASCII and Unicode,
+ this specification leaves the the problem of transcoding between the
+ local character set and Unicode up to the application. If different
+ applications (or different versions of one application) implement
+ different transcoding rules, they could interpret the same name
+ differently and contact different servers. This problem is not
+ solved by security protocols like TLS that do not take local
+ character sets into account.
+
+ Because this document normatively refers to [NAMEPREP], [PUNYCODE],
+ and [STRINGPREP], it includes the security considerations from those
+ documents as well.
+
+ If or when this specification is updated to use a more recent Unicode
+ normalization table, the new normalization table will need to be
+ compared with the old to spot backwards incompatible changes. If
+ there are such changes, they will need to be handled somehow, or
+ there will be security as well as operational implications. Methods
+ to handle the conflicts could include keeping the old normalization,
+ or taking care of the conflicting characters by operational means, or
+ some other method.
+
+ Implementations MUST NOT use more recent normalization tables than
+ the one referenced from this document, even though more recent tables
+ may be provided by operating systems. If an application is unsure of
+ which version of the normalization tables are in the operating
+
+
+
+Faltstrom, et al. Standards Track [Page 19]
+\f
+RFC 3490 IDNA March 2003
+
+
+ system, the application needs to include the normalization tables
+ itself. Using normalization tables other than the one referenced
+ from this specification could have security and operational
+ implications.
+
+ To help prevent confusion between characters that are visually
+ similar, it is suggested that implementations provide visual
+ indications where a domain name contains multiple scripts. Such
+ mechanisms can also be used to show when a name contains a mixture of
+ simplified and traditional Chinese characters, or to distinguish zero
+ and one from O and l. DNS zone adminstrators may impose restrictions
+ (subject to the limitations in section 2) that try to minimize
+ homographs.
+
+ Domain names (or portions of them) are sometimes compared against a
+ set of privileged or anti-privileged domains. In such situations it
+ is especially important that the comparisons be done properly, as
+ specified in section 3.1 requirement 4. For labels already in ASCII
+ form, the proper comparison reduces to the same case-insensitive
+ ASCII comparison that has always been used for ASCII labels.
+
+ The introduction of IDNA means that any existing labels that start
+ with the ACE prefix and would be altered by ToUnicode will
+ automatically be ACE labels, and will be considered equivalent to
+ non-ASCII labels, whether or not that was the intent of the zone
+ adminstrator or registrant.
+
+11. IANA Considerations
+
+ IANA has assigned the ACE prefix in consultation with the IESG.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 20]
+\f
+RFC 3490 IDNA March 2003
+
+
+12. Authors' Addresses
+
+ Patrik Faltstrom
+ Cisco Systems
+ Arstaangsvagen 31 J
+ S-117 43 Stockholm Sweden
+
+ EMail: paf@cisco.com
+
+
+ Paul Hoffman
+ Internet Mail Consortium and VPN Consortium
+ 127 Segre Place
+ Santa Cruz, CA 95060 USA
+
+ EMail: phoffman@imc.org
+
+
+ Adam M. Costello
+ University of California, Berkeley
+
+ URL: http://www.nicemice.net/amc/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 21]
+\f
+RFC 3490 IDNA March 2003
+
+
+13. Full Copyright Statement
+
+ Copyright (C) The Internet Society (2003). All Rights Reserved.
+
+ This document and translations of it may be copied and furnished to
+ others, and derivative works that comment on or otherwise explain it
+ or assist in its implementation may be prepared, copied, published
+ and distributed, in whole or in part, without restriction of any
+ kind, provided that the above copyright notice and this paragraph are
+ included on all such copies and derivative works. However, this
+ document itself may not be modified in any way, such as by removing
+ the copyright notice or references to the Internet Society or other
+ Internet organizations, except as needed for the purpose of
+ developing Internet standards in which case the procedures for
+ copyrights defined in the Internet Standards process must be
+ followed, or as required to translate it into languages other than
+ English.
+
+ The limited permissions granted above are perpetual and will not be
+ revoked by the Internet Society or its successors or assigns.
+
+ This document and the information contained herein is provided on an
+ "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
+ TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
+ BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
+ HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
+ MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Acknowledgement
+
+ Funding for the RFC Editor function is currently provided by the
+ Internet Society.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Faltstrom, et al. Standards Track [Page 22]
+\f
--- /dev/null
+
+
+
+
+
+
+Network Working Group A. Costello
+Request for Comments: 3492 Univ. of California, Berkeley
+Category: Standards Track March 2003
+
+
+ Punycode: A Bootstring encoding of Unicode
+ for Internationalized Domain Names in Applications (IDNA)
+
+Status of this Memo
+
+ This document specifies an Internet standards track protocol for the
+ Internet community, and requests discussion and suggestions for
+ improvements. Please refer to the current edition of the "Internet
+ Official Protocol Standards" (STD 1) for the standardization state
+ and status of this protocol. Distribution of this memo is unlimited.
+
+Copyright Notice
+
+ Copyright (C) The Internet Society (2003). All Rights Reserved.
+
+Abstract
+
+ Punycode is a simple and efficient transfer encoding syntax designed
+ for use with Internationalized Domain Names in Applications (IDNA).
+ It uniquely and reversibly transforms a Unicode string into an ASCII
+ string. ASCII characters in the Unicode string are represented
+ literally, and non-ASCII characters are represented by ASCII
+ characters that are allowed in host name labels (letters, digits, and
+ hyphens). This document defines a general algorithm called
+ Bootstring that allows a string of basic code points to uniquely
+ represent any string of code points drawn from a larger set.
+ Punycode is an instance of Bootstring that uses particular parameter
+ values specified by this document, appropriate for IDNA.
+
+Table of Contents
+
+ 1. Introduction...............................................2
+ 1.1 Features..............................................2
+ 1.2 Interaction of protocol parts.........................3
+ 2. Terminology................................................3
+ 3. Bootstring description.....................................4
+ 3.1 Basic code point segregation..........................4
+ 3.2 Insertion unsort coding...............................4
+ 3.3 Generalized variable-length integers..................5
+ 3.4 Bias adaptation.......................................7
+ 4. Bootstring parameters......................................8
+ 5. Parameter values for Punycode..............................8
+ 6. Bootstring algorithms......................................9
+
+
+
+Costello Standards Track [Page 1]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ 6.1 Bias adaptation function.............................10
+ 6.2 Decoding procedure...................................11
+ 6.3 Encoding procedure...................................12
+ 6.4 Overflow handling....................................13
+ 7. Punycode examples.........................................14
+ 7.1 Sample strings.......................................14
+ 7.2 Decoding traces......................................17
+ 7.3 Encoding traces......................................19
+ 8. Security Considerations...................................20
+ 9. References................................................21
+ 9.1 Normative References.................................21
+ 9.2 Informative References...............................21
+ A. Mixed-case annotation.....................................22
+ B. Disclaimer and license....................................22
+ C. Punycode sample implementation............................23
+ Author's Address.............................................34
+ Full Copyright Statement.....................................35
+
+1. Introduction
+
+ [IDNA] describes an architecture for supporting internationalized
+ domain names. Labels containing non-ASCII characters can be
+ represented by ACE labels, which begin with a special ACE prefix and
+ contain only ASCII characters. The remainder of the label after the
+ prefix is a Punycode encoding of a Unicode string satisfying certain
+ constraints. For the details of the prefix and constraints, see
+ [IDNA] and [NAMEPREP].
+
+ Punycode is an instance of a more general algorithm called
+ Bootstring, which allows strings composed from a small set of "basic"
+ code points to uniquely represent any string of code points drawn
+ from a larger set. Punycode is Bootstring with particular parameter
+ values appropriate for IDNA.
+
+1.1 Features
+
+ Bootstring has been designed to have the following features:
+
+ * Completeness: Every extended string (sequence of arbitrary code
+ points) can be represented by a basic string (sequence of basic
+ code points). Restrictions on what strings are allowed, and on
+ length, can be imposed by higher layers.
+
+ * Uniqueness: There is at most one basic string that represents a
+ given extended string.
+
+ * Reversibility: Any extended string mapped to a basic string can
+ be recovered from that basic string.
+
+
+
+Costello Standards Track [Page 2]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ * Efficient encoding: The ratio of basic string length to extended
+ string length is small. This is important in the context of
+ domain names because RFC 1034 [RFC1034] restricts the length of a
+ domain label to 63 characters.
+
+ * Simplicity: The encoding and decoding algorithms are reasonably
+ simple to implement. The goals of efficiency and simplicity are
+ at odds; Bootstring aims at a good balance between them.
+
+ * Readability: Basic code points appearing in the extended string
+ are represented as themselves in the basic string (although the
+ main purpose is to improve efficiency, not readability).
+
+ Punycode can also support an additional feature that is not used by
+ the ToASCII and ToUnicode operations of [IDNA]. When extended
+ strings are case-folded prior to encoding, the basic string can use
+ mixed case to tell how to convert the folded string into a mixed-case
+ string. See appendix A "Mixed-case annotation".
+
+1.2 Interaction of protocol parts
+
+ Punycode is used by the IDNA protocol [IDNA] for converting domain
+ labels into ASCII; it is not designed for any other purpose. It is
+ explicitly not designed for processing arbitrary free text.
+
+2. Terminology
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in BCP 14, RFC 2119
+ [RFC2119].
+
+ A code point is an integral value associated with a character in a
+ coded character set.
+
+ As in the Unicode Standard [UNICODE], Unicode code points are denoted
+ by "U+" followed by four to six hexadecimal digits, while a range of
+ code points is denoted by two hexadecimal numbers separated by "..",
+ with no prefixes.
+
+ The operators div and mod perform integer division; (x div y) is the
+ quotient of x divided by y, discarding the remainder, and (x mod y)
+ is the remainder, so (x div y) * y + (x mod y) == x. Bootstring uses
+ these operators only with nonnegative operands, so the quotient and
+ remainder are always nonnegative.
+
+ The break statement jumps out of the innermost loop (as in C).
+
+
+
+
+Costello Standards Track [Page 3]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ An overflow is an attempt to compute a value that exceeds the maximum
+ value of an integer variable.
+
+3. Bootstring description
+
+ Bootstring represents an arbitrary sequence of code points (the
+ "extended string") as a sequence of basic code points (the "basic
+ string"). This section describes the representation. Section 6
+ "Bootstring algorithms" presents the algorithms as pseudocode.
+ Sections 7.1 "Decoding traces" and 7.2 "Encoding traces" trace the
+ algorithms for sample inputs.
+
+ The following sections describe the four techniques used in
+ Bootstring. "Basic code point segregation" is a very simple and
+ efficient encoding for basic code points occurring in the extended
+ string: they are simply copied all at once. "Insertion unsort
+ coding" encodes the non-basic code points as deltas, and processes
+ the code points in numerical order rather than in order of
+ appearance, which typically results in smaller deltas. The deltas
+ are represented as "generalized variable-length integers", which use
+ basic code points to represent nonnegative integers. The parameters
+ of this integer representation are dynamically adjusted using "bias
+ adaptation", to improve efficiency when consecutive deltas have
+ similar magnitudes.
+
+3.1 Basic code point segregation
+
+ All basic code points appearing in the extended string are
+ represented literally at the beginning of the basic string, in their
+ original order, followed by a delimiter if (and only if) the number
+ of basic code points is nonzero. The delimiter is a particular basic
+ code point, which never appears in the remainder of the basic string.
+ The decoder can therefore find the end of the literal portion (if
+ there is one) by scanning for the last delimiter.
+
+3.2 Insertion unsort coding
+
+ The remainder of the basic string (after the last delimiter if there
+ is one) represents a sequence of nonnegative integral deltas as
+ generalized variable-length integers, described in section 3.3. The
+ meaning of the deltas is best understood in terms of the decoder.
+
+ The decoder builds the extended string incrementally. Initially, the
+ extended string is a copy of the literal portion of the basic string
+ (excluding the last delimiter). The decoder inserts non-basic code
+ points, one for each delta, into the extended string, ultimately
+ arriving at the final decoded string.
+
+
+
+
+Costello Standards Track [Page 4]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ At the heart of this process is a state machine with two state
+ variables: an index i and a counter n. The index i refers to a
+ position in the extended string; it ranges from 0 (the first
+ position) to the current length of the extended string (which refers
+ to a potential position beyond the current end). If the current
+ state is <n,i>, the next state is <n,i+1> if i is less than the
+ length of the extended string, or <n+1,0> if i equals the length of
+ the extended string. In other words, each state change causes i to
+ increment, wrapping around to zero if necessary, and n counts the
+ number of wrap-arounds.
+
+ Notice that the state always advances monotonically (there is no way
+ for the decoder to return to an earlier state). At each state, an
+ insertion is either performed or not performed. At most one
+ insertion is performed in a given state. An insertion inserts the
+ value of n at position i in the extended string. The deltas are a
+ run-length encoding of this sequence of events: they are the lengths
+ of the runs of non-insertion states preceeding the insertion states.
+ Hence, for each delta, the decoder performs delta state changes, then
+ an insertion, and then one more state change. (An implementation
+ need not perform each state change individually, but can instead use
+ division and remainder calculations to compute the next insertion
+ state directly.) It is an error if the inserted code point is a
+ basic code point (because basic code points were supposed to be
+ segregated as described in section 3.1).
+
+ The encoder's main task is to derive the sequence of deltas that will
+ cause the decoder to construct the desired string. It can do this by
+ repeatedly scanning the extended string for the next code point that
+ the decoder would need to insert, and counting the number of state
+ changes the decoder would need to perform, mindful of the fact that
+ the decoder's extended string will include only those code points
+ that have already been inserted. Section 6.3 "Encoding procedure"
+ gives a precise algorithm.
+
+3.3 Generalized variable-length integers
+
+ In a conventional integer representation the base is the number of
+ distinct symbols for digits, whose values are 0 through base-1. Let
+ digit_0 denote the least significant digit, digit_1 the next least
+ significant, and so on. The value represented is the sum over j of
+ digit_j * w(j), where w(j) = base^j is the weight (scale factor) for
+ position j. For example, in the base 8 integer 437, the digits are
+ 7, 3, and 4, and the weights are 1, 8, and 64, so the value is 7 +
+ 3*8 + 4*64 = 287. This representation has two disadvantages: First,
+ there are multiple encodings of each value (because there can be
+ extra zeros in the most significant positions), which is inconvenient
+
+
+
+
+Costello Standards Track [Page 5]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ when unique encodings are needed. Second, the integer is not self-
+ delimiting, so if multiple integers are concatenated the boundaries
+ between them are lost.
+
+ The generalized variable-length representation solves these two
+ problems. The digit values are still 0 through base-1, but now the
+ integer is self-delimiting by means of thresholds t(j), each of which
+ is in the range 0 through base-1. Exactly one digit, the most
+ significant, satisfies digit_j < t(j). Therefore, if several
+ integers are concatenated, it is easy to separate them, starting with
+ the first if they are little-endian (least significant digit first),
+ or starting with the last if they are big-endian (most significant
+ digit first). As before, the value is the sum over j of digit_j *
+ w(j), but the weights are different:
+
+ w(0) = 1
+ w(j) = w(j-1) * (base - t(j-1)) for j > 0
+
+ For example, consider the little-endian sequence of base 8 digits
+ 734251... Suppose the thresholds are 2, 3, 5, 5, 5, 5... This
+ implies that the weights are 1, 1*(8-2) = 6, 6*(8-3) = 30, 30*(8-5) =
+ 90, 90*(8-5) = 270, and so on. 7 is not less than 2, and 3 is not
+ less than 3, but 4 is less than 5, so 4 is the last digit. The value
+ of 734 is 7*1 + 3*6 + 4*30 = 145. The next integer is 251, with
+ value 2*1 + 5*6 + 1*30 = 62. Decoding this representation is very
+ similar to decoding a conventional integer: Start with a current
+ value of N = 0 and a weight w = 1. Fetch the next digit d and
+ increase N by d * w. If d is less than the current threshold (t)
+ then stop, otherwise increase w by a factor of (base - t), update t
+ for the next position, and repeat.
+
+ Encoding this representation is similar to encoding a conventional
+ integer: If N < t then output one digit for N and stop, otherwise
+ output the digit for t + ((N - t) mod (base - t)), then replace N
+ with (N - t) div (base - t), update t for the next position, and
+ repeat.
+
+ For any particular set of values of t(j), there is exactly one
+ generalized variable-length representation of each nonnegative
+ integral value.
+
+ Bootstring uses little-endian ordering so that the deltas can be
+ separated starting with the first. The t(j) values are defined in
+ terms of the constants base, tmin, and tmax, and a state variable
+ called bias:
+
+ t(j) = base * (j + 1) - bias,
+ clamped to the range tmin through tmax
+
+
+
+Costello Standards Track [Page 6]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ The clamping means that if the formula yields a value less than tmin
+ or greater than tmax, then t(j) = tmin or tmax, respectively. (In
+ the pseudocode in section 6 "Bootstring algorithms", the expression
+ base * (j + 1) is denoted by k for performance reasons.) These t(j)
+ values cause the representation to favor integers within a particular
+ range determined by the bias.
+
+3.4 Bias adaptation
+
+ After each delta is encoded or decoded, bias is set for the next
+ delta as follows:
+
+ 1. Delta is scaled in order to avoid overflow in the next step:
+
+ let delta = delta div 2
+
+ But when this is the very first delta, the divisor is not 2, but
+ instead a constant called damp. This compensates for the fact
+ that the second delta is usually much smaller than the first.
+
+ 2. Delta is increased to compensate for the fact that the next delta
+ will be inserting into a longer string:
+
+ let delta = delta + (delta div numpoints)
+
+ numpoints is the total number of code points encoded/decoded so
+ far (including the one corresponding to this delta itself, and
+ including the basic code points).
+
+ 3. Delta is repeatedly divided until it falls within a threshold, to
+ predict the minimum number of digits needed to represent the next
+ delta:
+
+ while delta > ((base - tmin) * tmax) div 2
+ do let delta = delta div (base - tmin)
+
+ 4. The bias is set:
+
+ let bias =
+ (base * the number of divisions performed in step 3) +
+ (((base - tmin + 1) * delta) div (delta + skew))
+
+ The motivation for this procedure is that the current delta
+ provides a hint about the likely size of the next delta, and so
+ t(j) is set to tmax for the more significant digits starting with
+ the one expected to be last, tmin for the less significant digits
+ up through the one expected to be third-last, and somewhere
+ between tmin and tmax for the digit expected to be second-last
+
+
+
+Costello Standards Track [Page 7]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ (balancing the hope of the expected-last digit being unnecessary
+ against the danger of it being insufficient).
+
+4. Bootstring parameters
+
+ Given a set of basic code points, one needs to be designated as the
+ delimiter. The base cannot be greater than the number of
+ distinguishable basic code points remaining. The digit-values in the
+ range 0 through base-1 need to be associated with distinct non-
+ delimiter basic code points. In some cases multiple code points need
+ to have the same digit-value; for example, uppercase and lowercase
+ versions of the same letter need to be equivalent if basic strings
+ are case-insensitive.
+
+ The initial value of n cannot be greater than the minimum non-basic
+ code point that could appear in extended strings.
+
+ The remaining five parameters (tmin, tmax, skew, damp, and the
+ initial value of bias) need to satisfy the following constraints:
+
+ 0 <= tmin <= tmax <= base-1
+ skew >= 1
+ damp >= 2
+ initial_bias mod base <= base - tmin
+
+ Provided the constraints are satisfied, these five parameters affect
+ efficiency but not correctness. They are best chosen empirically.
+
+ If support for mixed-case annotation is desired (see appendix A),
+ make sure that the code points corresponding to 0 through tmax-1 all
+ have both uppercase and lowercase forms.
+
+5. Parameter values for Punycode
+
+ Punycode uses the following Bootstring parameter values:
+
+ base = 36
+ tmin = 1
+ tmax = 26
+ skew = 38
+ damp = 700
+ initial_bias = 72
+ initial_n = 128 = 0x80
+
+ Although the only restriction Punycode imposes on the input integers
+ is that they be nonnegative, these parameters are especially designed
+ to work well with Unicode [UNICODE] code points, which are integers
+ in the range 0..10FFFF (but not D800..DFFF, which are reserved for
+
+
+
+Costello Standards Track [Page 8]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ use by the UTF-16 encoding of Unicode). The basic code points are
+ the ASCII [ASCII] code points (0..7F), of which U+002D (-) is the
+ delimiter, and some of the others have digit-values as follows:
+
+ code points digit-values
+ ------------ ----------------------
+ 41..5A (A-Z) = 0 to 25, respectively
+ 61..7A (a-z) = 0 to 25, respectively
+ 30..39 (0-9) = 26 to 35, respectively
+
+ Using hyphen-minus as the delimiter implies that the encoded string
+ can end with a hyphen-minus only if the Unicode string consists
+ entirely of basic code points, but IDNA forbids such strings from
+ being encoded. The encoded string can begin with a hyphen-minus, but
+ IDNA prepends a prefix. Therefore IDNA using Punycode conforms to
+ the RFC 952 rule that host name labels neither begin nor end with a
+ hyphen-minus [RFC952].
+
+ A decoder MUST recognize the letters in both uppercase and lowercase
+ forms (including mixtures of both forms). An encoder SHOULD output
+ only uppercase forms or only lowercase forms, unless it uses mixed-
+ case annotation (see appendix A).
+
+ Presumably most users will not manually write or type encoded strings
+ (as opposed to cutting and pasting them), but those who do will need
+ to be alert to the potential visual ambiguity between the following
+ sets of characters:
+
+ G 6
+ I l 1
+ O 0
+ S 5
+ U V
+ Z 2
+
+ Such ambiguities are usually resolved by context, but in a Punycode
+ encoded string there is no context apparent to humans.
+
+6. Bootstring algorithms
+
+ Some parts of the pseudocode can be omitted if the parameters satisfy
+ certain conditions (for which Punycode qualifies). These parts are
+ enclosed in {braces}, and notes immediately following the pseudocode
+ explain the conditions under which they can be omitted.
+
+
+
+
+
+
+
+Costello Standards Track [Page 9]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ Formally, code points are integers, and hence the pseudocode assumes
+ that arithmetic operations can be performed directly on code points.
+ In some programming languages, explicit conversion between code
+ points and integers might be necessary.
+
+6.1 Bias adaptation function
+
+ function adapt(delta,numpoints,firsttime):
+ if firsttime then let delta = delta div damp
+ else let delta = delta div 2
+ let delta = delta + (delta div numpoints)
+ let k = 0
+ while delta > ((base - tmin) * tmax) div 2 do begin
+ let delta = delta div (base - tmin)
+ let k = k + base
+ end
+ return k + (((base - tmin + 1) * delta) div (delta + skew))
+
+ It does not matter whether the modifications to delta and k inside
+ adapt() affect variables of the same name inside the
+ encoding/decoding procedures, because after calling adapt() the
+ caller does not read those variables before overwriting them.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 10]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+6.2 Decoding procedure
+
+ let n = initial_n
+ let i = 0
+ let bias = initial_bias
+ let output = an empty string indexed from 0
+ consume all code points before the last delimiter (if there is one)
+ and copy them to output, fail on any non-basic code point
+ if more than zero code points were consumed then consume one more
+ (which will be the last delimiter)
+ while the input is not exhausted do begin
+ let oldi = i
+ let w = 1
+ for k = base to infinity in steps of base do begin
+ consume a code point, or fail if there was none to consume
+ let digit = the code point's digit-value, fail if it has none
+ let i = i + digit * w, fail on overflow
+ let t = tmin if k <= bias {+ tmin}, or
+ tmax if k >= bias + tmax, or k - bias otherwise
+ if digit < t then break
+ let w = w * (base - t), fail on overflow
+ end
+ let bias = adapt(i - oldi, length(output) + 1, test oldi is 0?)
+ let n = n + i div (length(output) + 1), fail on overflow
+ let i = i mod (length(output) + 1)
+ {if n is a basic code point then fail}
+ insert n into output at position i
+ increment i
+ end
+
+ The full statement enclosed in braces (checking whether n is a basic
+ code point) can be omitted if initial_n exceeds all basic code points
+ (which is true for Punycode), because n is never less than initial_n.
+
+ In the assignment of t, where t is clamped to the range tmin through
+ tmax, "+ tmin" can always be omitted. This makes the clamping
+ calculation incorrect when bias < k < bias + tmin, but that cannot
+ happen because of the way bias is computed and because of the
+ constraints on the parameters.
+
+ Because the decoder state can only advance monotonically, and there
+ is only one representation of any delta, there is therefore only one
+ encoded string that can represent a given sequence of integers. The
+ only error conditions are invalid code points, unexpected end-of-
+ input, overflow, and basic code points encoded using deltas instead
+ of appearing literally. If the decoder fails on these errors as
+ shown above, then it cannot produce the same output for two distinct
+ inputs. Without this property it would have been necessary to re-
+
+
+
+Costello Standards Track [Page 11]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ encode the output and verify that it matches the input in order to
+ guarantee the uniqueness of the encoding.
+
+6.3 Encoding procedure
+
+ let n = initial_n
+ let delta = 0
+ let bias = initial_bias
+ let h = b = the number of basic code points in the input
+ copy them to the output in order, followed by a delimiter if b > 0
+ {if the input contains a non-basic code point < n then fail}
+ while h < length(input) do begin
+ let m = the minimum {non-basic} code point >= n in the input
+ let delta = delta + (m - n) * (h + 1), fail on overflow
+ let n = m
+ for each code point c in the input (in order) do begin
+ if c < n {or c is basic} then increment delta, fail on overflow
+ if c == n then begin
+ let q = delta
+ for k = base to infinity in steps of base do begin
+ let t = tmin if k <= bias {+ tmin}, or
+ tmax if k >= bias + tmax, or k - bias otherwise
+ if q < t then break
+ output the code point for digit t + ((q - t) mod (base - t))
+ let q = (q - t) div (base - t)
+ end
+ output the code point for digit q
+ let bias = adapt(delta, h + 1, test h equals b?)
+ let delta = 0
+ increment h
+ end
+ end
+ increment delta and n
+ end
+
+ The full statement enclosed in braces (checking whether the input
+ contains a non-basic code point less than n) can be omitted if all
+ code points less than initial_n are basic code points (which is true
+ for Punycode if code points are unsigned).
+
+ The brace-enclosed conditions "non-basic" and "or c is basic" can be
+ omitted if initial_n exceeds all basic code points (which is true for
+ Punycode), because the code point being tested is never less than
+ initial_n.
+
+ In the assignment of t, where t is clamped to the range tmin through
+ tmax, "+ tmin" can always be omitted. This makes the clamping
+ calculation incorrect when bias < k < bias + tmin, but that cannot
+
+
+
+Costello Standards Track [Page 12]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ happen because of the way bias is computed and because of the
+ constraints on the parameters.
+
+ The checks for overflow are necessary to avoid producing invalid
+ output when the input contains very large values or is very long.
+
+ The increment of delta at the bottom of the outer loop cannot
+ overflow because delta < length(input) before the increment, and
+ length(input) is already assumed to be representable. The increment
+ of n could overflow, but only if h == length(input), in which case
+ the procedure is finished anyway.
+
+6.4 Overflow handling
+
+ For IDNA, 26-bit unsigned integers are sufficient to handle all valid
+ IDNA labels without overflow, because any string that needed a 27-bit
+ delta would have to exceed either the code point limit (0..10FFFF) or
+ the label length limit (63 characters). However, overflow handling
+ is necessary because the inputs are not necessarily valid IDNA
+ labels.
+
+ If the programming language does not provide overflow detection, the
+ following technique can be used. Suppose A, B, and C are
+ representable nonnegative integers and C is nonzero. Then A + B
+ overflows if and only if B > maxint - A, and A + (B * C) overflows if
+ and only if B > (maxint - A) div C, where maxint is the greatest
+ integer for which maxint + 1 cannot be represented. Refer to
+ appendix C "Punycode sample implementation" for demonstrations of
+ this technique in the C language.
+
+ The decoding and encoding algorithms shown in sections 6.2 and 6.3
+ handle overflow by detecting it whenever it happens. Another
+ approach is to enforce limits on the inputs that prevent overflow
+ from happening. For example, if the encoder were to verify that no
+ input code points exceed M and that the input length does not exceed
+ L, then no delta could ever exceed (M - initial_n) * (L + 1), and
+ hence no overflow could occur if integer variables were capable of
+ representing values that large. This prevention approach would
+ impose more restrictions on the input than the detection approach
+ does, but might be considered simpler in some programming languages.
+
+ In theory, the decoder could use an analogous approach, limiting the
+ number of digits in a variable-length integer (that is, limiting the
+ number of iterations in the innermost loop). However, the number of
+ digits that suffice to represent a given delta can sometimes
+ represent much larger deltas (because of the adaptation), and hence
+ this approach would probably need integers wider than 32 bits.
+
+
+
+
+Costello Standards Track [Page 13]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ Yet another approach for the decoder is to allow overflow to occur,
+ but to check the final output string by re-encoding it and comparing
+ to the decoder input. If and only if they do not match (using a
+ case-insensitive ASCII comparison) overflow has occurred. This
+ delayed-detection approach would not impose any more restrictions on
+ the input than the immediate-detection approach does, and might be
+ considered simpler in some programming languages.
+
+ In fact, if the decoder is used only inside the IDNA ToUnicode
+ operation [IDNA], then it need not check for overflow at all, because
+ ToUnicode performs a higher level re-encoding and comparison, and a
+ mismatch has the same consequence as if the Punycode decoder had
+ failed.
+
+7. Punycode examples
+
+7.1 Sample strings
+
+ In the Punycode encodings below, the ACE prefix is not shown.
+ Backslashes show where line breaks have been inserted in strings too
+ long for one line.
+
+ The first several examples are all translations of the sentence "Why
+ can't they just speak in <language>?" (courtesy of Michael Kaplan's
+ "provincial" page [PROVINCIAL]). Word breaks and punctuation have
+ been removed, as is often done in domain names.
+
+ (A) Arabic (Egyptian):
+ u+0644 u+064A u+0647 u+0645 u+0627 u+0628 u+062A u+0643 u+0644
+ u+0645 u+0648 u+0634 u+0639 u+0631 u+0628 u+064A u+061F
+ Punycode: egbpdaj6bu4bxfgehfvwxn
+
+ (B) Chinese (simplified):
+ u+4ED6 u+4EEC u+4E3A u+4EC0 u+4E48 u+4E0D u+8BF4 u+4E2D u+6587
+ Punycode: ihqwcrb4cv8a8dqg056pqjye
+
+ (C) Chinese (traditional):
+ u+4ED6 u+5011 u+7232 u+4EC0 u+9EBD u+4E0D u+8AAA u+4E2D u+6587
+ Punycode: ihqwctvzc91f659drss3x8bo0yb
+
+ (D) Czech: Pro<ccaron>prost<ecaron>nemluv<iacute><ccaron>esky
+ U+0050 u+0072 u+006F u+010D u+0070 u+0072 u+006F u+0073 u+0074
+ u+011B u+006E u+0065 u+006D u+006C u+0075 u+0076 u+00ED u+010D
+ u+0065 u+0073 u+006B u+0079
+ Punycode: Proprostnemluvesky-uyb24dma41a
+
+
+
+
+
+
+Costello Standards Track [Page 14]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ (E) Hebrew:
+ u+05DC u+05DE u+05D4 u+05D4 u+05DD u+05E4 u+05E9 u+05D5 u+05D8
+ u+05DC u+05D0 u+05DE u+05D3 u+05D1 u+05E8 u+05D9 u+05DD u+05E2
+ u+05D1 u+05E8 u+05D9 u+05EA
+ Punycode: 4dbcagdahymbxekheh6e0a7fei0b
+
+ (F) Hindi (Devanagari):
+ u+092F u+0939 u+0932 u+094B u+0917 u+0939 u+093F u+0928 u+094D
+ u+0926 u+0940 u+0915 u+094D u+092F u+094B u+0902 u+0928 u+0939
+ u+0940 u+0902 u+092C u+094B u+0932 u+0938 u+0915 u+0924 u+0947
+ u+0939 u+0948 u+0902
+ Punycode: i1baa7eci9glrd9b2ae1bj0hfcgg6iyaf8o0a1dig0cd
+
+ (G) Japanese (kanji and hiragana):
+ u+306A u+305C u+307F u+3093 u+306A u+65E5 u+672C u+8A9E u+3092
+ u+8A71 u+3057 u+3066 u+304F u+308C u+306A u+3044 u+306E u+304B
+ Punycode: n8jok5ay5dzabd5bym9f0cm5685rrjetr6pdxa
+
+ (H) Korean (Hangul syllables):
+ u+C138 u+ACC4 u+C758 u+BAA8 u+B4E0 u+C0AC u+B78C u+B4E4 u+C774
+ u+D55C u+AD6D u+C5B4 u+B97C u+C774 u+D574 u+D55C u+B2E4 u+BA74
+ u+C5BC u+B9C8 u+B098 u+C88B u+C744 u+AE4C
+ Punycode: 989aomsvi5e83db1d2a355cv1e0vak1dwrv93d5xbh15a0dt30a5j\
+ psd879ccm6fea98c
+
+ (I) Russian (Cyrillic):
+ U+043F u+043E u+0447 u+0435 u+043C u+0443 u+0436 u+0435 u+043E
+ u+043D u+0438 u+043D u+0435 u+0433 u+043E u+0432 u+043E u+0440
+ u+044F u+0442 u+043F u+043E u+0440 u+0443 u+0441 u+0441 u+043A
+ u+0438
+ Punycode: b1abfaaepdrnnbgefbaDotcwatmq2g4l
+
+ (J) Spanish: Porqu<eacute>nopuedensimplementehablarenEspa<ntilde>ol
+ U+0050 u+006F u+0072 u+0071 u+0075 u+00E9 u+006E u+006F u+0070
+ u+0075 u+0065 u+0064 u+0065 u+006E u+0073 u+0069 u+006D u+0070
+ u+006C u+0065 u+006D u+0065 u+006E u+0074 u+0065 u+0068 u+0061
+ u+0062 u+006C u+0061 u+0072 u+0065 u+006E U+0045 u+0073 u+0070
+ u+0061 u+00F1 u+006F u+006C
+ Punycode: PorqunopuedensimplementehablarenEspaol-fmd56a
+
+ (K) Vietnamese:
+ T<adotbelow>isaoh<odotbelow>kh<ocirc>ngth<ecirchookabove>ch\
+ <ihookabove>n<oacute>iti<ecircacute>ngVi<ecircdotbelow>t
+ U+0054 u+1EA1 u+0069 u+0073 u+0061 u+006F u+0068 u+1ECD u+006B
+ u+0068 u+00F4 u+006E u+0067 u+0074 u+0068 u+1EC3 u+0063 u+0068
+ u+1EC9 u+006E u+00F3 u+0069 u+0074 u+0069 u+1EBF u+006E u+0067
+ U+0056 u+0069 u+1EC7 u+0074
+ Punycode: TisaohkhngthchnitingVit-kjcr8268qyxafd2f1b9g
+
+
+
+Costello Standards Track [Page 15]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ The next several examples are all names of Japanese music artists,
+ song titles, and TV programs, just because the author happens to have
+ them handy (but Japanese is useful for providing examples of single-
+ row text, two-row text, ideographic text, and various mixtures
+ thereof).
+
+ (L) 3<nen>B<gumi><kinpachi><sensei>
+ u+0033 u+5E74 U+0042 u+7D44 u+91D1 u+516B u+5148 u+751F
+ Punycode: 3B-ww4c5e180e575a65lsy2b
+
+ (M) <amuro><namie>-with-SUPER-MONKEYS
+ u+5B89 u+5BA4 u+5948 u+7F8E u+6075 u+002D u+0077 u+0069 u+0074
+ u+0068 u+002D U+0053 U+0055 U+0050 U+0045 U+0052 u+002D U+004D
+ U+004F U+004E U+004B U+0045 U+0059 U+0053
+ Punycode: -with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n
+
+ (N) Hello-Another-Way-<sorezore><no><basho>
+ U+0048 u+0065 u+006C u+006C u+006F u+002D U+0041 u+006E u+006F
+ u+0074 u+0068 u+0065 u+0072 u+002D U+0057 u+0061 u+0079 u+002D
+ u+305D u+308C u+305E u+308C u+306E u+5834 u+6240
+ Punycode: Hello-Another-Way--fc4qua05auwb3674vfr0b
+
+ (O) <hitotsu><yane><no><shita>2
+ u+3072 u+3068 u+3064 u+5C4B u+6839 u+306E u+4E0B u+0032
+ Punycode: 2-u9tlzr9756bt3uc0v
+
+ (P) Maji<de>Koi<suru>5<byou><mae>
+ U+004D u+0061 u+006A u+0069 u+3067 U+004B u+006F u+0069 u+3059
+ u+308B u+0035 u+79D2 u+524D
+ Punycode: MajiKoi5-783gue6qz075azm5e
+
+ (Q) <pafii>de<runba>
+ u+30D1 u+30D5 u+30A3 u+30FC u+0064 u+0065 u+30EB u+30F3 u+30D0
+ Punycode: de-jg4avhby1noc0d
+
+ (R) <sono><supiido><de>
+ u+305D u+306E u+30B9 u+30D4 u+30FC u+30C9 u+3067
+ Punycode: d9juau41awczczp
+
+ The last example is an ASCII string that breaks the existing rules
+ for host name labels. (It is not a realistic example for IDNA,
+ because IDNA never encodes pure ASCII labels.)
+
+ (S) -> $1.00 <-
+ u+002D u+003E u+0020 u+0024 u+0031 u+002E u+0030 u+0030 u+0020
+ u+003C u+002D
+ Punycode: -> $1.00 <--
+
+
+
+
+Costello Standards Track [Page 16]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+7.2 Decoding traces
+
+ In the following traces, the evolving state of the decoder is shown
+ as a sequence of hexadecimal values, representing the code points in
+ the extended string. An asterisk appears just after the most
+ recently inserted code point, indicating both n (the value preceeding
+ the asterisk) and i (the position of the value just after the
+ asterisk). Other numerical values are decimal.
+
+ Decoding trace of example B from section 7.1:
+
+ n is 128, i is 0, bias is 72
+ input is "ihqwcrb4cv8a8dqg056pqjye"
+ there is no delimiter, so extended string starts empty
+ delta "ihq" decodes to 19853
+ bias becomes 21
+ 4E0D *
+ delta "wc" decodes to 64
+ bias becomes 20
+ 4E0D 4E2D *
+ delta "rb" decodes to 37
+ bias becomes 13
+ 4E3A * 4E0D 4E2D
+ delta "4c" decodes to 56
+ bias becomes 17
+ 4E3A 4E48 * 4E0D 4E2D
+ delta "v8a" decodes to 599
+ bias becomes 32
+ 4E3A 4EC0 * 4E48 4E0D 4E2D
+ delta "8d" decodes to 130
+ bias becomes 23
+ 4ED6 * 4E3A 4EC0 4E48 4E0D 4E2D
+ delta "qg" decodes to 154
+ bias becomes 25
+ 4ED6 4EEC * 4E3A 4EC0 4E48 4E0D 4E2D
+ delta "056p" decodes to 46301
+ bias becomes 84
+ 4ED6 4EEC 4E3A 4EC0 4E48 4E0D 4E2D 6587 *
+ delta "qjye" decodes to 88531
+ bias becomes 90
+ 4ED6 4EEC 4E3A 4EC0 4E48 4E0D 8BF4 * 4E2D 6587
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 17]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ Decoding trace of example L from section 7.1:
+
+ n is 128, i is 0, bias is 72
+ input is "3B-ww4c5e180e575a65lsy2b"
+ literal portion is "3B-", so extended string starts as:
+ 0033 0042
+ delta "ww4c" decodes to 62042
+ bias becomes 27
+ 0033 0042 5148 *
+ delta "5e" decodes to 139
+ bias becomes 24
+ 0033 0042 516B * 5148
+ delta "180e" decodes to 16683
+ bias becomes 67
+ 0033 5E74 * 0042 516B 5148
+ delta "575a" decodes to 34821
+ bias becomes 82
+ 0033 5E74 0042 516B 5148 751F *
+ delta "65l" decodes to 14592
+ bias becomes 67
+ 0033 5E74 0042 7D44 * 516B 5148 751F
+ delta "sy2b" decodes to 42088
+ bias becomes 84
+ 0033 5E74 0042 7D44 91D1 * 516B 5148 751F
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 18]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+7.3 Encoding traces
+
+ In the following traces, code point values are hexadecimal, while
+ other numerical values are decimal.
+
+ Encoding trace of example B from section 7.1:
+
+ bias is 72
+ input is:
+ 4ED6 4EEC 4E3A 4EC0 4E48 4E0D 8BF4 4E2D 6587
+ there are no basic code points, so no literal portion
+ next code point to insert is 4E0D
+ needed delta is 19853, encodes as "ihq"
+ bias becomes 21
+ next code point to insert is 4E2D
+ needed delta is 64, encodes as "wc"
+ bias becomes 20
+ next code point to insert is 4E3A
+ needed delta is 37, encodes as "rb"
+ bias becomes 13
+ next code point to insert is 4E48
+ needed delta is 56, encodes as "4c"
+ bias becomes 17
+ next code point to insert is 4EC0
+ needed delta is 599, encodes as "v8a"
+ bias becomes 32
+ next code point to insert is 4ED6
+ needed delta is 130, encodes as "8d"
+ bias becomes 23
+ next code point to insert is 4EEC
+ needed delta is 154, encodes as "qg"
+ bias becomes 25
+ next code point to insert is 6587
+ needed delta is 46301, encodes as "056p"
+ bias becomes 84
+ next code point to insert is 8BF4
+ needed delta is 88531, encodes as "qjye"
+ bias becomes 90
+ output is "ihqwcrb4cv8a8dqg056pqjye"
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 19]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ Encoding trace of example L from section 7.1:
+
+ bias is 72
+ input is:
+ 0033 5E74 0042 7D44 91D1 516B 5148 751F
+ basic code points (0033, 0042) are copied to literal portion: "3B-"
+ next code point to insert is 5148
+ needed delta is 62042, encodes as "ww4c"
+ bias becomes 27
+ next code point to insert is 516B
+ needed delta is 139, encodes as "5e"
+ bias becomes 24
+ next code point to insert is 5E74
+ needed delta is 16683, encodes as "180e"
+ bias becomes 67
+ next code point to insert is 751F
+ needed delta is 34821, encodes as "575a"
+ bias becomes 82
+ next code point to insert is 7D44
+ needed delta is 14592, encodes as "65l"
+ bias becomes 67
+ next code point to insert is 91D1
+ needed delta is 42088, encodes as "sy2b"
+ bias becomes 84
+ output is "3B-ww4c5e180e575a65lsy2b"
+
+8. Security Considerations
+
+ Users expect each domain name in DNS to be controlled by a single
+ authority. If a Unicode string intended for use as a domain label
+ could map to multiple ACE labels, then an internationalized domain
+ name could map to multiple ASCII domain names, each controlled by a
+ different authority, some of which could be spoofs that hijack
+ service requests intended for another. Therefore Punycode is
+ designed so that each Unicode string has a unique encoding.
+
+ However, there can still be multiple Unicode representations of the
+ "same" text, for various definitions of "same". This problem is
+ addressed to some extent by the Unicode standard under the topic of
+ canonicalization, and this work is leveraged for domain names by
+ Nameprep [NAMEPREP].
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 20]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+9. References
+
+9.1 Normative References
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119, March 1997.
+
+9.2 Informative References
+
+ [RFC952] Harrenstien, K., Stahl, M. and E. Feinler, "DOD Internet
+ Host Table Specification", RFC 952, October 1985.
+
+ [RFC1034] Mockapetris, P., "Domain Names - Concepts and
+ Facilities", STD 13, RFC 1034, November 1987.
+
+ [IDNA] Faltstrom, P., Hoffman, P. and A. Costello,
+ "Internationalizing Domain Names in Applications
+ (IDNA)", RFC 3490, March 2003.
+
+ [NAMEPREP] Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep
+ Profile for Internationalized Domain Names (IDN)", RFC
+ 3491, March 2003.
+
+ [ASCII] Cerf, V., "ASCII format for Network Interchange", RFC
+ 20, October 1969.
+
+ [PROVINCIAL] Kaplan, M., "The 'anyone can be provincial!' page",
+ http://www.trigeminal.com/samples/provincial.html.
+
+ [UNICODE] The Unicode Consortium, "The Unicode Standard",
+ http://www.unicode.org/unicode/standard/standard.html.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 21]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+A. Mixed-case annotation
+
+ In order to use Punycode to represent case-insensitive strings,
+ higher layers need to case-fold the strings prior to Punycode
+ encoding. The encoded string can use mixed case as an annotation
+ telling how to convert the folded string into a mixed-case string for
+ display purposes. Note, however, that mixed-case annotation is not
+ used by the ToASCII and ToUnicode operations specified in [IDNA], and
+ therefore implementors of IDNA can disregard this appendix.
+
+ Basic code points can use mixed case directly, because the decoder
+ copies them verbatim, leaving lowercase code points lowercase, and
+ leaving uppercase code points uppercase. Each non-basic code point
+ is represented by a delta, which is represented by a sequence of
+ basic code points, the last of which provides the annotation. If it
+ is uppercase, it is a suggestion to map the non-basic code point to
+ uppercase (if possible); if it is lowercase, it is a suggestion to
+ map the non-basic code point to lowercase (if possible).
+
+ These annotations do not alter the code points returned by decoders;
+ the annotations are returned separately, for the caller to use or
+ ignore. Encoders can accept annotations in addition to code points,
+ but the annotations do not alter the output, except to influence the
+ uppercase/lowercase form of ASCII letters.
+
+ Punycode encoders and decoders need not support these annotations,
+ and higher layers need not use them.
+
+B. Disclaimer and license
+
+ Regarding this entire document or any portion of it (including the
+ pseudocode and C code), the author makes no guarantees and is not
+ responsible for any damage resulting from its use. The author grants
+ irrevocable permission to anyone to use, modify, and distribute it in
+ any way that does not diminish the rights of anyone else to use,
+ modify, and distribute it, provided that redistributed derivative
+ works do not contain misleading author or version information.
+ Derivative works need not be licensed under similar terms.
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 22]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+C. Punycode sample implementation
+
+/*
+punycode.c from RFC 3492
+http://www.nicemice.net/idn/
+Adam M. Costello
+http://www.nicemice.net/amc/
+
+This is ANSI C code (C89) implementing Punycode (RFC 3492).
+
+*/
+
+
+/************************************************************/
+/* Public interface (would normally go in its own .h file): */
+
+#include <limits.h>
+
+enum punycode_status {
+ punycode_success,
+ punycode_bad_input, /* Input is invalid. */
+ punycode_big_output, /* Output would exceed the space provided. */
+ punycode_overflow /* Input needs wider integers to process. */
+};
+
+#if UINT_MAX >= (1 << 26) - 1
+typedef unsigned int punycode_uint;
+#else
+typedef unsigned long punycode_uint;
+#endif
+
+enum punycode_status punycode_encode(
+ punycode_uint input_length,
+ const punycode_uint input[],
+ const unsigned char case_flags[],
+ punycode_uint *output_length,
+ char output[] );
+
+ /* punycode_encode() converts Unicode to Punycode. The input */
+ /* is represented as an array of Unicode code points (not code */
+ /* units; surrogate pairs are not allowed), and the output */
+ /* will be represented as an array of ASCII code points. The */
+ /* output string is *not* null-terminated; it will contain */
+ /* zeros if and only if the input contains zeros. (Of course */
+ /* the caller can leave room for a terminator and add one if */
+ /* needed.) The input_length is the number of code points in */
+ /* the input. The output_length is an in/out argument: the */
+ /* caller passes in the maximum number of code points that it */
+
+
+
+Costello Standards Track [Page 23]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ /* can receive, and on successful return it will contain the */
+ /* number of code points actually output. The case_flags array */
+ /* holds input_length boolean values, where nonzero suggests that */
+ /* the corresponding Unicode character be forced to uppercase */
+ /* after being decoded (if possible), and zero suggests that */
+ /* it be forced to lowercase (if possible). ASCII code points */
+ /* are encoded literally, except that ASCII letters are forced */
+ /* to uppercase or lowercase according to the corresponding */
+ /* uppercase flags. If case_flags is a null pointer then ASCII */
+ /* letters are left as they are, and other code points are */
+ /* treated as if their uppercase flags were zero. The return */
+ /* value can be any of the punycode_status values defined above */
+ /* except punycode_bad_input; if not punycode_success, then */
+ /* output_size and output might contain garbage. */
+
+enum punycode_status punycode_decode(
+ punycode_uint input_length,
+ const char input[],
+ punycode_uint *output_length,
+ punycode_uint output[],
+ unsigned char case_flags[] );
+
+ /* punycode_decode() converts Punycode to Unicode. The input is */
+ /* represented as an array of ASCII code points, and the output */
+ /* will be represented as an array of Unicode code points. The */
+ /* input_length is the number of code points in the input. The */
+ /* output_length is an in/out argument: the caller passes in */
+ /* the maximum number of code points that it can receive, and */
+ /* on successful return it will contain the actual number of */
+ /* code points output. The case_flags array needs room for at */
+ /* least output_length values, or it can be a null pointer if the */
+ /* case information is not needed. A nonzero flag suggests that */
+ /* the corresponding Unicode character be forced to uppercase */
+ /* by the caller (if possible), while zero suggests that it be */
+ /* forced to lowercase (if possible). ASCII code points are */
+ /* output already in the proper case, but their flags will be set */
+ /* appropriately so that applying the flags would be harmless. */
+ /* The return value can be any of the punycode_status values */
+ /* defined above; if not punycode_success, then output_length, */
+ /* output, and case_flags might contain garbage. On success, the */
+ /* decoder will never need to write an output_length greater than */
+ /* input_length, because of how the encoding is defined. */
+
+/**********************************************************/
+/* Implementation (would normally go in its own .c file): */
+
+#include <string.h>
+
+
+
+
+Costello Standards Track [Page 24]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+/*** Bootstring parameters for Punycode ***/
+
+enum { base = 36, tmin = 1, tmax = 26, skew = 38, damp = 700,
+ initial_bias = 72, initial_n = 0x80, delimiter = 0x2D };
+
+/* basic(cp) tests whether cp is a basic code point: */
+#define basic(cp) ((punycode_uint)(cp) < 0x80)
+
+/* delim(cp) tests whether cp is a delimiter: */
+#define delim(cp) ((cp) == delimiter)
+
+/* decode_digit(cp) returns the numeric value of a basic code */
+/* point (for use in representing integers) in the range 0 to */
+/* base-1, or base if cp is does not represent a value. */
+
+static punycode_uint decode_digit(punycode_uint cp)
+{
+ return cp - 48 < 10 ? cp - 22 : cp - 65 < 26 ? cp - 65 :
+ cp - 97 < 26 ? cp - 97 : base;
+}
+
+/* encode_digit(d,flag) returns the basic code point whose value */
+/* (when used for representing integers) is d, which needs to be in */
+/* the range 0 to base-1. The lowercase form is used unless flag is */
+/* nonzero, in which case the uppercase form is used. The behavior */
+/* is undefined if flag is nonzero and digit d has no uppercase form. */
+
+static char encode_digit(punycode_uint d, int flag)
+{
+ return d + 22 + 75 * (d < 26) - ((flag != 0) << 5);
+ /* 0..25 map to ASCII a..z or A..Z */
+ /* 26..35 map to ASCII 0..9 */
+}
+
+/* flagged(bcp) tests whether a basic code point is flagged */
+/* (uppercase). The behavior is undefined if bcp is not a */
+/* basic code point. */
+
+#define flagged(bcp) ((punycode_uint)(bcp) - 65 < 26)
+
+/* encode_basic(bcp,flag) forces a basic code point to lowercase */
+/* if flag is zero, uppercase if flag is nonzero, and returns */
+/* the resulting code point. The code point is unchanged if it */
+/* is caseless. The behavior is undefined if bcp is not a basic */
+/* code point. */
+
+static char encode_basic(punycode_uint bcp, int flag)
+{
+
+
+
+Costello Standards Track [Page 25]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ bcp -= (bcp - 97 < 26) << 5;
+ return bcp + ((!flag && (bcp - 65 < 26)) << 5);
+}
+
+/*** Platform-specific constants ***/
+
+/* maxint is the maximum value of a punycode_uint variable: */
+static const punycode_uint maxint = -1;
+/* Because maxint is unsigned, -1 becomes the maximum value. */
+
+/*** Bias adaptation function ***/
+
+static punycode_uint adapt(
+ punycode_uint delta, punycode_uint numpoints, int firsttime )
+{
+ punycode_uint k;
+
+ delta = firsttime ? delta / damp : delta >> 1;
+ /* delta >> 1 is a faster way of doing delta / 2 */
+ delta += delta / numpoints;
+
+ for (k = 0; delta > ((base - tmin) * tmax) / 2; k += base) {
+ delta /= base - tmin;
+ }
+
+ return k + (base - tmin + 1) * delta / (delta + skew);
+}
+
+/*** Main encode function ***/
+
+enum punycode_status punycode_encode(
+ punycode_uint input_length,
+ const punycode_uint input[],
+ const unsigned char case_flags[],
+ punycode_uint *output_length,
+ char output[] )
+{
+ punycode_uint n, delta, h, b, out, max_out, bias, j, m, q, k, t;
+
+ /* Initialize the state: */
+
+ n = initial_n;
+ delta = out = 0;
+ max_out = *output_length;
+ bias = initial_bias;
+
+ /* Handle the basic code points: */
+
+
+
+
+Costello Standards Track [Page 26]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ for (j = 0; j < input_length; ++j) {
+ if (basic(input[j])) {
+ if (max_out - out < 2) return punycode_big_output;
+ output[out++] =
+ case_flags ? encode_basic(input[j], case_flags[j]) : input[j];
+ }
+ /* else if (input[j] < n) return punycode_bad_input; */
+ /* (not needed for Punycode with unsigned code points) */
+ }
+
+ h = b = out;
+
+ /* h is the number of code points that have been handled, b is the */
+ /* number of basic code points, and out is the number of characters */
+ /* that have been output. */
+
+ if (b > 0) output[out++] = delimiter;
+
+ /* Main encoding loop: */
+
+ while (h < input_length) {
+ /* All non-basic code points < n have been */
+ /* handled already. Find the next larger one: */
+
+ for (m = maxint, j = 0; j < input_length; ++j) {
+ /* if (basic(input[j])) continue; */
+ /* (not needed for Punycode) */
+ if (input[j] >= n && input[j] < m) m = input[j];
+ }
+
+ /* Increase delta enough to advance the decoder's */
+ /* <n,i> state to <m,0>, but guard against overflow: */
+
+ if (m - n > (maxint - delta) / (h + 1)) return punycode_overflow;
+ delta += (m - n) * (h + 1);
+ n = m;
+
+ for (j = 0; j < input_length; ++j) {
+ /* Punycode does not need to check whether input[j] is basic: */
+ if (input[j] < n /* || basic(input[j]) */ ) {
+ if (++delta == 0) return punycode_overflow;
+ }
+
+ if (input[j] == n) {
+ /* Represent delta as a generalized variable-length integer: */
+
+ for (q = delta, k = base; ; k += base) {
+ if (out >= max_out) return punycode_big_output;
+
+
+
+Costello Standards Track [Page 27]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ t = k <= bias /* + tmin */ ? tmin : /* +tmin not needed */
+ k >= bias + tmax ? tmax : k - bias;
+ if (q < t) break;
+ output[out++] = encode_digit(t + (q - t) % (base - t), 0);
+ q = (q - t) / (base - t);
+ }
+
+ output[out++] = encode_digit(q, case_flags && case_flags[j]);
+ bias = adapt(delta, h + 1, h == b);
+ delta = 0;
+ ++h;
+ }
+ }
+
+ ++delta, ++n;
+ }
+
+ *output_length = out;
+ return punycode_success;
+}
+
+/*** Main decode function ***/
+
+enum punycode_status punycode_decode(
+ punycode_uint input_length,
+ const char input[],
+ punycode_uint *output_length,
+ punycode_uint output[],
+ unsigned char case_flags[] )
+{
+ punycode_uint n, out, i, max_out, bias,
+ b, j, in, oldi, w, k, digit, t;
+
+ /* Initialize the state: */
+
+ n = initial_n;
+ out = i = 0;
+ max_out = *output_length;
+ bias = initial_bias;
+
+ /* Handle the basic code points: Let b be the number of input code */
+ /* points before the last delimiter, or 0 if there is none, then */
+ /* copy the first b code points to the output. */
+
+ for (b = j = 0; j < input_length; ++j) if (delim(input[j])) b = j;
+ if (b > max_out) return punycode_big_output;
+
+ for (j = 0; j < b; ++j) {
+
+
+
+Costello Standards Track [Page 28]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ if (case_flags) case_flags[out] = flagged(input[j]);
+ if (!basic(input[j])) return punycode_bad_input;
+ output[out++] = input[j];
+ }
+
+ /* Main decoding loop: Start just after the last delimiter if any */
+ /* basic code points were copied; start at the beginning otherwise. */
+
+ for (in = b > 0 ? b + 1 : 0; in < input_length; ++out) {
+
+ /* in is the index of the next character to be consumed, and */
+ /* out is the number of code points in the output array. */
+
+ /* Decode a generalized variable-length integer into delta, */
+ /* which gets added to i. The overflow checking is easier */
+ /* if we increase i as we go, then subtract off its starting */
+ /* value at the end to obtain delta. */
+
+ for (oldi = i, w = 1, k = base; ; k += base) {
+ if (in >= input_length) return punycode_bad_input;
+ digit = decode_digit(input[in++]);
+ if (digit >= base) return punycode_bad_input;
+ if (digit > (maxint - i) / w) return punycode_overflow;
+ i += digit * w;
+ t = k <= bias /* + tmin */ ? tmin : /* +tmin not needed */
+ k >= bias + tmax ? tmax : k - bias;
+ if (digit < t) break;
+ if (w > maxint / (base - t)) return punycode_overflow;
+ w *= (base - t);
+ }
+
+ bias = adapt(i - oldi, out + 1, oldi == 0);
+
+ /* i was supposed to wrap around from out+1 to 0, */
+ /* incrementing n each time, so we'll fix that now: */
+
+ if (i / (out + 1) > maxint - n) return punycode_overflow;
+ n += i / (out + 1);
+ i %= (out + 1);
+
+ /* Insert n at position i of the output: */
+
+ /* not needed for Punycode: */
+ /* if (decode_digit(n) <= base) return punycode_invalid_input; */
+ if (out >= max_out) return punycode_big_output;
+
+ if (case_flags) {
+ memmove(case_flags + i + 1, case_flags + i, out - i);
+
+
+
+Costello Standards Track [Page 29]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ /* Case of last character determines uppercase flag: */
+ case_flags[i] = flagged(input[in - 1]);
+ }
+
+ memmove(output + i + 1, output + i, (out - i) * sizeof *output);
+ output[i++] = n;
+ }
+
+ *output_length = out;
+ return punycode_success;
+}
+
+/******************************************************************/
+/* Wrapper for testing (would normally go in a separate .c file): */
+
+#include <assert.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+/* For testing, we'll just set some compile-time limits rather than */
+/* use malloc(), and set a compile-time option rather than using a */
+/* command-line option. */
+
+enum {
+ unicode_max_length = 256,
+ ace_max_length = 256
+};
+
+static void usage(char **argv)
+{
+ fprintf(stderr,
+ "\n"
+ "%s -e reads code points and writes a Punycode string.\n"
+ "%s -d reads a Punycode string and writes code points.\n"
+ "\n"
+ "Input and output are plain text in the native character set.\n"
+ "Code points are in the form u+hex separated by whitespace.\n"
+ "Although the specification allows Punycode strings to contain\n"
+ "any characters from the ASCII repertoire, this test code\n"
+ "supports only the printable characters, and needs the Punycode\n"
+ "string to be followed by a newline.\n"
+ "The case of the u in u+hex is the force-to-uppercase flag.\n"
+ , argv[0], argv[0]);
+ exit(EXIT_FAILURE);
+}
+
+static void fail(const char *msg)
+
+
+
+Costello Standards Track [Page 30]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+{
+ fputs(msg,stderr);
+ exit(EXIT_FAILURE);
+}
+
+static const char too_big[] =
+ "input or output is too large, recompile with larger limits\n";
+static const char invalid_input[] = "invalid input\n";
+static const char overflow[] = "arithmetic overflow\n";
+static const char io_error[] = "I/O error\n";
+
+/* The following string is used to convert printable */
+/* characters between ASCII and the native charset: */
+
+static const char print_ascii[] =
+ "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
+ "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
+ " !\"#$%&'()*+,-./"
+ "0123456789:;<=>?"
+ "@ABCDEFGHIJKLMNO"
+ "PQRSTUVWXYZ[\\]^_"
+ "`abcdefghijklmno"
+ "pqrstuvwxyz{|}~\n";
+
+int main(int argc, char **argv)
+{
+ enum punycode_status status;
+ int r;
+ unsigned int input_length, output_length, j;
+ unsigned char case_flags[unicode_max_length];
+
+ if (argc != 2) usage(argv);
+ if (argv[1][0] != '-') usage(argv);
+ if (argv[1][2] != 0) usage(argv);
+
+ if (argv[1][1] == 'e') {
+ punycode_uint input[unicode_max_length];
+ unsigned long codept;
+ char output[ace_max_length+1], uplus[3];
+ int c;
+
+ /* Read the input code points: */
+
+ input_length = 0;
+
+ for (;;) {
+ r = scanf("%2s%lx", uplus, &codept);
+ if (ferror(stdin)) fail(io_error);
+
+
+
+Costello Standards Track [Page 31]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ if (r == EOF || r == 0) break;
+
+ if (r != 2 || uplus[1] != '+' || codept > (punycode_uint)-1) {
+ fail(invalid_input);
+ }
+
+ if (input_length == unicode_max_length) fail(too_big);
+
+ if (uplus[0] == 'u') case_flags[input_length] = 0;
+ else if (uplus[0] == 'U') case_flags[input_length] = 1;
+ else fail(invalid_input);
+
+ input[input_length++] = codept;
+ }
+
+ /* Encode: */
+
+ output_length = ace_max_length;
+ status = punycode_encode(input_length, input, case_flags,
+ &output_length, output);
+ if (status == punycode_bad_input) fail(invalid_input);
+ if (status == punycode_big_output) fail(too_big);
+ if (status == punycode_overflow) fail(overflow);
+ assert(status == punycode_success);
+
+ /* Convert to native charset and output: */
+
+ for (j = 0; j < output_length; ++j) {
+ c = output[j];
+ assert(c >= 0 && c <= 127);
+ if (print_ascii[c] == 0) fail(invalid_input);
+ output[j] = print_ascii[c];
+ }
+
+ output[j] = 0;
+ r = puts(output);
+ if (r == EOF) fail(io_error);
+ return EXIT_SUCCESS;
+ }
+
+ if (argv[1][1] == 'd') {
+ char input[ace_max_length+2], *p, *pp;
+ punycode_uint output[unicode_max_length];
+
+ /* Read the Punycode input string and convert to ASCII: */
+
+ fgets(input, ace_max_length+2, stdin);
+ if (ferror(stdin)) fail(io_error);
+
+
+
+Costello Standards Track [Page 32]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+ if (feof(stdin)) fail(invalid_input);
+ input_length = strlen(input) - 1;
+ if (input[input_length] != '\n') fail(too_big);
+ input[input_length] = 0;
+
+ for (p = input; *p != 0; ++p) {
+ pp = strchr(print_ascii, *p);
+ if (pp == 0) fail(invalid_input);
+ *p = pp - print_ascii;
+ }
+
+ /* Decode: */
+
+ output_length = unicode_max_length;
+ status = punycode_decode(input_length, input, &output_length,
+ output, case_flags);
+ if (status == punycode_bad_input) fail(invalid_input);
+ if (status == punycode_big_output) fail(too_big);
+ if (status == punycode_overflow) fail(overflow);
+ assert(status == punycode_success);
+
+ /* Output the result: */
+
+ for (j = 0; j < output_length; ++j) {
+ r = printf("%s+%04lX\n",
+ case_flags[j] ? "U" : "u",
+ (unsigned long) output[j] );
+ if (r < 0) fail(io_error);
+ }
+
+ return EXIT_SUCCESS;
+ }
+
+ usage(argv);
+ return EXIT_SUCCESS; /* not reached, but quiets compiler warning */
+}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 33]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+Author's Address
+
+ Adam M. Costello
+ University of California, Berkeley
+ http://www.nicemice.net/amc/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 34]
+\f
+RFC 3492 IDNA Punycode March 2003
+
+
+Full Copyright Statement
+
+ Copyright (C) The Internet Society (2003). All Rights Reserved.
+
+ This document and translations of it may be copied and furnished to
+ others, and derivative works that comment on or otherwise explain it
+ or assist in its implementation may be prepared, copied, published
+ and distributed, in whole or in part, without restriction of any
+ kind, provided that the above copyright notice and this paragraph are
+ included on all such copies and derivative works. However, this
+ document itself may not be modified in any way, such as by removing
+ the copyright notice or references to the Internet Society or other
+ Internet organizations, except as needed for the purpose of
+ developing Internet standards in which case the procedures for
+ copyrights defined in the Internet Standards process must be
+ followed, or as required to translate it into languages other than
+ English.
+
+ The limited permissions granted above are perpetual and will not be
+ revoked by the Internet Society or its successors or assigns.
+
+ This document and the information contained herein is provided on an
+ "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
+ TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
+ BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
+ HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
+ MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+Acknowledgement
+
+ Funding for the RFC Editor function is currently provided by the
+ Internet Society.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Costello Standards Track [Page 35]
+\f
projects / thirdparty / bind9.git / commitdiff
