From: Christian Heimes Date: Tue, 26 Mar 2013 16:47:23 +0000 (+0100) Subject: Issue 17538: Document XML vulnerabilties X-Git-Tag: v3.4.0a1~1083^2 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=768f6a53601a6c4e0b914aaedb977dd2ca97532a;p=thirdparty%2FPython%2Fcpython.git Issue 17538: Document XML vulnerabilties --- 768f6a53601a6c4e0b914aaedb977dd2ca97532a diff --cc Doc/library/xml.etree.elementtree.rst index 144e344397ab,dc9ebb999781..e429f04d7274 --- a/Doc/library/xml.etree.elementtree.rst +++ b/Doc/library/xml.etree.elementtree.rst @@@ -5,318 -5,52 +5,325 @@@ :synopsis: Implementation of the ElementTree API. .. moduleauthor:: Fredrik Lundh -**Source code:** :source:`Lib/xml/etree/ElementTree.py` +The :mod:`xml.etree.ElementTree` module implements a simple and efficient API +for parsing and creating XML data. --------------- - -The :class:`Element` type is a flexible container object, designed to store -hierarchical data structures in memory. The type can be described as a cross -between a list and a dictionary. +.. versionchanged:: 3.3 + This module will use a fast implementation whenever available. + The :mod:`xml.etree.cElementTree` module is deprecated. + + .. warning:: + + The :mod:`xml.etree.ElementTree` module is not secure against + maliciously constructed data. If you need to parse untrusted or + unauthenticated data see :ref:`xml-vulnerabilities`. + +Tutorial +-------- + +This is a short tutorial for using :mod:`xml.etree.ElementTree` (``ET`` in +short). The goal is to demonstrate some of the building blocks and basic +concepts of the module. + +XML tree and elements +^^^^^^^^^^^^^^^^^^^^^ + +XML is an inherently hierarchical data format, and the most natural way to +represent it is with a tree. ``ET`` has two classes for this purpose - +:class:`ElementTree` represents the whole XML document as a tree, and +:class:`Element` represents a single node in this tree. Interactions with +the whole document (reading and writing to/from files) are usually done +on the :class:`ElementTree` level. Interactions with a single XML element +and its sub-elements are done on the :class:`Element` level. + +.. _elementtree-parsing-xml: + +Parsing XML +^^^^^^^^^^^ + +We'll be using the following XML document as the sample data for this section: + +.. code-block:: xml + + + + + 1 + 2008 + 141100 + + + + + 4 + 2011 + 59900 + + + + 68 + 2011 + 13600 + + + + + +We can import this data by reading from a file:: + + import xml.etree.ElementTree as ET + tree = ET.parse('country_data.xml') + root = tree.getroot() + +Or directly from a string:: + + root = ET.fromstring(country_data_as_string) + +:func:`fromstring` parses XML from a string directly into an :class:`Element`, +which is the root element of the parsed tree. Other parsing functions may +create an :class:`ElementTree`. Check the documentation to be sure. + +As an :class:`Element`, ``root`` has a tag and a dictionary of attributes:: + + >>> root.tag + 'data' + >>> root.attrib + {} + +It also has children nodes over which we can iterate:: + + >>> for child in root: + ... print(child.tag, child.attrib) + ... + country {'name': 'Liechtenstein'} + country {'name': 'Singapore'} + country {'name': 'Panama'} + +Children are nested, and we can access specific child nodes by index:: + + >>> root[0][1].text + '2008' + +Finding interesting elements +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:class:`Element` has some useful methods that help iterate recursively over all +the sub-tree below it (its children, their children, and so on). For example, +:meth:`Element.iter`:: + + >>> for neighbor in root.iter('neighbor'): + ... print(neighbor.attrib) + ... + {'name': 'Austria', 'direction': 'E'} + {'name': 'Switzerland', 'direction': 'W'} + {'name': 'Malaysia', 'direction': 'N'} + {'name': 'Costa Rica', 'direction': 'W'} + {'name': 'Colombia', 'direction': 'E'} + +:meth:`Element.findall` finds only elements with a tag which are direct +children of the current element. :meth:`Element.find` finds the *first* child +with a particular tag, and :meth:`Element.text` accesses the element's text +content. :meth:`Element.get` accesses the element's attributes:: + + >>> for country in root.findall('country'): + ... rank = country.find('rank').text + ... name = country.get('name') + ... print(name, rank) + ... + Liechtenstein 1 + Singapore 4 + Panama 68 + +More sophisticated specification of which elements to look for is possible by +using :ref:`XPath `. + +Modifying an XML File +^^^^^^^^^^^^^^^^^^^^^ + +:class:`ElementTree` provides a simple way to build XML documents and write them to files. +The :meth:`ElementTree.write` method serves this purpose. + +Once created, an :class:`Element` object may be manipulated by directly changing +its fields (such as :attr:`Element.text`), adding and modifying attributes +(:meth:`Element.set` method), as well as adding new children (for example +with :meth:`Element.append`). + +Let's say we want to add one to each country's rank, and add an ``updated`` +attribute to the rank element:: + + >>> for rank in root.iter('rank'): + ... new_rank = int(rank.text) + 1 + ... rank.text = str(new_rank) + ... rank.set('updated', 'yes') + ... + >>> tree.write('output.xml') + +Our XML now looks like this: + +.. code-block:: xml + + + + + 2 + 2008 + 141100 + + + + + 5 + 2011 + 59900 + + + + 69 + 2011 + 13600 + + + + + +We can remove elements using :meth:`Element.remove`. Let's say we want to +remove all countries with a rank higher than 50:: + + >>> for country in root.findall('country'): + ... rank = int(country.find('rank').text) + ... if rank > 50: + ... root.remove(country) + ... + >>> tree.write('output.xml') + +Our XML now looks like this: + +.. code-block:: xml + + + + + 2 + 2008 + 141100 + + + + + 5 + 2011 + 59900 + + + + +Building XML documents +^^^^^^^^^^^^^^^^^^^^^^ + +The :func:`SubElement` function also provides a convenient way to create new +sub-elements for a given element:: + + >>> a = ET.Element('a') + >>> b = ET.SubElement(a, 'b') + >>> c = ET.SubElement(a, 'c') + >>> d = ET.SubElement(c, 'd') + >>> ET.dump(a) + + +Additional resources +^^^^^^^^^^^^^^^^^^^^ -Each element has a number of properties associated with it: - -* a tag which is a string identifying what kind of data this element represents - (the element type, in other words). - -* a number of attributes, stored in a Python dictionary. - -* a text string. - -* an optional tail string. - -* a number of child elements, stored in a Python sequence - -To create an element instance, use the :class:`Element` constructor or the -:func:`SubElement` factory function. - -The :class:`ElementTree` class can be used to wrap an element structure, and -convert it from and to XML. +See http://effbot.org/zone/element-index.htm for tutorials and links to other +docs. -A C implementation of this API is available as :mod:`xml.etree.cElementTree`. -See http://effbot.org/zone/element-index.htm for tutorials and links to other -docs. Fredrik Lundh's page is also the location of the development version of -the xml.etree.ElementTree. +.. _elementtree-xpath: -.. versionchanged:: 3.2 - The ElementTree API is updated to 1.3. For more information, see - `Introducing ElementTree 1.3 - `_. +XPath support +------------- +This module provides limited support for +`XPath expressions `_ for locating elements in a +tree. The goal is to support a small subset of the abbreviated syntax; a full +XPath engine is outside the scope of the module. + +Example +^^^^^^^ + +Here's an example that demonstrates some of the XPath capabilities of the +module. We'll be using the ``countrydata`` XML document from the +:ref:`Parsing XML ` section:: + + import xml.etree.ElementTree as ET + + root = ET.fromstring(countrydata) + + # Top-level elements + root.findall(".") + + # All 'neighbor' grand-children of 'country' children of the top-level + # elements + root.findall("./country/neighbor") + + # Nodes with name='Singapore' that have a 'year' child + root.findall(".//year/..[@name='Singapore']") + + # 'year' nodes that are children of nodes with name='Singapore' + root.findall(".//*[@name='Singapore']/year") + + # All 'neighbor' nodes that are the second child of their parent + root.findall(".//neighbor[2]") + +Supported XPath syntax +^^^^^^^^^^^^^^^^^^^^^^ + ++-----------------------+------------------------------------------------------+ +| Syntax | Meaning | ++=======================+======================================================+ +| ``tag`` | Selects all child elements with the given tag. | +| | For example, ``spam`` selects all child elements | +| | named ``spam``, ``spam/egg`` selects all | +| | grandchildren named ``egg`` in all children named | +| | ``spam``. | ++-----------------------+------------------------------------------------------+ +| ``*`` | Selects all child elements. For example, ``*/egg`` | +| | selects all grandchildren named ``egg``. | ++-----------------------+------------------------------------------------------+ +| ``.`` | Selects the current node. This is mostly useful | +| | at the beginning of the path, to indicate that it's | +| | a relative path. | ++-----------------------+------------------------------------------------------+ +| ``//`` | Selects all subelements, on all levels beneath the | +| | current element. For example, ``.//egg`` selects | +| | all ``egg`` elements in the entire tree. | ++-----------------------+------------------------------------------------------+ +| ``..`` | Selects the parent element. Returns ``None`` if the | +| | path attempts to reach the ancestors of the start | +| | element (the element ``find`` was called on). | ++-----------------------+------------------------------------------------------+ +| ``[@attrib]`` | Selects all elements that have the given attribute. | ++-----------------------+------------------------------------------------------+ +| ``[@attrib='value']`` | Selects all elements for which the given attribute | +| | has the given value. The value cannot contain | +| | quotes. | ++-----------------------+------------------------------------------------------+ +| ``[tag]`` | Selects all elements that have a child named | +| | ``tag``. Only immediate children are supported. | ++-----------------------+------------------------------------------------------+ +| ``[position]`` | Selects all elements that are located at the given | +| | position. The position can be either an integer | +| | (1 is the first position), the expression ``last()`` | +| | (for the last position), or a position relative to | +| | the last position (e.g. ``last()-1``). | ++-----------------------+------------------------------------------------------+ + +Predicates (expressions within square brackets) must be preceded by a tag +name, an asterisk, or another predicate. ``position`` predicates must be +preceded by a tag name. + +Reference +--------- .. _elementtree-functions: diff --cc Doc/library/xml.rst index 21b2e2382fa7,f84af58be10d..b86d51a3b920 --- a/Doc/library/xml.rst +++ b/Doc/library/xml.rst @@@ -3,8 -3,20 +3,21 @@@ XML Processing Modules ====================== + .. module:: xml + :synopsis: Package containing XML processing modules + .. sectionauthor:: Christian Heimes + .. sectionauthor:: Georg Brandl + + Python's interfaces for processing XML are grouped in the ``xml`` package. + .. warning:: + + The XML modules are not secure against erroneous or maliciously + constructed data. If you need to parse untrusted or unauthenticated data see + :ref:`xml-vulnerabilities`. + ++ It is important to note that modules in the :mod:`xml` package require that there be at least one SAX-compliant XML parser available. The Expat parser is included with Python, so the :mod:`xml.parsers.expat` module will always be @@@ -27,3 -39,93 +40,94 @@@ The XML handling submodules are * :mod:`xml.sax`: SAX2 base classes and convenience functions * :mod:`xml.parsers.expat`: the Expat parser binding + + + .. _xml-vulnerabilities: + + XML vulnerabilities + =================== + + The XML processing modules are not secure against maliciously constructed data. + An attacker can abuse vulnerabilities for e.g. denial of service attacks, to + access local files, to generate network connections to other machines, or + to or circumvent firewalls. The attacks on XML abuse unfamiliar features + like inline `DTD`_ (document type definition) with entities. + + + ========================= ======== ========= ========= ======== ========= + kind sax etree minidom pulldom xmlrpc + ========================= ======== ========= ========= ======== ========= + billion laughs **True** **True** **True** **True** **True** + quadratic blowup **True** **True** **True** **True** **True** + external entity expansion **True** False (1) False (2) **True** False (3) + DTD retrieval **True** False False **True** False + decompression bomb False False False False **True** + ========================= ======== ========= ========= ======== ========= + + 1. :mod:`xml.etree.ElementTree` doesn't expand external entities and raises a + ParserError when an entity occurs. + 2. :mod:`xml.dom.minidom` doesn't expand external entities and simply returns + the unexpanded entity verbatim. + 3. :mod:`xmlrpclib` doesn't expand external entities and omits them. + + + billion laughs / exponential entity expansion + The `Billion Laughs`_ attack -- also known as exponential entity expansion -- + uses multiple levels of nested entities. Each entity refers to another entity + several times, the final entity definition contains a small string. Eventually + the small string is expanded to several gigabytes. The exponential expansion + consumes lots of CPU time, too. + + quadratic blowup entity expansion + A quadratic blowup attack is similar to a `Billion Laughs`_ attack; it abuses + entity expansion, too. Instead of nested entities it repeats one large entity + with a couple of thousand chars over and over again. The attack isn't as + efficient as the exponential case but it avoids triggering countermeasures of + parsers against heavily nested entities. + + external entity expansion + Entity declarations can contain more than just text for replacement. They can + also point to external resources by public identifiers or system identifiers. + System identifiers are standard URIs or can refer to local files. The XML + parser retrieves the resource with e.g. HTTP or FTP requests and embeds the + content into the XML document. + + DTD retrieval + Some XML libraries like Python's mod:'xml.dom.pulldom' retrieve document type + definitions from remote or local locations. The feature has similar + implications as the external entity expansion issue. + + decompression bomb + The issue of decompression bombs (aka `ZIP bomb`_) apply to all XML libraries + that can parse compressed XML stream like gzipped HTTP streams or LZMA-ed + files. For an attacker it can reduce the amount of transmitted data by three + magnitudes or more. + + The documentation of `defusedxml`_ on PyPI has further information about + all known attack vectors with examples and references. + + defused packages + ---------------- + + `defusedxml`_ is a pure Python package with modified subclasses of all stdlib + XML parsers that prevent any potentially malicious operation. The courses of + action are recommended for any server code that parses untrusted XML data. The + package also ships with example exploits and an extended documentation on more + XML exploits like xpath injection. + + `defusedexpat`_ provides a modified libexpat and patched replacment + :mod:`pyexpat` extension module with countermeasures against entity expansion + DoS attacks. Defusedexpat still allows a sane and configurable amount of entity + expansions. The modifications will be merged into future releases of Python. + + The workarounds and modifications are not included in patch releases as they + break backward compatibility. After all inline DTD and entity expansion are + well-definied XML features. + + + .. _defusedxml: + .. _defusedexpat: + .. _Billion Laughs: http://en.wikipedia.org/wiki/Billion_laughs + .. _ZIP bomb: http://en.wikipedia.org/wiki/Zip_bomb + .. _DTD: http://en.wikipedia.org/wiki/Document_Type_Definition ++ diff --cc Doc/library/xmlrpc.client.rst index 1871c998ec8d,4592fb40a815..3a536559f7e7 --- a/Doc/library/xmlrpc.client.rst +++ b/Doc/library/xmlrpc.client.rst @@@ -21,12 -21,14 +21,19 @@@ supports writing XML-RPC client code; i between conformable Python objects and XML on the wire. + .. warning:: + + The :mod:`xmlrpc.client` module is not secure against maliciously + constructed data. If you need to parse untrusted or unauthenticated data see + :ref:`xml-vulnerabilities`. + + -.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, allow_none=False, use_datetime=False) +.. class:: ServerProxy(uri, transport=None, encoding=None, verbose=False, \ + allow_none=False, use_datetime=False, \ + use_builtin_types=False) + + .. versionchanged:: 3.3 + The *use_builtin_types* flag was added. A :class:`ServerProxy` instance is an object that manages communication with a remote XML-RPC server. The required first argument is a URI (Uniform Resource diff --cc Doc/library/xmlrpc.server.rst index 6493fd4d93b1,6b4c202ca0bb..18fee2fc1d36 --- a/Doc/library/xmlrpc.server.rst +++ b/Doc/library/xmlrpc.server.rst @@@ -16,9 -16,14 +16,16 @@@ servers written in Python. Servers ca :class:`CGIXMLRPCRequestHandler`. + .. warning:: + + The :mod:`xmlrpc.client` module is not secure against maliciously + constructed data. If you need to parse untrusted or unauthenticated data see + :ref:`xml-vulnerabilities`. + + -.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler, logRequests=True, allow_none=False, encoding=None, bind_and_activate=True) +.. class:: SimpleXMLRPCServer(addr, requestHandler=SimpleXMLRPCRequestHandler,\ + logRequests=True, allow_none=False, encoding=None,\ + bind_and_activate=True, use_builtin_types=False) Create a new server instance. This class provides methods for registration of functions that can be called by the XML-RPC protocol. The *requestHandler* diff --cc Misc/NEWS index 9efed13e26ed,5bdca7ff546c..7034bcd78523 --- a/Misc/NEWS +++ b/Misc/NEWS @@@ -815,152 -3645,348 +815,154 @@@ Test Build ----- -- Issue #10325: Fix two issues in the fallback definitions for PY_ULLONG_MAX and - PY_LLONG_MAX that made them unsuitable for use in preprocessor conditionals. - -Documentation -------------- - -- Issue #10299: List the built-in functions in a table in functions.rst. - - -What's New in Python 3.2 Alpha 4? -================================= - -*Release date: 13-Nov-2010* - -Core and Builtins ------------------ - -- Issue #10372: Import the warnings module only after the IO library is - initialized, so as to avoid bootstrap issues with the '-W' option. - -- Issue #10293: Remove obsolete field in the PyMemoryView structure, unused - undocumented value PyBUF_SHADOW, and strangely-looking code in - PyMemoryView_GetContiguous. - -- Issue #6081: Add str.format_map(), similar to ``str.format(**mapping)``. +- Issue #17550: Fix the --enable-profiling configure switch. -- If FileIO.__init__ fails, close the file descriptor. +- Issue #17425: Build with openssl 1.0.1d on Windows. -- Issue #10221: dict.pop(k) now has a key error message that includes the - missing key (same message d[k] returns for missing keys). - -- Issue #5437: A preallocated MemoryError instance should not keep traceback - data (including local variables caught in the stack trace) alive infinitely. - -- Issue #10186: Fix the SyntaxError caret when the offset is equal to the length - of the offending line. - -- Issue #10089: Add support for arbitrary -X options on the command line. They - can be retrieved through a new attribute ``sys._xoptions``. - -- Issue #4388: On Mac OS X, decode command line arguments from UTF-8, instead of - the locale encoding. If the LANG (and LC_ALL and LC_CTYPE) environment - variable is not set, the locale encoding is ISO-8859-1, whereas most programs - (including Python) expect UTF-8. Python already uses UTF-8 for the filesystem - encoding and to encode command line arguments on this OS. - -- Issue #9713, #10114: Parser functions (e.g. PyParser_ASTFromFile) expect - filenames encoded to the filesystem encoding with the surrogateescape error - handler (to support undecodable bytes), instead of UTF-8 in strict mode. - -- Issue #9997: Don't let the name "top" have special significance in scope - resolution. - -- Issue #9862: Compensate for broken PIPE_BUF in AIX by hard coding its value as - the default 512 when compiling on AIX. - -- Use locale encoding instead of UTF-8 to encode and decode filenames if - Py_FileSystemDefaultEncoding is not set. - -- Issue #10095: fp_setreadl() doesn't reopen the file, instead reuse the file - descriptor. - -- Issue #9418: Moved private string methods ``_formatter_parser`` and - ``_formatter_field_name_split`` into a new ``_string`` module. - -- Issue #9992: Remove PYTHONFSENCODING environment variable. - -Library -------- - -- Issue #10465: fix broken delegating of attributes by gzip._PaddedFile. - -- Issue #10356: Decimal.__hash__(-1) should return -2. - -- Issue #1553375: logging: Added stack_info kwarg to display stack information. - -- Issue #5111: IPv6 Host in the Header is wrapped inside [ ]. Patch by Chandru. - -- Fix Fraction.__hash__ so that Fraction.__hash__(-1) is -2. (See also issue - #10356.) - -- Issue #4471: Add the IMAP.starttls() method to enable encryption on standard - IMAP4 connections. Original patch by Lorenzo M. Catucci. - -- Issue #1466065: Add 'validate' option to base64.b64decode to raise an error if - there are non-base64 alphabet characters in the input. - -- Issue #10386: Add __all__ to token module; this simplifies importing in - tokenize module and prevents leaking of private names through ``import *``. - -- Issue #4471: Properly shutdown socket in IMAP.shutdown(). Patch by Lorenzo - M. Catucci. - -- Fix IMAP.login() to work properly. - -- Issue #9244: multiprocessing pool worker processes could terminate - unexpectedly if the return value of a task could not be pickled. Only the - ``repr`` of such errors are now sent back, wrapped in an - ``MaybeEncodingError`` exception. - -- Issue #9244: The ``apply_async()`` and ``map_async()`` methods of - ``multiprocessing.Pool`` now accepts a ``error_callback`` argument. This can - be a callback with the signature ``callback(exc)``, which will be called if - the target raises an exception. - -- Issue #10022: The dictionary returned by the ``getpeercert()`` method of SSL - sockets now has additional items such as ``issuer`` and ``notBefore``. - -- ``usenetrc`` is now false by default for NNTP objects. - -- Issue #1926: Add support for NNTP over SSL on port 563, as well as STARTTLS. - Patch by Andrew Vant. - -- Issue #10335: Add tokenize.open(), detect the file encoding using - tokenize.detect_encoding() and open it in read only mode. - -- Issue #10321: Add support for binary data to smtplib.SMTP.sendmail, and a new - method send_message to send an email.message.Message object. - -- Issue #6011: sysconfig and distutils.sysconfig use the surrogateescape error - handler to parse the Makefile file. Avoid a UnicodeDecodeError if the source - code directory name contains a non-ASCII character and the locale encoding is - ASCII. - -- Issue #10329: The trace module writes reports using the input Python script - encoding, instead of the locale encoding. Patch written by Alexander - Belopolsky. - -- Issue #10126: Fix distutils' test_build when Python was built with - --enable-shared. - -- Issue #9281: Prevent race condition with mkdir in distutils. Patch by - Arfrever. - -- Issue #10229: Fix caching error in gettext. - -- Issue #10252: Close file objects in a timely manner in distutils code and - tests. Patch by Brian Brazil, completed by Ãric Araujo. - -- Issue #10180: Pickling file objects is now explicitly forbidden, since - unpickling them produced nonsensical results. - -- Issue #10311: The signal module now restores errno before returning from its - low-level signal handler. Patch by Hallvard B Furuseth. - -- Issue #10282: Add a ``nntp_implementation`` attribute to NNTP objects. - -- Issue #10283: Add a ``group_pattern`` argument to NNTP.list(). - -- Issue #10155: Add IISCGIHandler to wsgiref.handlers to support IIS CGI - environment better, and to correct unicode environment values for WSGI 1.0.1. - -- Issue #10281: nntplib now returns None for absent fields in the OVER/XOVER - response, instead of raising an exception. - -- wsgiref now implements and validates PEP 3333, rather than an experimental - extension of PEP 333. (Note: earlier versions of Python 3.x may have - incorrectly validated some non-compliant applications as WSGI compliant; if - your app validates with Python <3.2b1+, but not on this version, it is likely - the case that your app was not compliant.) - -- Issue #10280: NNTP.nntp_version should reflect the highest version advertised - by the server. - -- Issue #10184: Touch directories only once when extracting a tarfile. - -- Issue #10199: New package, ``turtledemo`` now contains selected demo scripts - that were formerly found under Demo/turtle. - -- Issue #10265: Close file objects explicitly in sunau. Patch by Brian Brazil. - -- Issue #10266: uu.decode didn't close in_file explicitly when it was given as a - filename. Patch by Brian Brazil. - -- Issue #10110: Queue objects didn't recognize full queues when the maxsize - parameter had been reduced. - -- Issue #10160: Speed up operator.attrgetter. Patch by Christos Georgiou. - -- logging: Added style option to basicConfig() to allow %, {} or $-formatting. - -- Issue #5729: json.dumps() now supports using a string such as '\t' for - pretty-printing multilevel objects. - -- Issue #10253: FileIO leaks a file descriptor when trying to open a file for - append that isn't seekable. Patch by Brian Brazil. - -- Support context manager protocol for file-like objects returned by mailbox - ``get_file()`` methods. - -- Issue #10246: uu.encode didn't close file objects explicitly when filenames - were given to it. Patch by Brian Brazil. - -- Issue #10198: fix duplicate header written to wave files when writeframes() is - called without data. - -- Close file objects in modulefinder in a timely manner. - -- Close a io.TextIOWrapper object in email.parser in a timely manner. - -- Close a file object in distutils.sysconfig in a timely manner. +- Issue #16754: Fix the incorrect shared library extension on linux. Introduce + two makefile macros SHLIB_SUFFIX and EXT_SUFFIX. SO now has the value of + SHLIB_SUFFIX again (as in 2.x and 3.1). The SO macro is removed in 3.4. -- Close a file object in pkgutil in a timely manner. +- Issue #5033: Fix building of the sqlite3 extension module when the + SQLite library version has "beta" in it. Patch by Andreas Pelme. -- Issue #10233: Close file objects in a timely manner in the tarfile module and - its test suite. +- Issue #17228: Fix building without pymalloc. -- Issue #10093: ResourceWarnings are now issued when files and sockets are - deallocated without explicit closing. These warnings are silenced by default, - except in pydebug mode. +- Issue #3718: Use AC_ARG_VAR to set MACHDEP in configure.ac. -- tarfile.py: Add support for all missing variants of the GNU sparse extensions - and create files with holes when extracting sparse members. +- Issue #17031: Fix running regen in cross builds. -- Issue #10218: Return timeout status from ``Condition.wait`` in threading. +- Issue #3754: fix typo in pthread AC_CACHE_VAL. -- Issue #7351: Add ``zipfile.BadZipFile`` spelling of the exception name and - deprecate the old name ``zipfile.BadZipfile``. +- Issue #15484: Fix _PYTHON_PROJECT_BASE for srcdir != builddir builds; + use _PYTHON_PROJECT_BASE in distutils/sysconfig.py. -- Issue #5027: The standard ``xml`` namespace is now understood by - xml.sax.saxutils.XMLGenerator as being bound to - http://www.w3.org/XML/1998/namespace. Patch by Troy J. Farrell. +- Issue #17029: Let h2py search the multiarch system include directory. -- Issue #5975: Add csv.unix_dialect class. +- Issue #16953: Fix socket module compilation on platforms with + HAVE_BROKEN_POLL. Patch by Jeffrey Armstrong. -- Issue #7761: telnetlib.interact failures on Windows fixed. +- Issue #16836: Enable IPv6 support even if IPv6 is disabled on the build host. -- logging: Added style option to Formatter to allow %, {} or $-formatting. +- Cross compiling needs host and build settings. configure no longer + creates a broken PYTHON_FOR_BUILD variable when --build is missing. -- Issue #5178: Added tempfile.TemporaryDirectory class that can be used as a - context manager. +- Fix cross compiling issue in setup.py, ensure that lib_dirs and inc_dirs are + defined in cross compiling mode, too. -- Issue #1349106: Generator (and BytesGenerator) flatten method and Header - encode method now support a 'linesep' argument. +- Issue #16593: Have BSD 'make -s' do the right thing, thanks to Daniel Shahaf -- Issue #5639: Add a *server_hostname* argument to ``SSLContext.wrap_socket`` in - order to support the TLS SNI extension. ``HTTPSConnection`` and ``urlopen()`` - also use this argument, so that HTTPS virtual hosts are now supported. +- Issue #16262: fix out-of-src-tree builds, if mercurial is not installed. -- Issue #10166: Avoid recursion in pstats Stats.add() for many stats items. +- Issue #15298: ensure _sysconfigdata is generated in build directory, not + source directory. -- Issue #10163: Skip unreadable registry keys during mimetypes initialization. +- Issue #15833: Fix a regression in 3.3 that resulted in exceptions being + raised if importlib failed to write byte-compiled files. This affected + attempts to build Python out-of-tree from a read-only source directory. -- logging: Made StreamHandler terminator configurable. +- Issue #15923: Fix a mistake in ``asdl_c.py`` that resulted in a TypeError + after 2801bf875a24 (see #15801). -- logging: Allowed filters to be just callables. +- Issue #15819: Make sure we can build Python out-of-tree from a read-only + source directory. (Somewhat related to issue #9860.) -- logging: Added tests for _logRecordClass changes. +- Issue #15587: Enable Tk high-resolution text rendering on Macs with + Retina displays. Applies to Tkinter apps, such as IDLE, on OS X + framework builds linked with Cocoa Tk 8.5. -- Issue #10092: Properly reset locale in calendar.Locale*Calendar classes. +- Issue #17161: make install now also installs a python3 man page. -- logging: Added _logRecordClass, getLogRecordClass, setLogRecordClass to - increase flexibility of LogRecord creation. +Tools/Demos +----------- -- Issue #5117: Case normalization was needed on ntpath.relpath(). Also fixed - root directory issue on posixpath.relpath(). (Ported working fixes from - ntpath.) +- Issue #13301: use ast.literal_eval() instead of eval() in Tools/i18n/msgfmt.py + Patch by Serhiy Storchaka. -- Issue #1343: xml.sax.saxutils.XMLGenerator now has an option - short_empty_elements to direct it to use self-closing tags when appropriate. +Documentation +------------- -- Issue #9807 (part 1): Expose the ABI flags in sys.abiflags. Add --abiflags - switch to python-config for command line access. ++- Issue 17538: Document XML vulnerabilties + -- Issue #6098: Don't claim DOM level 3 conformance in minidom. +- Issue #16642: sched.scheduler timefunc initial default is time.monotonic. + Patch by Ramchandra Apte -- Issue #5762: Fix AttributeError raised by ``xml.dom.minidom`` when an empty - XML namespace attribute is encountered. +- Issue #17047: remove doubled words in docs and docstrings + reported by Serhiy Storchaka and Matthew Barnett. -- Issue #2830: Add the ``html.escape()`` function, which quotes all problematic - characters by default. Deprecate ``cgi.escape()``. +- Issue #15465: Document the versioning macros in the C API docs rather than + the standard library docs. Patch by Kushal Das. -- Issue #9409: Fix the regex to match all kind of filenames, for interactive - debugging in doctests. +- Issue #16406: Combine the pages for uploading and registering to PyPI. -- Issue #9183: ``datetime.timezone(datetime.timedelta(0))`` will now return the - same instance as ``datetime.timezone.utc``. +- Issue #16403: Document how distutils uses the maintainer field in + PKG-INFO. Patch by Jyrki Pulliainen. -- Issue #7523: Add SOCK_CLOEXEC and SOCK_NONBLOCK to the socket module, where - supported by the system. Patch by Nikita Vetoshkin. +- Issue #16695: Document how glob handles filenames starting with a + dot. Initial patch by Jyrki Pulliainen. -- Issue #10063: file:// scheme will stop accessing remote hosts via ftp - protocol. file:// urls had fallback to access remote hosts via ftp. This was - not correct, change is made to raise a URLError when a remote host is tried to - access via file:// scheme. +- Issue #8890: Stop advertising an insecure practice by replacing uses + of the /tmp directory with better alternatives in the documentation. + Patch by Geoff Wilson. -- Issue #1710703: Write structures for an empty ZIP archive when a ZipFile is - created in modes 'a' or 'w' and then closed without adding any files. Raise - BadZipfile (rather than IOError) when opening small non-ZIP files. +- Issue #17203: add long option names to unittest discovery docs. -- Issue #10041: The signature of optional arguments in socket.makefile() didn't - match that of io.open(), and they also didn't get forwarded properly to - TextIOWrapper in text mode. Patch by Kai Zhu. +- Issue #13094: add "Why do lambdas defined in a loop with different values + all return the same result?" programming FAQ. -- Issue #9003: http.client.HTTPSConnection, urllib.request.HTTPSHandler and - urllib.request.urlopen now take optional arguments to allow for server - certificate checking, as recommended in public uses of HTTPS. +- Issue #14901: Update portions of the Windows FAQ. + Patch by Ashish Nitin Patil. -- Issue #6612: Fix site and sysconfig to catch os.getcwd() error, eg. if the - current directory was deleted. Patch written by W. Trevor King. +- Issue #16267: Better document the 3.3+ approach to combining + @abstractmethod with @staticmethod, @classmethod and @property -- Issue #3873: Speed up unpickling from file objects that have a peek() method. +- Issue #15209: Clarify exception chaining description in exceptions module + documentation -- Issue #10075: Add a session_stats() method to SSLContext objects. +- Issue #15990: Improve argument/parameter documentation. -- Issue #9948: Fixed problem of losing filename case information. +- Issue #16209: Move the documentation for the str built-in function to a new + str class entry in the "Text Sequence Type" section. -Extension Modules ------------------ +- Issue #13538: Improve str() and object.__str__() documentation. -- Issue #5109: array.array constructor will now use fast code when - initial data is provided in an array object with correct type. +- Issue #16489: Make it clearer that importlib.find_loader() requires any and + all packages to be separately imported. -- Issue #6317: Now winsound.PlaySound only accepts unicode. +- Issue #16400: Update the description of which versions of a given package + PyPI displays. -- Issue #6317: Now winsound.PlaySound can accept non ascii filename. +- Issue #15677: Document that zlib and gzip accept a compression level of 0 to + mean 'no compression'. Patch by Brian Brazil. -- Issue #9377: Use Unicode API for gethostname on Windows. +- Issue #8040: added a version switcher to the documentation. Patch by + Yury Selivanov. -- Issue #10143: Update "os.pathconf" values. +- Additional comments and some style changes in the concurrent.futures URL + retrieval example -- Issue #6518: Support context manager protcol for ossaudiodev types. +- Issue #16115: Improve subprocess.Popen() documentation around args, shell, + and executable arguments. -- Issue #678250: Make mmap flush a noop on ACCESS_READ and ACCESS_COPY. +- Issue #15533: Clarify docs and add tests for `subprocess.Popen()`'s cwd + argument. -- Issue #9054: Fix a crash occurring when using the pyexpat module with expat - version 2.0.1. +- Issue #15979: Improve timeit documentation. -- Issue #5355: Provide mappings from Expat error numbers to string descriptions - and backwards, in order to actually make it possible to analyze error codes - provided by ExpatError. +- Issue #16036: Improve documentation of built-in `int()`'s signature and + arguments. -- The Unicode database was updated to 6.0.0. +- Issue #15935: Clarification of `argparse` docs, re: add_argument() type and + default arguments. Patch contributed by Chris Jerdonek. -C-API ------ +- Issue #11964: Document a change in v3.2 to the behavior of the indent + parameter of json encoding operations. -- Issue #10288: The deprecated family of "char"-handling macros - (ISLOWER()/ISUPPER()/etc) have now been removed: use Py_ISLOWER() etc instead. +- Issue #15116: Remove references to appscript as it is no longer being + supported. -- Issue #9778: Hash values are now always the size of pointers. A new Py_hash_t - type has been introduced. +- Issue #15116: Remove references to appscript as it is no longer being + supported. Tools/Demos -----------