Issue #2134: Clarify token.OP handling rationale in tokenize documentation.

diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 577d7cca4c99cbb3996940d7fbae971b82cea803..70919ca998c4807ca5751fee857585d54389369d 100644 (file)
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -15,6 +15,12 @@ implemented in Python.  The scanner in this module returns comments as tokens
 as well, making it useful for implementing "pretty-printers," including
 colorizers for on-screen displays.
 
+To simplify token stream handling, all :ref:`operators` and :ref:`delimiters`
+tokens are returned using the generic :data:`token.OP` token type.  The exact
+type can be determined by checking the token ``string`` field on the
+:term:`named tuple` returned from :func:`tokenize.tokenize` for the character
+sequence that identifies a specific operator token.
+
 The primary entry point is a :term:`generator`:
 
 .. function:: tokenize(readline)

diff --git a/Misc/NEWS b/Misc/NEWS
index 7d8cb7b40f815342ee8b9ce9593fc4b3911e16e1..ed6c1e0340f2ef143324970b7091ef3a87afb9ad 100644 (file)
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -418,6 +418,9 @@ Extension Modules
 Documentation
 -------------
 
+- Issue #2134: The tokenize documentation has been clarified to explain why
+  all operator and delimiter tokens are treated as token.OP tokens.
+
 - Issue #13513: Fix io.IOBase documentation to correctly link to the
   io.IOBase.readline method instead of the readline module.