From: Pablo Galindo Salgado <Pablogsal@gmail.com>
Date: Fri, 9 Jun 2023 15:59:37 +0000 (+0100)
Subject: Clarify the supported cases in the tokenize module (#105569)
X-Git-Tag: v3.13.0a1~1815
X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=1dd267af642ed6df05a1c106e9dafb8252d826e6;p=thirdparty%2FPython%2Fcpython.git

Clarify the supported cases in the tokenize module (#105569)
---

diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst
index 41222a771d1b..bffe93006edc 100644
--- a/Doc/library/tokenize.rst
+++ b/Doc/library/tokenize.rst
@@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type.  The exact
 type can be determined by checking the ``exact_type`` property on the
 :term:`named tuple` returned from :func:`tokenize.tokenize`.
 
+
+.. warning::
+
+   Note that the functions in this module are only designed to parse
+   syntactically valid Python code (code that does not raise when parsed
+   using :func:`ast.parse`).  The behavior of the functions in this module is
+   **undefined** when providing invalid Python code and it can change at any
+   point.
+
 Tokenizing Input
 ----------------