From: Miss Islington (bot) <31488909+miss-islington@users.noreply.github.com> Date: Fri, 9 Jun 2023 16:00:59 +0000 (-0700) Subject: [3.10] Clarify the supported cases in the tokenize module (GH-105569) (#105575) X-Git-Tag: v3.10.13~14 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=45de31db9cc9be945702f3a7ca35bbb9f98476af;p=thirdparty%2FPython%2Fcpython.git [3.10] Clarify the supported cases in the tokenize module (GH-105569) (#105575) Co-authored-by: Pablo Galindo Salgado --- diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst index 11f569df2e7c..a903bff1da8f 100644 --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -22,6 +22,15 @@ the generic :data:`~token.OP` token type. The exact type can be determined by checking the ``exact_type`` property on the :term:`named tuple` returned from :func:`tokenize.tokenize`. + +.. warning:: + + Note that the functions in this module are only designed to parse + syntactically valid Python code (code that does not raise when parsed + using :func:`ast.parse`). The behavior of the functions in this module is + **undefined** when providing invalid Python code and it can change at any + point. + Tokenizing Input ----------------