]> git.ipfire.org Git - thirdparty/Python/cpython.git/commit
gh-92651: Remove the Include/token.h header file (#92652)
authorVictor Stinner <vstinner@python.org>
Wed, 11 May 2022 21:22:50 +0000 (23:22 +0200)
committerGitHub <noreply@github.com>
Wed, 11 May 2022 21:22:50 +0000 (23:22 +0200)
commitda5727a120e426ffaf68bf3a8016491205bd2f80
treea1d551bbc55b420de3f7c68b69a05fdf080e83f2
parentb69297ea23c0ab9866ae8bd26a347a9b5df567a6
gh-92651: Remove the Include/token.h header file (#92652)

Remove the token.h header file. There was never any public tokenizer
C API. The token.h header file was only designed to be used by Python
internals.

Move Include/token.h to Include/internal/pycore_token.h. Including
this header file now requires that the Py_BUILD_CORE macro is
defined. It no longer checks for the Py_LIMITED_API macro.

Rename functions:

* PyToken_OneChar() => _PyToken_OneChar()
* PyToken_TwoChars() => _PyToken_TwoChars()
* PyToken_ThreeChars() => _PyToken_ThreeChars()
13 files changed:
Doc/whatsnew/3.12.rst
Include/internal/pycore_token.h [moved from Include/token.h with 87% similarity]
Makefile.pre.in
Misc/NEWS.d/next/C API/2022-05-11-02-33-10.gh-issue-92651.FIXLf0.rst [new file with mode: 0644]
PCbuild/pythoncore.vcxproj
PCbuild/pythoncore.vcxproj.filters
PCbuild/regen.targets
Parser/pegen.h
Parser/token.c
Parser/tokenizer.c
Parser/tokenizer.h
Python/pythonrun.c
Tools/scripts/generate_token.py