]> git.ipfire.org Git - thirdparty/Python/cpython.git/commit
[3.12] gh-119118: Fix performance regression in tokenize module (GH-119615) (#119683)
authorMiss Islington (bot) <31488909+miss-islington@users.noreply.github.com>
Tue, 28 May 2024 20:49:02 +0000 (22:49 +0200)
committerGitHub <noreply@github.com>
Tue, 28 May 2024 20:49:02 +0000 (22:49 +0200)
commit4a0af0cfdcc0b81da5d78dc219df4985c4403f9c
tree18efcf4c071e3beab5ce26947e4ab90ad02ab66e
parent7f06cd335e91b0ec79c4608f23018c2a785332b1
[3.12] gh-119118: Fix performance regression in tokenize module (GH-119615) (#119683)

- Cache line object to avoid creating a Unicode object
  for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
  smallest buffer possible to measure the difference.

(cherry picked from commit d87b0151062e36e67f9e42e1595fba5bf23a485c)

Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Misc/NEWS.d/next/Library/2024-05-28-12-15-03.gh-issue-119118.FMKz1F.rst [new file with mode: 0644]
Parser/pegen.c
Parser/pegen.h
Python/Python-tokenize.c