]> git.ipfire.org Git - thirdparty/Python/cpython.git/commit
gh-119118: Fix performance regression in tokenize module (#119615)
authorLysandros Nikolaou <lisandrosnik@gmail.com>
Tue, 28 May 2024 19:17:49 +0000 (21:17 +0200)
committerGitHub <noreply@github.com>
Tue, 28 May 2024 19:17:49 +0000 (19:17 +0000)
commitd87b0151062e36e67f9e42e1595fba5bf23a485c
tree1a552ad552d5a2cbdcad3ae58d33cbd4879c3217
parentae9140f32a1630838374f1af402291d4649a0be0
gh-119118: Fix performance regression in tokenize module (#119615)

* gh-119118: Fix performance regression in tokenize module

- Cache line object to avoid creating a Unicode object
  for all of the tokens in the same line.
- Speed up byte offset to column offset conversion by using the
  smallest buffer possible to measure the difference.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
Misc/NEWS.d/next/Library/2024-05-28-12-15-03.gh-issue-119118.FMKz1F.rst [new file with mode: 0644]
Parser/pegen.c
Parser/pegen.h
Python/Python-tokenize.c