]> git.ipfire.org Git - thirdparty/Python/cpython.git/commit
gh-88375, gh-111788: Fix parsing errors and normalization in robotparser (GH-138502)
authorSerhiy Storchaka <storchaka@gmail.com>
Fri, 5 Sep 2025 15:58:42 +0000 (18:58 +0300)
committerGitHub <noreply@github.com>
Fri, 5 Sep 2025 15:58:42 +0000 (18:58 +0300)
commitcb7ef18d70a0bc1363799e2dfa475db457155e43
treedd70e098942c1584691b57b64f35625dc669eb51
parented522ed211b7b9ea9d85a93b5d6ca79b2df3ef58
gh-88375, gh-111788: Fix parsing errors and normalization in robotparser (GH-138502)

* Don't fail trying to parse weird patterns.
* Don't fail trying to decode non-UTF-8 "robots.txt" files.
* No longer ignore trailing "?" in patterns and URLs.
* Distinguish raw special characters "?", "=" and "&" from the
  percent-encoded ones.
* Remove tests that do nothing.
Lib/test/test_robotparser.py
Lib/urllib/robotparser.py
Misc/NEWS.d/next/Library/2025-09-04-15-18-11.gh-issue-111788.tuTEM5.rst [new file with mode: 0644]
Misc/NEWS.d/next/Library/2025-09-05-15-35-59.gh-issue-88375.dC491a.rst [new file with mode: 0644]