[3.14] gh-88375, gh-111788: Fix parsing errors and normalization in robotparser ...

[3.14] gh-88375, gh-111788: Fix parsing errors and normalization in robotparser (GH-138502) (GH-138548)

* Don't fail trying to parse weird patterns.
* Don't fail trying to decode non-UTF-8 "robots.txt" files.
* No longer ignore trailing "?" in patterns and URLs.
* Distinguish raw special characters "?", "=" and "&" from the
  percent-encoded ones.
* Remove tests that do nothing.
(cherry picked from commit cb7ef18d70a0bc1363799e2dfa475db457155e43)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>