[3.14] gh-138907: Support RFC 9309 in robotparser (GH-138908) (GH-149374)

[3.14] gh-138907: Support RFC 9309 in robotparser (GH-138908) (GH-149374)

* empty lines are always ignored instead of separating groups
* the "user-agent" line after a rule starts a new group
* groups matching the same user agent are now merged
* the rule with the longest match wins instead of the first matching rule
* in case of equal matches, the “Allow” rule wins over “Disallow”
* special characters “$” and “*” are now supported in rules
* prefer full match for user agent

(cherry picked from commit bc285e583286c739e553e49c19fd946cb63432c7)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>