gh-83273: Rewrite csv.Sniffer dialect detection using trial parsing (GH-153694)
Guess the dialect by parsing the sample with every plausible
combination of delimiter, quotechar and escapechar, using the actual
CSV parser in strict mode, and choosing the combination which splits
the sample into rows with the most consistent number of fields. The
old heuristics, which guessed the delimiter from characters adjacent
to quotes and from character frequencies, are removed.
A large sample is parsed incrementally: first only its beginning,
then, after eliminating the combinations which are clearly worse than
the leader, a several times larger part, and so on.
* csv.Sniffer can now detect escapechar='\\'.
* Explicitly requested delimiters are no longer restricted to ASCII.
* A delimiter inside a quoted field no longer wins over the actual
delimiter, and sniffing no longer takes quadratic time on quoted
samples.
* The sample can be cut off at an arbitrary point: in the middle of a
row, of a quoted field or of an escaped sequence.
* Only '\r', '\n' and '\r\n' are treated as row separators, so
characters like '\x1c' can now be detected as a delimiter.
* A preamble (title or comment lines) before the data does not prevent
detection if the data rows outnumber the preamble lines.
* A sample consisting of a single column of quoted fields now raises
csv.Error instead of guessing a delimiter from the content of the
fields, and Sniffer.has_header() no longer raises ValueError for
such samples.
* doublequote is detected by comparing the two readings of the sample.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>