[3.15] gh-109638: Fix exponential time in csv.Sniffer for doubled quotes

[serhiy-storchaka]

The regular expression which detects a doubled quote character lets its runs match the quote character too, so every quote can be taken either as part of a run or as one of the three matched quotes. With three unbounded quantifiers over a region of quotes and delimiters the search is O(n^5) -- 254 s at n=100 in the report.

>>> sample = '"",' * 100 + '"' * 100 + '0' + '"' * 100 + '0'
>>> csv.Sniffer().sniff(sample)   # minutes

The runs now exclude the quote character and are matched possessively, which leaves exactly one way to read any input.

   n      before      after
  60    6.1300 s    0.0002 s
2000        --      0.0009 s

The runs also no longer exclude the delimiter and the line break, which the original did for no reason a reader shares -- so a doubled quote is now found in a field containing them as well, e.g. ,"All-Weather Dining Table, Round 48""",. That is the bulk of the behaviour change: 42 of 560 corpus files, and against ground truth doublequote=True improves from 25.8% to 36.7% and doublequote=False from 77.0% to 78.7%, with delimiter and quotechar unchanged.

Note #109639 proposed an atomic group on the first run only. It removes the time but also the detection: doublequote becomes unconditionally False (0 of 126 annotated files, vs 32 for 3.15).

main is not affected: the sniffer was rewritten there in gh-83273.

• Issue: csv.Sniffer regular expression has significant backtracing #109638

[serhiy-storchaka]

Note that the widened detection -- a doubled quote is now found in a field which contains the delimiter or a line break -- changes results on 42 of 560 files in the CSVsniffer corpora. Against that corpus's ground truth the change is positive on both sides: doublequote=True 25.4% -> 37.3%, doublequote=False 77.0% -> 78.5%.

There are two alternatives. #154865 changes no results, but leaves the search polynomial (O(n^5) -> O(n^4), still four and a half minutes on a 2 KB sample). Or, since 3.15 is still in beta, the new engine can be backported from main, which fixes this issue along with seven others; it changes results on 122 of 560 files.