Skip to content

Commit 4c342c5

Browse files
committed
gh-111487: Fix csv.Sniffer rejecting a delimiter at 90% consistency
_guess_delimiter counts a candidate delimiter's consistency down from 100% to a documented minimum threshold of 90%, but stepped the counter by repeatedly subtracting 0.01 from a float. The accumulated rounding error left it at 0.8999999999999999 after ten steps, so the loop exited before making the final pass and a delimiter present on exactly 90% of the rows was never accepted. Count down in whole percent and compare with exact integer arithmetic instead. `total` no longer needs to be a float, and the loop keeps the original behaviour of not running once a delimiter has been found in an earlier chunk.
1 parent 23b2b40 commit 4c342c5

3 files changed

Lines changed: 29 additions & 8 deletions

File tree

Lib/csv.py

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -404,18 +404,21 @@ def _guess_delimiter(self, data, delimiters):
404404

405405
# build a list of possible delimiters
406406
modeList = modes.items()
407-
total = float(min(chunkLength * iteration, len(data)))
408-
# (rows of consistent data) / (number of rows) = 100%
409-
consistency = 1.0
410-
# minimum consistency threshold
411-
threshold = 0.9
412-
while len(delims) == 0 and consistency >= threshold:
407+
total = min(chunkLength * iteration, len(data))
408+
# (rows of consistent data) / (number of rows) = 100%, counted
409+
# down in whole percent to a minimum consistency threshold of 90%.
410+
# The comparison uses exact integer arithmetic: stepping a float
411+
# down by 0.01 accumulated rounding error, so the counter fell to
412+
# 0.8999999999999999 instead of 0.9 and the final pass at the
413+
# threshold never ran (gh-111487).
414+
for consistency in range(100, 89, -1):
415+
if delims:
416+
break
413417
for k, v in modeList:
414418
if v[0] > 0 and v[1] > 0:
415-
if ((v[1]/total) >= consistency and
419+
if (v[1] * 100 >= consistency * total and
416420
(delimiters is None or k in delimiters)):
417421
delims[k] = v
418-
consistency -= 0.01
419422

420423
if len(delims) == 1:
421424
delim = list(delims.keys())[0]

Lib/test/test_csv.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1521,6 +1521,20 @@ def test_guess_delimiter_crlf_not_chosen(self):
15211521
self.assertEqual(sniffer.sniff(sample).delimiter, "|")
15221522
self.assertNotEqual(sniffer.sniff(sample).delimiter, "\r")
15231523

1524+
def test_guess_delimiter_at_minimum_consistency(self):
1525+
# gh-111487: a delimiter occurring on exactly 90% of the rows -- the
1526+
# documented minimum consistency -- used to be rejected, because the
1527+
# counter was stepped down by repeatedly subtracting 0.01 from a float
1528+
# and so never reached 0.9 exactly.
1529+
# Every row uses a different letter, so only ";" is consistent; the
1530+
# short row is in the first chunk, which keeps that chunk below the
1531+
# threshold and defers the decision to the 20-row chunk (18/20 == 90%).
1532+
rows = [(chr(ord('a') + i) + ';') * 3 + chr(ord('a') + i)
1533+
for i in range(20)]
1534+
rows[4] = 'e;e;e'
1535+
sniffer = csv.Sniffer()
1536+
self.assertEqual(sniffer.sniff('\n'.join(rows)).delimiter, ';')
1537+
15241538
def test_zero_mode_tie_order_independence(self):
15251539
sniffer = csv.Sniffer()
15261540
# ":" appears in half the rows (1, 0, 1, 0) - a tie between
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Fix :meth:`csv.Sniffer.sniff` rejecting a delimiter that occurs on exactly 90%
2+
of the rows, the documented minimum consistency. The consistency counter was
3+
stepped down by repeatedly subtracting ``0.01`` from a float and so never
4+
reached that threshold exactly; it now uses exact integer arithmetic.

0 commit comments

Comments
 (0)