Ensuring original text is preserved in CHUNKING_REGEX

[tylerganter]

Description

The original default CHUNKING_REGEX wasn't preserving the original text, repeating characters in ,.;。？！ were reduced to a singe instance, so in a scenario like [text](../path/to/file.md) this can get reconstructed as [text](./path/to/file.md) which will not match the original text. This then results in SentenceSplitter.start_char_idx = None because it fails to find the chunk text in the source document text.

Fixes # (issue)

Version Bump?

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Simple Regex Text Preservation Test

import re

text = '[**this is a link](.........../path/to/file.md) some\nmore;;text'

# Original pattern
original_pattern = "[^,.;。？！]+[,.;。？！]?"
original_splits = re.findall(original_pattern, text)
print(f"Original pattern: {original_pattern}")
print(f"Original splits: {original_splits}")
print(f"Joined result: {''.join(original_splits)}")
print(f"Matches original? {''.join(original_splits) == text}")
# Original pattern: [^,.;。？！]+[,.;。？！]?
# Original splits: ['[**this is a link](.', '/path/to/file.', 'md) some\nmore;', 'text']
# Joined result: [**this is a link](./path/to/file.md) some
# more;text
# Matches original? False

# Enhanced pattern that preserves the text while maintaining similar behavior
# This pattern: 
# 1. Captures sequences of non-punctuation followed by optional punctuation: [^,.;。？！]+[,.;。？！]?
# 2. OR captures single punctuation characters by themselves: [,.;。？！]
enhanced_pattern = r"[^,.;。？！]+[,.;。？！]?|[,.;。？！]"
enhanced_splits = re.findall(enhanced_pattern, text)
print(f"\nEnhanced pattern: {enhanced_pattern}")
print(f"Enhanced splits: {enhanced_splits}")
print(f"Joined result: {''.join(enhanced_splits)}")
print(f"Matches original? {''.join(enhanced_splits) == text}")
# Enhanced pattern: [^,.;。？！]+[,.;。？！]?|[,.;。？！]
# Enhanced splits: ['[**this is a link](.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '.', '/path/to/file.', 'md) some\nmore;', ';', 'text']
# Joined result: [**this is a link](.........../path/to/file.md) some
# more;;text
# Matches original? True

# Test cases to verify the solution
test_cases = [
    "Simple text with no punctuation",
    "Text with, some; punctuation.",
    "Text with......multiple punctuation",
    "Text with\nnewlines\npreserved",
    "Text with,.;。？！ all specified punctuation",
    "Text with consecutive....periods",
    "Text with;;semicolons",
    "[**Markdown-style](links) preserved"
]

print("\n--- Testing with various cases ---")
for case in test_cases:
    splits = re.findall(enhanced_pattern, case)
    reconstructed = ''.join(splits)
    print(f"Case: {case!r}")
    print(f"Splits: {splits}")
    print(f"Preserved: {reconstructed == case}")
    if reconstructed != case:
        print(f"  Expected: {case!r}")
        print(f"  Got: {reconstructed!r}")
    print()
# --- Testing with various cases ---
# Case: 'Simple text with no punctuation'
# Splits: ['Simple text with no punctuation']
# Preserved: True

# Case: 'Text with, some; punctuation.'
# Splits: ['Text with,', ' some;', ' punctuation.']
# Preserved: True

# Case: 'Text with......multiple punctuation'
# Splits: ['Text with.', '.', '.', '.', '.', '.', 'multiple punctuation']
# Preserved: True

# Case: 'Text with\nnewlines\npreserved'
# Splits: ['Text with\nnewlines\npreserved']
# Preserved: True

# Case: 'Text with,.;。？！ all specified punctuation'
# Splits: ['Text with,', '.', ';', '。', '？', '！', ' all specified punctuation']
# Preserved: True

# Case: 'Text with consecutive....periods'
# Splits: ['Text with consecutive.', '.', '.', '.', 'periods']
# Preserved: True

# Case: 'Text with;;semicolons'
# Splits: ['Text with;', ';', 'semicolons']
# Preserved: True

# Case: '[**Markdown-style](links) preserved'
# Splits: ['[**Markdown-style](links) preserved']
# Preserved: True

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[logan-markewich]

This test isn't really testing for how the content is split right? Shouldn't it check if the text has repeated .. or not?