feat(analyzer): opt-in merge of adjacent same-type entities (#1090)#2191
Open
watsos wants to merge 8 commits into
Open
feat(analyzer): opt-in merge of adjacent same-type entities (#1090)#2191watsos wants to merge 8 commits into
watsos wants to merge 8 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an opt-in mechanism in presidio-analyzer to merge adjacent, same-entity spans separated only by whitespace (e.g., NER splitting “Dave” and “Jones” into two PERSON results), while keeping merging scoped to explicitly requested entity types to avoid unintended fusions for pattern-based recognizers.
Changes:
- Added
EntityRecognizer.merge_adjacent_text_entities(results, text, entity_types)to fuse eligible adjacent spans and preserve metadata/explanation from the max-scoring span. - Exposed this via a new optional
AnalyzerEngine.analyze(..., merge_adjacent_entities=None)parameter and applied it after deduplication. - Added unit tests for merge behavior (basic merge, metadata preservation, chaining, non-merge scenarios, overlap/unsorted/empty inputs).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| presidio-analyzer/presidio_analyzer/entity_recognizer.py | Implements adjacent-span merging utility scoped to selected entity types. |
| presidio-analyzer/presidio_analyzer/analyzer_engine.py | Wires merging into AnalyzerEngine.analyze() behind an opt-in parameter. |
| presidio-analyzer/tests/test_entity_recognizer.py | Adds test coverage for merge logic and edge cases. |
Comment on lines
+365
to
+366
| merged_results.append(current) | ||
| return merged_results |
…sos/presidio into feature/merge-adjacent-entities
Author
|
Fixed — merge now returns results sorted by the same (-score, start, -length) key as remove_duplicates. Pushed in 0d87a8f |
Comment on lines
179
to
184
| allow_list: Optional[List[str]] = None, | ||
| allow_list_match: Optional[str] = "exact", | ||
| regex_flags: Optional[int] = re.DOTALL | re.MULTILINE | re.IGNORECASE, | ||
| nlp_artifacts: Optional[NlpArtifacts] = None, | ||
| merge_adjacent_entities: Optional[List[str]] = None, | ||
| ) -> List[RecognizerResult]: |
Comment on lines
+1282
to
+1288
| class DaveRecognizer(EntityRecognizer, ABC): | ||
| def analyze(self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts): | ||
| return [RecognizerResult("PERSON", 0, 4, 0.6)] | ||
|
|
||
| class JonesRecognizer(EntityRecognizer, ABC): | ||
| def analyze(self, text: str, entities: List[str], nlp_artifacts: NlpArtifacts): | ||
| return [RecognizerResult("PERSON", 5, 10, 0.85)] |
…to test recognizers
Agree with suggestion. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Adds
EntityRecognizer.merge_adjacent_text_entities(results, text, entity_types)— merges adjacent same-type spans separated only by whitespace (e.g. spaCy detecting "Dave" and "Jones" as two PERSON spans instead of one).Wired into
AnalyzerEngine.analyze()as a new opt-in parameter,merge_adjacent_entities, defaulting toNone(off). Callers pass the specific entity types they want merged (e.g.["PERSON"]); everything else is untouched.Why opt-in and type-scoped: An earlier attempt at this (#2046) applied merging unconditionally to every recognizer's output, which would silently fuse genuinely distinct adjacent matches for pattern-based recognizers (e.g. two phone numbers or emails on consecutive lines). Scoping to explicit entity types avoids that failure mode while still solving the NER-fragmentation problem.
Details:
max(score_a, score_b), and keepsanalysis_explanation/recognition_metadatafrom whichever span contributed that winning score.nxt.start >= current.endbefore checking the gap, so overlapping spans are never merged.entity_types) leaves results untouched, overlapping spans not merged, unsorted input, empty input. 31/31 passing.Issue reference
Fixes #1090
Checklist