Skip to content

Latest commit

 

History

History
350 lines (306 loc) · 9.53 KB

README.md

File metadata and controls

350 lines (306 loc) · 9.53 KB
Raw

CI codecov

razdel — rule-based system for Russian sentence and word tokenization.

Usage

>>> from razdel import tokenize

>>> tokens = list(tokenize('Кружка-термос на 0.5л (50/64 см³, 516;...)'))
>>> tokens
[Substring(0, 13, 'Кружка-термос'),
 Substring(14, 16, 'на'),
 Substring(17, 20, '0.5'),
 Substring(20, 21, 'л'),
 Substring(22, 23, '(')
 ...]
 
>>> [_.text for _ in tokens]
['Кружка-термос', 'на', '0.5', 'л', '(', '50/64', 'см³', ',', '516', ';', '...', ')']
>>> from razdel import sentenize

>>> text = '''
... - "Так в чем же дело?" - "Не ра-ду-ют".
... И т. д. и т. п. В общем, вся газета
... '''

>>> list(sentenize(text))
[Substring(1, 23, '- "Так в чем же дело?"'),
 Substring(24, 40, '- "Не ра-ду-ют".'),
 Substring(41, 56, 'И т. д. и т. п.'),
 Substring(57, 76, 'В общем, вся газета')]

Installation

razdel supports Python 3.5+ and PyPy 3.

$ pip install razdel

Documentation

Materials are in Russian:

Evaluation

Unfortunately, there is no single correct way to split text into sentences and tokens. For example, one may split «Как же так?! Захар...» — воскликнут Пронин. into three sentences ["«Как же так?!", "Захар...»", "— воскликнут Пронин."] while razdel splits it into two ["«Как же так?!", "Захар...» — воскликнут Пронин."]. What would be the correct way to tokenizer т.е.? One may split in into т.|е., razdel splits into т|.|е|..

razdel tries to mimic segmentation of these 4 datasets: SynTagRus, OpenCorpora, GICRYA and RNC. These datasets mainly consist of news and fiction. razdel rules are optimized for these kinds of texts. Library may perform worse on other domains like social media, scientific articles, legal documents.

We measure absolute number of errors. There are a lot of trivial cases in the tokenization task. For example, text чуть-чуть?! is not non-trivial, one may split it into чуть|-|чуть|?|! while the correct tokenization is чуть-чуть|?!, such examples are rare. Vast majority of cases are trivial, for example text в 5 часов ... is correctly tokenized even via Python native str.split into в| |5| |часов| |.... Due to the large number of trivial case overall quality of all segmenators is high, it is hard to compare differentiate between for examlpe 99.33%, 99.95% and 99.88%, so we report the absolute number of errors.

errors — number of errors per 1000 tokens/sentencies. For example, consider etalon segmentation is что-то|?, prediction is что|-|то?, then the number of errors is 3: 1 for missing split то? + 2 for extra splits что|-|то.

time — seconds taken to process whole dataset.

spacy_tokenize, aatimofeev and others a defined in naeval/segment/models.py, for links to models see Naeval registry. Tables are computed in naeval/segment/main.ipynb.

Tokens

corpora syntag gicrya rnc
errors time errors time errors time errors time
re.findall(\w+|\d+|\p+) 24 0.5 16 0.5 19 0.4 60 0.4
spacy 26 6.2 13 5.8 14 4.1 32 3.9
nltk.word_tokenize 60 3.4 256 3.3 75 2.7 199 2.9
mystem 23 5.0 15 4.7 19 3.7 14 3.9
mosestokenizer 11 2.1 8 1.9 15 1.6 16 1.7
segtok.word_tokenize 16 2.3 8 2.3 14 1.8 9 1.8
aatimofeev/spacy_russian_tokenizer 17 48.7 4 51.1 5 39.5 20 52.2
koziev/rutokenizer 15 1.1 8 1.0 23 0.8 68 0.9
razdel.tokenize 9 2.9 9 2.8 3 2.0 16 2.2

Sentences

corpora syntag gicrya rnc
errors time errors time errors time errors time
re.split([.?!…]) 114 0.9 53 0.6 63 0.7 130 1.0
segtok.split_single 106 17.8 36 13.4 1001 1.1 912 2.8
mosestokenizer 238 8.9 182 5.7 80 6.4 287 7.4
nltk.sent_tokenize 92 10.1 36 5.3 44 5.6 183 8.9
deeppavlov/rusenttokenize 57 10.9 10 7.9 56 6.8 119 7.0
razdel.sentenize 52 6.1 7 3.9 72 4.5 59 7.5

Support

Development

Test:

pip install -e .
pip install -r requirements/ci.txt
make test
make int  # 2000 integration tests

Package:

make version
git push
git push --tags

make clean wheel upload

mystem errors on syntag:

# see naeval/data
cat syntag_tokens.txt | razdel-ctl sample 1000 | razdel-ctl gen | razdel-ctl diff --show moses_tokenize | less

Non-trivial token tests:

pv data/*_tokens.txt | razdel-ctl gen --recall | razdel-ctl diff space_tokenize > tests.txt
pv data/*_tokens.txt | razdel-ctl gen --precision | razdel-ctl diff re_tokenize >> tests.txt

Update integration tests:

cd razdel/tests/data/
pv sents.txt | razdel-ctl up sentenize > t; mv t sents.txt

razdel and moses diff:

cat data/*_tokens.txt | razdel-ctl sample 1000 | razdel-ctl gen | razdel-ctl up tokenize | razdel-ctl diff moses_tokenize | less

razdel performance:

cat data/*_tokens.txt | razdel-ctl sample 10000 | pv -l | razdel-ctl gen | razdel-ctl diff tokenize | wc -l