+
+Splitting sentences by concatenating over-split substrings based on their language
+powered by [`wtpsplit`](https://github.com/segment-any-text/wtpsplit) and language detection ([`fast-langdetect`](https://github.com/LlmKira/fast-langdetect) and [`langdetect`](https://github.com/Mimino666/langdetect))
+
+
+
+
+
+
[](https://badge.fury.io/py/split-lang)
[](https://pepy.tech/project/split-lang)
[](https://pepy.tech/project/split-lang)
-[](https://colab.research.google.com/github/DoodleBears/split-lang/blob/main/split-lang-demo.ipynb)
+[](https://colab.research.google.com/github/DoodleBears/split-lang/blob/main/split-lang-demo.ipynb)
-[](https://github.com/ellerbrock/open-source-badge/)
+[](https://github.com/DoodleBears/split-lang/blob/main/LICENSE)
[](https://wakatime.com/badge/user/5728d95a-5cfb-4acb-b600-e34c2fc231b6/project/e06e0a00-9ba1-453d-8c62-a0b2604aaaad)
-Splitting sentences by concatenating over-split substrings based on their language
+
+
-powered by [`wtpsplit`](https://github.com/segment-any-text/wtpsplit) and [`fast-langdetect`](https://github.com/LlmKira/fast-langdetect) and [`langdetect`](https://github.com/Mimino666/langdetect)
-## 1.1. Idea
+
+# 1.1. Idea
**Stage 1**: rule-based split using punctuation
- `hello, how are you` -> `hello` | `,` | `how are you`
@@ -41,8 +57,7 @@ powered by [`wtpsplit`](https://github.com/segment-any-text/wtpsplit) and [`fast
Vielen Dank merci beaucoup for your help.
```
-- [1. `split-lang`](#1-split-lang)
- - [1.1. Idea](#11-idea)
+- [1.1. Idea](#11-idea)
- [2. Motivation](#2-motivation)
- [3. Usage](#3-usage)
- [3.1. Installation](#31-installation)
@@ -51,6 +66,7 @@ Vielen Dank merci beaucoup for your help.
- [3.3. Advanced](#33-advanced)
- [3.3.1. `threshold`](#331-threshold)
- [3.3.2. usage of `lang_map` (for better result)](#332-usage-of-lang_map-for-better-result)
+- [Acknowledgement](#acknowledgement)
# 3. Usage
@@ -189,3 +205,9 @@ LANG_MAP = {
}
DEFAULT_LANG = "en"
```
+
+# Acknowledgement
+
+- Inspired by [LlmKira/fast-langdetect](https://github.com/LlmKira/fast-langdetect)
+- Text segmentation depends on [segment-any-text/wtpsplit](https://github.com/segment-any-text/wtpsplit)
+- Language detection depends on [zafercavdar/fasttext-langdetect](https://github.com/zafercavdar/fasttext-langdetect) and [Mimino666/langdetect](https://github.com/Mimino666/langdetect) (fix miss detecting Chinese as Korean in [DoodleBears/langdetect](https://github.com/DoodleBears/langdetect))
\ No newline at end of file
+
+