Design tokenization

[turbopape]

No description provided.

[perkss]

Has any work been done on this?

[turbopape]

Hey , No not really... Would you be interested ? Le mar. 7 nov. 2017 à 2:44 PM, Stuart <[email protected]> a écrit :

Has any work been done on this? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACpoH8c9o2CDQTclnkTix4Hp6XaCmavzks5s0F7LgaJpZM4Mb30g> .

-- [image: --] Rafik Naccache [image: https://]about.me/rafik_naccache <https://about.me/rafik_naccache?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>

[perkss]

Yes I would be, lets move this forward let me know some initial thoughts for this? Could just tokenize simply on grammer to begin with

[turbopape]

Yeah sure, please go ahead with this, no need for fancy hyphenation etc for starters :) Thank you ! 2017-11-07 21:20 GMT+01:00 Stuart <[email protected]>:

…

Yes I would be, lets move this forward let me know some initial thoughts for this? Could just tokenize simply on grammer to begin with — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACpoH8qLFClsiNYVYPrZGt5B4HU17Fc5ks5s0LuigaJpZM4Mb30g> .

-- [image: --] Rafik Naccache [image: https://]about.me/rafik_naccache <https://about.me/rafik_naccache?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>