section_tokenization.md

Tokenization

Notes:

Tokenization

a book about information retrieval

↓

[a, book, about, information, retrieval]

"Information Retrieval": Also available as e-book!

↓

[Information Retrieval]? [Information, Retrieval]?

[e-book]? [eBook]? [e, book]?

Notes:

Challenge

Inverted index can only find exact tokens

Term	Doc IDs
book	#1, #2, #3
information	#1, #2, #3
retrieval	#1
search	#2

e-book will return no results!

Try in Elasticsearch.

Notes:

Challenge

• How can books find book?
• wi-fi ↔ wifi?
• ­ Jack's ↔ Jack?
• ­ MMT ↔ Multimediatechnology?
• ­ U.S.A. ↔ USA?
• ­ running ↔ run?

Notes: What are other examples?

Text analysis

• Analyze docs and query
• Add, remove, change terms

Try improved tokenization in Elasticsearch.

Notes:

Nomenclature

Token

• Character sequence, meaningful semantic unit
• No analysis yet
• the, routers, the

Term

• Index tokens
• router