Rebuilt docs

Author: x-tabdeveloping

docs/_build/doctrees/environment.pickle

@@ -5,3 +5,6 @@ API reference
 
 .. automodule:: neofuzz.process
     :members:
+
+.. automodule:: neofuzz.tokenization
+    :members:

docs/_build/doctrees/metadata.doctree

@@ -7,6 +7,7 @@ Sometimes it is, however beneficial to be able to access metadata about the entr
 The most sensible way to handle this is to store your metadata in a table that is in the same order as the corpus.
 
 .. code-block:: python
+
    import pandas as pd
 
    corpus: list[str] = [...]
@@ -19,6 +20,7 @@ The most sensible way to handle this is to store your metadata in a table that i
 Then you can use the query() method to retrieve indices and distances instead of passages:
 
 .. code-block:: python
+
    from neofuzz import Process
  
    process = Process(...)

docs/_build/doctrees/persistence.doctree

@@ -7,6 +7,7 @@ Neofuzz can serialize indexed Process objects for you using `joblib`.
 You can save indexed processes like so:
 
 .. code-block:: python
+
    from neofuzz import char_ngram_process
    from neofuzz.tokenization import SubWordVectorizer
  
@@ -19,6 +20,7 @@ You can save indexed processes like so:
 And then load them in a production environment:
 
 .. code-block:: python
+
    from neofuzz import Process
  
    process = Process.from_disk("process.joblib")

docs/_build/html/_sources/api_reference.rst.txt

@@ -216,7 +216,32 @@ <h1>Handling Metadata<a class="headerlink" href="#handling-metadata" title="Perm
 <p>Neofuzz makes it easy to do fuzzy search in text corpora.
 Sometimes it is, however beneficial to be able to access metadata about the entries retrieved in fuzzy search.</p>
 <p>The most sensible way to handle this is to store your metadata in a table that is in the same order as the corpus.</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
+
+<span class="n">corpus</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
+<span class="n">metadata</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
+
+<span class="c1"># The tenth element in both corresponds to the same entry</span>
+<span class="n">tenth_text</span> <span class="o">=</span> <span class="n">corpus</span><span class="p">[</span><span class="mi">9</span><span class="p">]</span>
+<span class="n">tenth_metadata_entry</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="mi">9</span><span class="p">]</span>
+</pre></div>
+</div>
 <p>Then you can use the query() method to retrieve indices and distances instead of passages:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">neofuzz</span> <span class="kn">import</span> <span class="n">Process</span>
+
+<span class="n">process</span> <span class="o">=</span> <span class="n">Process</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
+<span class="n">process</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
+
+<span class="c1"># Both results will be arrays shaped (len(search_terms), limit)</span>
+<span class="n">indices</span><span class="p">,</span> <span class="n">distances</span> <span class="o">=</span> <span class="n">process</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">search_terms</span><span class="o">=</span><span class="p">[</span><span class="s2">&quot;Search term 1&quot;</span><span class="p">,</span> <span class="s2">&quot;Search term 2&quot;</span><span class="p">],</span> <span class="n">limit</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
+
+<span class="n">results_for_term1</span> <span class="o">=</span> <span class="p">[</span><span class="n">corpus</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>
+<span class="n">metadata_for_term1</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="n">indices</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>
+
+<span class="n">results_for_term2</span> <span class="o">=</span> <span class="p">[</span><span class="n">corpus</span><span class="p">[</span><span class="n">idx</span><span class="p">]</span> <span class="k">for</span> <span class="n">idx</span> <span class="ow">in</span> <span class="n">indices</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span>
+<span class="n">metadata_for_term2</span> <span class="o">=</span> <span class="n">metadata</span><span class="o">.</span><span class="n">iloc</span><span class="p">[</span><span class="n">indices</span><span class="p">[</span><span class="mi">1</span><span class="p">]]</span>
+</pre></div>
+</div>
 </section>
 
         </article>

docs/_build/html/_sources/metadata.rst.txt

@@ -216,7 +216,21 @@ <h1>Persistence<a class="headerlink" href="#persistence" title="Permalink to thi
 <p>You might want to persist processes to disk and reuses them in production pipelines.
 Neofuzz can serialize indexed Process objects for you using <cite>joblib</cite>.</p>
 <p>You can save indexed processes like so:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">neofuzz</span> <span class="kn">import</span> <span class="n">char_ngram_process</span>
+<span class="kn">from</span> <span class="nn">neofuzz.tokenization</span> <span class="kn">import</span> <span class="n">SubWordVectorizer</span>
+
+<span class="n">process</span> <span class="o">=</span> <span class="n">char_ngram_process</span><span class="p">()</span>
+<span class="n">process</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="n">corpus</span><span class="p">)</span>
+
+<span class="n">process</span><span class="o">.</span><span class="n">to_disk</span><span class="p">(</span><span class="s2">&quot;process.joblib&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
 <p>And then load them in a production environment:</p>
+<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">neofuzz</span> <span class="kn">import</span> <span class="n">Process</span>
+
+<span class="n">process</span> <span class="o">=</span> <span class="n">Process</span><span class="o">.</span><span class="n">from_disk</span><span class="p">(</span><span class="s2">&quot;process.joblib&quot;</span><span class="p">)</span>
+</pre></div>
+</div>
 </section>
 
         </article>

docs/_build/html/_sources/persistence.rst.txt

@@ -7,6 +7,7 @@ Sometimes it is, however beneficial to be able to access metadata about the entr
 The most sensible way to handle this is to store your metadata in a table that is in the same order as the corpus.
 
 .. code-block:: python
+
    import pandas as pd
 
    corpus: list[str] = [...]
@@ -19,6 +20,7 @@ The most sensible way to handle this is to store your metadata in a table that i
 Then you can use the query() method to retrieve indices and distances instead of passages:
 
 .. code-block:: python
+
    from neofuzz import Process
  
    process = Process(...)