Add changes for 3211eff

Reference/code.html

@@ -947,7 +947,7 @@
 
 <dl class="py data">
 <dt class="sig sig-object py" id="sarracenia.config.str_options">
-<span class="sig-prename descclassname"><span class="pre">sarracenia.config.</span></span><span class="sig-name descname"><span class="pre">str_options</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">['action',</span> <span class="pre">'admin',</span> <span class="pre">'baseDir',</span> <span class="pre">'broker',</span> <span class="pre">'cluster',</span> <span class="pre">'directory',</span> <span class="pre">'exchange',</span> <span class="pre">'exchange_suffix',</span> <span class="pre">'feeder',</span> <span class="pre">'filename',</span> <span class="pre">'flatten',</span> <span class="pre">'flowMain',</span> <span class="pre">'header',</span> <span class="pre">'hostname',</span> <span class="pre">'identity',</span> <span class="pre">'inlineEncoding',</span> <span class="pre">'logLevel',</span> <span class="pre">'pollUrl',</span> <span class="pre">'post_baseUrl',</span> <span class="pre">'post_baseDir',</span> <span class="pre">'post_broker',</span> <span class="pre">'post_exchange',</span> <span class="pre">'post_exchangeSuffix',</span> <span class="pre">'post_format',</span> <span class="pre">'post_topic',</span> <span class="pre">'queueName',</span> <span class="pre">'sendTo',</span> <span class="pre">'rename',</span> <span class="pre">'report_exchange',</span> <span class="pre">'source',</span> <span class="pre">'strip',</span> <span class="pre">'timezone',</span> <span class="pre">'nodupe_ttl',</span> <span class="pre">'nodupe_driver',</span> <span class="pre">'nodupe_basis',</span> <span class="pre">'tlsRigour',</span> <span class="pre">'topic']</span></em><a class="headerlink" href="#sarracenia.config.str_options" title="Link to this definition"></a></dt>
+<span class="sig-prename descclassname"><span class="pre">sarracenia.config.</span></span><span class="sig-name descname"><span class="pre">str_options</span></span><em class="property"><span class="w"> </span><span class="p"><span class="pre">=</span></span><span class="w"> </span><span class="pre">['action',</span> <span class="pre">'admin',</span> <span class="pre">'baseDir',</span> <span class="pre">'broker',</span> <span class="pre">'cluster',</span> <span class="pre">'directory',</span> <span class="pre">'exchange',</span> <span class="pre">'exchange_suffix',</span> <span class="pre">'feeder',</span> <span class="pre">'filename',</span> <span class="pre">'flatten',</span> <span class="pre">'flowMain',</span> <span class="pre">'header',</span> <span class="pre">'hostname',</span> <span class="pre">'identity',</span> <span class="pre">'inlineEncoding',</span> <span class="pre">'logFormat',</span> <span class="pre">'logLevel',</span> <span class="pre">'pollUrl',</span> <span class="pre">'post_baseUrl',</span> <span class="pre">'post_baseDir',</span> <span class="pre">'post_broker',</span> <span class="pre">'post_exchange',</span> <span class="pre">'post_exchangeSuffix',</span> <span class="pre">'post_format',</span> <span class="pre">'post_topic',</span> <span class="pre">'queueName',</span> <span class="pre">'sendTo',</span> <span class="pre">'rename',</span> <span class="pre">'report_exchange',</span> <span class="pre">'source',</span> <span class="pre">'strip',</span> <span class="pre">'timezone',</span> <span class="pre">'nodupe_ttl',</span> <span class="pre">'nodupe_driver',</span> <span class="pre">'nodupe_basis',</span> <span class="pre">'tlsRigour',</span> <span class="pre">'topic']</span></em><a class="headerlink" href="#sarracenia.config.str_options" title="Link to this definition"></a></dt>
 <dd><p>for backward compatibility,</p>
 <p>convert some old plugins that are hard to get working with
 v2wrapper, into v3 plugin.</p>

Reference/sr3_options.7.html

@@ -1175,17 +1175,15 @@ <h4>nodupe_ttl &lt;off|on|999[smhdw]&gt;<a class="headerlink" href="#nodupe-ttl-
 a time will result in 300 seconds (or 5 minutes) being the expiry interval.</p>
 <p>Default value in a Poll is 8 hours, should be longer than nodupe_fileAgeMax to prevent
 re-ingesting files that have aged out of the duplicate suppression cache.</p>
-<p><strong>Use of the cache is incompatible with the default *parts 0* strategy</strong>, one must specify an
-alternate strategy.  One must use either a fixed blocksize, or always never partition files.
-One must avoid the dynamic algorithm that will change the partition size used as a file grows.</p>
 <p><strong>Note that the duplicate suppresion store is local to each instance</strong>. When N
 instances share a queue, the first time a posting is received, it could be
 picked by one instance, and if a duplicate one is received it would likely
 be picked up by another instance. <strong>For effective duplicate suppression with instances</strong>,
 one must <strong>deploy two layers of subscribers</strong>. Use
 a <strong>first layer of subscribers (shovels)</strong> with duplicate suppression turned
-off and output with <em>post_exchangeSplit</em>, which route notification messages by checksum to
-a <strong>second layer of subscibers (winnow) whose duplicate suppression caches are active.</strong></p>
+off and output with <em>post_exchangeSplit</em>, which route notification with the same checksum to
+the same member of a <strong>second layer of subscribers (winnow) whose duplicate suppression caches
+are active.</strong></p>
 </section>
 <section id="outlet-post-json-url-default-post">
 <h4>outlet post|json|url (default: post)<a class="headerlink" href="#outlet-post-json-url-default-post" title="Link to this heading"></a></h4>
@@ -1426,10 +1424,6 @@ <h4>realpathPost &lt;flag&gt; (Experimental)<a class="headerlink" href="#realpat
 given. This option also enforces traversing of symbolic links.</p>
 <p>This option is being used to investigate some use cases, and may disappear in future.</p>
 </section>
-<section id="sendto-url">
-<h4>sendTo &lt;url&gt;<a class="headerlink" href="#sendto-url" title="Link to this heading"></a></h4>
-<p>Specification of a remote resource to deliver to in a sender.</p>
-</section>
 <section id="rename-path">
 <h4>rename &lt;path&gt;<a class="headerlink" href="#rename-path" title="Link to this heading"></a></h4>
 <p>With the <em>rename</em> option, the user can suggest a destination path for its files. If the given
@@ -1564,6 +1558,31 @@ <h4>scheduled_interval,scheduled_hour,scheduled_minute<a class="headerlink" href
 <p>which will have the poll run each day at: 01:14, 01:17, 01:29, then the same minutes
 after each of 4h, 5h and 23h.</p>
 </section>
+<section id="sendto-url">
+<h4>sendTo &lt;url&gt;<a class="headerlink" href="#sendto-url" title="Link to this heading"></a></h4>
+<p>Specification of a remote resource to deliver to in a sender.</p>
+</section>
+<section id="set-developer">
+<h4>set (DEVELOPER)<a class="headerlink" href="#set-developer" title="Link to this heading"></a></h4>
+<p>The <em>set</em> option is used, usually by developers, to define settings
+for particular classes in the source code. the most prominent usage
+would be to set the logLevel higher for a particular class of interest.</p>
+<p>Use of this option is more easily done with the source code handy.
+an example:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">set</span> <span class="n">sarracenia</span><span class="o">.</span><span class="n">moth</span><span class="o">.</span><span class="n">amqp</span><span class="o">.</span><span class="n">AMQP</span><span class="o">.</span><span class="n">logLevel</span> <span class="n">debug</span>
+<span class="nb">set</span> <span class="n">sarracenia</span><span class="o">.</span><span class="n">moth</span><span class="o">.</span><span class="n">mqtt</span><span class="o">.</span><span class="n">MQTT</span><span class="o">.</span><span class="n">logLevel</span> <span class="n">debug</span>
+</pre></div>
+</div>
+<p>So <em>sarracenia.moth.amqp.AMQP</em> refers to the class to which the setting
+is applied. There is a <em>class AMQP</em> in the python file
+sarracenia/moth/amqp.py (relative to root of source.)</p>
+<p>The <em>logLevel</em> is the setting to applied but only within
+that class. The <em>set</em> option requires an implementation in the source
+code to implement it for each class.  All <em>flowcb</em>’s have the neeeded
+support. The <code class="docutils literal notranslate"><span class="pre">moth</span></code> and transfer classes have a specific implementation
+for logLevel.</p>
+<p>Other classes may be hit or miss in terms of implementing the <em>set</em> semantic.</p>
+</section>
 <section id="shim-defer-posting-to-exit-experimental">
 <h4>shim_defer_posting_to_exit (EXPERIMENTAL)<a class="headerlink" href="#shim-defer-posting-to-exit-experimental" title="Link to this heading"></a></h4>
 <blockquote>

_modules/sarracenia/config.html

@@ -197,6 +197,7 @@ <h1>Source code for sarracenia.config</h1><div class="highlight"><pre>
     <span class="s1">&#39;inline&#39;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
     <span class="s1">&#39;inlineOnly&#39;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
     <span class="s1">&#39;identity_method&#39;</span><span class="p">:</span> <span class="s1">&#39;sha512&#39;</span><span class="p">,</span>
+    <span class="s1">&#39;logFormat&#39;</span><span class="p">:</span> <span class="s1">&#39;</span><span class="si">%(asctime)s</span><span class="s1"> [</span><span class="si">%(levelname)s</span><span class="s1">] </span><span class="si">%(name)s</span><span class="s1"> </span><span class="si">%(funcName)s</span><span class="s1"> </span><span class="si">%(message)s</span><span class="s1">&#39;</span><span class="p">,</span>
     <span class="s1">&#39;logMetrics&#39;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
     <span class="s1">&#39;logStdout&#39;</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
     <span class="s1">&#39;metrics_writeInterval&#39;</span><span class="p">:</span> <span class="mi">5</span><span class="p">,</span>
@@ -276,7 +277,7 @@ <h1>Source code for sarracenia.config</h1><div class="highlight"><pre>
 <span class="n">str_options</span> <span class="o">=</span> <span class="p">[</span>
     <span class="s1">&#39;action&#39;</span><span class="p">,</span> <span class="s1">&#39;admin&#39;</span><span class="p">,</span> <span class="s1">&#39;baseDir&#39;</span><span class="p">,</span> <span class="s1">&#39;broker&#39;</span><span class="p">,</span> <span class="s1">&#39;cluster&#39;</span><span class="p">,</span> <span class="s1">&#39;directory&#39;</span><span class="p">,</span> <span class="s1">&#39;exchange&#39;</span><span class="p">,</span>
     <span class="s1">&#39;exchange_suffix&#39;</span><span class="p">,</span> <span class="s1">&#39;feeder&#39;</span><span class="p">,</span> <span class="s1">&#39;filename&#39;</span><span class="p">,</span> <span class="s1">&#39;flatten&#39;</span><span class="p">,</span> <span class="s1">&#39;flowMain&#39;</span><span class="p">,</span> <span class="s1">&#39;header&#39;</span><span class="p">,</span> 
-    <span class="s1">&#39;hostname&#39;</span><span class="p">,</span> <span class="s1">&#39;identity&#39;</span><span class="p">,</span> <span class="s1">&#39;inlineEncoding&#39;</span><span class="p">,</span> <span class="s1">&#39;logLevel&#39;</span><span class="p">,</span> 
+    <span class="s1">&#39;hostname&#39;</span><span class="p">,</span> <span class="s1">&#39;identity&#39;</span><span class="p">,</span> <span class="s1">&#39;inlineEncoding&#39;</span><span class="p">,</span> <span class="s1">&#39;logFormat&#39;</span><span class="p">,</span> <span class="s1">&#39;logLevel&#39;</span><span class="p">,</span> 
     <span class="s1">&#39;pollUrl&#39;</span><span class="p">,</span> <span class="s1">&#39;post_baseUrl&#39;</span><span class="p">,</span> <span class="s1">&#39;post_baseDir&#39;</span><span class="p">,</span> <span class="s1">&#39;post_broker&#39;</span><span class="p">,</span> <span class="s1">&#39;post_exchange&#39;</span><span class="p">,</span>
     <span class="s1">&#39;post_exchangeSuffix&#39;</span><span class="p">,</span> <span class="s1">&#39;post_format&#39;</span><span class="p">,</span> <span class="s1">&#39;post_topic&#39;</span><span class="p">,</span> <span class="s1">&#39;queueName&#39;</span><span class="p">,</span> <span class="s1">&#39;sendTo&#39;</span><span class="p">,</span> <span class="s1">&#39;rename&#39;</span><span class="p">,</span>
     <span class="s1">&#39;report_exchange&#39;</span><span class="p">,</span> <span class="s1">&#39;source&#39;</span><span class="p">,</span> <span class="s1">&#39;strip&#39;</span><span class="p">,</span> <span class="s1">&#39;timezone&#39;</span><span class="p">,</span> <span class="s1">&#39;nodupe_ttl&#39;</span><span class="p">,</span> <span class="s1">&#39;nodupe_driver&#39;</span><span class="p">,</span> 
@@ -3001,7 +3002,8 @@ <h1>Source code for sarracenia.config</h1><div class="highlight"><pre>
                 <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;Exception details:&quot;</span><span class="p">,</span> <span class="n">exc_info</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
                 <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span>
 
-        <span class="n">log_format</span> <span class="o">=</span> <span class="s1">&#39;</span><span class="si">%(asctime)s</span><span class="s1"> [</span><span class="si">%(levelname)s</span><span class="s1">] </span><span class="si">%(name)s</span><span class="s1"> </span><span class="si">%(funcName)s</span><span class="s1"> </span><span class="si">%(message)s</span><span class="s1">&#39;</span>
+        <span class="c1">#log_format = &#39;%(asctime)s [%(levelname)s] %(name)s %(funcName)s %(message)s&#39;</span>
+        <span class="n">log_format</span> <span class="o">=</span> <span class="n">cfg_preparse</span><span class="o">.</span><span class="n">logFormat</span>
         <span class="k">if</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span><span class="o">.</span><span class="n">hasHandlers</span><span class="p">():</span>
             <span class="k">for</span> <span class="n">h</span> <span class="ow">in</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">()</span><span class="o">.</span><span class="n">handlers</span><span class="p">:</span>
                 <span class="n">h</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>