Add changes for 13a4416

Contribution/Development.html

@@ -181,7 +181,7 @@ <h1>MetPX-Sarracenia Developer’s Guide<a class="headerlink" href="#metpx-sarra
 <dd class="field-odd"><p>UNKNOWN</p>
 </dd>
 <dt class="field-even">date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 </dl>
 <section id="tools-you-need">

Contribution/man_page_template.html

@@ -131,7 +131,7 @@ <h2>sr_title<a class="headerlink" href="#sr-title" title="Link to this heading">
 <dd class="field-odd"><p>1</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Explanation/CommandLineGuide.html

@@ -1092,6 +1092,17 @@ <h2>POLLING<a class="headerlink" href="#polling" title="Link to this heading">
 post per file. The file’s size is taken from the directory “ls”… but its
 checksum cannot be determined, so the default identity method is “cod”, asking
 clients to calculate the identity Checksum On Download.</p>
+<p>To set when to poll, use the <em>scheduled_interval</em> or <em>scheduled_hour</em> and <em>scheduled_minute</em>
+settings. for example:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scheduled_interval</span> <span class="mi">30</span><span class="n">m</span>
+</pre></div>
+</div>
+<p>to poll the remote resources every thirty minutes. Alternatively:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scheduled_hour</span> <span class="mi">1</span><span class="p">,</span><span class="mi">13</span><span class="p">,</span><span class="mi">19</span>
+<span class="n">scheduled_minute</span> <span class="mi">27</span>
+</pre></div>
+</div>
+<p>specifies that poll be run at 1:27, 13:27, and 19:27 each day.</p>
 <p>By default, sr_poll sends its post notification message to the broker with default exchange
 (the prefix <em>xs_</em> followed by the broker username). The <em>post_broker</em> is mandatory.
 It can be given incomplete if it is well defined in the credentials.conf file.</p>
@@ -1232,7 +1243,8 @@ <h3>POLL<a class="headerlink" href="#poll" title="Link to this heading"></a><
 present, modified or created in the remote directory, the program will
 notify about the new product.</p>
 <p>The notification protocol is defined here <a class="reference external" href="../Reference/sr_post.7.html">sr_post(7)</a></p>
-<p><strong>poll</strong> connects to a <em>broker</em>.  Every <em>sleep</em> seconds, it connects to
+<p><strong>poll</strong> connects to a <em>broker</em>.  Every <em>scheduled_interval</em> seconds (or can used
+combination of <em>scheduled_hour</em> and <em>scheduled_minute</em>) , it connects to
 a <em>pollUrl</em> (sftp, ftp, ftps). For each of the <em>directory</em> defined, it lists
 the contents.  Polling is only intended to be used for recently modified
 files. The <em>fileAgeMax</em> option eliminates files that are too old

Explanation/SarraPluginDev.html

@@ -151,7 +151,7 @@ <h3>Revision Record<a class="headerlink" href="#revision-record" title="Link to
 <dd class="field-odd"><p>UNKNOWN</p>
 </dd>
 <dt class="field-even">date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 </dl>
 </section>
@@ -652,7 +652,10 @@ <h3>Flow Callback Points<a class="headerlink" href="#flow-callback-points" title
 </tr>
 <tr class="row-odd"><td><p>gather(self)</p></td>
 <td><p>gather messages from a source, returns a list of
-messages.</p></td>
+messages.
+can also return a tuple where the first element
+is a boolean flag keep_going indicating whether
+to stop gather processing.</p></td>
 </tr>
 <tr class="row-even"><td><p>on_housekeeping
 (self)</p></td>

How2Guides/Email_Ingesting_With_Sarracenia.html

@@ -167,7 +167,7 @@ <h3>Extending Polling Protocols<a class="headerlink" href="#extending-polling-pr
 post_broker amqp://tsource@${FLOWBROKER}
 post_exchange xs_tsource
 
-sleep 60
+scheduled_interval 60
 
 pollUrl &lt;scheme&gt;://&lt;user&gt;@&lt;host&gt;:&lt;port&gt;/
 

How2Guides/FlowCallbacks.html

@@ -320,7 +320,11 @@ <h2>Entry Points<a class="headerlink" href="#entry-points" title="Link to this h
 
 
  <span class="k">def</span> <span class="nf">gather</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
-     <span class="n">Task</span><span class="p">:</span> <span class="n">gather</span> <span class="n">notification</span> <span class="n">messages</span> <span class="kn">from</span> <span class="nn">a</span> <span class="n">source</span><span class="o">...</span> <span class="k">return</span> <span class="n">a</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">notification</span> <span class="n">messages</span><span class="o">.</span>
+     <span class="n">Task</span><span class="p">:</span> <span class="n">gather</span> <span class="n">notification</span> <span class="n">messages</span> <span class="kn">from</span> <span class="nn">a</span> <span class="n">source</span><span class="o">...</span> <span class="k">return</span> <span class="n">either</span><span class="p">:</span>
+           <span class="o">*</span> <span class="n">a</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">notification</span> <span class="n">messages</span><span class="p">,</span> <span class="ow">or</span>
+           <span class="o">*</span> <span class="n">a</span> <span class="nb">tuple</span><span class="p">,</span> <span class="p">(</span><span class="nb">bool</span><span class="p">:</span><span class="n">keep_going</span><span class="p">,</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">messages</span><span class="p">)</span>
+           <span class="o">*</span> <span class="n">to</span> <span class="n">curtail</span> <span class="n">further</span> <span class="n">gathers</span> <span class="ow">in</span> <span class="n">this</span> <span class="n">cycle</span><span class="o">.</span>
+
      <span class="k">return</span> <span class="p">[]</span>
 
  <span class="k">def</span> <span class="nf">metrics_report</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>

How2Guides/Hydro_Examples.html

@@ -161,7 +161,7 @@ <h3>NOAA CO-OPS API<a class="headerlink" href="#noaa-co-ops-api" title="Link to
 and Currents website</a>. For example, if you want to access the
 water temperature data from the last hour in Honolulu, you can navigate to <cite>https://tidesandcurrents.noaa.gov/api/datagetter?range=1&amp;station=1612340&amp;product=water_temperature&amp;units=metric&amp;time_zone=gmt&amp;application=web_services&amp;format=csv</cite>.
 A new observation gets recorded every six minutes, so if you wanted to advertise solely new data through
-Sarracenia, you would configure an sr_poll instance to connect to the API, sleep every hour, and build
+Sarracenia, you would configure an sr_poll instance to connect to the API, set a one hour <em>scheduled_interval</em> , and build
 it a GET request to announce every time it woke up (this operates under the potentially misguided assumption
 that the data source is maintaining their end of the bargain). To download this shiny new file, you would connect
 an sr_subscribe to the same exchange it got announced on, and it would retrieve the URL, which a <em>do_download</em>

How2Guides/source.html

@@ -145,7 +145,7 @@ <h3>Revision Record<a class="headerlink" href="#revision-record" title="Link to
 <dd class="field-odd"><p>UNKNOWN</p>
 </dd>
 <dt class="field-even">date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 </dl>
 <p>A Sarracenia data pump is a web (or sftp) server with notifications for subscribers

How2Guides/subscriber.html

@@ -143,7 +143,7 @@ <h3>Revision Record<a class="headerlink" href="#revision-record" title="Link to
 <dd class="field-odd"><p>UNKNOWN</p>
 </dd>
 <dt class="field-even">date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 </dl>
 </section>

Reference/code.html

@@ -1696,13 +1696,6 @@
 <dd><p>The cfg is should be an sarra/config object.</p>
 </dd></dl>
 
-<dl class="py method">
-<dt class="sig sig-object py" id="sarracenia.flow.poll.Poll.do">
-<span class="sig-name descname"><span class="pre">do</span></span><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="../_modules/sarracenia/flow/poll.html#Poll.do"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#sarracenia.flow.poll.Poll.do" title="Link to this definition"></a></dt>
-<dd><p>stub to do the work: does nothing, marking everything done.
-to be replaced in child classes that do transforms or transfers.</p>
-</dd></dl>
-
 </dd></dl>
 
 </section>

Reference/flowcb.html

@@ -464,13 +464,22 @@
 <div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Task</span><span class="p">:</span> <span class="n">acknowledge</span> <span class="n">messages</span> <span class="kn">from</span> <span class="nn">a</span> <span class="n">gather</span> <span class="n">source</span><span class="o">.</span>
 </pre></div>
 </div>
-<p>def gather(self, messageCountMax) -&gt; list:</p>
-<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Task</span><span class="p">:</span> <span class="n">gather</span> <span class="n">messages</span> <span class="kn">from</span> <span class="nn">a</span> <span class="n">source</span><span class="o">...</span> <span class="k">return</span> <span class="n">a</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">messages</span>
+<p>def gather(self, messageCountMax) -&gt; (gather_more, messages)</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Task</span><span class="p">:</span> <span class="n">gather</span> <span class="n">messages</span> <span class="kn">from</span> <span class="nn">a</span> <span class="n">source</span><span class="o">...</span> <span class="k">return</span> <span class="n">a</span> <span class="nb">tuple</span><span class="p">:</span>
 
-      <span class="ow">in</span> <span class="n">a</span> <span class="n">poll</span><span class="p">,</span> <span class="n">gather</span> <span class="ow">is</span> <span class="n">always</span> <span class="n">called</span><span class="p">,</span> <span class="n">regardless</span> <span class="n">of</span> <span class="n">vip</span> <span class="n">posession</span><span class="o">.</span>
-      <span class="ow">in</span> <span class="nb">all</span> <span class="n">other</span> <span class="n">components</span><span class="p">,</span> <span class="n">gather</span> <span class="ow">is</span> <span class="n">only</span> <span class="n">called</span> <span class="n">when</span> <span class="ow">in</span> <span class="n">posession</span>
+      <span class="o">*</span> <span class="n">gather_more</span> <span class="o">...</span> <span class="nb">bool</span> <span class="n">whether</span> <span class="n">to</span> <span class="k">continue</span> <span class="n">gathering</span>
+      <span class="o">*</span> <span class="n">messages</span> <span class="o">...</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">messages</span>
+
+      <span class="ow">or</span> <span class="n">just</span> <span class="k">return</span> <span class="n">a</span> <span class="nb">list</span> <span class="n">of</span> <span class="n">messages</span><span class="o">.</span>
+
+      <span class="n">In</span> <span class="n">a</span> <span class="n">poll</span><span class="p">,</span> <span class="n">gather</span> <span class="ow">is</span> <span class="n">always</span> <span class="n">called</span><span class="p">,</span> <span class="n">regardless</span> <span class="n">of</span> <span class="n">vip</span> <span class="n">posession</span><span class="o">.</span>
+
+      <span class="n">In</span> <span class="nb">all</span> <span class="n">other</span> <span class="n">components</span><span class="p">,</span> <span class="n">gather</span> <span class="ow">is</span> <span class="n">only</span> <span class="n">called</span> <span class="n">when</span> <span class="ow">in</span> <span class="n">posession</span>
       <span class="n">of</span> <span class="n">the</span> <span class="n">vip</span><span class="o">.</span>
-<span class="k">return</span> <span class="p">[]</span>
+
+<span class="k">return</span> <span class="p">(</span><span class="kc">True</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
+ <span class="n">OR</span>
+<span class="k">return</span> <span class="nb">list</span>
 </pre></div>
 </div>
 <p>def after_accept(self,worklist) -&gt; None:</p>
@@ -1607,7 +1616,14 @@
 <dl class="py method">
 <dt class="sig sig-object py" id="sarracenia.flowcb.gather.message.Message.gather">
 <span class="sig-name descname"><span class="pre">gather</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">messageCountMax</span></span></em><span class="sig-paren">)</span> <span class="sig-return"><span class="sig-return-icon">&#x2192;</span> <span class="sig-return-typehint"><span class="pre">list</span></span></span><a class="reference internal" href="../_modules/sarracenia/flowcb/gather/message.html#Message.gather"><span class="viewcode-link"><span class="pre">[source]</span></span></a><a class="headerlink" href="#sarracenia.flowcb.gather.message.Message.gather" title="Link to this definition"></a></dt>
-<dd><p>return a current list of messages.</p>
+<dd><dl class="field-list simple">
+<dt class="field-odd">Returns<span class="colon">:</span></dt>
+<dd class="field-odd"><p>a list of messages obtained from this source.</p>
+</dd>
+<dt class="field-even">Return type<span class="colon">:</span></dt>
+<dd class="field-even"><p>True … you can gather from other sources. and</p>
+</dd>
+</dl>
 </dd></dl>
 
 </dd></dl>

Reference/sr3.1.html

@@ -130,7 +130,7 @@ <h2>sr3 Sarracenia CLI<a class="headerlink" href="#sr3-sarracenia-cli" title="Li
 <dd class="field-odd"><p>1</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Reference/sr3_cpump.1.html

@@ -127,7 +127,7 @@ <h2>sr_shovel in C<a class="headerlink" href="#sr-shovel-in-c" title="Link to th
 <dd class="field-odd"><p>1</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Reference/sr3_credentials.7.html

@@ -124,7 +124,7 @@ <h2>SR3 Credential File Format<a class="headerlink" href="#sr3-credential-file-f
 <dd class="field-odd"><p>7</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Reference/sr3_options.7.html

@@ -128,7 +128,7 @@ <h2>SR3 Configuration File Format<a class="headerlink" href="#sr3-configuration-
 <dd class="field-odd"><p>7</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>
@@ -1477,6 +1477,23 @@ <h4>sanity_log_dead &lt;interval&gt; (default: 1.5*housekeeping)<a class="header
 <p>The <strong>sanity_log_dead</strong> option sets how long to consider too long before restarting
 a component.</p>
 </section>
+<section id="scheduled-interval-scheduled-hour-scheduled-minute">
+<h4>scheduled_interval,scheduled_hour,scheduled_minute<a class="headerlink" href="#scheduled-interval-scheduled-hour-scheduled-minute" title="Link to this heading"></a></h4>
+<p>When working with scheduled flows, such as polls, one can configure a duration
+(no units defaults to seconds, suffixes: m-minute, h-hour) at which to run a
+given activity:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scheduled_interval</span> <span class="mi">30</span>
+</pre></div>
+</div>
+<p>run the flow or poll every 30 seconds.  If no duration is set, then the
+flowcb.scheduled.Scheduled class will look for the other two time specifiers:</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">scheduled_hour</span> <span class="mi">1</span><span class="p">,</span><span class="mi">4</span><span class="p">,</span><span class="mi">5</span><span class="p">,</span><span class="mi">23</span>
+<span class="n">scheduled_minute</span> <span class="mi">14</span><span class="p">,</span><span class="mi">17</span><span class="p">,</span><span class="mi">29</span>
+</pre></div>
+</div>
+<p>which will have the poll run each day at: 01:14, 01:17, 01:29, then the same minutes
+after each of 4h, 5h and 23h.</p>
+</section>
 <section id="shim-defer-posting-to-exit-experimental">
 <h4>shim_defer_posting_to_exit (EXPERIMENTAL)<a class="headerlink" href="#shim-defer-posting-to-exit-experimental" title="Link to this heading"></a></h4>
 <blockquote>
@@ -1512,10 +1529,12 @@ <h4>shim_skip_parent_open_files (EXPERIMENTAL)<a class="headerlink" href="#shim-
 </section>
 <section id="sleep-time">
 <h4>sleep &lt;time&gt;<a class="headerlink" href="#sleep-time" title="Link to this heading"></a></h4>
-<p>The time to wait between generating events.  When files are written frequently, it is counter productive
+<p>The time to wait between generating events. When files are written frequently, it is counter productive
 to produce a post for every change, as it can produce a continuous stream of changes where the transfers
-cannot be done quickly enough to keep up.  In such circumstances, one can group all changes made to a file
+cannot be done quickly enough to keep up. In such circumstances, one can group all changes made to a file
 in <em>sleep</em> time, and produce a single post.</p>
+<p>When sleep is set &gt; 0 for use with a <em>poll</em> it has the effect to setting <em>scheduled_interval</em>  to that value
+for compatibility reasons.  It is better for poll to use <em>scheduled</em> settings explicitly going forward.</p>
 </section>
 <section id="statehost-false-true-default-false">
 <h4>statehost &lt;False|True&gt; ( default: False )<a class="headerlink" href="#statehost-false-true-default-false" title="Link to this heading"></a></h4>

Reference/sr3_post.1.html

@@ -128,7 +128,7 @@ <h2>Publish the Availability of Files<a class="headerlink" href="#publish-the-av
 <dd class="field-odd"><p>1</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Reference/sr_post.7.html

@@ -133,7 +133,7 @@ <h2>Sarracenia v03 Notification Message Format/Protocol<a class="headerlink" hre
 <dd class="field-odd"><p>7</p>
 </dd>
 <dt class="field-even">Date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 <dt class="field-odd">Version<span class="colon">:</span></dt>
 <dd class="field-odd"><p>UNKNOWN</p>

Tutorials/Install.html

@@ -134,7 +134,7 @@ <h2>Revision Record<a class="headerlink" href="#revision-record" title="Link to
 <dd class="field-odd"><p>UNKNOWN</p>
 </dd>
 <dt class="field-even">date<span class="colon">:</span></dt>
-<dd class="field-even"><p>Mar 28, 2024</p>
+<dd class="field-even"><p>Apr 02, 2024</p>
 </dd>
 </dl>
 </section>