Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V03 issue677 #682

Closed
wants to merge 36 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b82c8d5
incrementing away from release.
petersilva May 14, 2023
a33b578
Making How2 section more legible.
petersilva May 14, 2023
a9b01f3
adding invocation of maintenance CI/CD action for #669
petersilva May 14, 2023
18e24b7
trying to get doc generation to use python3
petersilva May 14, 2023
042d379
moving scheduled around so it is alphabetic.
petersilva May 15, 2023
f9a71e2
trying to fix sphinx build/publish.
petersilva May 17, 2023
fb9dd04
improving Explanation index.
petersilva May 17, 2023
f9c0184
documentation tweaks to philosphy.
petersilva May 17, 2023
56aa92d
trying to fix github sphinx failure
petersilva May 17, 2023
ba621ce
reducing overwhelming tofc
petersilva May 17, 2023
52893e5
Update docs.yml
petersilva May 17, 2023
0c25416
suggested fix for sphinx doc processing from tomk
petersilva May 18, 2023
dd4cfc6
Merge branch 'v03_wip' of https://github.com/MetPX/Sarracenia into v0…
petersilva May 18, 2023
533071f
trying to fix sphinx doc generation.
petersilva May 18, 2023
8eb1957
re-arranging the index for more frequent first.
petersilva May 18, 2023
26df1ac
Update docs.yml
petersilva May 18, 2023
67b0c03
more misc. doc fixes (errors from make)
petersilva May 18, 2023
49ec880
fix docs CI
tomkralidis May 18, 2023
bdb66b9
some fixes from sphinx build errors.
petersilva May 18, 2023
a220db1
Merge branch 'v03_wip' of https://github.com/MetPX/Sarracenia into v0…
petersilva May 18, 2023
2d853bb
build docs on every commit.
petersilva May 18, 2023
3509b82
fixups based on sphinx error messages, also dia update for sr3
petersilva May 19, 2023
7f1b469
reverting all recent changes to docs.yml
petersilva May 19, 2023
0dc2399
more fixups for docs from sphinx build errors. also move action to 20.04
petersilva May 19, 2023
e8bc492
try sudo pip
petersilva May 19, 2023
dbbc458
trying to install paramiko only once
petersilva May 19, 2023
485c815
do not install sarracenia itself using root, only the deps.
petersilva May 19, 2023
2a462d5
trying again... which python is used? also 22.04
petersilva May 19, 2023
9a0723a
ugh... pip3 not pip.
petersilva May 19, 2023
1b03901
checkout ?
petersilva May 19, 2023
64cfaf3
do not install sarracenia package... will autodoc work anyways?
petersilva May 19, 2023
f72162b
ok paramiko error gone now, but version stuff still broken.
petersilva May 19, 2023
943b244
try again how many .. required?
petersilva May 19, 2023
809dd11
more random stuff.
petersilva May 19, 2023
3901f5b
attempting to fix doc generation
petersilva May 24, 2023
5536767
documentation correction for #677
petersilva May 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 17 additions & 8 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,33 +3,42 @@ on:
push:
branches:
- v03_wip
paths:
- 'docs/**'

jobs:
gh-pages:
name: Publish Github Pages
runs-on: ubuntu-latest
runs-on: ubuntu-20.04
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
permissions:
pages: write
id-token: write

steps:
- name: Setup locale
run: |
sudo locale-gen en_US.UTF-8
sudo update-locale
- name: Install dependencies
run: |
sudo apt-get update -y && sudo apt-get install -y pandoc dia python3-paramiko
sudo apt-get update -y && sudo apt-get install -y pandoc
- name: Setup python
uses: actions/setup-python@v3
- name: Checkout
uses: actions/checkout@master
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: install Sarracenia
run: |
pip install -r requirements.txt
pip install .
python3 -V
which python3
which pip3
which python
export PATH=${HOME}/.local/bin:${PATH}
echo `pwd`
- name: Build and Commit
uses: sphinx-notes/pages@v2
uses: sphinx-notes/pages@v3
with:
documentation_path: ./docs/source
requirements_path: ./requirements-dev.txt
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/flow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
runs-on: ${{ matrix.osver }}

name: ${{ matrix.which_test }} test on ${{ matrix.osver }}
timeout-minutes: 30
timeout-minutes: 40

steps:
- uses: actions/checkout@v3
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:

steps:
- name: Checkout branch
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v1
Expand Down
49 changes: 49 additions & 0 deletions .github/workflows/maint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
name: sr_insects test maintenance functions.

on:
pull_request:
types: [opened, edited, reopened]
push:
branches:
- v03_wip
- main

workflow_dispatch:
inputs:
debug_enabled:
type: boolean
description: 'Run the build with tmate debugging enabled (https://github.com/marketplace/actions/debugging-with-tmate)'
required: false
default: false

jobs:

run_sr_insects_tests:

strategy:
# Don't cancel the entire matrix when one job fails
fail-fast: false
matrix:
osver: [ "ubuntu-20.04", "ubuntu-22.04" ]

runs-on: ${{ matrix.osver }}

name: Maintenance test on ${{ matrix.osver }}
timeout-minutes: 30

steps:
- uses: actions/checkout@v3

- name: Install dependencies
run: |
travis/flow_autoconfig.sh
travis/ssh_localhost.sh

- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event.inputs.debug_enabled }}
#if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled }}

- name: Add and Remove configs,
run: |
cd ${HOME}/sr_insects/static_flow; ./flow_maint_test.sh
2 changes: 1 addition & 1 deletion .github/workflows/publish-to-pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:

steps:
- name: Checkout source
uses: actions/checkout@v2
uses: actions/checkout@v3

- name: Set up Python
uses: actions/setup-python@v3
Expand Down
6 changes: 6 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
metpx-sr3 (3.00.40) UNRELEASED; urgency=medium

* incrementing away from release.

-- Peter <[email protected]> Sun, 14 May 2023 11:01:45 -0400

metpx-sr3 (3.00.39) unstable; urgency=medium

* fix #667 had not been completely merged. adding.
Expand Down
11 changes: 11 additions & 0 deletions docs/source/Contribution/Philosophy/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@

Sarracenia Design Philosophy
============================

.. toctree::
:maxdepth: 1
:caption: Contents:

How to run the examples <README.html>
Amdahl applied <Amdahl_Applied.ipynb>
CAP Theorem <CAP_Theorem_Applied.ipynb>
10 changes: 5 additions & 5 deletions docs/source/Contribution/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,19 @@ Contributing to Sarracenia
==========================

.. toctree::
:maxdepth: 2
:maxdepth: 1
:caption: Contents:

AMQPprimer
deltas
Development
Documentation
Evolution
man_page_template.rst
mqtt_issues
v03
on_part_assembly
Evolution
Original (2015) Basic Idea <BasicIdea>
Original (2015) Design <Design>
v03
Amdahl Applied (philosophy)<Philosophy/Amdahl_Applied.ipynb>
deltas
Philosophy<Philosophy/index>

10 changes: 5 additions & 5 deletions docs/source/Explanation/CommandLineGuide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ the state of all components. It then makes the change requested.
**sr3** *options* *action* [ *component/configuration* ... ]

sr3 components are used to publish to and download files from websites or file servers
that provide `sr_post(7) <../Reference/sr_post.7.rst>`_ protocol notifications. Such sites
that provide `sr_post(7) <../Reference/sr_post.7.html>`_ protocol notifications. Such sites
publish notification messages for each file as soon as it is available. Clients connect to a
*broker* (often the same as the server itself) and subscribe to the notifications.
The *sr_post* notifications provide true push notices for web-accessible folders (WAF),
Expand All @@ -41,7 +41,7 @@ to reduce the size of individual components. The components are:
- winnow - copy notification messages, suppressing duplicates.

All of these components accept the same options, with the same effects.
There is also `sr3_cpump(1) <../Reference/sr3_cpump.1.rst>`_ which is a C version that implements a
There is also `sr3_cpump(1) <../Reference/sr3_cpump.1.html>`_ which is a C version that implements a
subset of the options here, but where they are implemented, they have the same effect.

The **sr3** command usually takes two arguments: an action followed by a list
Expand Down Expand Up @@ -939,7 +939,7 @@ By default, sr_poll sends its post notification message to the broker with defau
It can be given incomplete if it is well defined in the credentials.conf file.

Refer to `sr3_post(1) <../Reference/sr3_post.1.html>`_ - to understand the complete notification process.
Refer to `sr_post(7) <../Reference/sr_post.7.rst>`_ - to understand the complete notification format.
Refer to `sr_post(7) <../Reference/sr_post.7.html>`_ - to understand the complete notification format.


These options set what files the user wants to be notified for and where
Expand Down Expand Up @@ -1106,7 +1106,7 @@ check in various directories for some files. When a file is
present, modified or created in the remote directory, the program will
notify about the new product.

The notification protocol is defined here `sr_post(7) <../Reference/sr_post.7.rst>`_
The notification protocol is defined here `sr_post(7) <../Reference/sr_post.7.html>`_

**poll** connects to a *broker*. Every *sleep* seconds, it connects to
a *pollUrl* (sftp, ftp, ftps). For each of the *directory* defined, it lists
Expand Down Expand Up @@ -2187,7 +2187,7 @@ the network separately, and in parallel. When files change, transfers are
optimized by only sending parts which have changed.

The *outlet* option allows the final output to be other than a post.
See `sr3_cpump(1) <sr3_cpump.1.rst>`_ for details.
See `sr3_cpump(1) <sr3_cpump.1.html>`_ for details.

[-pbd|--post_baseDir <path>] (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
Binary file modified docs/source/Explanation/Concepts/AMQP4Sarra.dia
Binary file not shown.
46 changes: 29 additions & 17 deletions docs/source/Explanation/Concepts/AMQP4Sarra.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/source/Explanation/DetectFileReady.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ File Detection Strategy Table
|Alternate | - Adds delay in every transfer. |sources. |
|setting | - Vulnerable to network failures. | |
| | - Vulnerable to clock skew. |(ok choice with PDS) |
|nodupe_\ | | |
|nodupe\_\ | | |
|fileAgeMin | |If a process is re-writing a file |
| | |often, can use mtime to smooth out |
| | |the i/o pattern, by slowing posts. |
Expand Down
1 change: 0 additions & 1 deletion docs/source/Explanation/History/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,3 @@ History
messages_v02
messages_v03
sr3_Announcement
talks/SarraIntro/index
2 changes: 1 addition & 1 deletion docs/source/Explanation/History/mesh_gts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ In practice, without careful design & implementation, the performance can
devolve to N**2.

.. [2] Algorithmic complexity expresses how many computer operations are
needed to perform a function. For example, to search through a list of
needed to perform a function. For example, to search through a list of
N items sequentially would require a computer to perform N operations,
we then say the algorithm if on the order of N, or O(N). If one sorts
the list, a binary search takes only log N operations to find a
Expand Down
12 changes: 4 additions & 8 deletions docs/source/Explanation/Sundew_Migration/filter_conversion.rst
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
==========
MIGRATION
==========

---------------------------------------------
=============================================
Sundew filter migration to sarracenia (PXATX)
---------------------------------------------
=============================================

:Manual section: 1
:Date: @Date@
Expand Down Expand Up @@ -36,7 +32,7 @@ files into one product all of this into a sarra plugin...
**sarracenia** supports a lot of possible plugins in all of its programs.
If we take the simpler form of filter one to one, and we want to translated
this into a **sarra** process we need to receive the posting of a local product,
generate a resulting product and post the notification for it. It is easy to
generate a resulting product and post the notification for it. It is easy to
see that can be done using an **sr-sarra** process.


Expand Down Expand Up @@ -226,7 +222,7 @@ Examining **on_file_converted** you will find an after_accept function
that removes filter extensions from the filename. This was required because
old sundew clients needed to receive sarracenia converted products without
their specific extension name. When this is required, the **on_file_converted**
plugin can be added to the sender config. So example, a converted product
plugin can be added to the sender config. So example, a converted product
to PNG, in sarra would have a .png extension. Should it be required to send
it to a sundew client with option *filename NONE* without the plugin
the client would receive *WHATFN.png:...:...* with the plugin, it receives
Expand Down
4 changes: 2 additions & 2 deletions docs/source/Explanation/Sundew_Migration/index.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Explanation
===========
Sundew Migration Guide
======================

.. toctree::
:maxdepth: 2
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
==========
MIGRATION
==========

-------------------------------------------
===========================================
Sundew pull migration to sarracenia (PXATX)
-------------------------------------------
===========================================

:Manual section: 1
:Date: @Date@
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
==========
MIGRATION
==========

---------------------------------------------
=============================================
Sundew sender migration to sarracenia (PXATX)
---------------------------------------------
=============================================

:Manual section: 1
:Date: @Date@
Expand Down
Loading