v3.00.54rc1 Wololo! and Voluble Status

The Wololo! release speeds up the v2 to sr3 converter substantially, and added much more helpful flow states
to the sr3 status display. In place of just running or stopped, there is now: idle, lagging, slow, retry, rejecting, cpuSlow
indicating various ways flows could be in difficulty or require tuning. HTTP scheduled flows can now
query the resource using metadata calls for more effective polling. Many fixes for issues people ran into during deployments.
Also fixes for problems with statehost setting (important for HPC deployments.)

BUGS

• #1115 .. so there will be an rc2.

What's Changed

• Updates to NASA Auth plugins and examples by @reidsunderland in #1042
• Issue1036 - Add unit tests for AM code by @andreleblanc11 in #1044
• Fix sanity not starting missing instances by @reidsunderland in #1053
• Build Unit test boilerplate by @gcglinton in #1052
• multi-instances will be partial, not missing by @petersilva in #1059
• Dev doc refactor release instructions by @petersilva in #1057
• to convert overwriting, one must Wololo by @petersilva in #1055
• Fix #1008 and update documentation by @mshak2 in #1056
• Fix bug caused by different variable names by @reidsunderland in #1060
• Unit Test Engineering wmo00 by @petersilva in #1048
• Unit Test Engineering of the sarracenia/config.py module. by @petersilva in #1058
• Fixes search function and logo, and adds favicon for doc website by @mshak2 in #1062
• Fix #1064 by adding try/except by @reidsunderland in #1065
• #980 Expand scope of isProblem. by @andreleblanc11 in #1046
• #1036 Add more AM unit tests for latest bug findings by @andreleblanc11 in #1068
• Issue824 morestates and issue966 and... other stuff by @petersilva in #1067
• please stop infinite loop by calling self by @petersilva in #1070
• disabled configs were showing as "unknown" by @petersilva in #1073
• Fix #1074 by @andreleblanc11 in #1075
• Bug fix for AM sender by @andreleblanc11 in #1078
• Doc updates by @petersilva in #1072
• Fix mkdir failures on hpc by @petersilva in #1076
• Fix #1079 by @andreleblanc11 in #1081
• remove plugins that are useless and will not run. by @petersilva in #1082
• Add post_baseUrl to delete on post, changes to logging by @reidsunderland in #1050
• adding new_pathWrite() as per Reid suggestion by @petersilva in #1049
• Issue1080 2 by @petersilva in #1083
• Add scheduled flow with http metadata by @reidsunderland in #1084
• Issue1035 retry too much, reduce cpu to some extent... by @petersilva in #1086
• fix typo rel_relPath -> new_relPath by @reidsunderland in #1092
• Scheduled flow with HTTP metadata: add variable expansion for the path by @reidsunderland in #1088
• honour statehost in one mkdir case by @petersilva in #1087
• fix #1091 download and/or sender plugin error crashes by @petersilva in #1093
• Doc feedback by @petersilva in #1094
• issue #982 detail in cleanup was not working when statehost set by @petersilva in #1096
• more robust validation of return value from destfnscript by @petersilva in #1085
• Add directory to send, etc. errors by @reidsunderland in #1099
• only run after_post after we actually post by @petersilva in #1101
• fix #1095 crash in sender when size missing from message by @petersilva in #1102
• Issue1089 fileAgeMin and fileAgeMax should be honoured by all components by @petersilva in #1100
• Finalize Azure Transfer tests by @gcglinton in #1051
• fix crash where 'dictionary changed size during iteration' by @reidsunderland in #1109
• Change chmod -> utime, i think it was a typo? by @reidsunderland in #1108
• Issue1104 msg rate cpu by @petersilva in #1105
• Random fixes by @petersilva in #1103
• Restoring blockmanifest transfers by @petersilva in #1110
• Issue1097 fractional rates by @petersilva in #1106

Full Changelog: v3.00.53pre2...v3.00.54rc1

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.54rc1/
Note: uploaded pypi package from ubuntu 22.04... installation on older OS & python versions may
be compromised (error, will be corrected in next release.)

• (Coming Soon) RPM package for fedora core 39 (fc39)

• (Coming Soon) RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.

v3.0.53

What's new

• Introduces cloud transfer drivers S3 and Azure
• improved poll behaviour to consume messages when in passive (noVip) mode.
• better sender errors.
• various updates for WMO (WMO-00 support, WIS fixes.)

What's Changed

• Add topicPrefix to docs to fix #961 by @reidsunderland in #964
• Fixed "Downloading using CLI" tutorial page by @mshak2 in #962
• Remove threading from AMQP Consumer code by @reidsunderland in #965
• Run housekeeping during long sleeps by @reidsunderland in #969
• fix #967 define hostname as a string option, and stuff works by @petersilva in #970
• Add missing tokIsYear method from Sundew. Prevents some erronous bulletins not being filtered. by @andreleblanc11 in #973
• fix #971 Add missing to list of possible process status by @petersilva in #972
• Re-enable AMQP consumer flow tests by @reidsunderland in #977
• Reject float types for Julian day value in getTime by @andreleblanc11 in #979
• Housekeeping logging changes by @reidsunderland in #978
• Rename tests to have consistent naming by @reidsunderland in #981
• Add S3 Sender plugin by @reidsunderland in #983
• NASA PODAAC checksum change by @reidsunderland in #988
• Issue986 - stop converting directory to path in poll. by @petersilva in #987
• Issue992 - re-parse values using correct type for add_options from flow_callbacks. by @petersilva in #999
• No download message when not downloading by @petersilva in #998
• fix #982 by checking for statehost in cleanup by @petersilva in #996
• Issue994 unit tests broken and failing a lot (but weirdly not always.) by @petersilva in #995
• Clarify a bit more how each component works in the documentation by @andreleblanc11 in #993
• Issue997 - Eliminate retry file race condition and make AM work more like a sarracenia flow by @andreleblanc11 in #1005
• Issue974 yet another scheduled_poll refactor PR by @petersilva in #991
• Transfer documentation and a fix for all tests... by @petersilva in #1009
• #982 statehost config option should be used to see pick state directory by @petersilva in #1003
• Native S3 transfer driver by @gcglinton in #1010
• Make persistent a configurable option (sr3) by @reidsunderland in #1013
• Azure transfer driver by @gcglinton in #1019
• EUMETSAT poll plugin update by @reidsunderland in #1027
• implement topicCopy for #1023 by @petersilva in #1025
• fix #947, only report on .conf or .inc files by @petersilva in #1014
• Issue1022 by @petersilva in #1024
• Wis2 integration - MQTT publishing using sr3 stack by @mshak2 in #1026
• Issue1029 wmo00 accumulated file format support by @petersilva in #1030
• Issue908 - Improve general am/gather code and properly kill children zombies. by @andreleblanc11 in #1031
• flakey flow test failing when polls killed with -9 cannot remove novip state file by @petersilva in #1032
• Issue980 - Add verifyHeader method to prevent erronous headers from passing through by @andreleblanc11 in #1033
• updatePaths work without being supplied new_dir and new_path by @petersilva in #1034
• locally installed plugins are not selected because we prepend system … by @petersilva in #1038
• Exchange split override by @petersilva in #1041
• updatePaths no longer fails when called with just options by @petersilva in #1039
• Better sender io errors (download also...) by @petersilva in #1037

Full Changelog: v3.00.52...v3.0.53

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.53/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Enterprise Linux Version 8.9 (rhel8_9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.53pre2 - Cloud Transfers Enabled

• Introduces (experimental) S3 and Azure Transfer classes (native support for polling and transfer to and from cloud storage.) from (@gcglinton !). but also many other improvements.
• heavily re-factored and improved AM server.
• heavily re-factored poll
• improvements in callback api.

What's Changed

• Add topicPrefix to docs to fix #961 by @reidsunderland in #964
• Fixed "Downloading using CLI" tutorial page by @mshak2 in #962
• Remove threading from AMQP Consumer code by @reidsunderland in #965
• Run housekeeping during long sleeps by @reidsunderland in #969
• fix #967 define hostname as a string option, and stuff works by @petersilva in #970
• Add missing tokIsYear method from Sundew. Prevents some erronous bulletins not being filtered. by @andreleblanc11 in #973
• fix #971 Add missing to list of possible process status by @petersilva in #972
• Re-enable AMQP consumer flow tests by @reidsunderland in #977
• Reject float types for Julian day value in getTime by @andreleblanc11 in #979
• Housekeeping logging changes by @reidsunderland in #978
• Rename tests to have consistent naming by @reidsunderland in #981
• Add S3 Sender plugin by @reidsunderland in #983
• NASA PODAAC checksum change by @reidsunderland in #988
• Issue986 - stop converting directory to path in poll. by @petersilva in #987
• Issue992 - re-parse values using correct type for add_options from flow_callbacks. by @petersilva in #999
• No download message when not downloading by @petersilva in #998
• fix #982 by checking for statehost in cleanup by @petersilva in #996
• Issue994 unit tests broken and failing alot (but weirdly not always.) by @petersilva in #995
• Clarify a bit more how each component works in the documentation by @andreleblanc11 in #993
• Issue997 - Eliminate retry file race condition and make AM work more like a sarracenia flow by @andreleblanc11 in #1005
• Issue974 yet another scheduled_poll refactor PR by @petersilva in #991
• Transfer documentation and a fix for all tests... by @petersilva in #1009
• #982 statehost config option should be used to see pick state directory by @petersilva in #1003
• Native S3 transfer driver by @gcglinton in #1010
• Make persistent a configurable option (sr3) by @reidsunderland in #1013
• Azure transfer driver by @gcglinton in #1019
• EUMETSAT poll plugin update by @reidsunderland in #1027
• implement topicCopy for #1023 by @petersilva in #1025
• fix #947, only report on .conf or .inc files by @petersilva in #1014
• Issue1022 by @petersilva in #1024
• Wis2 integration - MQTT publishing using sr3 stack by @mshak2 in #1026
• Issue1029 wmo00 accumulated file format support by @petersilva in #1030
• Issue908 - Improve general am/gather code and properly kill children zombies. by @andreleblanc11 in #1031
• flakey flow test failing when polls killed with -9 cannot remove novip state file by @petersilva in #1032
• Issue980 - Add verifyHeader method to prevent erronous headers from passing through by @andreleblanc11 in #1033
• updatePaths work without being supplied new_dir and new_path by @petersilva in #1034
• locally installed plugins are not selected because we prepend system … by @petersilva in #1038
• Exchange split override by @petersilva in #1041
• updatePaths no longer fails when called with just options by @petersilva in #1039
• Better sender io errors (download also...) by @petersilva in #1037

Full Changelog: v3.00.52...v3.00.53pre2

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.53rc2/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Enterprise Linux Version 8.9 (rhel8_9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.53rc1 - S3 driver, AM server, Poll enhancements

• New (experimental) S3 Transfer Protocol driver.
• heavily re-factored and improved AM server.
• heavily re-factored poll

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

What's Changed

• Add topicPrefix to docs to fix #961 by @reidsunderland in #964
• Fixed "Downloading using CLI" tutorial page by @mshak2 in #962
• Remove threading from AMQP Consumer code by @reidsunderland in #965
• Run housekeeping during long sleeps by @reidsunderland in #969
• fix #967 define hostname as a string option, and stuff works by @petersilva in #970
• Add missing tokIsYear method from Sundew. Prevents some erronous bulletins not being filtered. by @andreleblanc11 in #973
• fix #971 Add missing to list of possible process status by @petersilva in #972
• Re-enable AMQP consumer flow tests by @reidsunderland in #977
• Reject float types for Julian day value in getTime by @andreleblanc11 in #979
• Housekeeping logging changes by @reidsunderland in #978
• Rename tests to have consistent naming by @reidsunderland in #981
• Add S3 Sender plugin by @reidsunderland in #983
• NASA PODAAC checksum change by @reidsunderland in #988
• Issue986 - stop converting directory to path in poll. by @petersilva in #987
• Issue992 - re-parse values using correct type for add_options from flow_callbacks. by @petersilva in #999
• No download message when not downloading by @petersilva in #998
• fix #982 by checking for statehost in cleanup by @petersilva in #996
• Issue994 unit tests broken and failing alot (but weirdly not always.) by @petersilva in #995
• Clarify a bit more how each component works in the documentation by @andreleblanc11 in #993
• Issue997 - Eliminate retry file race condition and make AM work more like a sarracenia flow by @andreleblanc11 in #1005
• Issue974 yet another scheduled_poll refactor PR by @petersilva in #991
• Transfer documentation and a fix for all tests... by @petersilva in #1009
• #982 statehost config option should be used to see pick state directory by @petersilva in #1003
• Native S3 transfer driver by @gcglinton in #1010
• Make persistent a configurable option (sr3) by @reidsunderland in #1013

Full Changelog: v3.00.52...v3.00.53rc1

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.53rc1/
Note: uploaded pypi package from ubuntu 22.04... installation on older OS & python versions may
be compromised (error, will be corrected in next release.)

• (Coming Soon) RPM package for fedora core 39 (fc39)

• (Coming Soon) RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.

v2.24.04 prevent message loss on broker restart

NOTE: version 2 stream is the legacy support version of Sarracenia. For any new work, please look at metpx-sr3 instead.

Release Notes

Primary rationale for this release is to make it easier to restart nodes in clusters. We noticed there is message loss on broker restart. Switching to persistent messaging by default with options to re-instate old behaviour. Likely to validate performance
by turning options on and off and comparing.

• from PR #1012 enable publishing of persistent messages (withstand broker restart), combination of: durable and persistent
• fixed msg_fdelay plugin to work as documented.
• part of #639 take root of paths from baseUrl (formerly ignored) compatibility with sr3 (not used in v2 until now.)
• part of #639 ignore mkdir and rmdir events without errors (unimplemented in v2)

Noticed that v2.22.10 was the last release publicized on github. There were interim releases on launchpad, which were
never publicized on github. Including those for completeness.

following changes from v2.22.10p1, v2.23.06, v2.23.07:

• fixes for #715 missing attribute failure to process message
• fix #703 replacing "integrity" with "identity" in v03 format (compatibility with sr3)
• from PR #655 graceful degradation when dateparser module missing.
• fix #645 dependency for systemd startup on network-online
• fix #382 sr cleanup to only remove unused exchanges.
• dd_subscribe removed (wasn't working, not really supported.)
• minor fixes.
• v2 fallback code so it partially works even in watchdog module is missing.

BUG: the version tag was not properly updated, so in pypi, this version is v2.23.12 If installing from launchpad,
the package will be externally named 2.24.04, but post installation, it will also claim to be v2.23.12.

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

Publishing only to launchpad pre-release repository for (tentatively) one week. Once the pre-release phase
is done, will upload to stable repo, and pypi.

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-release
python: https://pypi.org/project/metpx-sarracenia/

v3.00.52

What's Changed

• correct work description for different running components by @petersilva in #921
• fix #922 avoid mixing up post_exchange with exchange in display by @petersilva in #923
• changing defaults and documenting the consideration by @petersilva in #924
• fix #907 admin declare exchange by @petersilva in #930
• Update subscriber.rst by @petersilva in #929
• Issue920 fix inflight mtime not working. by @petersilva in #926
• Issue912 retry refilter flag by @petersilva in #928
• Issue920 cleanup and #912 add _isRetry by @petersilva in #932
• make rename field processing contingent on download option being set by @petersilva in #936
• Issue918 - Add plugin to rename AM bulletins properly by @andreleblanc11 in #933
• adding mention of scheduled flows as an option when migrating v2 polls by @petersilva in #938
• fix #940 brackets wrong by @petersilva in #941
• Issue942 - sarra consumer writing wrong files because 'rename' field should have been deleted. by @petersilva in #944
• Issue934 2 reducing references to v2 (sr_subscribe, sr_watch, etc...) in documentation. by @petersilva in #937
• Lots of AM corrections (Renamer + AHL corrections in gather) by @andreleblanc11 in #946
• Am final touches before first release by @andreleblanc11 in #949
• Edit for typos and clarity by @mshak2 in #950
• Move AM renamer to gather. Allows us to properly use sundew variable expansions. by @andreleblanc11 in #952
• Issue953 PRECONDITION_FAILED unknown delivery tag by @petersilva in #954
• Properly handle trailing spaces at end of bulletin headers by @andreleblanc11 in #956
• Change raise_signal to os.kill for #948 by @reidsunderland in #957
• Add msg['post_baseUrl'] processing for #951 by @reidsunderland in #958
• Add plugin to poll the Copernicus Marine Data Store service by @reidsunderland in #959
• fix #955 check cache dir exists before cleaning it. by @petersilva in #960
• Add fix to handle improperly parsed station names from getStation by @andreleblanc11 in #963

Full Changelog: v3.00.51...v3.00.52

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.52/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.

v3.00.52rc4

What's Changed

• Change raise_signal to os.kill for #948 by @reidsunderland in #957
• Add msg['post_baseUrl'] processing for #951 by @reidsunderland in #958
• Add plugin to poll the Copernicus Marine Data Store service by @reidsunderland in #959
• fix #955 check cache dir exists before cleaning it. by @petersilva in #960
• Add fix to handle unproperly parsed station names from getStation by @andreleblanc11 in #963

Full Changelog: v3.00.52rc3...v3.00.52rc4

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc4/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.52rc3

What's Changed

• Edit for typos and clarity by @mshak2 in #950
• Move AM renamer to gather. Allows us to properly use sundew variable expansions. by @andreleblanc11 in #952
• Issue953 PRECONDITION_FAILED unknown delivery tag by @petersilva in #954
• Properly handle trailing spaces at end of bulletin headers by @andreleblanc11 in #956

New Contributors

• @mshak2 made their first contribution in #950

Full Changelog: v3.00.52rc2...v3.00.52rc3

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc3/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.52rc2

What's Changed

• adding mention of scheduled flows as an option when migrating v2 polls by @petersilva in #938
• fix #940 brackets wrong by @petersilva in #941
• Issue942 - sarra consumer writing wrong files because 'rename' field should have been deleted. by @petersilva in #944
• Issue934 2 reducing references to v2 (sr_subscribe, sr_watch, etc...) in documentation. by @petersilva in #937
• Lots of AM corrections (Renamer + AHL corrections in gather) by @andreleblanc11 in #946
• Am final touches before first release by @andreleblanc11 in #949

Full Changelog: v3.0.52rc1...v3.00.52rc2

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc2/

• (Coming Soon) RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.0.52rc1 watch working with inflight times.

What's Changed

• correct work description for different running components by @petersilva in #921
• fix #922 avoid mixing up post_exchange with exchange in display by @petersilva in #923
• changing defaults and documenting the consideration by @petersilva in #924
• fix #907 admin declare exchange by @petersilva in #930
• Update subscriber.rst by @petersilva in #929
• Issue920 fix inflight mtime not working. by @petersilva in #926
• Issue912 retry refilter flag by @petersilva in #928
• Issue920 cleanup and #912 add _isRetry by @petersilva in #932
• make rename field processing contingent on download option being set by @petersilva in #936
• Issue918 - Add plugin to rename AM bulletins properly by @andreleblanc11 in #933

Full Changelog: v3.00.51rc7...v3.0.52rc1

Guide:

https://metpx.github.io/sarracenia/Tutorials/Install.html

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.52rc1/

• RPM package for fedora core 39 (fc39)

• RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily avialable. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.