Merge pull request #1103 from MetPX/random_fixes

Random fixes

debian/changelog

@@ -1,17 +1,27 @@
 metpx-sr3 (3.00.54) UNRELEASED; urgency=medium
 
-  * merge PR #1067 (closing #824 )
+  * messages reviewed & consolidated: #1094, #1099 (de-cluttering.)
+  * add http metadata to scheduled flows. #1084
+  * only run after_post when actually posted #1101
+  * when plugins go bad, report better, recover better: #1085, #1091, 
+  * performance improvements #1083, #1086
+  * crashes/problems with statehost #1076, #1087, #1096
+  * fix #1104 message rate per cpu second and cpuS state.
+  * fix #1097 better parsing of low fractional rates. 
+  * added logFormat option.
+  * sr3 status flow state detail improved.
     * adds lag,rtry,slow,reje states to status display.
     * adds checks for running process to cleanup, fail if running.
     * adds cleanup to remove, don't remove if cleanup fails.
     * adds progressive logs to transfers  (closing #966) 
-  * several fixes for sender crashes resulting from changes in try/except scope.
+  * several fixes for sender crashes resulting from changes in try/except
+    scope. #1091, #1095,  
   * add #1054 can now convert multiple configs, and overwrite (with --wololo)
   * fixed #1064 poll crash.
   * fixed #927 sanity not restarting crashed polls.
-  * bug fixes and unit tests for AM
+  * bug fixes and unit tests for AM ( #1036, #1074, #1078, #1079 )
   * many unit tests added, improved coverage (though still poor.)
-  * many other plugin improvements.
+  * many other improvements and fixes in core and plugins.
   * search function restored on web-site documentation.
 
  -- SSC-5CD2310S60 <[email protected]>  Fri, 17 May 2024 12:29:22 -0400

docs/source/Contribution/Development.rst

@@ -142,6 +142,15 @@ Both v2 and v3 are supported on the stable branch of sr_insects.  That branch sh
 used to support all development in both versions.... 
 
 
+Unit tests
+~~~~~~~~~~
+
+The tests/ sub-directory contains a woefully incomplete but growing set of unit tests
+using the *pytest* framework. These tests are only exercised on Ubuntu 22.04 at the moment.
+consult tests/README.md for how to run them. Unit tests passing (or a very good explanation
+of why they *temporarily* fail) should be another gate before merging to the main *development* branch.
+
+
 Local Installation
 ------------------
 
@@ -291,6 +300,14 @@ multipass launch -m 8G bionic
 '''
 
 can run developer tests as per multipass as described above.
+Need to edit:
+
+* requirements.txt (remove paramiko and watchdog)
+* setup.py (remove install_requires for paramiko and watchdog)
+
+These packages have comedically complex dependencies. Install them separately
+and they will be usable by sr3, but listing them as a requirement breaks things.
+
 
 Python Wheel
 ~~~~~~~~~~~~

docs/source/Reference/sr3_options.7.rst

@@ -1054,6 +1054,15 @@ other values: on_start, on_stop, post, gather, ... etc... It is comma separated,
 if the list starts with a plus sign (+) then the selected events are appended to current value.
 A minus signe (-) can be used to remove events from the set.
 
+LogFormat ( default: %(asctime)s [%(levelname)s] %(name)s %(funcName)s %(message)s )
+------------------------------------------------------------------------------------
+
+The *LogFormat* option is passed directly to python logging mechanisms and can be used
+to control what is written to log files.  The format is documented here:
+
+* https://docs.python.org/3/library/logging.html#logrecord-attributes
+
+
 logLevel ( default: info )
 --------------------------
 

docs/source/fr/Contribution/Développement.rst

@@ -127,6 +127,15 @@ Le référentiel sr_insects a sa propre base de données de problèmes, et le tr
 Les versions 2 et 3 sont prises en charge sur la branche principale de sr_insects. Cette branche devrait être
 utilisé pour prendre en charge tout le développement dans les deux versions...
 
+Tests unitaires
+~~~~~~~~~~
+
+Le sous-répertoire tests/ contient un ensemble malheureusement incomplet mais croissant de tests unitaires
+qui utilisent le framework *pytest*. Ces tests requiert Ubuntu 22.04 pour le moment.
+Consultez tests/README.md pour savoir comment les exécuter. Réussite des tests unitaires (ou une très bonne explication
+pourquoi ils échouent *temporairement*) devrait être une autre porte avant de fusionner avec la branche *developpement* principale.
+
+
 Installation locale
 -------------------
 

docs/source/fr/Reference/sr3_options.7.rst

@@ -1037,8 +1037,18 @@ messages de journal. Autres valeurs : on_start, on_stop, post, gather, ... etc..
 On peut débuter la valeur avec un plus (+) pour signifier un ajout au valeurs actuels.
 la valeur moins (-) signifie la soustraction des valeurs de l´ensemble actuel. 
 
+LogFormat ( default: %(asctime)s [%(levelname)s] %(name)s %(funcName)s %(message)s )
+------------------------------------------------------------------------------------
+
+L'option *LogFormat* est passée directement au mécanismes de contrôle des journalisation
+de python. Le format est documenté ici:
+
+* https://docs.python.org/fr/3/library/logging.html#logrecord-attributes
+
+
 logLevel ( défaut: info )
 -------------------------
+
 Niveau de journalisation exprimé par la journalisation de python. Les valeurs possibles sont :
 critical, error, info, warning, debug.
 

sarracenia/__init__.py

@@ -358,8 +358,8 @@ def durationToSeconds(str_value, default=None) -> float:
 
 
 known_report_codes = {
-    201:
-    "Download successful. (variations: Downloaded, Inserted, Published, Copied, or Linked)",
+    201: "Download successful. (variations: Downloaded, Inserted, Published, Copied, or Linked)",
+    202: "Accepted. mkdir skipped as it already exists", 
     203: "Non-Authoritative Information: transformed during download.",
     205:
     "Reset Content: truncated. File is shorter than originally expected (changed length during transfer) This only arises during multi-part transfers.",
@@ -368,6 +368,7 @@ def durationToSeconds(str_value, default=None) -> float:
     "Not modified (Checksum validated, unchanged, so no download resulted.)",
     307: "Insertion deferred (writing to temporary part file for the moment.)",
     417: "Expectation Failed: invalid notification message (corrupt headers)",
+    422: "Unprocessable Content: could not determine path to transfer to",
     499: "Failure: Not Copied. SFTP/FTP/HTTP download problem",
     #FIXME : should  not have 503 error code 3 times in a row
     # 503: "Service unavailable. delete (File removal not currently supported.)",

sarracenia/config.py

@@ -1043,11 +1043,10 @@ def _parse_set_string( self, v:str, old_value: set ) -> set:
             if v == 'None': 
                 sv=set([])
             else:
-                if v[0] in [ '+', '-']:
+                op='r'
+                while v[0] in [ '+', '-']:
                     op=v[0]
                     v=v[1:]
-                else:
-                    op='r'
 
                 if ',' in v: 
                     sv=set(v.split(','))
@@ -1058,6 +1057,7 @@ def _parse_set_string( self, v:str, old_value: set ) -> set:
                     sv= old_value | sv
                 elif op == '-' :
                     sv= old_value - sv
+
         return sv
 
     def add_option(self, option, kind='list', default_value=None, all_values=None ):

sarracenia/flowcb/accept/wmotypesuffix.py

@@ -43,6 +43,10 @@ def __find_type(self, TT):
 
     def after_accept(self, worklist):
         for message in worklist.incoming:
+
+            if 'fileOp' in message and 'directory' in message['fileOp']:
+                continue
+
             type_suffix = self.__find_type(message['new_file'][0:2])
             ## FIXME confused as to how this could ever be true since find_type never returns "UNKNOWN"
             #if type_suffix == 'UNKNOWN':
@@ -53,6 +57,13 @@ def after_accept(self, worklist):
                 continue
 
             message['new_file'] = message['new_file'] + type_suffix
+
             if 'rename' in message:
-                message['rename'] = message['rename'] + type_suffix
+                message['rename'] += type_suffix
+
+            if 'fileOp' in message and 'rename' in message['fileOp']:
+                message['fileOp']['rename'] += type_suffix
+
             # TODO else -> worklist.rejected.append(message) ?? should this be happening at any point?
+
+

sarracenia/sr.py

@@ -2817,7 +2817,8 @@ def convert1(self,cfg):
                         if 'none' in line[1].lower():
                             v=line[1]
                         else:
-                            line[1]= '+' + line[1]
+                            if line[1][0] not in ['+','-']:
+                                line[1]= '+' + line[1]
                             v=line[1]
 
                     if k == 'continue':