V03 make *filetype* an sr3 extra feature, because of dependency issues.

docs/source/Contribution/Development.rst

@@ -137,16 +137,18 @@ Local Installation
 
 There are many different ways to install python packages on a computer. Different developers
 will prefer different methods, and all the methods need to be tested prior to each release.
+Sarracenia can work with either mqtt or amqp (most mature and stable) message passing libraries.
+Install one of those first. in these examples, we use amqp.
 
 * **Wheel** when people are running different operating systems (non-ubuntu, non-debian) people will be installing wheels, typically that have been uploaded to pypi.python.org.  On the other hand, it is a bit of a pain/noise to upload every development version, so we only upload releases, so testing of wheels is done by building local wheels. Need to build a new wheel every time a change is made.
 
-* **pip install (not -e)** would pull a wheel down from pypi.python.org. Generally not used during development of Sarracenia itself.
+* **pip install metpx-sr3[amqp]** would pull a wheel down from pypi.python.org. Generally not used during development of Sarracenia itself.
+  one could also pull in all possible dependencies with **pip install metpx-sr3[all]**
+* **pip install -e .[amqp] ... lets you edit the source code of the installed package, ideal for debugging problems, because it allows live changes to the application without having to go through building and installing a new package.
 
-* **pip install -e** ... lets you edit the source code of the installed package, ideal for debugging problems, because it allows live changes to the application without having to go through building and installing a new package.
+* **apt install metpx-sr3** install debian package from repositories, similarly to pip install (not -e), normally dev snapshots are not uploaded to repositories, so while this would be the normal way for users of ubuntu servers, it is not available during development of the package itself. Also need **apt install python3-amqp**
 
-* **apt install** install debian package from repositories, similarly to pip install (not -e), normally dev snapshots are not uploaded to repositories, so while this would be the normal way for users of ubuntu servers, it is not available during development of the package itself.
-
-* **dpkg -i** builds a debian package for local installation. This is how packages are tested prior to upload to repositories.  It can also be used to support development (have to run dpkg -i for each package change.)
+* **dpkg -i** builds a debian package for local installation. This is how packages are tested prior to upload to repositories.  It can also be used to support development (have to run dpkg -i for each package change.) also need **apt install python3-amqp**
 
 The sr_insects tests invokes the version of metpx-sarracenia that is installed on the system,
 and not what is in the development tree.  It is necessary to install the package on 
@@ -998,7 +1000,7 @@ to identify more issues. sample run to 100,000 entries::
   maximum of the shovels is: 100008
 
 
-While it is runnig one can run flow_check.sh at any time::
+While it is running one can run flow_check.sh at any time::
 
   NB retries for sr_subscribe t_f30 0
   NB retries for sr_sender 18

docs/source/Tutorials/Install.rst

@@ -51,6 +51,7 @@ On Ubuntu 22.04 and derivatives::
   sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx
   sudo apt update
   sudo apt install metpx-sr3  # main python package.
+  sudo apt install python3-magic # optional support putting file type content-type message headers.
   sudo apt install metpx-sr3c # optional C client.
   sudo apt install python3-amqp  # optionally support rabbitmq brokers
   sudo apt install python3-paho-mqtt  # optionally support MQTT brokers
@@ -139,6 +140,7 @@ For example, on fedora 28 mandatories::
 Optional ones::
 
   $ sudo dnf install python3-amqp      # optionally support rabbitmq brokers
+  $ sudo dnf install python3-magic      # optionally support content-type header in messages.
   $ sudo dnf install python3-netifaces # optionally support vip directive for HA.
   $ sudo dnf install python3-paho-mqtt # optionally support mqtt brokers
 
@@ -174,6 +176,10 @@ one could also add the extras::
 
   $ pip install metpx-sr3[amqp,mqtt,vip]  
 
+for all the extras, there is a shortcut::
+
+  $ pip install metpx-sr3[all]  
+
 and to upgrade after the initial installation::
 
   $ pip install metpx-sr3

docs/source/fr/Contribution/Développement.rst

@@ -133,28 +133,35 @@ Installation locale
 
 Il existe de nombreuses façons d’installer des paquets python sur un ordinateur. Différents développeurs
 préféreront différentes méthodes, et toutes les méthodes doivent être testées avant chaque version.
+Avant d´installer le paquet il faut généralement une librarie pour communiquer avec le courtier
+de messages (généralement rabbitmq/AMQP, mais ca peut être MQTT également)
 
 * **Wheel** Lorsque les gens utilisent différents systèmes d’exploitation (non-Ubuntu, non-Debian),
   les gens installent des wheel, généralement qui ont été téléchargées sur pypi.python.org.  D’un
   autre côté, c’est un peu pénible / bruyant de télécharger chaque version de développement, donc
   nous ne téléchargeons que des versions, donc les tests de wheel se font en construisant des roues
   locales. Besoin de construire une nouvelle wheel chaque fois qu’un changement est apporté.
+  *pip install amqp* sera également nécessaire pour le support rabbitmq.
 
 * **pip install (pas -e)** tirerait une wheel vers le bas de pypi.python.org. Généralement pas utilisé
   pendant le développement de Sarracenia lui-même.
+  *pip install amqp* sera également nécessaire pour le support rabbitmq.
 
 * **pip install -e** ... vous permet de modifier le code source du package installé, idéal pour les
   problèmes de débogage, car il permet des modifications en direct de l’application sans avoir à passer
   par la construction et l’installation d’un nouveau package.
+  *pip install amqp* sera également nécessaire pour le support rabbitmq.
 
 * **apt install** installer le paquet Debian à partir de dépôts, de la même manière que pip install (pas -e),
   normalement les instantanés de développement ne sont pas téléchargés vers des dépôts, donc bien que ce soit
   la manière normale pour les utilisateurs de serveurs Ubuntu, il n’est pas disponible pendant le développement
   du paquet lui-même.
+  *apt install python3-amqp* sera également nécessaire pour le support rabbitmq.
 
 * **dpkg -i** construit un paquet Debian pour l’installation locale. C’est ainsi que les packages sont testés
   avant d’être téléchargés vers des référentiels.  Il peut également être utilisé pour soutenir le développement
   (il faut exécuter dpkg -i pour chaque changement de paquet).
+  *apt install python3-amqp* sera également nécessaire pour le support rabbitmq.
 
 Le test sr_insects appelle la version de metpx-sarracenia installée sur le système,
 et non ce qui est dans l’arbre de développement.  Il est nécessaire d’installer le paquet sur

docs/source/fr/Tutoriel/Installer.rst

@@ -48,12 +48,13 @@ Sur Ubuntu 22.04 et dérivés du même::
 
   sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx
   sudo apt update
-  sudo apt install metpx-sr3  # main python package.
-  sudo apt install metpx-sr3c # optional C client.
-  sudo apt install python3-amqp  # optionally support rabbitmq brokers
-  sudo apt install python3-paho-mqtt  # optionally support MQTT brokers
-  sudo apt install python3-netifaces # optionally support the vip directive (HA failover.)
-  sudo apt install python3-dateparser python3-pytz # optionally support ftp polling.
+  sudo apt install metpx-sr3  # pacquet principale.
+  sudo apt install metpx-sr3c # client binaire (en C) .
+  sudo apt install python3-amqp  # support optionnel pour les courtiers AMWP (rabbitmq)
+  sudo apt install python3-magic  # support optionnel pour les entêtes "content-type" dans les messages
+  sudo apt install python3-paho-mqtt  # support optionnel pour les courtiers MQTT 
+  sudo apt install python3-netifaces # support optionnel pour les vip (haut-disponibilité)
+  sudo apt install python3-dateparser python3-pytz # support optionnel pour les sondages ftp. 
 
 Si les paquets ne sont pas disponibles, on peut les remplacer en utilisant python install package (pip)
 Actuellement, seuls les paquets Debian incluent des pages de manuel. Les guides sont seulement
@@ -127,6 +128,7 @@ Par exemple, sur fedora 28 obligatoirement::
 Facultatifs::
 
   $ sudo dnf install python3-amqp   # optionally support rabbitmq brokers
+  $ sudo dnf install python3-magic   # optionally support content-type headers in files.
   $ sudo dnf install python3-netifaces # optionally support vip directive for HA.
   $ sudo dnf install python3-paho-mqtt # optionally support mqtt brokers
 
@@ -161,6 +163,10 @@ on pourrait aussi ajouter les extras::
 
   $ pip install metpx-sr3[amqp,mqtt,vip]  
 
+Si veut avoir tous les extras::
+
+  $ pip install metpx-sr3[all]  
+
 et à mettre à niveau après l’installation initiale::
 
   $ pip install metpx-sr3

sarracenia/__init__.py

@@ -31,7 +31,6 @@
 import datetime
 import importlib.util
 import logging
-import magic
 import os
 import os.path
 import paramiko
@@ -475,11 +474,15 @@ def fromFileData(path, o, lstat=None):
         if lstat :
             if os_stat.S_ISREG(lstat.st_mode):
                 m.__computeIdentity(path, o)
-                try:
-                    t = magic.from_file(path,mime=True)
-                    m['contentType'] = t
-                except Exception as ex:
-                    logging.info("trying to determine mime-type. Exception details:", exc_info=True)
+                if extras['filetypes']['present']:
+                    try:
+                        t = magic.from_file(path,mime=True)
+                        m['contentType'] = t
+                    except Exception as ex:
+                        logging.info("trying to determine mime-type. Exception details:", exc_info=True)
+                #else:
+                #    m['contentType'] = 'application/octet-stream' # https://www.rfc-editor.org/rfc/rfc2046.txt (default when clueless)
+                # I think setting a bad value is worse than none, so just omitting.
             elif os_stat.S_ISDIR(lstat.st_mode):
                 m['contentType'] = 'text/directory' # source: https://www.w3.org/2002/12/cal/rfc2425.html
             elif os_stat.S_ISLNK(lstat.st_mode):
@@ -804,6 +807,7 @@ def getContent(msg):
 
   amqp - ability to communicate with AMQP (rabbitmq) brokers
   mqtt - ability to communicate with MQTT brokers
+  filetypes - ability to
   ftppoll - ability to poll FTP servers
   vip  - enable vip (Virtual IP) settings to implement singleton processing
          for high availability support.
@@ -816,6 +820,7 @@ def getContent(msg):
    'ftppoll' : { 'modules_needed': ['dateparser', 'pytz'], 'present': False, 'lament' : 'will not be able to poll with ftp' },
    'humanize' : { 'modules_needed': ['humanize' ], 'present': False, 'lament': 'humans will have to read larger, uglier numbers' },
    'mqtt' : { 'modules_needed': ['paho.mqtt.client'], 'present': False, 'lament': 'will not be able to connect to mqtt brokers' },
+   'filetypes' : { 'modules_needed': ['magic'], 'present': False, 'lament': 'will not be able to set content headers' },
    'vip'  : { 'modules_needed': ['netifaces'] , 'present': False, 'lament': 'will not be able to use the vip option for high availability clustering' },
    'watch'  : { 'modules_needed': ['watchdog'] , 'present': False, 'lament': 'cannot watch directories' }
 }
@@ -837,6 +842,9 @@ def getContent(msg):
 
 # Some sort of graceful fallback, or good messaging for when dependencies are missing.
 
+if extras['filetypes']['present']:
+   import magic
+
 if extras['mqtt']['present']:
    import paho.mqtt.client
    if not hasattr( paho.mqtt.client, 'MQTTv5' ):

sarracenia/flow/__init__.py

@@ -1,7 +1,6 @@
 import copy
 import importlib
 import logging
-import magic
 import os
 import re
 
@@ -58,6 +57,9 @@
     'vip': None
 }
 
+if sarracenia.extras['filetypes']['present']:
+    import magic
+
 if sarracenia.extras['vip']['present']:
     import netifaces
 
@@ -1868,7 +1870,7 @@ def download(self, msg, options) -> bool:
                 os.rename(new_inflight_path, new_file)
 
             # older versions don't include the contentType, so patch it here.
-            if 'contentType' not in msg:
+            if sarracenia.extras['filetypes']['present'] and 'contentType' not in msg:
                 msg['contentType'] = magic.from_file(new_file,mime=True)
 
             self.metrics['flow']['transferRxBytes'] += len_written
@@ -1957,7 +1959,8 @@ def send(self, msg, options):
             local_path = '/' + msg['relPath']
 
         # older versions don't include the contentType, so patch it here.
-        if 'contentType' not in msg and not 'fileOp' in msg:
+        if sarracenia.extras['filetypes']['present'] and \
+           ('contentType' not in msg) and (not 'fileOp' in msg):
             msg['contentType'] = magic.from_file(local_path,mime=True)
 
         local_dir = os.path.dirname(local_path).replace('\\', '/')

setup.py

@@ -1,4 +1,5 @@
 import codecs
+import itertools
 import os
 import re
 import sys
@@ -35,6 +36,16 @@ def read(*parts):
 packages = find_packages()
 print("packages = %s" % packages)
 
+extras = {
+       'amqp' : [ "amqp" ],
+       'filetypes': [ "python-magic" ], 
+       'ftppoll' : ['dateparser' ],
+       'mqtt': [ 'paho.mqtt>=1.5.1' ],
+       'vip': [ 'netifaces' ],
+       'redis': [ 'redis' ]
+    } 
+extras['all'] = list(itertools.chain.from_iterable(extras.values()))
+
 setup(
     name='metpx-sr3',
     python_requires='>=3.6',
@@ -83,13 +94,11 @@ def read(*parts):
         'Topic :: System :: Logging',
     ],
     install_requires=[
-        "appdirs", "humanfriendly", "humanize", "jsonpickle", "python-magic", "paramiko",
+        "appdirs", "humanfriendly", "humanize", "jsonpickle", "paramiko",
         "psutil>=5.3.0", "watchdog"
     ],
-    extras_require = {
-       'amqp' : [ "amqp" ],
-       'ftppoll' : ['dateparser' ],
-       'mqtt': [ 'paho.mqtt>=1.5.1' ],
-       'vip': [ 'netifaces' ],
-       'redis': [ 'redis' ]
-    }) 
+    extras_require = extras
+    )
+
+
+

travis/flow_autoconfig.sh

@@ -9,7 +9,7 @@ sudo apt-key adv --keyserver "hkps.pool.sks-keyservers.net" --recv-keys "0x6B73A
 sudo add-apt-repository -y ppa:ssc-hpc-chp-spc/metpx
 sudo apt update
 sudo apt upgrade
-sudo apt -y install python3-setuptools 
+sudo apt -y install python3-setuptools python3-magic
 sudo apt -y install metpx-libsr3c metpx-libsr3c-dev metpx-sr3c
 sudo apt -y install metpx-libsr3c metpx-libsr3c-dev metpx-sr3c
 sudo apt -y install erlang-nox erlang-diameter erlang-eldap findutils git librabbitmq4 net-tools openssh-client openssh-server python3-pip rabbitmq-server xattr wget 
@@ -20,7 +20,7 @@ pip3 install pyftpdlib paramiko net-tools
 # The dependencies that are installed using apt are only available to system default Python versions (e.g. Python 3.8 on Ubuntu 20.04)
 # If we are testing on a non-default Python version, we need to ensure these dependencies are still installed, so we use pip.
 # See issue #407, #445.
-for PKG in amqp appdirs dateparser python-magic watchdog netifaces humanize jsonpickle paho-mqtt psutil xattr ; do
+for PKG in amqp appdirs dateparser watchdog netifaces humanize jsonpickle paho-mqtt psutil xattr ; do
     PKG_INSTALLED="`pip3 list | grep ${PKG}`"
     if [ "$?" == "0" ] ; then
         echo "$PKG is already installed"