diff --git a/docs/source/Contribution/Development.rst b/docs/source/Contribution/Development.rst index 7822878b8..e0b47855b 100644 --- a/docs/source/Contribution/Development.rst +++ b/docs/source/Contribution/Development.rst @@ -137,16 +137,18 @@ Local Installation There are many different ways to install python packages on a computer. Different developers will prefer different methods, and all the methods need to be tested prior to each release. +Sarracenia can work with either mqtt or amqp (most mature and stable) message passing libraries. +Install one of those first. in these examples, we use amqp. * **Wheel** when people are running different operating systems (non-ubuntu, non-debian) people will be installing wheels, typically that have been uploaded to pypi.python.org. On the other hand, it is a bit of a pain/noise to upload every development version, so we only upload releases, so testing of wheels is done by building local wheels. Need to build a new wheel every time a change is made. -* **pip install (not -e)** would pull a wheel down from pypi.python.org. Generally not used during development of Sarracenia itself. +* **pip install metpx-sr3[amqp]** would pull a wheel down from pypi.python.org. Generally not used during development of Sarracenia itself. + one could also pull in all possible dependencies with **pip install metpx-sr3[all]** +* **pip install -e .[amqp] ... lets you edit the source code of the installed package, ideal for debugging problems, because it allows live changes to the application without having to go through building and installing a new package. -* **pip install -e** ... lets you edit the source code of the installed package, ideal for debugging problems, because it allows live changes to the application without having to go through building and installing a new package. +* **apt install metpx-sr3** install debian package from repositories, similarly to pip install (not -e), normally dev snapshots are not uploaded to repositories, so while this would be the normal way for users of ubuntu servers, it is not available during development of the package itself. Also need **apt install python3-amqp** -* **apt install** install debian package from repositories, similarly to pip install (not -e), normally dev snapshots are not uploaded to repositories, so while this would be the normal way for users of ubuntu servers, it is not available during development of the package itself. - -* **dpkg -i** builds a debian package for local installation. This is how packages are tested prior to upload to repositories. It can also be used to support development (have to run dpkg -i for each package change.) +* **dpkg -i** builds a debian package for local installation. This is how packages are tested prior to upload to repositories. It can also be used to support development (have to run dpkg -i for each package change.) also need **apt install python3-amqp** The sr_insects tests invokes the version of metpx-sarracenia that is installed on the system, and not what is in the development tree. It is necessary to install the package on @@ -998,7 +1000,7 @@ to identify more issues. sample run to 100,000 entries:: maximum of the shovels is: 100008 -While it is runnig one can run flow_check.sh at any time:: +While it is running one can run flow_check.sh at any time:: NB retries for sr_subscribe t_f30 0 NB retries for sr_sender 18 diff --git a/docs/source/Tutorials/Install.rst b/docs/source/Tutorials/Install.rst index 48b6136ae..6ddd579dd 100644 --- a/docs/source/Tutorials/Install.rst +++ b/docs/source/Tutorials/Install.rst @@ -51,6 +51,7 @@ On Ubuntu 22.04 and derivatives:: sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx sudo apt update sudo apt install metpx-sr3 # main python package. + sudo apt install python3-magic # optional support putting file type content-type message headers. sudo apt install metpx-sr3c # optional C client. sudo apt install python3-amqp # optionally support rabbitmq brokers sudo apt install python3-paho-mqtt # optionally support MQTT brokers @@ -139,6 +140,7 @@ For example, on fedora 28 mandatories:: Optional ones:: $ sudo dnf install python3-amqp # optionally support rabbitmq brokers + $ sudo dnf install python3-magic # optionally support content-type header in messages. $ sudo dnf install python3-netifaces # optionally support vip directive for HA. $ sudo dnf install python3-paho-mqtt # optionally support mqtt brokers @@ -174,6 +176,10 @@ one could also add the extras:: $ pip install metpx-sr3[amqp,mqtt,vip] +for all the extras, there is a shortcut:: + + $ pip install metpx-sr3[all] + and to upgrade after the initial installation:: $ pip install metpx-sr3 diff --git "a/docs/source/fr/Contribution/D\303\251veloppement.rst" "b/docs/source/fr/Contribution/D\303\251veloppement.rst" index eb94d68f5..282921eb0 100644 --- "a/docs/source/fr/Contribution/D\303\251veloppement.rst" +++ "b/docs/source/fr/Contribution/D\303\251veloppement.rst" @@ -133,28 +133,35 @@ Installation locale Il existe de nombreuses façons d’installer des paquets python sur un ordinateur. Différents développeurs préféreront différentes méthodes, et toutes les méthodes doivent être testées avant chaque version. +Avant d´installer le paquet il faut généralement une librarie pour communiquer avec le courtier +de messages (généralement rabbitmq/AMQP, mais ca peut être MQTT également) * **Wheel** Lorsque les gens utilisent différents systèmes d’exploitation (non-Ubuntu, non-Debian), les gens installent des wheel, généralement qui ont été téléchargées sur pypi.python.org. D’un autre côté, c’est un peu pénible / bruyant de télécharger chaque version de développement, donc nous ne téléchargeons que des versions, donc les tests de wheel se font en construisant des roues locales. Besoin de construire une nouvelle wheel chaque fois qu’un changement est apporté. + *pip install amqp* sera également nécessaire pour le support rabbitmq. * **pip install (pas -e)** tirerait une wheel vers le bas de pypi.python.org. Généralement pas utilisé pendant le développement de Sarracenia lui-même. + *pip install amqp* sera également nécessaire pour le support rabbitmq. * **pip install -e** ... vous permet de modifier le code source du package installé, idéal pour les problèmes de débogage, car il permet des modifications en direct de l’application sans avoir à passer par la construction et l’installation d’un nouveau package. + *pip install amqp* sera également nécessaire pour le support rabbitmq. * **apt install** installer le paquet Debian à partir de dépôts, de la même manière que pip install (pas -e), normalement les instantanés de développement ne sont pas téléchargés vers des dépôts, donc bien que ce soit la manière normale pour les utilisateurs de serveurs Ubuntu, il n’est pas disponible pendant le développement du paquet lui-même. + *apt install python3-amqp* sera également nécessaire pour le support rabbitmq. * **dpkg -i** construit un paquet Debian pour l’installation locale. C’est ainsi que les packages sont testés avant d’être téléchargés vers des référentiels. Il peut également être utilisé pour soutenir le développement (il faut exécuter dpkg -i pour chaque changement de paquet). + *apt install python3-amqp* sera également nécessaire pour le support rabbitmq. Le test sr_insects appelle la version de metpx-sarracenia installée sur le système, et non ce qui est dans l’arbre de développement. Il est nécessaire d’installer le paquet sur diff --git a/docs/source/fr/Tutoriel/Installer.rst b/docs/source/fr/Tutoriel/Installer.rst index e0add8af6..4c938538a 100644 --- a/docs/source/fr/Tutoriel/Installer.rst +++ b/docs/source/fr/Tutoriel/Installer.rst @@ -48,12 +48,13 @@ Sur Ubuntu 22.04 et dérivés du même:: sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx sudo apt update - sudo apt install metpx-sr3 # main python package. - sudo apt install metpx-sr3c # optional C client. - sudo apt install python3-amqp # optionally support rabbitmq brokers - sudo apt install python3-paho-mqtt # optionally support MQTT brokers - sudo apt install python3-netifaces # optionally support the vip directive (HA failover.) - sudo apt install python3-dateparser python3-pytz # optionally support ftp polling. + sudo apt install metpx-sr3 # pacquet principale. + sudo apt install metpx-sr3c # client binaire (en C) . + sudo apt install python3-amqp # support optionnel pour les courtiers AMWP (rabbitmq) + sudo apt install python3-magic # support optionnel pour les entêtes "content-type" dans les messages + sudo apt install python3-paho-mqtt # support optionnel pour les courtiers MQTT + sudo apt install python3-netifaces # support optionnel pour les vip (haut-disponibilité) + sudo apt install python3-dateparser python3-pytz # support optionnel pour les sondages ftp. Si les paquets ne sont pas disponibles, on peut les remplacer en utilisant python install package (pip) Actuellement, seuls les paquets Debian incluent des pages de manuel. Les guides sont seulement @@ -127,6 +128,7 @@ Par exemple, sur fedora 28 obligatoirement:: Facultatifs:: $ sudo dnf install python3-amqp # optionally support rabbitmq brokers + $ sudo dnf install python3-magic # optionally support content-type headers in files. $ sudo dnf install python3-netifaces # optionally support vip directive for HA. $ sudo dnf install python3-paho-mqtt # optionally support mqtt brokers @@ -161,6 +163,10 @@ on pourrait aussi ajouter les extras:: $ pip install metpx-sr3[amqp,mqtt,vip] +Si veut avoir tous les extras:: + + $ pip install metpx-sr3[all] + et à mettre à niveau après l’installation initiale:: $ pip install metpx-sr3 diff --git a/sarracenia/__init__.py b/sarracenia/__init__.py index 14e3337dd..f8392678a 100755 --- a/sarracenia/__init__.py +++ b/sarracenia/__init__.py @@ -31,7 +31,6 @@ import datetime import importlib.util import logging -import magic import os import os.path import paramiko @@ -475,11 +474,15 @@ def fromFileData(path, o, lstat=None): if lstat : if os_stat.S_ISREG(lstat.st_mode): m.__computeIdentity(path, o) - try: - t = magic.from_file(path,mime=True) - m['contentType'] = t - except Exception as ex: - logging.info("trying to determine mime-type. Exception details:", exc_info=True) + if extras['filetypes']['present']: + try: + t = magic.from_file(path,mime=True) + m['contentType'] = t + except Exception as ex: + logging.info("trying to determine mime-type. Exception details:", exc_info=True) + #else: + # m['contentType'] = 'application/octet-stream' # https://www.rfc-editor.org/rfc/rfc2046.txt (default when clueless) + # I think setting a bad value is worse than none, so just omitting. elif os_stat.S_ISDIR(lstat.st_mode): m['contentType'] = 'text/directory' # source: https://www.w3.org/2002/12/cal/rfc2425.html elif os_stat.S_ISLNK(lstat.st_mode): @@ -804,6 +807,7 @@ def getContent(msg): amqp - ability to communicate with AMQP (rabbitmq) brokers mqtt - ability to communicate with MQTT brokers + filetypes - ability to ftppoll - ability to poll FTP servers vip - enable vip (Virtual IP) settings to implement singleton processing for high availability support. @@ -816,6 +820,7 @@ def getContent(msg): 'ftppoll' : { 'modules_needed': ['dateparser', 'pytz'], 'present': False, 'lament' : 'will not be able to poll with ftp' }, 'humanize' : { 'modules_needed': ['humanize' ], 'present': False, 'lament': 'humans will have to read larger, uglier numbers' }, 'mqtt' : { 'modules_needed': ['paho.mqtt.client'], 'present': False, 'lament': 'will not be able to connect to mqtt brokers' }, + 'filetypes' : { 'modules_needed': ['magic'], 'present': False, 'lament': 'will not be able to set content headers' }, 'vip' : { 'modules_needed': ['netifaces'] , 'present': False, 'lament': 'will not be able to use the vip option for high availability clustering' }, 'watch' : { 'modules_needed': ['watchdog'] , 'present': False, 'lament': 'cannot watch directories' } } @@ -837,6 +842,9 @@ def getContent(msg): # Some sort of graceful fallback, or good messaging for when dependencies are missing. +if extras['filetypes']['present']: + import magic + if extras['mqtt']['present']: import paho.mqtt.client if not hasattr( paho.mqtt.client, 'MQTTv5' ): diff --git a/sarracenia/flow/__init__.py b/sarracenia/flow/__init__.py index f3acbdc76..61f406ca8 100644 --- a/sarracenia/flow/__init__.py +++ b/sarracenia/flow/__init__.py @@ -1,7 +1,6 @@ import copy import importlib import logging -import magic import os import re @@ -58,6 +57,9 @@ 'vip': None } +if sarracenia.extras['filetypes']['present']: + import magic + if sarracenia.extras['vip']['present']: import netifaces @@ -1868,7 +1870,7 @@ def download(self, msg, options) -> bool: os.rename(new_inflight_path, new_file) # older versions don't include the contentType, so patch it here. - if 'contentType' not in msg: + if sarracenia.extras['filetypes']['present'] and 'contentType' not in msg: msg['contentType'] = magic.from_file(new_file,mime=True) self.metrics['flow']['transferRxBytes'] += len_written @@ -1957,7 +1959,8 @@ def send(self, msg, options): local_path = '/' + msg['relPath'] # older versions don't include the contentType, so patch it here. - if 'contentType' not in msg and not 'fileOp' in msg: + if sarracenia.extras['filetypes']['present'] and \ + ('contentType' not in msg) and (not 'fileOp' in msg): msg['contentType'] = magic.from_file(local_path,mime=True) local_dir = os.path.dirname(local_path).replace('\\', '/') diff --git a/setup.py b/setup.py index 10908b46c..3eaa69d33 100755 --- a/setup.py +++ b/setup.py @@ -1,4 +1,5 @@ import codecs +import itertools import os import re import sys @@ -35,6 +36,16 @@ def read(*parts): packages = find_packages() print("packages = %s" % packages) +extras = { + 'amqp' : [ "amqp" ], + 'filetypes': [ "python-magic" ], + 'ftppoll' : ['dateparser' ], + 'mqtt': [ 'paho.mqtt>=1.5.1' ], + 'vip': [ 'netifaces' ], + 'redis': [ 'redis' ] + } +extras['all'] = list(itertools.chain.from_iterable(extras.values())) + setup( name='metpx-sr3', python_requires='>=3.6', @@ -83,13 +94,11 @@ def read(*parts): 'Topic :: System :: Logging', ], install_requires=[ - "appdirs", "humanfriendly", "humanize", "jsonpickle", "python-magic", "paramiko", + "appdirs", "humanfriendly", "humanize", "jsonpickle", "paramiko", "psutil>=5.3.0", "watchdog" ], - extras_require = { - 'amqp' : [ "amqp" ], - 'ftppoll' : ['dateparser' ], - 'mqtt': [ 'paho.mqtt>=1.5.1' ], - 'vip': [ 'netifaces' ], - 'redis': [ 'redis' ] - }) + extras_require = extras + ) + + + diff --git a/travis/flow_autoconfig.sh b/travis/flow_autoconfig.sh index 6894b9721..05225765f 100755 --- a/travis/flow_autoconfig.sh +++ b/travis/flow_autoconfig.sh @@ -9,7 +9,7 @@ sudo apt-key adv --keyserver "hkps.pool.sks-keyservers.net" --recv-keys "0x6B73A sudo add-apt-repository -y ppa:ssc-hpc-chp-spc/metpx sudo apt update sudo apt upgrade -sudo apt -y install python3-setuptools +sudo apt -y install python3-setuptools python3-magic sudo apt -y install metpx-libsr3c metpx-libsr3c-dev metpx-sr3c sudo apt -y install metpx-libsr3c metpx-libsr3c-dev metpx-sr3c sudo apt -y install erlang-nox erlang-diameter erlang-eldap findutils git librabbitmq4 net-tools openssh-client openssh-server python3-pip rabbitmq-server xattr wget @@ -20,7 +20,7 @@ pip3 install pyftpdlib paramiko net-tools # The dependencies that are installed using apt are only available to system default Python versions (e.g. Python 3.8 on Ubuntu 20.04) # If we are testing on a non-default Python version, we need to ensure these dependencies are still installed, so we use pip. # See issue #407, #445. -for PKG in amqp appdirs dateparser python-magic watchdog netifaces humanize jsonpickle paho-mqtt psutil xattr ; do +for PKG in amqp appdirs dateparser watchdog netifaces humanize jsonpickle paho-mqtt psutil xattr ; do PKG_INSTALLED="`pip3 list | grep ${PKG}`" if [ "$?" == "0" ] ; then echo "$PKG is already installed"