Merge pull request #74 from CybercentreCanada/update/stages+categories

Update/stages+categories

docs/developer_manual/services/advanced/service_manifest.md

@@ -9,7 +9,7 @@ The table below shows all the elements that the manifest file can contain, inclu
 | Field name | Value type | Required? | Description |
 |:---|:---|:---|:---|
 | accepts | Keyword | No <br> Default: `.*` | Regexes applied to Assemblyline style file type string. For example, `.*` will allow the service to accept all types of files. |
-| category | Keyword | No <br> Default: `Static Analysis` | Which category is the service part of? Must be one of `Antivirus`, `Dynamic Analysis`, `External`, `Extraction`, `Filtering`, `Networking`, or `Static Analysis`. |
+| category | Keyword | No <br> Default: `Static Analysis` | Which category is the service part of? Must be one of `Antivirus`, `Dynamic Analysis`, `External`, `Extraction`, `Filtering`, `Internet Connected`, `Networking`, or `Static Analysis`. |
 | config | Mapping of Any | No | Dictionary of service configuration variables. The key names can be any Keyword and the value can be of Any type. |
 | default_result_classification | Classification string | No <br> Default: `UNRESTRICTED` | The default classification for the results generated by the service. If no classification is provided for a result section, this default classification is used. |
 | dependencies | Mapping of [Dependency Config](#dependency-config) | No | Refer to the [dependency config](#dependency-config) section. |
@@ -24,7 +24,7 @@ The table below shows all the elements that the manifest file can contain, inclu
 | name | Keyword | Yes | Name of the service. |
 | privileged | Boolean | No <br> Default: `false` | Allow service to have direct access to core for processing. <br>  **Note: Should only be enabled on services that perform static analysis.** |
 | rejects | Keyword | No <br> Default: <code>empty&#124;metadata/.\*</code>| Regexes applied to Assemblyline style file type string. For example, <code>empty&#124;metadata/.\*</code> will reject all empty and metadata files. |
-| stage | Keyword | No <br> Default: `CORE` | At which stage should the service run. Must be one of: (1) `FILTER`, (2) `EXTRACT`,  (3) `CORE`, (4) `SECONDARY`, (5) `POST`. Note that stages are executed in the numbered order shown. |
+| stage | Keyword | No <br> Default: `CORE` | At which stage should the service run. Must be one of: (1) `FILTER`, (2) `EXTRACT`,  (3) `CORE`, (4) `SECONDARY`, (5) `POST`, (6) `REVIEW`. Note that stages are executed in the numbered order shown. |
 | submission_params | List of [Submission Params](#submission-params) | No | List of submission param(s) that define parameters that the user can change about the service for each of its scans. Refer to the [submission_params](#submission-params) section. |
 | timeout | Integer | No <br> Default: `60` | Maximum execution time the service has before the task is timed out. |
 | update_config | [Update Config](#update-config) | No | Refer to the [update config](#update-config) section. |

docs/developer_manual/services/developing_an_assemblyline_service.md

@@ -82,7 +82,7 @@ In your service directory, you will add the YAML configuration file `service_man
     accepts: .*
     rejects: empty
 
-    # At which stage the service should run (one of FILTER, EXTRACT, CORE, SECONDARY, POST)
+    # At which stage the service should run (one of FILTER, EXTRACT, CORE, SECONDARY, POST, REVIEW)
     # NOTE: Stages are executed in the order defined in the list
     stage: CORE
     # Which category the service is part of (one of Antivirus, Dynamic Analysis, External, Extraction, Filtering, Internet Connected, Networking, Static Analysis)

docs/installation/configuration/config_file.md

@@ -1,6 +1,6 @@
 # Configuration YAML file
 
-Assemblyline 4 configuration is done using a YAML file (`config.yml`) which is deployed to all containers when they are launched. 
+Assemblyline 4 configuration is done using a YAML file (`config.yml`) which is deployed to all containers when they are launched.
 
 ## Specification and defaults
 
@@ -94,7 +94,7 @@ The full specification of the file [is defined here](https://github.com/Cybercen
             client_secret: null
             jwks_uri: https://www.googleapis.com/oauth2/v3/certs
             user_get: v1/userinfo
-    
+
     core:
       alerter:
         alert_ttl: 90
@@ -174,15 +174,15 @@ The full specification of the file [is defined here](https://github.com/Cybercen
           growth: 60
           min_instances: 0
           shrink: 30
-    
+
     datasources:
       al:
         classpath: assemblyline.datasource.al.AL
         config: {}
       alert:
         classpath: assemblyline.datasource.alert.Alert
         config: {}
-    
+
     datastore:
       hosts:
       - http://elastic:devpass@localhost
@@ -201,13 +201,13 @@ The full specification of the file [is defined here](https://github.com/Cybercen
           submission: *id002
         update_archive: false
       type: elasticsearch
-    
+
     filestore:
       cache:
       - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-cache&use_ssl=False
       storage:
       - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-storage&use_ssl=False
-    
+
     logging:
       export_interval: 5
       heartbeat_file: /tmp/heartbeat
@@ -219,7 +219,7 @@ The full specification of the file [is defined here](https://github.com/Cybercen
       log_to_syslog: false
       syslog_host: localhost
       syslog_port: 514
-    
+
     services:
       allow_insecure_registry: false
       categories:
@@ -228,6 +228,7 @@ The full specification of the file [is defined here](https://github.com/Cybercen
       - External
       - Extraction
       - Filtering
+      - Internet Connected
       - Networking
       - Static Analysis
       cpu_reservation: 0.25
@@ -241,7 +242,8 @@ The full specification of the file [is defined here](https://github.com/Cybercen
       - CORE
       - SECONDARY
       - POST
-
+      - REVIEW
+
     submission:
       default_max_extracted: 500
       default_max_supplementary: 500
@@ -271,12 +273,12 @@ The full specification of the file [is defined here](https://github.com/Cybercen
         - network.dynamic.ip
         - network.dynamic.domain
         - network.dynamic.uri
-    
+
     system:
       constants: assemblyline.common.constants
       organisation: ACME
       type: production
-    
+
     ui:
       allow_malicious_hinting: false
       allow_raw_downloads: true
@@ -326,7 +328,7 @@ The configuration file is built in layers:
 
 ## Changing the configuration file
 
-If you want to change the `config.yml` file that will be deployed in the containers, it will have to be done through the `configuration` section found in the `values.yml` file of your deployment. 
+If you want to change the `config.yml` file that will be deployed in the containers, it will have to be done through the `configuration` section found in the `values.yml` file of your deployment.
 
 !!! example
     Let's say that you would want to change the log level in the system to `ERROR` an up.
@@ -341,13 +343,13 @@ If you want to change the `config.yml` file that will be deployed in the contain
     ```
 
     Then you would simply deploy that new `values.yaml` file using the `helm upgrade` command specific to your deployment:
-    
+
     * [Cluster deployment update](../../cluster/#update-your-deployment)
     * [Appliance deployment update](../../appliance/#updating-the-current-deployment)
 
 ## Exhaustive configuration file documentation
 
-All parameters of each configuration section will be thoroughly documented in their respective pages. 
+All parameters of each configuration section will be thoroughly documented in their respective pages.
 
 Here are the links to the different section documentations:
 

docs/installation/configuration/services.md

@@ -19,6 +19,7 @@ Since this section is quite simple, we will list the default configuration at th
       - External
       - Extraction
       - Filtering
+      - Internet Connected
       - Networking
       - Static Analysis
 
@@ -43,6 +44,7 @@ Since this section is quite simple, we will list the default configuration at th
       - CORE
       - SECONDARY
       - POST
+      - REVIEW
     ...
     ```
 

docs/overview/services.fr.md

@@ -8,8 +8,8 @@ Cette liste contient tous les services inclus et maintenue avec Assemblyline:
 | ------------------| -- | -------------------- | ------------- |
 | APIVector         | Windows binaries | Extrait les imports de fichiers PE files ou mémoire pour générer un vecteur de classification. | [link](https://github.com/CybercentreCanada/assemblyline-service-apivector) |
 | APKaye            | Android APK | Les APK sont décompilés et inspectés. Les indicateurs de réseau et les informations trouvées dans le fichier manifeste APK sont affichés | [link](https://github.com/CybercentreCanada/assemblyline-service-apkaye) |
-| Anti-virus        | Anti-virus | Client ICAP générique utilisant plusieurs solution d'Anti-virus commerciales | [link](https://github.com/CybercentreCanada/assemblyline-service-antivirus) |
-| BatchDeobfuscator | Deobfuscation | Résolution de variables locales et globales d'un fichier batch | [link](https://github.com/CybercentreCanada/assemblyline-service-batchdeobfuscator) |
+| AntiVirus        | Anti-virus | Client ICAP générique utilisant plusieurs solution d'Anti-virus commerciales | [link](https://github.com/CybercentreCanada/assemblyline-service-antivirus) |
+| Batchdeobfuscator | Deobfuscation | Résolution de variables locales et globales d'un fichier batch | [link](https://github.com/CybercentreCanada/assemblyline-service-batchdeobfuscator) |
 | CAPA              | Windows binaries | Intégration de l'outil public [CAPA](https://github.com/mandiant/capa) | [link](https://github.com/CybercentreCanada/assemblyline-service-capa) |
 | Characterize      | Analyze d'entropy | Calcule l'entropy des fichiers et extrait les meta-donnée Exif. | [link](https://github.com/CybercentreCanada/assemblyline-service-characterize) |
 | ConfigExtractor   | Extraction | Extrait la configuration de malware connu, pour trouvé des liste de C2s, cle d'encryption etc. | [link](https://github.com/CybercentreCanada/assemblyline-service-configextractor) |
@@ -22,10 +22,9 @@ Cette liste contient tous les services inclus et maintenue avec Assemblyline:
 | Extract           | Compressed file | Extrait la plus part des type de compression (like ZIP, RAR, 7z, ...) | [link](https://github.com/CybercentreCanada/assemblyline-service-extract)|
 | Floss             | IoC extraction  | Extrait des chaîne de charracters obfusqué avec [FireEye Labs Obfuscated String Solver](https://github.com/fireeye/flare-floss) | [link](https://github.com/CybercentreCanada/assemblyline-service-floss)|
 | FrankenStrings    | IoC extraction | Ce service effectue des extractions de fichiers et d'IOC à l'aide de la correspondance de modèles, d'un décodeur d'encodage simple et de désobfuscateurs de script | [link](https://github.com/CybercentreCanada/assemblyline-service-frankenstrings)|
-| IntezerDynamic    | File genome identification | Interface entre Intezer Analyze API 2.0, soumet le fichier pour analyse si le hachage n'est pas présent dans la base de données Intezer | [link](https://github.com/CybercentreCanada/assemblyline-service-intezer-dynamic)|
-| IntezerStatic     | File genome identification | Interface entre Intezer Analyze API 2.0, effectue des recherches de hachage du fichier soumis | [link](https://github.com/CybercentreCanada/assemblyline-service-intezer-static)|
+| Intezer    | File genome identification | Interface entre Intezer Analyze API 2.0, soumet le fichier pour analyse si le hachage n'est pas présent dans la base de données Intezer | [link](https://github.com/CybercentreCanada/assemblyline-service-intezer)|
 | IPArse            | Apple IOS | Analyze de fichier Apple IOS | [link](https://github.com/CybercentreCanada/assemblyline-service-iparse)|
-| JSJaws            | Javascript | Analyze de fichier Javascript | [link](https://github.com/CybercentreCanada/assemblyline-service-jsjaws)|
+| JsJaws            | Javascript | Analyze de fichier Javascript | [link](https://github.com/CybercentreCanada/assemblyline-service-jsjaws)|
 | MetaPeek          | Meta data analysis | Détect les signe malicieux dans les meta-données et les noms de fichier (double extension etc) | [link](https://github.com/CybercentreCanada/assemblyline-service-metapeek)|
 | Oletools          | Office documents | Ce service analyze les fichiers Office et extrait des indicateurs de compromis avec [Python library py-oletools](https://github.com/decalage2/oletools) by Philippe Lagadec - http://www.decalage.info | [link](https://github.com/CybercentreCanada/assemblyline-service-oletools)|
 | Overpower         | PowerShell | Déobfusque les fichier powershell |[link](https://github.com/CybercentreCanada/assemblyline-service-overpower) |

docs/overview/services.md

@@ -9,7 +9,7 @@ This is the list of all the services that are bundled with Assemblyline and that
 | APIVector         | Windows binaries | Extracts library imports from windows PE files or memory dump to generate api vector classification. | [link](https://github.com/CybercentreCanada/assemblyline-service-apivector) |
 | APKaye            | Android APK | APKs are decompiled and inspected. Network indicators and information found in the APK manifest file are displayed | [link](https://github.com/CybercentreCanada/assemblyline-service-apkaye) |
 | AntiVirus        | Anti-virus | Generic ICAP client to integrate with most Anti-virus enterprise scanners | [link](https://github.com/CybercentreCanada/assemblyline-service-antivirus) |
-| BatchDeobfuscator | Deobfuscation | Deobfuscate batch file through variable resolution | [link](https://github.com/CybercentreCanada/assemblyline-service-batchdeobfuscator) |
+| Batchdeobfuscator | Deobfuscation | Deobfuscate batch file through variable resolution | [link](https://github.com/CybercentreCanada/assemblyline-service-batchdeobfuscator) |
 | CAPA              | Windows binaries | [CAPA](https://github.com/mandiant/capa) open-source tool integration | [link](https://github.com/CybercentreCanada/assemblyline-service-capa) |
 | Characterize      | Entropy analysis | Partitions the file and calculates visual entropy for each partition, extract Exif metadata | [link](https://github.com/CybercentreCanada/assemblyline-service-characterize) |
 | ConfigExtractor   | IoC extraction | Extract malware configuration file, allowing to get list of C2, encryption material etc. | [link](https://github.com/CybercentreCanada/assemblyline-service-configextractor) |