Merge pull request #110 from CybercentreCanada/malware_archive

Malware Archive

docs/installation/configuration/datastore.md

@@ -12,6 +12,17 @@ Since this section is quite simple, we will list the default configuration at th
       hosts:
       - http://elastic:devpass@localhost
 
+      # Datastore Archive feature configuration
+      archive:
+        # Are we enabling Achiving features across indices?
+        enabled: true
+
+        # List of indices the ILM Applies to
+        indices:
+        - file
+        - submission
+        - result
+
       # Index Lifecycle management configuration block
       ilm:
         # After how many days do documents go in the ILM managed indexes

docs/installation/configuration/filestore.md

@@ -16,6 +16,10 @@ Since this section is quite simple, we will list the default configuration at th
     # more files but to look them up less often.
 
     filestore:
+      # List of filestores used for malware archive
+      archive:
+      - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-archive&use_ssl=False
+
       # List of URLs to connect to the cache filestore
       cache:
       - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-cache&use_ssl=False

docs/installation/configuration/malware_archive.md

@@ -0,0 +1,30 @@
+# Malware Archive section
+
+Configuring the Malware Archive functionality is done by modifying the data storage section (`datastore:`) and the file storage section (`filestore:`). Refer to the [Malware Archive](../overview/architecture/#keeping-files-forever-malware-archive) to get an understanding of how this feature works.
+
+Enable the `archive:` option in the configuration to show the "Archive" link in the left navbar, activate the Archiver core component and create the archive indices in the database. The `indices:` list defines which indices will have an archive counterpart. This key is not required given its default values of `file`, `submission`, and `result`.
+
+???+ example "Datastore section configuration example"
+    ```yaml
+    ...
+    datastore:
+      # Datastore Archive feature configuration
+      archive:
+        # Are we enabling Achiving features across indices?
+        enabled: true
+
+        # List of indices the ILM Applies to
+        indices:
+        - file
+        - submission
+        - result
+
+    filestore:
+      # List of filestores used for malware archive
+      archive:
+      - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-archive&use_ssl=False
+    ...
+    ```
+
+!!! tip
+    Refer to the [changing the configuration file](../config_file/#changing-the-configuration-file) documentation for more details on where and how to change the configuration of the system.

docs/overview/architecture.md

@@ -193,11 +193,11 @@ When the services launch, they pull their signature set from the service updater
 
 ## Keeping files forever (Malware Archive)
 
-The Assemblyline team is working on new feature called the Malware Archive that will keep files forever in the system by moving them to another filestore and index, so they are not deleted. To support this, a core component was added to move the file and the analysis over to the Malware Archive.
+Malware Archive is a new feature for Assemblyline that allows users to preserve important documents forever. To accomplish this, archived filestore and datastore indices have been defined where the stored documents do not have an expiry date and will not be deleted by the expiry process. The new core component - "the Archiver" - was added to move the file and the analysis over to the Malware Archive.
 
 ### [Archiver](https://github.com/CybercentreCanada/assemblyline-core/tree/master/assemblyline_core/archiver)
 
-The archiver receives messages about a file or submission that should be kept forever and moved to the Malware Archive. It takes in those messages and copies the actual files into a permanent file store then moves associated analysis data to the archive indices.
+The Archiver process is the core component that archives the files and documents. When a user or a system requests a submission to be archived, a message is created in the Redis (volatile) message broker. The Archiver listens to those messages and is tasked with copying the file to the archived filestore and the related submissions in the datastore to their archive indices.
 
 ## Work online, continue offline
 

docs/user_manual/malware_archive.md

@@ -0,0 +1,40 @@
+# Using Malware Archive
+
+## Overview
+
+The Malware Archive feature allows users to preserve important submission information indefinitely. Most documents in Assemblyline are subjected to the Expiry process which will delete them after they reach their expiry date. This is done to avoid bloating the system with unimportant data. However, if a user wants to preserve a relevant submission for future use, they can archive it to keep it forever!
+
+To start using Malware Archive, make sure you have the correct [configuration](../installation/configuration/malware_archive.md).
+
+If you want to understand how Malware Archive works, a detailed description can be found in the [System Architecture](../overview/architecture.md#keeping-files-forever-malware-archive)
+
+## Archiving a submission
+
+To archive a submission, go to its detail page and click on the “Archive” button to submit a request to archive this submission. If the archive functionality is enabled and working properly, you should see a successful snackbar message appear from the bottom.
+
+![File submission](./images/malware_archive1.png)
+
+## View archived files
+
+The next step is to navigate to the Malware Archive page using the left navbar. This interface is based around files instead of submissions. Therefore, all the files that are part of the submission, including supplementary files, generated during the analysis will be archived and be found in this search interface.
+
+![File submission](./images/malware_archive2.png)
+
+## Searching through the archived files
+
+This search interface offers methods to quickly filter and allow users to find the relevant files. Actions such as clicking on the three quick action buttons on the right side of the search bar, on a graph element, or a label in the table, will add a filter chip below the search bar. You can click on that chip to get the opposite effect marked in red.
+
+![File submission](./images/malware_archive3.png)
+
+## Analysing an archived file
+
+Clicking on a file in the table will open its archived detail page. To make this more convenient, we have included familiar and new sections needed to analyze this file:
+
+- Details: This section summarize the file's information.
+- Detections: This section contains the detailed results of this file's analysis by Assemblyline.
+- Tags: This section displays all the heuristics and tags in a tabular format. This table allows users to sort and filter the data using the headers. Clicking on a row submits a request to find results sharing this tag type and value.
+- Relations: The goal of this section is to find similar results sharing similar properties.
+- ASCII, Strings, Hex: Instead of navigating to another page, we have included the file viewers in these sections.
+- Community: This section contains all the user-provided actions such as labelling this file and adding comments. Note that labels and comments are also search parameters.
+
+![File submission](./images/malware_archive4.png)

mkdocs.yml

@@ -42,6 +42,7 @@ nav:
   - user_manual/results.md
   - user_manual/searching.md
   - user_manual/verdicts.md
+  - user_manual/malware_archive.md
 - Installation:
   - Getting Started:
     - installation/deployment.md