Merge pull request #118 from CybercentreCanada/retrohunt

Retrohunt Documentation

docs/installation/configuration/config_file.md

@@ -184,6 +184,8 @@ The full specification of the file [is defined here](https://github.com/Cybercen
         config: {}
 
     datastore:
+      archive:
+        enabled: false
       hosts:
       - http://elastic:devpass@localhost
       ilm:
@@ -203,6 +205,8 @@ The full specification of the file [is defined here](https://github.com/Cybercen
       type: elasticsearch
 
     filestore:
+      archive:
+      - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-archive&use_ssl=False
       cache:
       - s3://al_storage_key:Ch@ngeTh!sPa33w0rd@localhost:9000?s3_bucket=al-cache&use_ssl=False
       storage:
@@ -220,6 +224,14 @@ The full specification of the file [is defined here](https://github.com/Cybercen
       syslog_host: localhost
       syslog_port: 514
 
+    retrohunt:
+      enabled: false
+      dtl: 30
+      max_dtl: 0
+      url: https://hauntedhouse:4443
+      api_key: ChangeThisDefaultRetroHuntAPIKey!
+      tls_verify: true
+
     services:
       allow_insecure_registry: false
       categories:
@@ -359,6 +371,7 @@ Here are the links to the different section documentations:
 * [Database (datastore:)](../datastore)
 * [File storage (filestore:)](../filestore)
 * [Logging (logging:)](../logging)
+* [Retrohunt (retrohunt:)](../retrohunt)
 * [Services (services:)](../services)
 * [Submission (submission:)](../submission)
 * [System (system:)](../system)

docs/installation/configuration/retrohunt.md

@@ -0,0 +1,34 @@
+# Retrohunt section
+
+This section goes over how to configure the deployment instance to ensure the Retrohunt component is configured properly.
+
+The Retrohunt section (`retrohunt:`) of the configuration file contains all the different parameters that you can change.
+
+Refer to the [Retrohunt architecture](../../overview/architecture.md/#yara-back-in-time-retrohunt) to get an understanding of how this feature works.
+
+???+ example "Retrohunt configuration example"
+    ```yaml
+    ...
+    retrohunt:
+      # Is the Retrohunt functionality enabled on the frontend?
+      enabled: false
+
+      # Number of days retrohunt jobs will remain in the system by default
+      dtl: 30
+
+      # Maximum number of days retrohunt jobs will remain in the system
+      max_dtl: 0
+
+      # Base URL for service API
+      url: https://hauntedhouse:4443
+
+      # Service API Key
+      api_key: ChangeThisDefaultRetroHuntAPIKey!
+
+      # Should tls certificates be verified
+      tls_verify: true
+    ...
+    ```
+
+!!! tip
+    Refer to the [changing the configuration file](../config_file/#changing-the-configuration-file) documentation for more details on where and how to change the configuration of the system.

docs/overview/architecture.md

@@ -193,12 +193,20 @@ When the services launch, they pull their signature set from the service updater
 
 ## Keeping files forever (Malware Archive)
 
-Malware Archive is a new feature for Assemblyline that allows users to preserve important documents forever. To accomplish this, archived filestore and datastore indices have been defined where the stored documents do not have an expiry date and will not be deleted by the expiry process. The new core component - "the Archiver" - was added to move the file and the analysis over to the Malware Archive.
+Malware Archive is a new feature for Assemblyline that allows users to preserve important documents forever. To accomplish this, archived filestore and datastore indices have been defined where the stored documents do not have an expiry date and will not be deleted by the expiry process. The new core component `Archiver` was added to move the file and the analysis over to the Malware Archive.
 
 ### [Archiver](https://github.com/CybercentreCanada/assemblyline-core/tree/master/assemblyline_core/archiver)
 
 The Archiver process is the core component that archives the files and documents. When a user or a system requests a submission to be archived, a message is created in the Redis (volatile) message broker. The Archiver listens to those messages and is tasked with copying the file to the archived filestore and the related submissions in the datastore to their archive indices.
 
+## YARA back in time (Retrohunt)
+
+Retrohunt is a new feature that allows users to scan the collection of submitted files in Assemblyline using their own YARA rules. When a user submits a request to create a new search job, the UI will task the new component called `Haunted House` to process the hunting of the files and inform the user on the progress of the job. The resulting file hits of the search job are stored in the Datastore which makes them easily accessible and minimizes the interaction with Haunted House.
+
+### [Haunted House](https://github.com/CybercentreCanada/haunted-house)
+
+Haunted House is composed of three components called the "Python Client", the "Server" and the "Worker", the latter two being written in Rust. The Python Client is the intermediary between the Assemblyline UI and the Rust Server. It receives the requests from the UI to either start a new search, repeat an existing search or receive status updates on an in-progress job. The Rust Server is responsible for tasking a multitude of Workers to match the YARA rules provided to the files.
+
 ## Work online, continue offline
 
 An Assemblyline instance can continue the work that another Assemblyline instance has started. This allows you to start a scan on an instance of Assemblyline that is online (connected to the internet) and gather as much information possible on a file using these internet sources. Then ship the full analysis bundle to another instance of Assemblyline that is on an offline network (NOT connected to the internet) and continue the analysis with extra service that you may not want to deploy on your online version.

docs/user_manual/retrohunt.md

@@ -0,0 +1,43 @@
+# Using Retrohunt
+
+## Overview
+
+The Retrohunt feature allows users to scan the historical collection of files in Assemblyline using YARA rules in order to detect early versions of an attack using newer rule sets to understand how an attack has evolved over time.
+
+To start using Retrohunt, make sure you have the correct [configuration](../installation/configuration/retrohunt.md).
+
+If you want to understand how Retrohunt works, a detailed description can be found in the [System Architecture](../overview/architecture.md#yara-back-in-time-retrohunt)
+
+## Retrohunt Interface
+
+Click the "Retrohunt" left navbar link to access the main Retrohunt interface. This page displays all the Retrohunt searches that users have created while taking into consideration their classification. You can filter these searches using your own search parameters, and it also provides quick search actions (on the right side of the search bar) to display jobs that were completed in the last 24h and your own jobs.
+
+![Retrohunt](./images/retrohunt1.png)
+
+## Creating a new Retrohunt search
+
+To create a new search job, click the green "plus" button on the top right of the main interface to open the Create Retrohunt Search page. Enter the following details:
+
+- **Search information**: Provide a description for the search, select which indices to search on (search space) and enter the expiry date (days to live).
+- **Maximum file classification**: This property defines the scope of the included files in the search, but note that even with higher classification levels, users with a lower classification will not be able to see all the resulting hits after the search has completed.
+- **YARA rule**: When writing a YARA rule, starting to type `rule` will show a snippet that will create a basic rule template. You must write a valid rule, otherwise submitting the request to create a search will fail.
+
+When you're done, click the "ADD RETROHUNT JOB" and confirm the job to create it. If no errors have occurred, a green snack bar should appear from the bottom and the interface should redirect to its Retrohunt Detail page. That page will show a progress bar denoting the status of the search. After the job has completed, it will show the resulting hits of the search.
+
+![Retrohunt](./images/retrohunt2.png)
+
+## Viewing the hits of a Retrohunt search
+
+The Retrohunt Detail page displays information in a tabular format.
+
+- **Details**: This tab shows the search information and displays the resulting hits of the search. You may reduce the scope of the hits by entering a search query or by clicking on a column in the distribution graphs to add a filter value. Clicking on a table row will open the detail page of that file.
+- **YARA Rule**: The submitted YARA rule is found in this tab.
+- **Errors**: This tab is only accessible to administrators as it shows the warnings and errors that occurred during the Retrohunt search process, which may be used for debugging purposes.
+
+![Retrohunt](./images/retrohunt3.png)
+
+## Repeating a Retrohunt search
+
+The Retrohunt feature offers the ability to repeat completed jobs using the same YARA rule. To repeat a search, click on the "Repeat this Retrohunt search" button on the top right side of the Retrohunt Detail page to open the "Repeat Retrohunt Search" dialog box. You may change the `maximum file classification` and the `days to live`, but note that selecting a lower value will not update that property even though the interface doesn't prevent you from selecting them.
+
+![Retrohunt](./images/retrohunt4.png)

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
   - user_manual/searching.md
   - user_manual/verdicts.md
   - user_manual/malware_archive.md
+  - user_manual/retrohunt.md
   - user_manual/llm.md
 - Installation:
   - Getting Started:
@@ -62,8 +63,9 @@ nav:
     - installation/configuration/datasources.md
     - installation/configuration/datastore.md
     - installation/configuration/filestore.md
-    - installation/configuration/malware_archive.md
     - installation/configuration/logging.md
+    - installation/configuration/malware_archive.md
+    - installation/configuration/retrohunt.md
     - installation/configuration/services.md
     - installation/configuration/submission.md
     - installation/configuration/system.md