Pipeline: Release 4.5.0.stable20 documentation

CybercentreCanada · Apr 29, 2024 · 6dd3f04 · 6dd3f04
1 parent 331e3b5
commit 6dd3f04
Show file tree

Hide file tree

Showing 3 changed files with 118 additions and 1 deletion.
diff --git a/docs/odm/models/actions.md b/docs/odm/models/actions.md
@@ -12,6 +12,7 @@
 | raise_alert | Boolean | Raise an alert when this action is triggered | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
 | resubmit | [ResubmitOptions](/assemblyline4_docs/odm/models/actions/#resubmitoptions) | Resubmission configuration | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
 | archive_submission | Boolean | Archive the submission when this action is triggered | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| use_archive_alternate_dtl | Boolean | Should we use the alternate dtl while archiving? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
 
 
 [comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)

diff --git a/docs/odm/models/config.md b/docs/odm/models/config.md
@@ -256,7 +256,50 @@
 
 | Field | Type | Description | Required | Default |
 | :--- | :--- | :--- | :--- | :--- |
+| alternate_dtl | Integer | Alternate number of days to keep the data in the malware archive. (0: Disabled, will keep data forever) | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `0` |
+| metadata | Mapping [String, [ArchiverMetadata](/assemblyline4_docs/odm/models/config/#archivermetadata)] | Proxy configuration that is passed to Python Requests | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
 | minimum_required_services | List [Keyword] | List of minimum required service before archiving takes place | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
+| webhook | [Webhook](/assemblyline4_docs/odm/models/config/#webhook) | Webhook to call before triggering the archiving process | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| use_metadata | Boolean | Should the UI ask form metadata to be filed out when archiving | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| use_webhook | Boolean | None | <div style="width:100px">:material-minus-box-outline: Optional</div> | `False` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+#### ArchiverMetadata
+> Malware Archive Configuration
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| default | Keyword | None | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| editable | Boolean | Can the user provide a custom value | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| values | List [Keyword] | List of possible values to pick from | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+#### Webhook
+> Webhook Configuration
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| password | Keyword | Password used to authenticate with source | <div style="width:100px">:material-minus-box-outline: Optional</div> | `` |
+| ca_cert | Keyword | CA cert for source | <div style="width:100px">:material-minus-box-outline: Optional</div> | `` |
+| ssl_ignore_errors | Boolean | Ignore SSL errors when reaching out to source? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| proxy | Keyword | Proxy server for source | <div style="width:100px">:material-minus-box-outline: Optional</div> | `` |
+| method | Keyword | HTTP method used to access webhook | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `POST` |
+| uri | Keyword | URI to source | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| username | Keyword | Username used to authenticate with source | <div style="width:100px">:material-minus-box-outline: Optional</div> | `` |
+| headers | List [[NamedValue](/assemblyline4_docs/odm/models/config/#namedvalue)] | Headers | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
+| retries | Integer | None | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `3` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+##### NamedValue
+> Named Value
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| name | Keyword | Name | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| value | Keyword | Value | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
 
 
 [comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
@@ -652,11 +695,32 @@
 | max_file_size | Integer | Maximum size for files submitted in the system | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `104857600` |
 | max_metadata_length | Integer | Maximum length for each metadata values | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `4096` |
 | max_temp_data_length | Integer | Maximum length for each temporary data values | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `4096` |
-| sha256_sources | List [[Sha256Source](/assemblyline4_docs/odm/models/config/#sha256source)] | List of external source to fetch file via their SHA256 hashes | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
+| sha256_sources | List [[Sha256Source](/assemblyline4_docs/odm/models/config/#sha256source)] | List of external source to fetch file via their SHA256 hashes<br>:material-alert-outline: Use submission.file_sources which is an extension of this configuration | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
+| file_sources | List [[FileSource](/assemblyline4_docs/odm/models/config/#filesource)] | List of external source to fetch file | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[]` |
 | tag_types | [TagTypes](/assemblyline4_docs/odm/models/config/#tagtypes) | Tag types that show up in the submission summary | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | See [TagTypes](/assemblyline4_docs/odm/models/config/#tagtypes) for more details. |
 | verdicts | [Verdicts](/assemblyline4_docs/odm/models/config/#verdicts) | Minimum score value to get the specified verdict. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | See [Verdicts](/assemblyline4_docs/odm/models/config/#verdicts) for more details. |
 
 
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+### FileSource
+> A file source entry for remote fetching via string
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| name | Keyword | Name of the sha256 source | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| hash_types | List [Keyword] | Method(s) of fetching file from source by string input(ie. ['sha256', 'sha1', 'md5', 'tlsh', 'ssdeep']). This also supports custom types. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `['sha256']` |
+| hash_patterns | Mapping [String, Text] | Custom types to regex pattern definition for input detection/validation | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| classification | ClassificationString | None | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| data | Keyword | None | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| failure_pattern | Keyword | None | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| method | Enum | Method used to call the URL<br>Values:<br>`"GET", "POST"` | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `GET` |
+| url | Keyword | Url to fetch the file via SHA256 from (Uses replace pattern) | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| replace_pattern | Keyword | Pattern to replace in the URL with the SHA256 | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| headers | Mapping [String, Keyword] | Headers used to connect to the URL | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `{}` |
+| proxies | Mapping [String, Keyword] | Proxy used to connect to the URL | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `{}` |
+| verify | Boolean | Should the download function Verify SSL connections? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `True` |
+
+
 [comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
 ### Sha256Source
 > A source entry for the sha256 downloader
@@ -716,6 +780,7 @@
 | Field | Type | Description | Required | Default |
 | :--- | :--- | :--- | :--- | :--- |
 | ai | [AI](/assemblyline4_docs/odm/models/config/#ai) | AI support for the UI | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | See [AI](/assemblyline4_docs/odm/models/config/#ai) for more details. |
+| ai_backends | [AIBackends](/assemblyline4_docs/odm/models/config/#aibackends) | AI Multi-backends support for the UI | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | See [AIBackends](/assemblyline4_docs/odm/models/config/#aibackends) for more details. |
 | alerting_meta | [AlertingMeta](/assemblyline4_docs/odm/models/config/#alertingmeta) | Alerting metadata fields | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | See [AlertingMeta](/assemblyline4_docs/odm/models/config/#alertingmeta) for more details. |
 | allow_malicious_hinting | Boolean | Allow user to tell in advance the system that a file is malicious? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
 | allow_raw_downloads | Boolean | Allow user to download raw files? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `True` |
@@ -779,6 +844,56 @@
 | Field | Type | Description | Required | Default |
 | :--- | :--- | :--- | :--- | :--- |
 | system_message | Keyword | System message used for the query. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| task | Keyword | Task description sent to the AI | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `` |
+| max_tokens | Integer | Maximum ammount of token used for the response. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| options | Mapping [String, Any] | Other kwargs options directly passed to the API. | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+### AIBackends
+> AI Multi-Backend support configuration block
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| enabled | Boolean | Is AI support enabled? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| api_connections | List [[AIConnection](/assemblyline4_docs/odm/models/config/#aiconnection)] | List of API definitions use in the API Pool | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `[{'chat_url': 'https://api.openai.com/v1/chat/completions', 'api_type': 'openai', 'headers': {'Content-Type': 'application/json'}, 'model_name': 'gpt-3.5-turbo', 'proxies': None, 'verify': True}, {'chat_url': 'https://api.openai.com/v1/chat/completions', 'api_type': 'openai', 'headers': {'Content-Type': 'application/json'}, 'model_name': 'gpt-4', 'proxies': None, 'verify': True}]` |
+| function_params | [AIFunctionParameters](/assemblyline4_docs/odm/models/config/#aifunctionparameters) | Definition of each parameters used in the different AI functions | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+#### AIConnection
+> Connection information to an AI backend
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| api_type | Enum | Type of chat API we are communicating with<br>Values:<br>`"cohere", "openai"` | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| chat_url | Keyword | URL to the AI API | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| headers | Mapping [String, Keyword] | Headers used by the _call_ai_backend method | <div style="width:100px">:material-minus-box-outline: Optional</div> | `{}` |
+| model_name | Keyword | Name of the model to be used for the AI analysis. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| proxies | Mapping [String, Keyword] | Proxies used by the _call_ai_backend method | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
+| verify | Boolean | Should the SSL connection to the AI API be verified. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `True` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+#### AIFunctionParameters
+> Definition of each parameters used in the different AI functions
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| assistant | [AIQueryParams](/assemblyline4_docs/odm/models/config/#aiqueryparams) | Parameters used for Assamblyline Assistant | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| code | [AIQueryParams](/assemblyline4_docs/odm/models/config/#aiqueryparams) | Parameters used for code analysis | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| detailed_report | [AIQueryParams](/assemblyline4_docs/odm/models/config/#aiqueryparams) | Parameters used for detailed reports | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| executive_summary | [AIQueryParams](/assemblyline4_docs/odm/models/config/#aiqueryparams) | Parameters used for executive summaries | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+
+
+[comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)
+##### AIQueryParams
+> Parameters used during a AI query
+
+| Field | Type | Description | Required | Default |
+| :--- | :--- | :--- | :--- | :--- |
+| system_message | Keyword | System message used for the query. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
+| task | Keyword | Task description sent to the AI | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `` |
 | max_tokens | Integer | Maximum ammount of token used for the response. | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `None` |
 | options | Mapping [String, Any] | Other kwargs options directly passed to the API. | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
 

diff --git a/docs/odm/models/submission.md b/docs/odm/models/submission.md
@@ -67,6 +67,7 @@
 | initial_data | Text | Initialization for temporary submission data | <div style="width:100px">:material-minus-box-outline: Optional</div> | `None` |
 | auto_archive | Boolean | Does the submission automatically goes into the archive when completed? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
 | delete_after_archive | Boolean | When the submission is archived, should we delete it from hot storage right away? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
+| use_archive_alternate_dtl | Boolean | Should we use the alternate dtl while archiving? | <div style="width:100px">:material-checkbox-marked-outline: Yes</div> | `False` |
 
 
 [comment]: # (AUTOGENERATED MARKDOWN CONTENT. UPDATES TO ODM DOCUMENTATION SHOULD BE DONE THROUGH ASSEMBLYLINE-BASE REPO!)