4.3 Release

Author: toddbruner

content/about/history.md

@@ -26,7 +26,7 @@ I'd like to personally thank and recognize the amazing engineers that I have had
 ### Scot 3 Engineers
 * Nick Peterson
 * Nick Georgieff
-* Bryce Montoya
+* Bryce Montano
 * Javier Chavez
 
 ### Scot 4 Engineers

content/administration/ad.md

@@ -11,33 +11,36 @@ SCOT can use AzureAD to authenticate users.
 To configure this type of authentication fill in the following fields:
 
 Scopes
-: ?
+: The scopes to request when logging in (optional)
 
 Authority
-: The URI to the AD authority
+: The authority used to log in to Azure; this is usually in the format https://login.microsoftonline.com/<tenant_id> or https://login.microsoftonline.com/
 
 ClientID
-: ?
+: The client ID of your application in Azure Active Directory
 
 Access Roles
-: ?
+: If set, users will be required to have the given application role in order to log in
 
 Callback URL
-: ?
+: The callback url configured for your SCOT instance, this should usually be the base of the SCOT GUI, e.g. https://your-scot-instance.com/
 
 Client Secret
-: ?
+: A client secret configured for your application in Azure Active Directory
 
 Provider Name
-: ?
+: A name that identifies this authentication instance
 
 Auto-create Groups
-: Select this if you want SCOT to automatically create corresponding Roles.
+: When set, SCOT will auto-create roles matching the names of all configured application roles when a user logs in
+
 
 Un-Email Usernames
-: Select this if you do not want usernames in the form of "[email protected]"
+: Whether or not to attempt to convert the username provided by Azure out of email address format (dropping the portion after the @)
+
 
 Certificate Authority Bundle
-: ?
+: When set to a path, the given certificate bundle is used instead of the default certificate bundle. Useful if the network is behind an intercepting proxy.
+
 
 

content/administration/ldap.md

@@ -11,7 +11,7 @@ LDAP Authentication is set up by an admin user by going to the Administration Se
 Setting up LDAP for authentication can be very tricky, and it is recommended that you work with your local LDAP administrator to set the following settings.
 
 Server
-: the FQDN of the LDAP Server
+: The address of the LDAP server
 
 Bind User
 : the user with permission to bind to LDAP and perform queries
@@ -20,10 +20,10 @@ Bind Password
 : password for the Bind User
 
 Test User
-: ?
+: A user that can be looked up to test user queries
 
 Test Group
-: ?
+: A group that can be looked up to test group queries
 
 Group Base DN
 : the DN for retrieving a set of groups.  E.g. `ou=groups,ou=orgname1,dc=dcname1,dc=dcname2`
@@ -32,31 +32,31 @@ Group Filter
 : LDAP group lists can get rather large and when the total length of that list exceeds 1k characters, some LDAP implementations truncate the list.  By applying a filter, you can help ensure that the group list is under that limit.  We recommend picking a prefix to attach to all the groups you want to uses with SCOT.  So if you pick a prefix like wg-scot, all your SCOT related groups would look like: wg-scot-response, wg-scot-mgmt, etc.  The corresponding group filter would be `(| (cn=wg-scot*))`
 
 User Base DN
-: The DN for looking up a user's groups.  E.g. `ou=accounts,ou=ouname,dc=dcname1,dc=dcname2`
+: The base search domain for users
 
 User Filter
-: E.g. `uid=%s`
+: The filter to use when searching for users
 
 Provider Name
-: ?
+: A name that identifies this authentication instance
 
 Auto-create Groups
-: Check this box if you want SCOT to detect new groups with your prefix and to automatically create a corresponding Role.
+: When set, all discovered ldap groups will be created as roles in SCOT
 
 Username Attribute
-: ?
+: The ldap attribute containing a user's username (default: "uid")
 
 Group Name Attribute
-: ?
+: The ldap attribute containing the group's name (default: "cn")
 
 User Email Attribute
-: ?
+: The ldap attribute containing a user's email address
 
 User Group Attribute
-: ? 
+: The ldap attribute on a user containing that user's group memberships
 
 Group Member Attribute
-: ?
+: The ldap attribute on a group containing that group's users
 
 User Full Name Attribute
-: ? 
+: The ldap attribute containing a user's full nam

content/install/manual_install.md

@@ -215,7 +215,7 @@ It will take several minutes to download and spin up all the containers.  You ca
 watch kubectl -n scot4 get pods
 ```
 
-Once the display lookes like below, you can <ctrl-c> and end the watch program.
+Once the display lookes like below, you can ctrl-c and end the watch program.
 
 ```
 Every 2.0s: kubectl -n scot4 get pods                   dev24su: Tue Sep 10 14:41:32 2024

content/releases/4.1.md

@@ -25,6 +25,6 @@ weight = 2
 - Users can now delete Pivots if necessary.
 - The search engine is now indexing Subjects as well as entries.
 - Column filters can be negated by prefixing '!', e.g. '!Foo'  would exclude rows containing 'Foo' in that column.
-- Wrapping text with either <noflair></noflair> or <span class="noflair"></span> will prevent the Flair engine from operating on that block.
+- Wrapping text with either &lt;noflair&gt;&lt;/noflair&gt; or &lt;span class="noflair"&gt;&lt;/span&gt; will prevent the Flair engine from operating on that block.
 -
 

content/usage/Flair.md

@@ -6,31 +6,43 @@ weight = 3
 
 ![SCOT-Flair-Logo](/images/flair-page-logo.png)
 
-One of SCOT's unique features is the ability to automatically highlight and cross-reference key pieces of data within submissions to SCOT.  Flair refers to the highlighting and the addition of icons following that data. (The process that does the highlighting is also called "flair").  Entities are the pieces of data that are being highlighted.
+One of SCOT's unique features is the ability to automatically highlight and cross-reference key pieces of data within submissions to SCOT.  Flair refers to the highlighting and the addition of icons following that data. (The process that does the highlighting is also called "flair").  Entities are the pieces of data that are being highlighted. Entities can be thought of a collection of IOC (indicators of compromise) but are not limited to just IOC's.
+
+## What's it really doing?
+
+The Flair engine searches strings (HTML and plain text) with a set of regular expressions.  When one matches, it wraps that match in a HTML span that allows the front-end to recognize that text as an Entity.  The Flair engine also updates SCOT's database of Entities and their locations.  
+
+At display time, the front-end queries the SCOT database for icons (flair items) to place after the span.  Upon clicking on that span, SCOT will display all the locations that the entity has previously been found, as well as additional enrichment data that has been stored for the entity.  
 
 ## Core Entities
 
-Entities can be thought of a collection of IOC (indicators of compromise) but are not limited to just IOC's.  The core set of entities that the flair process detects include:
+The core set of entities that the flair process detects include:
 
-* IPv4 Addresses
-* IPv6 Addresses
-* Domain Names
-* Filenames with common extensions
 * CVE references
+* Browser User Agent strings (experimental)
 * CIDR blocks
+* IPv6 Addresses
+* IPv4 Addresses (with and without common obfuscation)
+* MD5, SHA1, and SHA256 Hashes
+* E-mail message ids
+* Domain Names (with and without common obfuscation)
+* Filenames with common extensions
 * UUIDs
 * Clsids
-* MD5, SHA1, and SHA256 Hashes
 * E-mail addresses
-* E-mail message ids
 * WinRegistry keys
 * jarm_hash
+* Country names (American English versions)
+* Internal Scot Links
+* SIDs
+
+See [Core Entity Descriptions](/usage/core_entity.md) for more information about the core entities.
 
 ## User Defined Entities
 
 In addition to the Core set, SCOT4 users can add "User Defined" Entities based on string matches.  While viewing an Entry, a user can click and highlight any string within the Entry.  A pop-up window will then ask the user to confirm creation of user defined entity and ask what [Entity Class](/about/terminology/#entity-class) to assign to this item.  
 
-Note: Choose your user defined entities carefully.  5 minutes after introducing this feature someone highlighted the word "the" (user defined flair has a 3 character minimum) and created an entity from it.  This brought the system to a crawl while it dutifully began finding all instances of "the" in the data and flaired them.  Cleanup was not fun. (contact [[email protected]](mailto:[email protected]) for details)  SCOT follows the Unix philosophy of giving you enough rope to shoot yourself in the foot. Consider yourself warned!
+Note: Choose your user defined entities carefully.  5 minutes after introducing this feature someone highlighted the word "the" (user defined flair has a 3 character minimum) and created an entity from it.  This dramatically slowed the system while it dutifully began finding all instances of "the" in the data and flaired them.  Cleanup was not fun. (contact [[email protected]](mailto:[email protected]) for details)  SCOT follows the Unix philosophy of giving you enough rope to shoot yourself in the foot. Consider yourself warned!
 
 ## Entity Icons
 
@@ -52,8 +64,3 @@ The "Recent Appearances" table allows you to see where else this IP address appe
 
 Finally, we see a series of Entries.  These Entries can be from Actions or from your team's manual input.  It is very useful to store the results of your research about this Entity here because it will be available to all users whenever this entity is encountered.
 
-
-
-
-
-

hugo.toml

@@ -1,4 +1,4 @@
-baseURL = 'http://scot.baltig-pages.sandia.gov/scot4/scot4-docs/'
+baseURL = 'https://sandialabs.github.io/scot4-docs/'
 languageCode = 'en-us'
 title = 'Sandia Cyber Omni Tracker'
 theme = 'hugo-theme-relearn'

themes/hugo-theme-relearn/.githooks/pre-push.py

@@ -10,10 +10,10 @@
 # Linux, Windows and MacOS)
 
 # #!/bin/sh
-# echo 'execute .githooks/pre-push.py' >> .githooks/hooks.log
 # python3 .githooks/pre-push.py
 
 from datetime import datetime
+import os
 import re
 import subprocess
 
@@ -29,14 +29,20 @@
 # an "#" (which are work in progress).
 
 def main():
+    script_name = "PRE-PUSH"
+    script_dir = os.path.dirname(os.path.abspath(__file__))
+    log_file = os.path.join(script_dir, "hooks.log")
     time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+    repo_root = subprocess.check_output(['git', 'rev-parse', '--show-toplevel'], universal_newlines=True).strip()
+    repo_name = os.path.basename(repo_root)
+
     local_branch = subprocess.check_output(['git', 'rev-parse', '--abbrev-ref', 'HEAD'], universal_newlines=True).strip()
     wip_prefix = '^#\\d+(?:\\b.*)$'
     if re.match(wip_prefix, local_branch):
-        print(f'{time}: Branch "{local_branch}" was not pushed because its name starts with a "#" which marks it as work in progress', file=open(".githooks/hooks.log", "a"))
-        print(f'Branch "{local_branch}" was not pushed because its name starts with a "#" which marks it as work in progress')
+        print(f'{time}: {repo_name} - {script_name} - Branch "{local_branch}" was not pushed because its name starts with a "#" which marks it as work in progress', file=open(log_file, "a"))
+        print(f'{script_name} - Branch "{local_branch}" was not pushed because its name starts with a "#" which marks it as work in progress')
         exit(1)
-    print(f'{time}: Branch "{local_branch}" was pushed', file=open(".githooks/hooks.log", "a"))
+    print(f'{time}: {repo_name} - {script_name} - Branch "{local_branch}" was pushed', file=open(log_file, "a"))
     exit(0)
 
 if __name__ == "__main__":

themes/hugo-theme-relearn/.github/actions/check_milestone/action.yaml

@@ -23,7 +23,7 @@ runs:
       with:
         query: |
           query {
-            search(first: 1, type: ISSUE, query: "user:${{ github.repository_owner }} repo:${{ github.event.repository.name }} milestone:${{ env.MILESTONE }} state:closed") {
+            search(first: 1, type: ISSUE, query: "repo:${{ github.repository_owner }}/${{ github.event.repository.name }} milestone:${{ env.MILESTONE }} state:closed") {
               issueCount
             }
           }
@@ -37,55 +37,60 @@ runs:
       with:
         query: |
           query {
-            search(first: 1, type: ISSUE, query: "user:${{ github.repository_owner }} repo:${{ github.event.repository.name }} milestone:${{ env.MILESTONE }} state:open") {
+            search(first: 1, type: ISSUE, query: "repo:${{ github.repository_owner }}/${{ github.event.repository.name }} milestone:${{ env.MILESTONE }} state:open") {
               issueCount
             }
           }
 
-    - name: Get old version number
-      id: oldvers
-      uses: andstor/file-reader-action@v1
+    - name: Get current major version number
+      id: majorvers
+      uses: azarc-io/regex-property-action@master
+      env:
+        MILESTONE: ${{ inputs.milestone }}
       with:
-        path: layouts/partials/version.txt
+        value: ${{ env.MILESTONE }}
+        regex: (\d+)\.\d+\.\d+
+        replacement: "$1"
 
-    - name: Get old main version number
-      id: oldmainvers
-      uses: ashley-taylor/regex-property-action@v1
+    - name: Get current minor version number
+      id: minorvers
+      uses: azarc-io/regex-property-action@master
+      env:
+        MILESTONE: ${{ inputs.milestone }}
       with:
-        value: ${{ steps.oldvers.outputs.contents }}
-        regex: (\d+)\.(\d+)\.\d+.*
-        replacement: '$1\.$2'
+        value: ${{ env.MILESTONE }}
+        regex: \d+\.(\d+)\.\d+
+        replacement: "$1"
 
     - name: Get current patch version number
       id: patchvers
-      uses: ashley-taylor/regex-property-action@v1
+      uses: azarc-io/regex-property-action@master
       env:
         MILESTONE: ${{ inputs.milestone }}
       with:
         value: ${{ env.MILESTONE }}
         regex: \d+\.\d+\.(\d+)
         replacement: "$1"
 
-    - name: Get migration notes
-      id: migrationnotes
-      uses: andstor/file-reader-action@v1
-      with:
-        path: exampleSite/content/basics/migration/_index.en.md
-
-    - name: Check for old migration notes
-      id: hasoldnotes
-      uses: ashley-taylor/regex-property-action@v1
-      with:
-        value: ${{ steps.migrationnotes.outputs.contents }}
-        regex: '.*?[\n\r\s]*<!--GH-ACTION-RELEASE-MILESTONE-->[\n\r\s]*-*\s*[\n\r\s]*?[\n\r]+##\s+${{ steps.oldmainvers.outputs.value }}\.0\s+.*?[\n\r][\n\r\s]*.*'
-        flags: gs
-        replacement: '1'
+    - name: Check if releasenotes exists
+      id: releasenotes
+      shell: bash
+      run: |
+        if [ -f "exampleSite/content/introduction/releasenotes/${{ steps.majorvers.outputs.value }}/${{ steps.minorvers.outputs.value }}.en.md" ]; then
+          echo "file_exists=true" >> $GITHUB_OUTPUT
+        else
+          echo "file_exists=false" >> $GITHUB_OUTPUT
+        fi
 
     - name: Set outcome
       id: outcome
       shell: bash
       run: |
-        if [ "${{ fromJSON(steps.closed_issues.outputs.data).search.issueCount > 0 && fromJSON(steps.open_issues.outputs.data).search.issueCount == 0 && ( (steps.patchvers.outputs.value!='0'&&steps.hasoldnotes.outputs.value=='1') || (steps.patchvers.outputs.value=='0'&&steps.hasoldnotes.outputs.value!='1') ) }}" = "true" ]; then
+        if [[ \
+          ${{ fromJSON(steps.closed_issues.outputs.data).search.issueCount }} -gt 0 && \
+          ${{ fromJSON(steps.open_issues.outputs.data).search.issueCount }} -eq 0 && \
+          ${{ steps.releasenotes.outputs.file_exists == 'true' }} \
+        ]]; then
           echo "outcome=success" >> $GITHUB_OUTPUT
         else
           echo "outcome=failure" >> $GITHUB_OUTPUT
@@ -94,12 +99,12 @@ runs:
     - name: Log results and exit
       shell: bash
       run: |
-        echo outcome            : ${{ steps.outcome.outputs.outcome }}
-        echo has closed issues  : ${{ fromJSON(steps.closed_issues.outputs.data).search.issueCount > 0 }}
-        echo has open issues    : ${{ fromJSON(steps.open_issues.outputs.data).search.issueCount > 0 }}
-        echo is patch version   : ${{ steps.patchvers.outputs.value != '0' }}
-        echo has old main notes : ${{ steps.hasoldnotes.outputs.value == '1' }}
-        echo are notes okay     : ${{ (steps.patchvers.outputs.value!='0'&&steps.hasoldnotes.outputs.value=='1') || (steps.patchvers.outputs.value=='0'&&steps.hasoldnotes.outputs.value!='1') }}
+        echo outcome                : ${{ steps.outcome.outputs.outcome }}
+        echo has closed issues      : ${{ fromJSON(steps.closed_issues.outputs.data).search.issueCount > 0 }}
+        echo   count                : ${{ fromJSON(steps.closed_issues.outputs.data).search.issueCount }}
+        echo has all issues closed  : ${{ fromJSON(steps.open_issues.outputs.data).search.issueCount == 0 }}
+        echo   count                : ${{ fromJSON(steps.open_issues.outputs.data).search.issueCount }}
+        echo has releasenotes       : ${{ steps.releasenotes.outputs.file_exists }}
         if [ "${{ steps.outcome.outputs.outcome }}" = "failure" ]; then
           exit 1
         fi