latest updates including revised install instructions

Author: toddbruner

content/install/install.md

@@ -12,26 +12,64 @@ weight = 2
 You must create the scot4 user prior to cloning the repository.
 
 ```
-useradd -m -s /bin/bash -c "SCOT4 User" scot4
+$ sudo useradd -m -s /bin/bash -c "SCOT4 User" scot4
+```
+
+Set the scot4 user's password
+
+```
+$ sudo passwd scot4
+Changing password for user scot4.
+New password: 
+Retype new password: 
+passwd: all authentication tokens updated successfully.
+```
+
+Add user to the proper group to allow sudo:
+
+RHEL:
+```
+$ sudo usermod -aG wheel scot4
+```
+Ubuntu:
+```
+$ sudo usermod -aG sudo scot4
 ```
 
 ### Clone Repository
 
 Clone the SCOT 4 repository as the scot4 user.
 
 ```
-su - scot4
-git clone https://github.com/sandialabs/scot4.git
+$ su - scot4
+$ pwd
+/home/scot4
+$ git clone https://github.com/sandialabs/scot4.git
+```
+
+### Optional, but recommended
+
+If you are behind a web proxy, it is recommended that you set the following variables in the /etc/environment file.  This is so sudo will pick up these variables during the install.
+
+```
+http_proxy=http://your.proxy.here
+https_proxy=https://your.proxy.here
+no_proxy=localhost,yourservername,other,exceptions,list,here
+HTTP_PROXY=$http_proxy
+HTTPS_PROXY=$https_proxy
+NO_PROXY=$no_proxy
 ```
 
+If you do not wish to alter you environment file, the install script will prompt you for your proxy settings.
+
 ### Run Helper
 
 The repository contains a helper script to automate most of the remaining tasks.
 
 ```
-su - scot4
-cd scot4
-./install.sh
+$ sudo su - scot4
+$ cd scot4
+$ sudo ./install.sh
 ```
 
 ### install.sh options
@@ -47,12 +85,13 @@ Here are the options supported:
                         database. You do not have to set this if you are using the 
                         default provided mysql database container.
 -e SURGE            set the surge limit for the API server. (Kubernetes)
--g                  pause script after displaying values of the script variables
 -h VERSION          set the version of Helm to download and install.  defaults to 
-                        version 3.14.3
+                        version 3.19.0
 -i IPADDR           set the IP address that the API server will listen to.  Defaults to
                         the first result of "ip -4 -o addr show scope global"
 -k TLS_KEY_FILE     the fully qualified filename for your TLS KEY File
+-m TRAEFIKVERSION   set the version of Traefik to download and install. defaults to 
+                        version 3.3.6
 -n NO_PROXY         the values to use for your NO_PROXY environment
 -P HTTPS_PROXY      the value to use for HTTPS_PROXY
 -p HTTP_PROXY       the value to use for HTTP_PROXY
@@ -81,7 +120,7 @@ Here are the options supported:
 Once the install program has completed, it will take a few minutes for the containers to download and spin up.  You can monitor progress with the following command:
 
 ```
- watch kubectl -n scot4 get pods
+$ watch kubectl -n scot4 get pods
 ```
 
 You will see the pods go through various init stages.  When the display looks like:
@@ -102,12 +141,23 @@ scot4-search-init-7llfw           0/1     Completed   0          20m
 
 you can then end the watch program (ctrl-c) and begin to use SCOT.
 
-### Should there be an Error
+### Should there be an installer error
+
+We have seen on rare occasions that the install of k3s can experience a problem.  When this occurs, you will most likely see this in the "testing k3s readiness" section of the installer.  Try the following steps:
+
+```
+$ sudo /usr/local/bin/k3s-uninstall.sh
+$ sudo rm -rf /etc/rancher/node
+$ sudo ./install.sh
+```
+
+### Should there be a Pod Error
 
 If a pod is displaying a backoff error condition, you can get more details about what is causing the problem by using the command:
 
 ```
-kubectl -n scot4 describe pod <pod-name-here> 
+$ kubectl -n scot4 describe pod <pod-name-here> 
 ```
 
 The most likely error would be some kind of failure to pull the container from the repository.  Make sure you are not having a network issue and retry the install once network issue has been resolved.
+

content/install/manual_install.md

@@ -16,14 +16,48 @@ Deployment of SCOT requires the existence of a scot4 user.
 sudo useradd -m -s /bin/bash -c "SCOT4 User" scot4
 ```
 
+### TLS Certificates
+
+For deploying SCOT for testing purposes, you can use self signed certificates.  If you are planning on using SCOT in production, you will need a valid certificate for the URL that you are planning on serving SCOT from.  
+
+To create self-signed certificate for testing:
+
+```
+$ export KEYFILENAME="/home/test/.ssl/scot4.key"
+$ export CSRFILENAME="/home/test/.ssl/scot4.csr"
+$ export CRTFILENAME="/home/test/.ssl/scot4.crt"
+$ openssl genrsa -out $KEYFILENAME 2048
+$ openssl req -key $KEYFILENAME -new -out $CSRFILENAME
+$ openssl x509 -signkey $KEYFILENAME -in $CSRFILENAME -req -days 365 -out $CRTFILENAME
+```
+
+Note: ensure that when prompted for the Common Name while generating the KEYFILE, that you enter the hostname for the server you are installing SCOT on. 
+
+### Disable Swap (as root)
+
+Disabling swap is recommended for k3s.
+
+```
+swapoff -a
+sed -i '/ swap / s/^/#/' /etc/fstab
+```
+
+### SELinux
+
+If you are running SELinux, which is default on RHEL like systems, you will need to put it into permissive mode.  This can be accomplished by entering `setenforce 0`.
 
 ### Install K3s (as root)
 
 K3s is the Kubernetes implementation we use.  Here's how to install it.  
 
+Non-SELinux:
 ```
 curl -sfLl https://get.k3s.io | INSTALL_K3S_EXEC="--prefer-bundled-bin --disable-cloud-controller" sh -
 ```
+SELinux:
+```
+curl -sfLl https://get.k3s.io | INSTALL_K3S_EXEC="--prefer-bundled-bin --disable-cloud-controller --selinux" sh -
+```
 Go to [k3s](https://docs.k3s.io/installation) for detailed installation instructions.
 
 Using a different implementation of Kubernetes is an exercise left to the reader.  
@@ -33,7 +67,7 @@ Using a different implementation of Kubernetes is an exercise left to the reader
 The installer downloads a specific version of Helm.  This is mainly because they don't have a *latest* alias on their downloads.  The Helper then extracts the tar file and installs the helm executable into the /usr/local/bin directory.
 
 ```
-HELM_VERSION="v3.14.3"
+HELM_VERSION="v3.19.0"
 HELM_TAR="helm-$HELM_VERSION-linux-amd64.tar.gz"
 curl -sfl -o $HELM_TAR https://get.helm.sh/$HELM_TAR
 tar zxvf /tmp/$HELM_TAR -C /tmp
@@ -42,23 +76,6 @@ mv /tmp/linux-amd64/helm /usr/local/bin/helm
 
 Addition Helm installation information can be found [here](https://helm.sh/docs/intro/install/)
 
-### TLS Certificates
-
-For deploying SCOT for testing purposes, you can use self signed certificates.  If you
-are planning on using SCOT in production, you will need a valid certificate for the URL
-that you are planning on serving SCOT from.  
-
-To create self-signed certificate for testing:
-
-```
-$ export KEYFILENAME="/home/test/.ssl/scot4.key"
-$ export CSRFILENAME="/home/test/.ssl/scot4.csr"
-$ export CRTFILENAME="/home/test/.ssl/scot4.crt"
-$ openssl genrsa -out $KEYFILENAME 2048
-$ openssl req -key $KEYFILENAME -new -out $CSRFILENAME
-$ openssl x509 -signkey $KEYFILENAME -in $CSRFILENAME -req -days 365 -out $CRTFILENAME
-```
-
 ### IP Address
 
 You will need to know the IP address of your server.  This command will help:
@@ -77,6 +94,7 @@ $ export NO_PROXY=localhost,127.0.0.1,.widget.com,172.16.,192.168.,*.local,.loca
 
 where $IPADDR is the IP address you discovered in the previous step.
 
+Go ahead and put those lines in your /etc/environment as well.
 
 ### Firewall Configuration (as root)
 
@@ -100,6 +118,20 @@ ufw allow from 10.43.0.0/16 to any
 
 *note*: a rule for port 6443 is not necessary for single node installs like SCOT.
 
+### Ensure that PyYAML is not Ancient (as scot4)
+
+The helper then makes sure the major version number of PyYAML is at least 5.  If it is older, then pip is used to upgrade the module.
+
+```
+PYYAMLVER=$(python3 -m pip freeze | grep -i pyyaml | awk -F== '{print $2}')
+PYYAMLMAJ=$(echo $PYYAMLVER | awk -F. '{print $1}')
+
+if [ "$PYYAMLMAJ" -lt "5" ]; then
+    echo "Upgrading PyYAML..."
+    python3 -m pip install --upgrade PyYAML
+fi
+```
+
 ### Install Tab Completions (as scot4)
 
 The following commands creates aliases and tab completions to make working on the command line easier.
@@ -113,34 +145,14 @@ echo "kubectl config set-context --current --namespace=scot4" >> /home/scot4/.ba
 
 These are not absolutely necessary, but make administrating your Kubernetes system easier.
 
-### Disable Swap (as root)
-
-Disabling swap is recommended for k3s.
-
-```
-swapoff -a
-sed -i '/ swap / s/^/#/' /etc/fstab
-```
-
 ### Create a scot4 namespace in Kubernetes 
 
 ```
 su -c 'kubectl create ns scot4' scot4
 ```
 
-### Ensure that PyYAML is not Ancient (as scot4)
-
-The helper then makes sure the major version number of PyYAML is at least 5.  If it is older, then pip is used to upgrade the module.
-
-```
-PYYAMLVER=$(python3 -m pip freeze | grep -i pyyaml | awk -F== '{print $2}')
-PYYAMLMAJ=$(echo $PYYAMLVER | awk -F. '{print $1}')
+If you receive an error creating this namespace, do not proceed until resolved.
 
-if [ "$PYYAMLMAJ" -lt "5" ]; then
-    echo "Upgrading PyYAML..."
-    python3 -m pip install --upgrade PyYAML
-fi
-```
 
 ### Merge Secrets into Kubernetes (as scot4)
 
@@ -153,6 +165,20 @@ kubectl -n scot4 apply -f scot4-chart/auto_gen_secrets.yaml
 kubectl -n scot4 apply -f scot4-chart/auto_gen_flair_secrets.yaml
 ```
 
+Again, do not proceed until any errors are resolved.
+
+### Merge Repository Secrets
+
+If you are using a custom repository, you will need to merge those secrets now.  Users pulling from GitHub may skip this step.
+
+```
+kubectl create secret docker-registry scot4-image-pull-secret \
+    --docker-server $REPO_HOST \
+    --docker-username $USER \
+    --docker-password $PASS \
+    --namespace $NS
+```
+
 ### Update values.yaml (as scot4)
 
 Update the `scot4-chart/OS_values.yaml` file with information about your environment. 
@@ -177,6 +203,8 @@ scot4.frontend.vueAppApiBase
 scot4.flair.frontendAccessibleURL
 : set to the same as scot4.api.externalApiUri
 
+You may also look at the script, `./inst_func/inst_update_values.sh` for ways to update the OS_values.yaml.
+
 ### Run Helm to Deploy (as scot4)
 
 If this is your first time installing, you will run the following command.  *NOTE*: Running this command will delete all data in your scot4 database.

content/install/migration.md

@@ -4,6 +4,8 @@ description = "Bringing forward your SCOT3 data"
 weight = 5
 +++
 
+# NOTE: Migration process is being revamped.  The process below is currently not working.  Please contact the team if you need to migrate data from a SCOT3 instance.
+
 If you are upgrading from SCOT 3 and wish to bring forward your data to SCOT 4, these are the steps to take.
 
 ## Backup SCOT 3

content/install/requirements.md

@@ -23,6 +23,23 @@ Assuming that you select to use an external database, and either an external obj
 | 8 to 32          | 8         | 32 GB  |
 | 32+              | 16        | 64 GB  |
 
+### Disk Partitioning
+If you partition your disk space, it is important to allocate a large /var partition because that is where all persistent storage will reside.
+
+There are several factors that will impact how much disk space you will need. Primarily, disk usage is driven by the amount of data being input into SCOT.  Also, configuring SCOT to use an external database server will greatly reduce your long term storage needs.  Likewise, if you use S3 to store uploaded files, storage needs can be reduced.  
+
+In the table below, we characterize the minimum space needed based on usage patterns and assuming that you will collocate you database and file storage on this system.   When in doubt, give /var as much space as you can afford.
+
+| Usage            | GB of Disk | Notes |
+| Proof of Concept | 128 | Little automated input of data, few users |
+| Light Activity   | 256 | Automated feeds of 10 to 100 items a day, User input of 0 - 100 items a day  |
+| Medium Activity  | 512 | Automated feeds of 100-500 items a day, User input of 100 - 300 items a day |
+| Heavy Activity   | 1TB | Automated feeds of 500+ items a day, User input 300+ items a day |
+
+In the table above, automated items refer to Alertgroup and Dispatch items created and any other automated input of data you might do via API.  User input is primary the creation of Entries.  
+
+Keep in mind, that your usage patterns may allow you to use less or require more space.  If at possible, configure you system so that you may adjust the /var partition to meet your needs as they change with your usage.  When planning also consider that SCOT does not do any data reduction, so usage will grow over time.  In other words, data you input into SCOT will stay there unless you manually delete it.
+
 ## Kubernetes
 
 We use [k3s](https://k3s.io) as our Kubernetes orchestrater, and [Helm](https://helm.sh) to define and manage the application.