update vpn and cocalc; also add some notes to the README for other people

Author: williamstein

README.md

@@ -4,13 +4,17 @@ URL: https://github.com/sagemathinc/cocalc-compute-docker
 
 [Compute Server Documentation](https://doc.cocalc.com/compute_server.html)
 
-There will be sections below with step-by-step instructions
+There will be sections below with step\-by\-step instructions
 about how to update, build and test the compute server Docker images and npm pacckages we manage. For things that aren't documented yet, you have to
-just read the source code, Makefiles and Dockerfiles. The Makefile is useful as a makefile, but it's not at all a traditional "bullet proof" makefile that ensure any relevant dependency is automatically built. It's a useful way to run scripts, as documented here, and that is all.
+just read the source code, Makefiles and Dockerfiles. The Makefile is useful as a makefile, but it's not at all a traditional "bullet proof" makefile that ensure any relevant dependency is automatically built. It's a useful way to run scripts, as documented here, and that is all.  Sometimes you'll just have to read the makefile to figure out how to build dependencies of a target.
+
+**IMPORTANT:** For Docker containers we always <u>_explicitly use version numbers_</u>, and NEVER rely on latest anywhere here or in CoCalc.   This is important to understand.  The version to use is explicitly listed in images.json, and in the admin panel of cocalc, you specify to use a particular version of images.json \(which could be in a fork of this repo!\).    For the cocalc npm package we do use the latest and test tags.
 
 ## Architectures: `x86_64` and `arm64`
 
-I've taken great pains to ensure we fully support both architectures. This adds complexity and extra work at every step, unfortunately.
+I've taken great pains to ensure we fully support both architectures. This adds complexity and extra work at every step, but is the right thing to do at this point.  
+
+In particular, often the pattern is something like \(1\) running `make x && make push-x` on x86\_64, then run the same on arm64, then on exactly one of the two platforms run `make assemble-x`.   In particular, you must setup separate x86\_64 and arm64 build hosts \-\- there is no way around it.  We do not try to use the internal qemu based emulation, since many of our images \(e.g., building sage\) are just way too complicated for that to work.
 
 ## How to update the cocalc npm package
 

images.json

@@ -13,7 +13,7 @@
         "tested": true
       },
       {
-        "version": "1.14.1",
+        "version": "1.14.2",
         "tag": "test",
         "tested": false
       }
@@ -65,7 +65,10 @@
     "icon": "files",
     "url": "https://github.com/sagemathinc/cocalc-compute-docker/tree/main/src/vpn",
     "source": "https://github.com/sagemathinc/cocalc-compute-docker/tree/main/src/vpn",
-    "versions": [{ "tag": "1.6", "tested": false }],
+    "versions": [
+      { "tag": "1.6", "tested": false },
+      { "tag": "1.7", "tested": false }
+    ],
     "description": "VPN - encrypted virtual private network (built on wireguard)"
   },
   "base": {

src/cocalc/check_in.py

@@ -28,7 +28,7 @@
 and when storage changes, write it /cocalc/conf/storage.json.
 
 """
-import datetime, json, subprocess, sys, time, requests
+import datetime, json, os, sys, time, requests
 import update_hosts
 from requests.auth import HTTPBasicAuth
 
@@ -105,26 +105,25 @@ def check_in():
 
 def run(cmd):
     print(f"Run '{cmd}'")
-    result = subprocess.run(cmd.split(),
-                            check=True,
-                            stdout=subprocess.PIPE,
-                            stderr=subprocess.PIPE)
-    print("Done - Command Output:", result.stdout.decode())
+    os.system(cmd)
 
 
 def update_vpn():
     image = json.loads(open('/cocalc/conf/vpn.json').read())['image']
     # Process latest vpn configuration
-    run(f'docker run -it --rm --network host --privileged -v /cocalc/conf:/cocalc/conf {image}'
+    run(f'docker run --rm --network host --privileged -v /cocalc/conf:/cocalc/conf {image}'
         )
     # Update /etc/hosts on the root VM
     update_hosts.update_hosts()
     # Update /etc/hosts in the compute docker container
-    run('docker exec -it compute sudo /cocalc/update_hosts.py')
+    run('docker exec compute sudo /cocalc/update_hosts.py')
     # NOTE: you can't just bind mount /etc/hosts into the container, and you can't just edit /etc/hosts
     # from a bind mounted /etc in a container -- i.e., every approach to *directly* using /etc/hosts
     # that I tried failed, and should fail (as it would lead to subtle bugs).  Being explicit with the update_hosts.py
     # command is much better.
+    if os.path.exists('/cocalc/conf/pings.sh'):
+        # launch pings in the background to keep us on the vpn from behind our firewall.
+        os.system("exec /cocalc/conf/pings.sh &")
 
 
 if __name__ == '__main__':

src/vpn/conf.py

@@ -111,7 +111,7 @@ def write_conf(compute_server_id, nodes):
         for peer in peers:
             pings += f"ping -r -I wg{compute_server_id} -n -i 15 -c 4 {peer['vpn_ip']} &\n"
         if len(peers) > 0:
-            open(f'pings.sh', 'w').write(pings)
+            open('pings.sh', 'w').write(pings)
 
 
 HOSTS_COMMENT = '### COCALC VPN -- EVERYTHING BELOW IS AUTOGENERATED -- DO NOT EDIT'

src/vpn/run.sh

@@ -17,6 +17,9 @@ cp -v hosts /cocalc/conf/hosts
 if [ -e pings.sh ]; then
    cp -v pings.sh /cocalc/conf
    chmod a+x /cocalc/conf/pings.sh
+else
+   # ensure it is not there if not needed
+   rm -f /cocalc/conf/pings.sh
 fi
 
 # actually do the configuration -- if up doesn't work, e.g., because