Skip to content
This repository has been archived by the owner on Jul 6, 2020. It is now read-only.

Runtime v2 #745

Draft
wants to merge 76 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
8a9845d
Started runtime rewrite
jonathan-j-lee Dec 30, 2018
7620fcc
Updated Dockerfile
jonathan-j-lee Oct 7, 2019
0ae1f6e
Removing old jank shared memory code
jonathan-j-lee Oct 7, 2019
f35ddbe
Updating documentation
jonathan-j-lee Oct 14, 2019
856041b
Started runtime node client
jonathan-j-lee Oct 15, 2019
0e09dc0
Updated Dockerfile to build libzmq with DRAFT support
jonathan-j-lee Oct 15, 2019
25acd60
Added device schema
jonathan-j-lee Oct 15, 2019
e1f0004
Added starting SDK API
jonathan-j-lee Oct 31, 2019
b7da1bd
Refactoring process supervisor
jonathan-j-lee Jan 10, 2020
f9c28fd
Can get hotplug events
jonathan-j-lee Jan 14, 2020
536b19f
Added rudimentary RPC implementation
jonathan-j-lee Jan 17, 2020
7ce14cc
Using child process for StudentCodeExecutor
jonathan-j-lee Jan 20, 2020
a91723b
Started Dawn CLI
jonathan-j-lee Jan 24, 2020
8990535
Using single process again for executor
jonathan-j-lee Jan 26, 2020
0bad58c
Fixed device initialization bug
jonathan-j-lee Jan 26, 2020
df0f580
Started documenting studentapi
jonathan-j-lee Jan 26, 2020
026a97c
Added TTL for devices
jonathan-j-lee Jan 27, 2020
7959f29
Removed old tests
jonathan-j-lee Jan 30, 2020
489f5e5
Implemented Runtime client
jonathan-j-lee Feb 1, 2020
c758061
Fixed multithreading issue with ZMQ sockets
jonathan-j-lee Feb 1, 2020
4d1526c
Fixed log capture (random crash in supervisor)
jonathan-j-lee Feb 2, 2020
c7173d8
Started work on Runtime node SDK
jonathan-j-lee Feb 2, 2020
32f796d
[runtime] Added basic linter program that creates dictionaries for ev…
0neDividedbyZer0 Feb 15, 2020
3e29e08
Added and fixed linter.py
0neDividedbyZer0 Feb 22, 2020
c821446
Included runtime client example
jonathan-j-lee Feb 29, 2020
46e85d6
Fixed None student code
jonathan-j-lee Mar 5, 2020
79bec8a
Added autopep8 autoformatter, autoformatted code
jonathan-j-lee Mar 5, 2020
29cb687
Set up sphinx documentation, added 'Getting Started' guide
jonathan-j-lee Feb 3, 2020
6f65db5
Documented workflow and how to use Git
jonathan-j-lee Feb 3, 2020
9fb2f74
Started adding architecture notes
jonathan-j-lee Feb 3, 2020
6ad9df5
Started section on shared memory
jonathan-j-lee Feb 3, 2020
db93206
Documented student API
jonathan-j-lee Mar 5, 2020
0ca5960
Added test coverage calculation
jonathan-j-lee Mar 5, 2020
73d5e70
Renamed routing module
jonathan-j-lee Mar 5, 2020
155842a
Added device aliases test
jonathan-j-lee Mar 5, 2020
302ac28
Added Gamepad studentapi tests
jonathan-j-lee Mar 6, 2020
4816c30
Finished student API tests
jonathan-j-lee Mar 6, 2020
743c037
Started documenting device lib
jonathan-j-lee Mar 6, 2020
87a6be2
Added more device library tests (sensor UID bytes, get field bytes)
jonathan-j-lee Mar 6, 2020
59dc1b8
Added device buffer cross-memory test
jonathan-j-lee Mar 6, 2020
0547bc2
Refactored RPC logic
jonathan-j-lee Mar 6, 2020
70d66fe
Require 80% test coverage in build, can send dev disable
jonathan-j-lee Mar 6, 2020
779af6d
Implemented RPC notifications
jonathan-j-lee Mar 6, 2020
53e55de
Implemented round-trip-time, refactoring for better encapsulation
jonathan-j-lee Mar 6, 2020
ac7db37
Action executor tests and docs
jonathan-j-lee Mar 6, 2020
e0d0785
Cleaning up legacy files
jonathan-j-lee Mar 6, 2020
8d9080f
Taking steps to fix the build
jonathan-j-lee Mar 7, 2020
1c24b8c
Merged changes from upstream, removing more legacy files
jonathan-j-lee Mar 7, 2020
90421c2
Upgraded all Dawn packages
jonathan-j-lee Feb 4, 2020
8265eb4
Started fixing eslint issues
jonathan-j-lee Feb 5, 2020
64fe8a3
Fixed lint issues
jonathan-j-lee Feb 5, 2020
324e87f
Can open Dawn with yarn run start
jonathan-j-lee Feb 5, 2020
851cf11
Removed outdated fake runtime
jonathan-j-lee Feb 5, 2020
fff1657
Actually fixed Dawn build
jonathan-j-lee Mar 7, 2020
2702eac
Upgraded node version to next LTS, more runtime docker build changes
jonathan-j-lee Mar 7, 2020
6b73647
Added linux headers to install
jonathan-j-lee Mar 7, 2020
5391d9d
Started Runtime-Dawn integration
jonathan-j-lee Mar 7, 2020
ff1a0ee
Can send/receive datagrams/logs from Dawn
jonathan-j-lee Mar 7, 2020
b34e477
Can communicate some data to/from Dawn
jonathan-j-lee Mar 7, 2020
3de8c7d
Working mode set
jonathan-j-lee Mar 7, 2020
c856b8d
Attempting to fix hibike build
jonathan-j-lee Mar 7, 2020
cd05652
Implemented device subscriptions
jonathan-j-lee Mar 7, 2020
981c38a
Revert Hibike version
jonathan-j-lee Mar 7, 2020
73019ed
Attempting to fix Dawn build and lint
jonathan-j-lee Mar 7, 2020
e3b05f0
More aggressively reduce size of production image
jonathan-j-lee Mar 7, 2020
7410837
Fixed host binding bug in runtime SDKs
jonathan-j-lee Mar 7, 2020
2e8918b
Updated dependencies
jonathan-j-lee Mar 7, 2020
9a44feb
Fixed student code module name
jonathan-j-lee Mar 7, 2020
b6374e7
Fixed null executor bug
jonathan-j-lee Mar 7, 2020
84d10e1
Disable uvloop, allow strip to fail on bad file formats
jonathan-j-lee Feb 16, 2020
3aa0bed
Fixing some bugs
jonathan-j-lee Mar 7, 2020
5e20f76
Added KoalaBear parameters
jonathan-j-lee Mar 7, 2020
570eb4d
Added KoalaBear parameters
jonathan-j-lee Mar 7, 2020
920cef4
Fixed Koala Bear ID
jonathan-j-lee Mar 9, 2020
f9c5361
Merge branch 'runtime/master' of github.com:pioneers/PieCentral into …
jonathan-j-lee May 23, 2020
e3bedbb
Considerable refactoring
jonathan-j-lee May 30, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -404,3 +404,6 @@ hibike/virtual_devices.txt
/dawn-*
/artifacts
.vagrant

# Exported Docker containers
runtime-*.tar.gz
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
8.12.0
13.10.1
15 changes: 7 additions & 8 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
dist: xenial
sudo: required
dist: bionic
os:
- linux
matrix:
jobs:
include:
- name: Dawn
language: node_js
Expand All @@ -13,21 +12,21 @@ matrix:
env: TEST_DIR=hibike/travis
- name: Runtime
language: python
python: 3.6
arch: arm64
python: 3.8
env: TEST_DIR=runtime
- name: Shepherd
language: python
python: 3.7
python: 3.8
env: TEST_DIR=shepherd
- name: Foundry
language: python
python: 3.7
python: 3.8
env: TEST_DIR=foundry
addons:
apt:
packages:
- python3-pip
- libasound2-dev
- socat
env:
global:
Expand Down Expand Up @@ -74,7 +73,7 @@ branches:
deploy:
provider: script
script: ./DevOps/pipeline/deploy-artifacts.sh "$APP_ID" "$TRAVIS_TAG" "$(pwd)/artifacts"
skip_cleanup: true
cleanup: true
on:
branch: master
tags: true
Expand Down
5 changes: 2 additions & 3 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
# Project managers are automatically requested for review
/dawn/ @Funplings @kdarken
/DevOps/ @herojelly @rs4ostrea
/foundry/ @herojelly @rs4ostrea
/ansible-protos/ @benliao1
/DevOps/ @rs4ostrea
/foundry/ @rs4ostrea
/runtime/ @benliao1
/hibike/ @benliao1
/shepherd/ @FIREdog5 @kaipinryankoh
48 changes: 48 additions & 0 deletions DevOps/fabric/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# fabric
Fabric is the latest iteration of the PiE kit operating system image.
In year 11 (Spring 2019), PiE transitioned from using the Beaglebone Black (BBB) as its kit single-board computer (SBC) to the Raspberry Pi 3B+.
The Raspberry Pi offers several advantages over the Beaglebone:
* The Pi's ARM Cortex-A53 has four physical cores (up from the single core on the BBB's ARM Cortex-A8) yielding increased parallelism and better performance.
The time-to-boot was also greatly reduced.
* The Pi loads its disk from a removable microSD card, which can be swapped easily to patch critical issues.
* The Pi has on-board WiFi, which cannot be dislodged by mechanical shocks.
This also eliminates the need to purchase WiFi dongles and build the associated drivers.
* The Pi community is more active, with better documentation and support.

## Building the Fabric Base Image
Download the [latest Raspbian Lite image](https://downloads.raspberrypi.org/raspbian_lite_latest).
At the time of this writing, this image was:
```
Release date: 2018-11-13
Kernel version: 4.14
SHA-256: 47ef1b2501d0e5002675a50b6868074e693f78829822eef64f3878487953234d
```
Using a app like [Etcher](https://www.balena.io/etcher/), burn this image to a microSD card.
This will create two partitions named `boot` and `rootfs`.
```bash
$ touch /path/to/boot/ssh
$ cp /path/to/fabric/wpa_supplicant.conf /path/to/boot
```

### Networking

### Raspi

```
Boot Options > Desktop / CLI > Console
Boot Options > Wait for Network at Boot > No
Localisation Options > Change Locale > en_US.UTF-8 UTF-8 (disable en_GB.UTF-8 UTF-8) > en_US.UTF-8.UTF-8
Localisation Options > Change Timezone > America > Los_Angeles
Localisation Options > Change Keyboard Layout > Logitech Generic Keyboard > Other > English (US) > The default for the keyboard layout > No compose key
Interfacing Options > Camera > No
Interfacing Options > SSH > Yes
Interfacing Options > VNC > No
Interfacing Options > SPI > No
Interfacing Options > I2C > No
Interfacing Options > Serial > Yes
Interfacing Options > 1-Wire > No
Interfacing Options > Remote GPIO > No
Interfacing Options > Advanced Options > Expand Filesystem
```

Finish, then reboot.
22 changes: 22 additions & 0 deletions DevOps/fabric/dhcpd.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# dhcpd.conf

option domain-name "whiterose.local";
option domain-name-servers 8.8.8.8, 8.8.4.4;
option subnet-mask 255.255.255.0;
option routers 192.168.100.100;
option broadcast-address 192.168.100.255;

default-lease-time 600;
max-lease-time 7200;

# Use this to send dhcp log messages to a different log file (you also
# have to hack syslog.conf to complete the redirection).
log-facility local7;

subnet 192.168.100.0 netmask 255.255.255.0 {
range 192.168.100.101 192.168.100.150;
host odroid-arch {
hardware ethernet 00:1e:06:36:22:b3;
fixed-address 192.168.100.101;
}
}
25 changes: 25 additions & 0 deletions DevOps/fabric/netbridge.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/bin/bash

set -e
ip link
read -e -p "Enter the interface to accept from: " accept
read -e -p "Enter the interface to forward to: " forward
read -e -p "Enter the remote's IP address: " remote_ip
read -e -p "Enter the user to log into the remote as: " remote_user

host_ip=$(ip addr show "$accept" | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)
echo "This machine's IP address on '$accept': $host_ip"

iptables -F
iptables -X

iptables -t nat -A POSTROUTING -o "$accept" -j MASQUERADE
iptables -A FORWARD -i "$accept" -o "$forward" -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i "$forward" -o "$accept" -j ACCEPT

iptables -t nat -A POSTROUTING -o "$forward" -j MASQUERADE
iptables -A FORWARD -i "$forward" -o "$accept" -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i "$accept" -o "$forward" -j ACCEPT

sysctl -w net.ipv4.ip_forward=1
ssh "$remote_user"@"$remote_ip" "sudo ip route add default via $host_ip"
50 changes: 50 additions & 0 deletions DevOps/fabric/odroid_arch_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/bin/bash

# Give the current user sudo privileges (no password).
function sudo-init {
echo "$1" | su root -c "
set -e
cp /etc/sudoers /tmp/sudoers.tmp
sed -ir 's/^# %wheel ALL=(ALL) NOPASSWD: ALL$/%wheel ALL=(ALL) NOPASSWD: ALL/g' /tmp/sudoers.tmp
diff --color /etc/sudoers /tmp/sudoers.tmp
visudo -c -f /tmp/sudoers.tmp
mv -f /tmp/sudoers.tmp /etc/sudoers
usermod -aG wheel $(whoami)
" 2> /dev/null
echo "'$(whoami)' has been added to the 'wheel' group."
}

function systemd-init {
$sudo timedatectl set-timezone "$1"
echo "The time is: $(date)"
$sudo whoami
}

# Initialize the pacman keyring, install core packages,
# and install yay (the AUR helper).
function pacman-init-install {
$sudo pacman-key --init
$sudo pacman-key --populate archlinuxarm
$sudo pacman --noconfirm -Syu
$sudo pacman --noconfirm -S $@
$sudo sudo acman --noconfirm -Scc
git clone https://aur.archlinux.org/yay.git
cd yay && makepkg --noconfirm -si && cd .. && rm -rf yay
}

# Set to 'sudo' to run commands as root,
# or empty to run as the current user.
sudo=sudo

set -e
sudo-init root # Default root password
systemd-init America/Los_Angeles
sudo pacman-init-install make binutils gcc sudo fakeroot \
go vim git tmux htop wavemon wget docker \
python3 python-pip

echo 'export PATH=$PATH:~/.local/bin' >> ~/.bashrc
pip install --user pipenv
git clone https://github.com/pioneers/PieCentral.git
cd PieCentral/runtime
pipenv install
2 changes: 1 addition & 1 deletion DevOps/frankfurter/resources/runtime.service
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ AssertPathExists=/home/ubuntu/PieCentral/hibike
Type=simple
WorkingDirectory=/home/ubuntu/PieCentral/runtime
Environment="PYTHONPATH=/home/ubuntu/PieCentral/hibike:/home/ubuntu/PieCentral/runtime"
ExecStart=/usr/bin/env python3 runtime.py
ExecStart=/usr/bin/env pipenv run python3 runtime.py

[Install]
WantedBy=sysinit.target
2 changes: 1 addition & 1 deletion DevOps/frankfurter/scripts/install/master_setup
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ function install_apt_packages {
sudo apt update -y
sudo apt upgrade -y
# FIXME: small issue with `/etc/issue` needing to be replaced
sudo apt install -y man-db make build-essential gcc git vim tmux htop curl zip unzip linux-headers-$(uname -r)
sudo apt install -y sudo net-tools libpython3.7-dev usbutils man-db make build-essential gcc git vim tmux htop curl zip unzip linux-headers-$(uname -r)
sudo apt install -y python3 python3-dev python3-pip # Python dependencies
sudo apt install -y devmem2
sudo apt clean -y
Expand Down
28 changes: 0 additions & 28 deletions Dockerfile

This file was deleted.

3 changes: 0 additions & 3 deletions ansible-protos/README.md

This file was deleted.

32 changes: 0 additions & 32 deletions ansible-protos/ansible.proto

This file was deleted.

25 changes: 0 additions & 25 deletions ansible-protos/notification.proto

This file was deleted.

30 changes: 0 additions & 30 deletions ansible-protos/runtime.proto

This file was deleted.

20 changes: 14 additions & 6 deletions dawn/.babelrc
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
{
"presets": [
["env", {
"targets": {
"electron": "1.7"
[
"@babel/preset-env",
{
"targets": {
"electron": "1.7"
}
}
}],
"react",
"stage-3"
],
"@babel/preset-react"
],
"plugins": [
"@babel/plugin-syntax-dynamic-import",
"@babel/plugin-syntax-import-meta",
"@babel/plugin-proposal-class-properties",
"@babel/plugin-proposal-json-strings"
]
}
1 change: 1 addition & 0 deletions dawn/.eslintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
}
},
"rules": {
"react/destructuring-assignment": "off",
"react/forbid-prop-types": "off",
"react/jsx-filename-extension": "off",
"react/no-string-refs": "off",
Expand Down
Loading