Documentation/binderhub

doc/RANCHER_PROXMOX.md

@@ -0,0 +1,102 @@
+Deploy FADI with Rancher and Proxmox
+=============
+
+* [1. Upload ISO on Proxmox Node](#1-Upload-IS0-on-Proxmox-Node)
+* [2. Install Rancher](#2-Install-Rancher)
+* [3. Add docker-machine driver](#1-Add-docker-machine-driver)
+* [4. Create Cluster With Rancher](#2-Create-Cluster-With-Rancher)
+     * [Create Node Template](#Create-Node-Template)
+     * [Create Cluster](#Create-Cluster)
+     * [Create The Nodes](#Create-The-Nodes)
+* [5. Manage the provisioning of the persistent volumes](#5-Manage-the-provisioning-of-the-persistent-volumes)
+* [6. Control Cluster from Local PC](#3-Control-Cluster-from-Local-PC)
+
+This page provides information on how to create a Kubernetes cluster on [Proxmox](https://www.proxmox.com/en/) as IaaS provider using [Rancher](https://rancher.com/).
+
+## 1. Upload RancherOS ISO on Proxmox
+
+<a href="https://www.proxmox.com/" alt="ProxMox"> <img src="../images/logos/Proxmox.png" width="150px" /></a>
+
+> "Proxmox VE is a complete open-source platform for enterprise virtualization. With the built-in web interface you can easily manage VMs and containers, software-defined storage and networking, high-availability clustering, and multiple out-of-the-box tools on a single solution."
+
+First, download the [rancheros-proxmoxve-autoformat.iso](https://github.com/rancher/os/releases/latest) image and upload it to one of your Proxmox nodes.
+
+## 2. Install Rancher
+
+We consider in this documentation that you already have deployed Rancher. However, we give you the instructions that we have followed to deploy our Rancher server: [https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/](https://rancher.com/docs/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/).
+
+## 3. Add docker-machine driver
+
+Then, you need to allow Rancher to access Proxmox. We have contributed to upgrade a existing [docker-machine driver](https://github.com/lnxbil/docker-machine-driver-proxmox-ve/releases/download/v3/docker-machine-driver-proxmoxve.linux-amd64
+) to make it compatible with Rancher.
+
+To add this driver in your Rancher, follow these steps :
+
+![Proxmox driver](../images/installation/proxmoxdriver.gif)
+
+Driver Url:
+```
+https://github.com/lnxbil/docker-machine-driver-proxmox-ve/releases/download/v3/docker-machine-driver-proxmoxve.linux-amd64
+```
+
+## 4. Create Cluster With Rancher
+
+<a href="https://rancher.com/" alt="Rancher"> <img src="../images/logos/rancher.png" width="150px" /></a>
+
+> "Rancher is open source software that combines everything an organization needs to adopt and run containers in production. Built on Kubernetes, Rancher makes it easy for DevOps teams to test, deploy and manage their applications."
+
+After connecting to rancher, you can follow these steps:
+
+### Create Node Template
+
+This is where you have to define the templates you want to use for your nodes (both masters and workers nodes). To do so, you can go to: `profile (top right corner)`  > `Node templates` > `Add Template` :
+
+Choose `Proxmoxve`
+![Proxmoxve](../images/installation/Proxmoxve.png)
+and then fill the rest of the fields like the IP of the Proxmox `i.e. proxmoxHost`, the username/password `i.e. proxmoxUserName, proxmoxUserPassword `, storage of the image file `vmImageFile ` which is in our case `local:iso/rancheros-proxmoxve-autoformat.isot` and coming down to the resources you want to allocate for your node `i.e. nodevmCpuCores, vmMemory, vmStorageSize `.
+
+### Create Cluster
+
+To create your cluster:
+
+ `Cluster`  > `Add Cluster` > `Proxmoxve`
+
+you will need to give a name to your cluster, then precise the nodes in the cluster, at first start with **one master node**, you give it a name, choose of the templates created earlier for that node and then tick all 3 boxes for `etcd`, `Control Plane` and `Worker`, then choose the kubernetes version and click `create`.
+
+> "you'll have to wait the `VM creation`, `the RancherOS install` and `the IP address retrieving` and that might take a while "
+
+ Once the master node gets its IP address, go to `Cluster`  > `Edit Cluster` and add an other worker node, untick the worker box from the master node and tick it in the new worker node. It should look to something like this:
+ ![Proxmoxve](../images/installation/workernode.png)
+
+If a second (or more) node (master or worker) is needed, you can either add an other one with a different template by following the same way we just did. You can also add as much nodes as you want using the same template by simply going to  `YourCluster (not global)`  > `nodes` > `+` and it will add an other node of the same kind:
+
+ ![Proxmoxve](../images/installation/addnode.png)
+
+## 5. Manage the provisioning of the persistent volumes.
+
+#### StorageOS
+
+Once all your nodes are up and running, it's time to deploy your services. But before, you need to set your default PVC for the persistent volumes. To do so, we first need to deploy the volume plugin `StorageOS`, go to `YourCluster (not global)`  > `system` > `apps` > `launch` and search for `StorageOS`. Make sure all the fields are filled correctly, like the following screenshot:
+
+![Proxmoxve](../images/installation/StorageOS.png)
+
+and now, launch it 🚀.
+
+> "launching apps usually takes several minutes, you're going to need to wait a few minutes till the "
+
+Be Careful that this service gives the posibility to allocate maximum 50Gi with the basic License.
+
+![Proxmoxve](../images/installation/StorageOS_limits.png)
+#### Manualy
+
+TBT
+
+### Deploy FADI
+![defaultpvc](../images/installation/defaultpvc.png)
+
+ selector set as default the StorageClass
+#### Longhorn
+7. edit the values yaml of fadi
+8. run fadi
+
+## 4. Control Cluster from Local PC

doc/README.md

@@ -6,5 +6,8 @@ FADI Documentation
 * [Reverse proxy](REVERSEPROXY.md) - Traefik reverse proxy configuration
 * [Security](SECURITY.md) - SSL setup
 * [TSimulus](TSIMULUS.md) - how to simulate sensors and generate realistic data with [TSimulus](https://github.com/cetic/TSimulus)
-
-For tutorials and examples, see the [examples section](../examples/README.md)
+* [Sample self-hosted infrastructure](RANCHER_PROXMOX.md) - How to install FADI on a self hosted infrastructure using 
+    * [Proxmox](https://www.proxmox.com/en/) as a self-hosted private cloud (IaaS) provider. It provides virtual machines for the various Kubernetes nodes.
+    * [Rancher](https://rancher.com/what-is-rancher/what-rancher-adds-to-kubernetes/) to manage (install, provision, maintain, upgrade, ...) several Kubernetes clusters, e.g. when needing several environments or several well separated tenant installations.
+
+For tutorials and examples, see the [examples section](../examples/README.md)

examples/binderhub/README.md

@@ -0,0 +1,108 @@
+BinderHub
+===========
+
+* [What is BinderHub](#what-is-binderhub)
+* [Add Binderhub to FADI](#add-binderhub-to-fadi)
+* [Basic example of BinderHub workflow](#basic-example-of-binderhub-workflow)
+    * [BinderHub Configuration](#binderhub-configuration)
+    * [Build the image](#building-the-image)
+    * [Publish the image](#publishing-the-image)
+    * [View the project in JupyterHub](#view-the-project-in-jupyterhub)
+    * [Launch the project](#launch-the-project)
+* [References](#references)            
+
+## What is Binderhub
+
+<a href="https://binderhub.readthedocs.io/en/latest/" title="BinderHub"><img src="images/binderhub.png" width="200px" alt="Binderhub" /></a>
+
+> *The primary goal of BinderHub is creating custom computing environments that can be used by many remote users. BinderHub enables an end user to easily specify a desired computing environment from a Git repo. BinderHub then serves the custom computing environment at a URL which users can access remotely.*
+
+> *BinderHub will build Docker images out of Git repositories, and then push them to a Docker registry so that JupyterHub can launch user servers based on these images*
+
+## Prerequisites
+
+We assume in this documentation that 
+
+* you have already a Kubernetes cluster deployed. If not, you can refer and follow our [Installation guide](https://github.com/cetic/fadi/blob/master/INSTALL.md) to install a local minikube cluster. 
+* you will have a [valid Docker account](https://hub.docker.com/signup/).
+
+## Add BinderHub to FADI
+
+Follow the following steps to install FADI with BinderHub on your cluster:
+
+1. Clone this repository and go to the [BinderHub example folder](/examples/binderhub):
+
+```bash
+git clone https://github.com/cetic/fadi.git fadi
+cd fadi/examples/binderhub
+```
+
+2. Edit the [`config.yaml`](/examples/binderhub/config.yaml) file to set your Docker credentials (you need a Docker account because the containerized notebook will be stored on the official Docker registry - [Docker Hub](https://hub.docker.com/signup/)) and the name of your project:
+
+```yaml
+config:
+    BinderHub:
+      use_registry: true
+      image_prefix: <DOCKER_ID>/<PROJECT_NAME>-
+registry:
+    username: <DOCKER_ID>
+    password: <DOCKER_PASSWORD>
+```
+
+> The prefix of an image is always built in the same way when you want to save a container image in the Docker Hub. First the *username*, *a backslash* and then the *name of the project*. In our case, BinderHub will be responsible for adding a tag to version the container image.
+
+3. Launch the Helm scripts, those will deploy all the FADI services and BinderHub on the cluster (this may take some time).
+```bash
+../helm/deploy.sh
+./deploy_binderhub.sh
+# see deploy.log for connection information to the various services
+```
+> The first script deploys FADI. It is important to do this from the **fadi/examples/binderhub** folder so that the **values.yaml** file is taken into account and that JupyterHub is not deployed. The second script will deploy BinderHub.
+
+## Basic example of BinderHub workflow
+
+This example is used to test the deployment of BinderHub with a project using a `requirement.txt` file.
+
+### BinderHub configuration
+
+The first step is to access the BinderHub page.
+
+If your Kubernetes cluster is deployed with **minikube**, the command `minikube service list` will allow you to recover the address to copy/paste in your browser. On the other hand, in the case of a bare-metal cluster, the command: `kubectl get svc -n binderhub` will allow you to recover the service port.
+As this service is of `NodePort` type, you can use the IP address of any node to reach the BinderHub home page.
+
+Once on the BinderHub page is opened, simply fill in the fields with the following inputs:
+
+- **GitHub repository name or url:** `https://github.com/binder-examples/requirements`
+- **branch:** `master`
+
+Finally, click on the `launch` button:
+
+![images/1_input.png](images/1_input.png)
+
+### Build the image
+
+From now on, everything is automated. BinderHub will create a container image based on what resides in the git project.
+
+![images/2_building.png](images/2_building.png)
+
+### Publish the image
+
+The image is now built, it will be saved in your Docker repository, e.g. ([hub.docker.com](https://hub.docker.com)). It will no longer be necessary to go back through the build stages to access this project. The duration of this step will be dependent on your internet connection (the image size is almost 600MB).
+
+![images/3_pushing.png](images/3_pushing.png)
+
+### View the project in JupyterHub
+
+The project will automatically be launched in JupyterHub once all the previous steps are completed.
+
+![images/4_jupyter.png](images/4_jupyter.png)
+
+### Launch the project
+
+You can now enjoy your work environment!
+
+![images/5_notebook.png](images/5_notebook.png)
+
+## References
+
+- [https://binderhub.readthedocs.io/en/latest/](https://binderhub.readthedocs.io/en/latest/)

examples/binderhub/config.yaml

@@ -0,0 +1,29 @@
+config:
+  BinderHub:
+    use_registry: true
+    image_prefix: <DOCKER_ID>/<PROJECT_NAME>-
+registry:
+  username: <DOCKER_ID>
+  password: <DOCKER_PASSWORD>
+
+
+dind:
+  enabled: true
+  daemonset:
+    image:
+      name: docker
+      tag: 18.09.2-dind
+
+jupyterhub:
+  hub:
+    services:
+      binder:
+        apiToken: 8675d9b1ff09ff2502886dfd4f0f36fd45c916372536aa09613cc9c5563d8d1d
+  proxy:
+    secretToken: 613e0ace7628f92bab45478873451f00e65977ca6a61d2f9255667b7bbd71d0e
+    db:
+      type: sqlite-memory
+    service:
+      type: NodePort
+      nodePorts:
+        http: 30902

examples/binderhub/deploy_binderhub.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# This script will deploy the various FADI services on a Kubernetes cluster using Helm and kubectl
+# See https://github.com/cetic/fadi/examples/binderhub/ for usage documentation
+# Usage: ./deploy_binderhub.sh [namespace]
+set -o errexit
+
+LOG_FILE="deploy.log"
+[ -e ${LOG_FILE} ] && rm ${LOG_FILE}
+exec > >(tee -a ${LOG_FILE} )
+exec 2> >(tee -a ${LOG_FILE} >&2)
+
+printf "\n\nFadi is deployed... Now helm will Install BinderHub in FADI...\n"
+# Install BinderHub in FADI
+kubectl get namespace binderhub 2> /dev/null || kubectl create namespace binderhub
+helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
+helm repo update
+helm upgrade --install binderhub jupyterhub/binderhub --version=0.2.0-n132.h1a8ce62  -f ./config.yaml --namespace binderhub
+
+sleep 5s
+# Get the node IP where JupyterHub is deployed
+nodeIP=$(kubectl get po -n binderhub -o wide |sed -n '/proxy/p' |awk '{ print $7 }')
+if [ $nodeIP = "minikube" ];then
+  nodeIP=$(minikube ip)
+fi
+
+# Upgrade the BinderHub release with hub_url
+printf "\n\n Found JupyterHub deployed at $nodeIP\n"
+helm upgrade binderhub jupyterhub/binderhub --version=0.2.0-n132.h1a8ce62 -f ./config.yaml --set config.BinderHub.hub_url=http://$nodeIP:30902 --namespace binderhub
+printf "\n\nInstallation successful, !\n"

examples/binderhub/values.yaml

@@ -0,0 +1,8 @@
+---
+# Default values for FADI are defined here: https://github.com/cetic/helm-fadi.
+# This is a YAML-formatted file.
+# Declare variables to be passed into your templates.
+# Values you define here will overwrite default values from helm-fadi.
+
+jupyterhub:
+  enabled: false