Merge pull request #22 from mlrepa/dev

Update MLPanel BASE edition - v0.2

README.md

@@ -1,234 +1,71 @@
-# mlpanel
+# MLPanel
 
-GitBook docs: https://mlrepa.gitbook.io/mlpanel/
+![MLPanel](https://gblobscdn.gitbook.com/assets%2F-Lzh1AArhTugENuEmRGS%2F-M6jmR5ihP5f3uoAqt1Y%2F-M6se7wKEtDD0HjtK3_O%2Fimage.png?alt=media&token=a5428f6c-2bfb-448d-8972-e25311515f4f)
 
-## Preparations
 
-### Create *config/.env*
-
-* **if you are going to use GCS bucket as artifact storage**, put into *config/* Google credentials json
+Docs: [MLPanel documentation](https://mlrepa.gitbook.io/mlpanel/)
 
-* create file *config/.env*:
-```.env
-
-# Common
-HOST_MODE=[local|remote]
-WORKSPACE=/path/to/workspace/folder/on/host
-GOOGLE_APPLICATION_CREDENTIALS=/home/config/<credentials.json>
-
-# Deploy
-GCP_PROJECT=vision-230607
-GCP_ZONE=us-east1-b
-GCP_MACHINE_TYPE=g1-small
-GCP_OS_IMAGE=ubuntu1804-docker
-GCP_BUCKET=<bucket>
-MODEL_DEPLOY_DOCKER_IMAGE=meisterurian/mlflow-deploy
-MODEL_DEPLOY_DEFAULT_PORT=5000
-MODEL_DEPLOY_FIREWALL_RULE=mlflow-deploy
-
-# Projects
-LOGLEVEL=<DEBUG|INFO|WARNING|WARN|ERROR|FATAL|CRITICAL>
-ARTIFACT_STORE=[mlruns|gs://<bucket>]
-MLFLOW_TRACKING_SERVERS_PORTS_RANGE=5000-5020
-
-# UI
-MLPANEL_UI_BRANCH=feature/ui-features
-REACT_APP_API_URL=http://0.0.0.0:8080/
-```
-
-
-### Description of environment variables
-
-#### Common
-
-#####  ***HOST_MODE***
-
-Required: No.
-
-Default: local.
-
-Defines ip address of tracking servers: 
-
-* if *local* then ip address = "0.0.0.0" (e.i. **mlpanel** is running and available locally);
-* if *remote* then ip address = <external_ip> (e.i. **mlpanel** is running on some remote server and available by external ip);
-external ip is detected automatically.
-
-
-##### ***WORKSPACE***
-
-Required: Yes.
-
-WORKSPACE sets path to workspace folder - any path on you disk (taking into account permission).
-
-The folder contains:
-
-* projects database;
-* deploy database;
-* folders for each project, each folder is named by the project id and contains:
-    * mlflow.db - mlflow database;
-    * artifact storage folder - if variable *ARTIFACT_STORE* points to local path.
-
-**Note**: if artifacts are stored locally then folder WORKSPACE can have huge size, it depends on 
-number of projects and artifacts (especially ML models) volume.
-
-
-##### ***GOOGLE_APPLICATION_CREDENTIALS***
-
-Required: No.
-
-Defines path to your Google credentials JSON (https://cloud.google.com/docs/authentication/getting-started) 
-inside docker container.
-
-Consists from two parts: fixed (/home/config/ - exact path of folder config inside container) and name of your JSON.  
-The JSON you must put to folder config/.
-
-**Note** define *GOOGLE_APPLICATION_CREDENTIALS* if you are going to use Google Cloud Storage 
-(https://cloud.google.com/storage) or/and deploy models on Google Compute Engine (GCE, https://cloud.google.com/compute).
-
-
-#### Deploy
-
-Now two types of deployments exist:
-
-* local - deploys model locally (as new MLflow model server process inside container);
-* gcp - start new GCE instance and run MLflow model server process on it.
-
-##### ***GCP_PROJECT***
-
-Required: No (Yes for deploying on GCE).
-
-Id of your project Google Cloud Platform. Define it if you are going to deploy models on GCE.
-
-
-##### ***GCP_ZONE***
-
-Required: No (Yes for deploying on GCE).
-
-Zone type for instances. More about zone: https://cloud.google.com/compute/docs/regions-zones.
-
-##### ***GCP_MACHINE_TYPE***
-
-Required: No (Yes for deploying on GCE).
-
-GCE machine type. More about machine types: https://cloud.google.com/compute/docs/machine-types.
-
-Minimum requirement of machine type is *g1-small*.
-
-##### ***GCP_OS_IMAGE***
-
-Required: No (Yes for deploying on GCE).
-
-Your private (custom) OS image. Docker must be installed inside OS image. 
-Working with OS images: https://cloud.google.com/compute/docs/images/create-delete-deprecate-private-images.
-
-
-##### ***GCP_BUCKET***
-
-Required: No (Yes for deploying on GCE).
-
-Google Storage bucket name.
-
-
-##### ***MODEL_DEPLOY_DOCKER_IMAGE***
-
-Required: No (Yes for deploying on GCE).
-
-Name of docker image which provides running of deployment container on GCE instance. 
-MLflow must be installed in the image.
-
-
-##### ***MODEL_DEPLOY_DEFAULT_PORT*** 
-
-Required: No (Yes for deploying on GCE).
-
-Port on which deployments are available.
-
-**Note**: all gcp deployment have the same port, but theirs external ip addresses are different.
-
-
-##### ***MODEL_DEPLOY_FIREWALL_RULE***
-
-Required: No (Yes for deploying on GCE).
-
-Name of firewall rule. Firewall rule is required to open some port(s) for communication with services running on 
-GCE instances. More about firewall: https://cloud.google.com/vpc/docs/using-firewalls.
-
-
-#### Projects
-
-##### ***LOGLEVEL***
-
-Required: No.
-
-Default: INFO.
-
-Logging level in service *projects*.
-
-
-##### ***ARTIFACT_STORE***
-
-Required: Yes.
-
-Path to artifact store. It can be:
+Examples: [MLPanel examples](https://mlrepa.gitbook.io/mlpanel/) 
 
-* name of folder - it means relative path in $WORKSPACE/<project_folder>, e.g.:
-```
-ARTIFACT_STORE=mlruns => full path of artifact store is $WORKSPACE/<project_folder>/mlruns.
-```
+Find us on [Discord](https://discord.gg/DAMrSAy)
 
-* Google Storage bucket in format **gs://<bucket_name>**; this case requires to 
-define variable *GOOGLE_APPLICATION_CREDENTIALS*.
 
+# Features 
 
-##### ***MLFLOW_TRACKING_SERVERS_PORTS_RANGE***
+- interface to collaborate on multiple Machine Learning and Data Science projects 
+- create and manage Projects
+- manage Machine Learning Experiments: metrics tracking and models management
+- 1-click-deploy
+- customizable 
 
-Required: Yes.
+![Projects Panel](https://lh5.googleusercontent.com/DLt_aeA71h5a6zZI6_DOEInAU-6I9cjRyshF7Tihywi_khMOlq_OX6ToXQ8kR-43KM8Li1ENEWp22R_dTBRpIu1C52jqky08uuihXC28NOwXzkfq3eTw-62JPesN5OJP1kHUhpCR5-w)
 
-Format: <start_port>-<end_port>.
 
-Range of ports for tracking servers. It also defines how many projects can be 
-created: one port corresponds to one project (tracking server).
+# How to start 
 
+1) look into [MLPanel Discord channel](https://discord.gg/DAMrSAy)
+2) [Install MLPanel](https://mlrepa.gitbook.io/mlpanel/get-started-1/install) 
+3) [Run examples](https://mlrepa.gitbook.io/mlpanel/tutorials/examples-with-jupyter-notebook) 
+4) Try MLPanel in your projects   
 
-#### UI
 
-##### ***MLPANEL_UI_BRANCH***
 
-Required: Yes.
+# Feature requests and feedback
 
-[mlpanel-ui](https://github.com/mlrepa/mlpanel-ui) branch. 
-It can be any existing branch name.
+The best way to send feedback is to file an issue or message on [Discord](https://discord.gg/DAMrSAy)
 
+If you are proposing a feature:
 
-##### ***REACT_APP_API_URL***
+* Give a short use case
+* Describe how it would work
+* Provide support materials like charts, diagrams, code samples 
+* Feel free to make a pull request  :) 
 
-Required: Yes.
-Fixed value: http://0.0.0.0:8080/.
+# Development
 
-URL to API which UI service calls.
 
+1. Fork [MLPanel](https://github.com/mlrepa/mlpanel)
+
+2. Clone your fork locally:
 
-## Build
+        git clone [email protected]:mlrepa/mlpanel.git
 
-```bash
-ln -sf config/.env && docker-compose build
-```
+3. Create a branch for local development::
 
-## Run services
+        git checkout -b name-of-feature-branch
 
-```bash
-docker-compose up
-```
+3. Add your awesome changes !!!
 
-### Enter UI
+4. Run tests 
 
-`http://<host>:3000`,
-where `host` = *localhost* or *remote server ip*
+        ./services/projects/tests/test.sh -vv
+        ./services/deploy/tests/test.sh -vv
 
 
-## Tests and examples
+5. Commit your changes and push your branch to GitHub::
 
-Read **README.md** files in *services/*:
+        git add .
+        git commit -m "Describe changes made"
+        git push origin name-of-feature-branch
 
-* [projects](services/projects/README.md);  
-* [deploy](services/deploy/README.md).
+6. Submit a pull request through the GitHub website.

build.sh

@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+source functions.sh
+handle_help build
+check_input_edition
+
+if [ 'base' == $EDITION ]; then
+
+  DOCKER_FOLDER="config/base/docker"
+
+  echo "***Base image***"
+  docker build -t mlrepa/mlpanel-base:v0.2 $DOCKER_FOLDER/base
+
+  echo "***Deploy image***"
+  docker build -t mlrepa/mlpanel-base-deploy:v0.2 $DOCKER_FOLDER/deploy
+
+  echo "***Projects image***"
+  docker build -t mlrepa/mlpanel-base-projects:v0.2 $DOCKER_FOLDER/projects
+
+  echo "***UI image***"
+  docker build -t mlrepa/mlpanel-base-ui:v0.2 $DOCKER_FOLDER/ui
+
+  echo "***Web image***"
+  docker build -t mlrepa/mlpanel-base-web:v0.2 $DOCKER_FOLDER/web
+
+else
+  echo "Edition $EDITION undefined"
+
+fi;

common/utils.py

@@ -10,6 +10,14 @@
 from starlette.responses import JSONResponse
 
 
+class ModelDoesNotExistError(Exception):
+    """Model does not exists"""
+
+
+class InvalidEnvVarValueError(Exception):
+    """Invalid env var value"""
+
+
 def get_rfc3339_time() -> Text:
     """Get timestamp in rfc3339 format
 
@@ -57,9 +65,6 @@ def build_error_response(status_code: HTTPStatus, e: Exception) -> JSONResponse:
         message=str(e)
     )
 
-    # Set CORS
-    error_resp.headers['Access-Control-Allow-Origin'] = '*'
-
     return error_resp
 
 
@@ -95,6 +100,10 @@ def kill(proc_pid: int) -> None:
     Arguments:
         proc_pid {int} -- process id
     """
+
+    if not psutil.pid_exists(proc_pid):
+        return
+
     process = psutil.Process(proc_pid)
 
     for proc in process.children(recursive=True):
@@ -118,5 +127,16 @@ def is_remote(path: Text) -> bool:
     return False
 
 
-class ModelDoesNotExistError(Exception):
-    """Model does not exists"""
+def validate_env_var(name: Text, value: Text) -> None:
+
+    if ' ' in value:
+        raise InvalidEnvVarValueError(f'Invalid value "{value}" of env var {name}')
+
+
+def get_utc_timestamp() -> Text:
+    """Get utc timestamp.
+    Returns:
+        Text: utc timestamp
+    """
+
+    return str(datetime.datetime.utcnow().timestamp() * 1000)