diff --git a/README.md b/README.md index e41484e..fde4f57 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,13 @@ Content This project has a sample for the three types of Carol Apps: Online, Batch and Web. +All Carol Apps on this project have been created through the cookie-cutter project: https://docs.carol.ai/docs/carol-app-dev + Online Carol App --- -Language supported: python. +Language supported: python, nodeJS, and anything that can be embedded in a Docker Image. Goal: Execute an application online. This kind of application will provide a custom back-end. @@ -15,25 +17,19 @@ Goal: Execute an application online. This kind of application will provide a cus Batch Carol App --- -Language supported: python. +Language supported: python, nodeJS, and anything that can be embedded in a Docker Image. Goal: Execute an application that will process the data in batch. The process can be scheduled to run automatically. -Batch Carol App with Docker ---- - -Language supported: python. - -Goal: Execute a Python application that will process the data in batch, and has the example to submit the build through Operator. The process can be scheduled to run automatically. - Web Carol App --- -Language supported: any HTML framework (T-Faces). +Language supported: any HTML framework. On this example, we are sharing a sample using Portinari (https://portinari.io/). Goal: Allow the Carol App to provide a customizable experience to the final user through Web technologies and frameworks. + Dynamic Report --- @@ -48,92 +44,16 @@ Languages supported: Python and Html/Javascript. Goal: Create an interactive data visualization web [page](dynamic-report-d3/dynamicReportD3Preview.png), built on top of [D3](https://d3js.org/) (v5), [DC](https://dc-js.github.io/dc.js/) and [Crossfilter](https://square.github.io/crossfilter/). This Dynamic Report runs like an online Carol app with Flask backend. The template is ready to execute Flask and serve endpoints (available in app/webapp.py). + How to package the app? --- -Carol App divide the resources into the three types: - -### Web Carol app - folder structure - -The Web resource, related to the Front-end, should be zipped, and the app content should be inside a folder named `site`, as follow: - -``` -site/ -├── NunitoSans-Bold.51066f4d1d33630cd761.ttf -├── NunitoSans-ExtraLight.bc36a8726e20804a94da.ttf -├── NunitoSans-Regular.fb98ed1700e8dfaf0764.ttf -├── TotvsIcon.1642008454153acd9ef7.svg -├── TotvsIcon.4e1607e6f3f53fbb5e01.woff -├── TotvsIcon.683aa55aa3feb2d437b7.ttf -├── TotvsIcon.ab04e6bef307c5a4847d.eot -├── assets -│ ├── arrows -│ │ ├── arrow-down.png -│ │ ├── arrow-up.png -│ │ └── arrow-zero.png -│ ├── campaigns.png -│ ├── face.png -│ ├── flags -│ │ ├── EN.png -│ │ ├── PT.png -│ │ ├── china.png -│ │ ├── en-uk.png -│ │ └── spain.png -│ ├── general.png -│ ├── icon.png -│ ├── images -│ │ ├── order-ascending.svg -│ │ ├── order-descending.svg -│ │ ├── order-unselected.svg -│ │ ├── totvs-avatar-default.svg -│ │ ├── totvs-loading-icon.svg -│ │ ├── totvs-logo-grey.svg -│ │ ├── totvs-logo-menu.svg -│ │ └── totvs-logo-page-login.svg -│ ├── logo.png -│ └── zoom.png -├── favicon.ico -├── index.html -├── inline.bundle.js -├── inline.bundle.js.map -├── main.bundle.js -├── main.bundle.js.map -├── polyfills.bundle.js -├── polyfills.bundle.js.map -├── scripts.bundle.js -├── scripts.bundle.js.map -├── styles.bundle.js -├── styles.bundle.js.map -├── vendor.bundle.js -└── vendor.bundle.js.map -``` - - -### Online Carol App - folder structure - -The content of an Online Carol App should be zipped, and the app content should be inside a folder named `ai-script`, as follow: - -``` -ai-script -├── manifest.json -├── requirements.yml -└── run_me.py -``` - -### Batch Carol App - folder structure - -``` -ai-script/ -├── logging.cfg -├── luigi.cfg -├── manifest.json -├── requirements.yml -└── test_script.py -``` +To package and deploy the Carol App, please, follow the documentation https://docs.carol.ai/docs/carol-app-dev Additional information --- -Both Carol App (Online/Batch) need pyCarol (https://github.com/totvslabs/pyCarol). -Python 3.6 is required for both Carol App (Online/Batch). +- Carol Documentation: docs.carol.ai +- Both Carol App (Online/Batch) need pyCarol (https://github.com/totvslabs/pyCarol). +- Python 3.6 is required for both Carol App (Online/Batch). diff --git a/batch-carolapp-docker/Dockerfile b/batch-carolapp-docker/Dockerfile deleted file mode 100644 index bdc5b5c..0000000 --- a/batch-carolapp-docker/Dockerfile +++ /dev/null @@ -1,10 +0,0 @@ -FROM totvslabs/pycarol - -RUN mkdir /app -WORKDIR /app -ADD requirements.txt /app/ - -RUN pip install -r requirements.txt -ADD app/* /app/ - -ENTRYPOINT ["/app/model_run.sh"] diff --git a/batch-carolapp-docker/README.md b/batch-carolapp-docker/README.md deleted file mode 100644 index cfd5f12..0000000 --- a/batch-carolapp-docker/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# batch-carolapp-docker - -This sample project contains one app: process.py. - - -# Project Structure -
-. -├ai-script -│ └──manifest.json -├app -│ ├──process.py -├operator -│ ├──Makefile -│ └──spec.json -├Dockerfile -└requirements.txt -- -## ai-script -### manifest.json - -This manifest contains the requirements to run the process app. - -## app -On this folder we can find the scripts used on the Carol App. - -### process.py -The processing script sends a "Hello World" log to the task using pyCarol. - -## operator -The files on this folder are an interface for the GPC Operator API - -### Makefile -The make file have three different methods to be called: -- build: Will build the Docker image and deploy it on GCP. E.g:
make build -e TOKEN=- -### spec.json -The definition on the spec.json will identify wich resources are nedded to run the build machine. diff --git a/batch-carolapp-docker/ai-script/manifest.json b/batch-carolapp-docker/ai-script/manifest.json deleted file mode 100644 index ea3acf9..0000000 --- a/batch-carolapp-docker/ai-script/manifest.json +++ /dev/null @@ -1,32 +0,0 @@ -{ - "batch": { - "processes": [ - { - "algorithmDescription": { - "en-US": "Processing.", - "pt-BR": "Processamento.", - "pt-br": "Processamento.", - }, - "algorithmName": "process", - "algorithmTitle": { - "en-US": "Processing", - "pt-BR": "Processamento" - }, - "instanceProperties": { - "keepInstance": "false", - "luigi": { - "workers": "1" - }, - "profile": "", - "properties": { - "dockerImage": "carolapp_sample_docker", - "instanceMemory": "250M", - "instanceSize": "10G", - "instanceVCPUs": "0.5" - } - }, - "name": "process" - } - ] - } -} diff --git a/batch-carolapp-docker/app/model_run.sh b/batch-carolapp-docker/app/model_run.sh deleted file mode 100755 index 088edf8..0000000 --- a/batch-carolapp-docker/app/model_run.sh +++ /dev/null @@ -1,13 +0,0 @@ -#!/bin/bash - -#set -e -export PYTHONUNBUFFERED=1 - -echo "Running ${FILENAME}.py" -python "${FILENAME}.py" - -if [ $? != 0 ] -then - python /app/task_failed.py - exit 1 -fi diff --git a/batch-carolapp-docker/app/task_failed.py b/batch-carolapp-docker/app/task_failed.py deleted file mode 100644 index 4a630a4..0000000 --- a/batch-carolapp-docker/app/task_failed.py +++ /dev/null @@ -1,59 +0,0 @@ -import os -import sys -sys.path.append('./pyCarol') - -from pycarol.auth.ApiKeyAuth import ApiKeyAuth -from pycarol.carol import Carol -from pycarol.apps import Apps -from pycarol.tasks import Tasks - -class docker_environ_vars(): - - def __init__(self): - self.caroltenant = '' - self.carolappoauth = '' - self.carolconnectorir = '' - self.longtaskid = '' - - def docker_enviroment_variables_validation(self): - fault_vars = [] - - if 'CAROLTENANT' not in os.environ: - fault_vars.append('CAROLTENANT') - if 'CAROLAPPOAUTH' not in os.environ: - fault_vars.append('CAROLAPPOAUTH') - if 'CAROLCONNECTORID' not in os.environ: - fault_vars.append('CAROLCONNECTORID') - if 'LONGTASKID' not in os.environ: - fault_vars.append('LONGTASKID') - if 'ENV_DOMAIN' not in os.environ: - fault_vars.append('ENV_DOMAIN') - - if len(fault_vars) > 0: - raise Exception('Some environment variable were not defined: ',fault_vars) - - def load_vars(self): - self.docker_enviroment_variables_validation() - self.caroltenant = os.environ['CAROLTENANT'] - self.carolappoauth = os.environ['CAROLAPPOAUTH'] - self.carolconnectorir = os.environ['CAROLCONNECTORID'] - self.longtaskid = os.environ['LONGTASKID'] - self.environment = os.environ['ENV_DOMAIN'] - -print("-------------------------------") -print("STARTING ERROR REPORT") -print("-------------------------------") -env_vars = docker_environ_vars() -env_vars.load_vars() - -carol = Carol(domain=env_vars.caroltenant, - app_name='', - auth=ApiKeyAuth(env_vars.carolappoauth), - connector_id=env_vars.carolconnectorir, - environment=env_vars.environment) - -tasks = Tasks(carol) -task = tasks.get_task(env_vars.longtaskid) -if task.mdm_task_status != 'FAILED' or task.mdm_task_processing: - task.add_log(log_message="Something went wrong.", log_level="ERROR") - task.set_progress(100) diff --git a/batch-carolapp-docker/operator/.gitignore b/batch-carolapp-docker/operator/.gitignore deleted file mode 100644 index 50993e6..0000000 --- a/batch-carolapp-docker/operator/.gitignore +++ /dev/null @@ -1 +0,0 @@ -spec.json_tmp diff --git a/batch-carolapp-docker/operator/Makefile b/batch-carolapp-docker/operator/Makefile deleted file mode 100644 index 41182d7..0000000 --- a/batch-carolapp-docker/operator/Makefile +++ /dev/null @@ -1,28 +0,0 @@ -IMAGE = carolapp_sample_docker -APP = sample -OPERATOR ?= api.operator.carol.ai -ENVIRONMENT ?= labs-ai-apps-production - -TOKEN ?= -USER ?= -ID ?= -GHTOKEN ?= - -.PHONY: spec -spec: - @APP=$(USER) IMAGE=$(IMAGE) ENVIRONMENT=$(ENVIRONMENT) envsubst < spec.json > spec.json_tmp - -.PHONY: build -build: spec - @curl -sXPOST -H "x-auth-token:$(TOKEN)" \ - "https://$(OPERATOR)/api/builds" -d@./spec.json_tmp - -.PHONY: get -get: - @curl -sXGET -H "x-auth-token:$(TOKEN)" \ - "https://$(OPERATOR)/api/builds/build/$(APP)/$(ID)" - -.PHONY: delete -delete: - @curl -sXDELETE -H "x-auth-token:$(TOKEN)" \ - "https://$(OPERATOR)/api/builds/build/$(APP)/$(ID)" diff --git a/batch-carolapp-docker/operator/spec.json b/batch-carolapp-docker/operator/spec.json deleted file mode 100644 index 1dfe1ce..0000000 --- a/batch-carolapp-docker/operator/spec.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "user": "${APP}", - "app": "build", - "cpu": 2, - "memory": 4000, - "source": { - "type": "git", - "path": "https://${GHTOKEN}@github.com/totvslabs/carolapp-samples", - "branch": "master" - }, - "dockerfile": "Dockerfile", - "cwd": "./batch-carolapp-docker", - "cache": true, - "target": { - "image": "gcr.io/${ENVIRONMENT}/${IMAGE}" - } -} diff --git a/batch-carolapp-docker/requirements.txt b/batch-carolapp-docker/requirements.txt deleted file mode 100644 index a777e60..0000000 --- a/batch-carolapp-docker/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -pycarol>=2.20 diff --git a/batch-carolapp-nop/ai-script/manifest.json b/batch-carolapp-nop/ai-script/manifest.json deleted file mode 100644 index c5a167f..0000000 --- a/batch-carolapp-nop/ai-script/manifest.json +++ /dev/null @@ -1,33 +0,0 @@ -{ - "batch": { - "processes": [ - { - "algorithmDescription": { - "en-US": "Hello World", - "pt-BR": "Hello World", - "pt-br": "Hello World" - }, - "algorithmName": "test_script", - "algorithmTitle": { - "en-US": "Hello World" - }, - "instanceProperties": { - "environments": { - "EXAMPLE_ENV_2": "value_2" - }, - "keepInstance": "false", - "luigi": { - "workers": "4" - }, - "profile": "", - "properties": { - "dockerImage": "carol/base-batch", - "instanceMemory": "2", - "instanceVCPUs": "4" - } - }, - "name": "helloworld" - } - ] - } -} \ No newline at end of file diff --git a/batch-carolapp-nop/ai-script/test_script.py b/batch-carolapp-nop/ai-script/test_script.py deleted file mode 100644 index a6cad0f..0000000 --- a/batch-carolapp-nop/ai-script/test_script.py +++ /dev/null @@ -1,39 +0,0 @@ -import os -import pycarol -import datetime -import random - -from pycarol import * -from pycarol.carol import * -from pycarol.tasks import * -from pycarol.query import * -from pycarol.auth.ApiKeyAuth import * -from pycarol.staging import * -from pycarol.connectors import Connectors - - -def run(): - print(">>>>> RUNNING THE METHOD RUN INSIDE EXECUTION") - - apiAuth = ApiKeyAuth(api_key = os.environ['CAROLAPPOAUTH']) - connectorId = os.environ['CAROLCONNECTORID'] - appName = os.environ['CAROLAPPNAME'] - - print("Connecting to Carol...") - carol_instance = Carol(os.environ['CAROLDOMAIN'], appName, apiAuth, connector_id=connectorId) - - print("Connected...") - tasks = Tasks(carol_instance) - if(os.environ['LONGTASKID'] != "TBD"): - task = tasks.get_task(os.environ['LONGTASKID']) - task.set_progress(50) - - print(f">>>>> HELLO WORLD! My Task ID is {os.environ['LONGTASKID']}") - - if(os.environ['LONGTASKID'] != "TBD"): - task.add_log(log_message="Everything went well.", log_level="INFO") - task.set_progress(100) - -if __name__ == '__main__': - print(">>>>> MAIN!") - run() diff --git a/batch-carolapp/README.md b/batch-carolapp/README.md deleted file mode 100644 index 92c51f6..0000000 --- a/batch-carolapp/README.md +++ /dev/null @@ -1,82 +0,0 @@ -Preparing the environment to run the application locally ---- - -Create the environment variables (`sintax for Mac`): - -``` -export CAROLTENANT=wssp -export CAROLAPPNAME=appdemo -export LONGTASKID=TBD -``` - -The `LONGTASKID` will not be considered since its value is `TBD` (take a look at the source code to see how it is being handled). - -Go to Carol, get the `connector ID` the application will send data to. In the provided application, it's being considered the connector `carolml`. Inside Carol, for the connector the application will send data, generate the `Connector Token`. With the connector token in hands, create this new environment variable: - -``` -export CAROLCONNECTORID=e89641b0e91011e8aec042010a801002 -export CAROLAPPOAUTH=d5f358e0b30a11e8a0caca8943f25e25 -``` - -Ps.: -- `e89641b0e91011e8aec042010a801002` is the `Connector ID` I got from Connector's page. -- `d5f358e0b30a11e8a0caca8943f25e25` is the `Connector Token` I generated on my side. - - -Running the application locally ---- - -Create a virtualenv based on the `requirements.yml` file: - -``` -conda env create -f requirements.yml -``` - -Some useful commands related to conda and virtualenv: - -Activate a specific virtualenv: -``` -conda activate pycarol -``` - -Deactivate a virtualenv: -``` -conda deactivate -``` - -List all virtualenv avaliable: -``` -conda env list -``` - -After creating and activating the virtualenv, run the following command: - -``` -python3 -m luigi --module test_script main.runMe --local-scheduler -``` - - -Running the application in Carol ---- - -To make the application compatible to Carol, you should replace the `connector ID` defined to run it locally to the default connector id: `0a0829172fc2433c9aa26460c31b78f0` - -On the provided sampe, it's this line: - -``` -carol_instance = Carol(os.environ['CAROLTENANT'], 'myapp', apiAuth, connector_id='0a0829172fc2433c9aa26460c31b78f0') -``` - - -Additional information ---- - -In case you want to keep the `AWS` instance, without destroying it after finishing the algorithm, change the manifest file setting the property `keepInstance` to true. - -Both Carol App (Online/Batch) need pyCarol (https://github.com/totvslabs/pyCarol). - - -Questions? ---- - -Send a message on `#carol-feedback` on Slack. diff --git a/batch-carolapp/ai-script/manifest.json b/batch-carolapp/ai-script/manifest.json deleted file mode 100644 index 39129d8..0000000 --- a/batch-carolapp/ai-script/manifest.json +++ /dev/null @@ -1,61 +0,0 @@ -{ - "batch": { - "processes": [ - { - "algorithmDescription": { - "en-US": "Predict the customer status", - "pt-BR": "Predict the customer status", - "pt-br": "Predict the customer status" - }, - "algorithmName": "test_script", - "algorithmTitle": { - "en-US": "Predict the customer status" - }, - "instanceProperties": { - "environments": { - "CONNECTORNAME": "carolml", - "EXAMPLE_ENV_2": "value_2" - }, - "keepInstance": "false", - "luigi": { - "workers": "4" - }, - "profile": "", - "properties": { - "dockerImage": "carol/base-batch", - "instanceMemory": "2", - "instanceVCPUs": "4" - } - }, - "name": "testbatch" - }, - { - "algorithmDescription": { - "en-US": "Generate custom report", - "pt-BR": "Generate custom report", - "pt-br": "Generate custom report" - }, - "algorithmName": "test_report", - "algorithmTitle": { - "en-US": "Generate a custom report" - }, - "instanceProperties": { - "environments": { - "CONNECTORNAME": "carolml" - }, - "keepInstance": "false", - "luigi": { - "workers": "4" - }, - "profile": "", - "properties": { - "dockerImage": "carol/base-batch", - "instanceMemory": "2", - "instanceVCPUs": "4" - } - }, - "name": "testreport" - } - ] - } -} \ No newline at end of file diff --git a/batch-carolapp/ai-script/report-template.html b/batch-carolapp/ai-script/report-template.html deleted file mode 100644 index b01da0b..0000000 --- a/batch-carolapp/ai-script/report-template.html +++ /dev/null @@ -1,43 +0,0 @@ - - - - --e GHTOKEN=
{6:d} | |
---|---|
Lower: | {0:C} |
Q1: | {1:C} |
Median: | {2:C} |
Mean: | {5:C} |
Q3: | {3:C} |
Upper: | {4:C} |
{4:d} | |
---|---|
Open: | {0:C} |
High: | {1:C} |
Low: | {2:C} |
Close: | {3:C} |