python-pfconclient

A Python3 client for pfcon's web API.

Overview

This repository provides a Python3 client for pfcon service's web API. The client provides both a Python programmatic interface and a standalone CLI tool called pfconclient.

Installation

$> pip install -U python-pfconclient

pfcon server preconditions

These preconditions are only necessary to be able to test the client against an actual instance of the pfcon server and run the automated tests.

Install latest Docker

Currently tested platforms:

• Ubuntu 18.04+ and MAC OS X 11.1+

Note: On a Linux machine make sure to add your computer user to the docker group. Consult this page https://docs.docker.com/engine/install/linux-postinstall/

Fire up the full set of pfcon services

Open a terminal and run the following commands in any working directory:

$> git clone https://github.com/FNNDSC/pfcon.git
$> cd pfcon
$> ./make.sh

You can later remove all the backend containers with:

$> cd pfcon
$> ./unmake.sh

Usage

Python programmatic interface

Instantiate the client:

from pfconclient import client

token = client.Client.get_auth_token('http://localhost:30006/api/v1/auth-token/', 'pfcon', 'pfcon1234')
cl = client.Client('http://localhost:30006/api/v1/', token)

Run fs plugin until finished using any local input directory and get the resulting files in a local output directory:

job_descriptors = {
    'args': ['--saveinputmeta', '--saveoutputmeta', '--dir', 'cube/uploads'],
    'args_path_flags': ['--dir'],  # list of flags with arguments of type 'path' or 'unextpath'
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simplefsapp',
    'entrypoint': ['python3', '/usr/local/bin/simplefsapp'],
    'type': 'fs'
}
job_id = 'chris-jid-1'
inputdir = '/tmp/sbin/in'
outputdir = '/tmp/sbin/out/chris-jid-1'
cl.run_job(job_id, job_descriptors, inputdir, outputdir)

Run ds plugin until finished using the local output directory of a previous plugin as its input directory and get the resulting files in a local output directory:

job_descriptors = {
    'args': ['--saveinputmeta', '--saveoutputmeta', '--prefix', 'lolo'],
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simpledsapp',
    'entrypoint': ['python3', '/usr/local/bin/simpledsapp'],
    'type': 'ds'
}
job_id = 'chris-jid-2'
inputdir = '/tmp/sbin/out/chris-jid-1'
outputdir = '/tmp/sbin/out/chris-jid-2'
cl.run_job(job_id, job_descriptors, inputdir, outputdir)

Visit the Python programmatic interface wiki page to learn more about the client's programmatic API.

Standalone CLI client tool

Get and print auth token with the auth subcommand:

$> pfconclient http://localhost:30006/api/v1/ auth --pfcon_user pfcon --pfcon_password pfcon1234

Run fs plugin until finished using any local input directory and get the resulting files in a local output directory:

$> pfconclient http://localhost:30006/api/v1/ -a <token> run --jid chris-jid-3 --args '--saveinputmeta --saveoutputmeta --dir cube/uploads' --args_path_flags='--dir' --auid cube --number_of_workers 1 --cpu_limit 1000 --memory_limit 200 --gpu_limit 0 --image fnndsc/pl-simplefsapp --selfexec simplefsapp --selfpath /usr/local/bin --execshell python3 --type fs /tmp/sbin/in /tmp/sbin/out/chris-jid-3

Run ds plugin until finished using the local output directory of a previous plugin as its input directory and get the resulting files in a local output directory:

$> pfconclient http://localhost:30006/api/v1/ -a <token> run --jid chris-jid-4 --args '--saveinputmeta --saveoutputmeta --prefix lolo' --auid cube --number_of_workers 1 --cpu_limit 1000 --memory_limit 200 --gpu_limit 0 --image fnndsc/pl-simpledsapp --selfexec simpledsapp --selfpath /usr/local/bin --execshell python3 --type ds /tmp/sbin/out/chris-jid-3 /tmp/sbin/out/chris-jid-4

Visit the standalone CLI client wiki page to learn more about the CLI client.

Arguments of type path or unextpath

If a plugin's args list contains flags with arguments of type path or unextpath then those flags should be included in the optional args_path_flags list. This string represents a list of flags. This way pfcon server will know that it has to substitute the local path specified by the flag by an actual path in the cloud.

Development and testing

Optionally setup a virtual environment

Install virtualenv and virtualenvwrapper using your OS package manager.

Create a directory for your virtual environments e.g.:

$> mkdir ~/Python_Envs

You might want to add the following lines to your .bashrc or .zshrc file:

VIRTUALENVWRAPPER_PYTHON=/usr/local/bin/python3
export WORKON_HOME=~/Python_Envs
source /usr/local/bin/virtualenvwrapper.sh

Then source the file and create a new Python3 virtual environment:

$> mkvirtualenv pfcon_client_env

To activate pfcon_client_env:

$> workon pfcon_client_env

To deactivate pfcon_client_env:

$> deactivate

Clone the repo

$> git clone https://github.com/FNNDSC/python-pfconclient.git

Run automated tests

$> cd python-pfconclient
$> workon pfcon_client_env
$> pip install -e ".[dev]"
$> pytest