Simple TensorFlow Serving Client

A simple, consolidated gRPC-based client wrapper for querying a hosted TensorFlow Model Server.

What it does?

• It simplifies working with protocol buffers and provides custom functions for working with protobuf APIs (i.e. messages and services) inside Tensorflow Serving, all without leaving the comfort of python.

• Implements gRPC client stubs for GetModelMetadata, GetModelStatus, HandleReloadConfig and Predict APIs (more to come, including some custom ones).

• It requires only the minimum dependencies, i.e. numpy, grpcio, protobuf making it very light to install on memory constrained edge devices, such as Raspberry Pi, NVIDIA Jetson, etc.

For a more detailed reference, click here.

Installation

Client

Clone the repository and run ;

cd stfsclient
pip install .

or

pip install git+https://github.com/jagans94/stfsclient.git

for just the package.

tensorflow_model_server

Note: Run as sudo; only works on Debian/Ubuntu

echo "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list && \
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -
apt update
apt-get install tensorflow-model-server

Tutorial

Refer here for the most up-to-date tutorial.

Benchmark

        The answer to the question **"Why use gRPC when TensorFlow Serving supports REST?"** 

​ Prediction latency on 10,000 images with batch sizes 1, 4, 8, and 16 on MNIST data set! :)

This is why! gRPC predict requests have a lot smaller latency profile and are approx. 6 times faster compared with REST based predict requests

Note: Code for bench marking can be found at extras/benchmarks.

Contributions

Want to add a feature that's not in here or implement one of these. Raise an Issue or better yet PR. ;)

Copyright

Copyright 2019 onward. Licensed under the MIT License; you may not use this project's files except in compliance with the License. A copy of the License is provided in the LICENSE file in this repository.