Moved Post-training Optimization Tool to open-source (openvinotoolkit…

…#7940)

* Moved POT to opensource

* Added OMZ as a submodule

* Exclude OMZ from ShellCheck

.gitattributes

@@ -64,3 +64,13 @@
 *.gif filter=lfs diff=lfs merge=lfs -text
 *.vsdx filter=lfs diff=lfs merge=lfs -text
 *.bmp filter=lfs diff=lfs merge=lfs -text
+
+#POT attributes
+tools/pot/tests/data/test_cases_refs/* filter=lfs diff=lfs merge=lfs -text
+/tools/pot/tests/** -pot_package
+/configs/accuracy_checker/** -pot_package
+/configs/quantization/** -pot_package
+/tools/pot/tools/auxilary/** -pot_package
+/tools/pot/tools/run_series_experiments.py -pot_package
+/tools/pot/.pylintrc -pot_package
+/tools/pot/README_dev.md -pot_package

.gitmodules

@@ -56,3 +56,6 @@
 [submodule "thirdparty/onednn_gpu"]
 	path = thirdparty/onednn_gpu
 	url = https://github.com/oneapi-src/oneDNN.git
+[submodule "tools/pot/thirdparty/open_model_zoo"]
+	path = tools/pot/thirdparty/open_model_zoo
+	url = https://github.com/openvinotoolkit/open_model_zoo.git

scripts/CMakeLists.txt

@@ -12,6 +12,7 @@ ie_shellcheck_process(DIRECTORY "${OpenVINO_SOURCE_DIR}"
                            "${OpenVINO_SOURCE_DIR}/thirdparty"
                            "${OpenVINO_SOURCE_DIR}/runtime/bindings/python/thirdparty/pybind11"
                            "${IE_MAIN_SOURCE_DIR}/thirdparty"
+                           "${OpenVINO_SOURCE_DIR}/tools/pot/thirdparty"               
                            "${TEMP}"
                            # TODO fix and enable back:
                            "${OpenVINO_SOURCE_DIR}/inference-engine/scripts/dependencies.sh"

tools/pot/.gitignore

@@ -0,0 +1,107 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# dotenv
+.env
+
+# virtualenv
+.venv
+venv/
+ENV/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+
+# PyCharm
+.idea
+
+# snapshots
+*.tar

tools/pot/.pylintrc

@@ -0,0 +1,29 @@
+[MASTER]
+disable = fixme,
+          invalid-name,
+          missing-docstring,
+          no-self-use,
+          too-few-public-methods,
+          too-many-arguments,
+          too-many-locals
+max-attributes=20
+max-line-length = 120
+ignore-docstrings = yes
+ignored-modules = mo,accuracy_checker,extensions,openvino.inference_engine,cv2,open_model_zoo.model_tools._configuration,open_model_zoo.model_tools._common
+ignore-patterns = ac_imports.py
+extension-pkg-whitelist = numpy
+
+[SIMILARITIES]
+min-similarity-lines = 19
+ignore-imports = yes
+
+[BASIC]
+good-names=logger,fn
+
+[DESIGN]
+max-statements=120
+max-branches=14
+max-nested-blocks=7
+
+[OPTIONS]
+generated-members=torch.*

tools/pot/CODEOWNERS

@@ -0,0 +1,5 @@
+# See help here: https://docs.gitlab.com/ee/user/project/code_owners.html
+
+# Control 3d party dependencies
+**/*requirements*.*  [email protected]
+**/setup.py  [email protected]

tools/pot/README.md

@@ -0,0 +1,59 @@
+# Post-Training Optimization Tool {#pot_README}
+
+## Introduction
+
+Post-training Optimization Tool (POT) is designed to accelerate the inference of deep learning models by applying
+special methods without model retraining or fine-tuning, like post-training quantization. Therefore, the tool does not
+require a training dataset or a pipeline. To apply post-training algorithms from the POT, you need:
+* A floating-point precision model, FP32 or FP16, converted into the OpenVINO™ Intermediate Representation (IR) format
+and run on CPU with the OpenVINO™.
+* A representative calibration dataset representing a use case scenario, for example, 300 images.
+
+Post-training Optimization Tool provides the following key
+features:
+
+* Two post-training 8-bit quantization algorithms: fast [DefaultQuantization](openvino/tools/pot/algorithms/quantization/default/README.md) and precise [AccuracyAwareQuantization](openvino/tools/pot/algorithms/quantization/accuracy_aware/README.md).
+* Compression for different hardware targets such as CPU and GPU.
+* Multiple domains: Computer Vision, Natural Language Processing, Recommendation Systems, Speech Recognition.
+* [API](openvino/tools/pot/api/README.md) that helps to apply optimization methods within a custom inference script written with OpenVINO Python* API.
+* Symmetric and asymmetric quantization schemes. For details, see the [Quantization](openvino/tools/pot/algorithms/quantization/README.md) section.
+* Per-channel quantization for Convolutional and Fully-Connected layers.
+* Global optimization of post-training quantization parameters using the [Tree-Structured Parzen Estimator](openvino/tools/pot/optimization/tpe/README.md).
+
+The tool is aimed to fully automate the model transformation process without a need to change the model on the user's side.
+The POT is available only in the Intel® distribution of OpenVINO™ toolkit and is not opensourced. For details
+about the low-precision flow in OpenVINO™, see the [Low Precision Optimization Guide](docs/LowPrecisionOptimizationGuide.md).
+
+For benchmarking results collected for the models optimized with POT tool, see [INT8 vs FP32 Comparison on Select Networks and Platforms](@ref openvino_docs_performance_int8_vs_fp32).
+
+Further documentation presumes that you are familiar with the basic Deep Learning concepts, such as model inference,
+dataset preparation, model optimization, as well as with the OpenVINO™ toolkit and its components such 
+as  [Model Optimizer](@ref openvino_docs_MO_DG_Deep_Learning_Model_Optimizer_DevGuide) 
+and [Accuracy Checker Tool](@ref omz_tools_accuracy_checker_README).
+
+## Use POT
+![](docs/images/workflow.png) 
+
+The POT provides three basic usage scenarios:
+* **[Command-line interface](docs/CLI.md)**: this is the recommended path if the model from OpenVINO™ 
+[Model Zoo](https://github.com/openvinotoolkit/open_model_zoo) or there is a valid [Accuracy Checker Tool](@ref omz_tools_accuracy_checker_README)
+configuration file for the model that allows validating model accuracy using [Accuracy Checker Tool](@ref omz_tools_accuracy_checker_README).
+* **[Python* API](openvino/tools/pot/api/README.md)**: it allows integrating optimization methods implemented in POT into
+a Python* inference script written with [Python* API](@ref openvino_inference_engine_ie_bridges_python_docs_api_overview). 
+This flow is recommended if it is not possible to use [Accuracy Checker Tool](@ref omz_tools_accuracy_checker_README)
+for validation on the dedicated dataset.
+* **[Deep Learning Workbench](@ref workbench_docs_Workbench_DG_Introduction) (DL Workbench)**: a web-based graphical environment 
+that enables you to optimize, fine-tune, analyze, visualize, and compare performance of deep learning models.
+
+Note: POT also supports optimization in the so-called [**Simplified mode**](@ref pot_configs_README) which is essentially a local
+implementation of the POT Python API aimed at quantizing Computer Vision with simple pre-processing and inference flow. But
+please note that this mode can lead to an inaccurate model after optimization due to the difference in the model preprocessing.
+
+To get started with POT, follow the [Installation Guide](docs/InstallationGuide.md).
+
+## See Also
+
+* [Low Precision Optimization Guide](docs/LowPrecisionOptimizationGuide.md)
+* [Post-Training Optimization Best Practices](docs/BestPractices.md)
+* [POT Frequently Asked Questions](docs/FrequentlyAskedQuestions.md) 
+* [INT8 Quantization by Using Web-Based Interface of the DL Workbench](https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Int_8_Quantization.html)

tools/pot/README_dev.md

@@ -0,0 +1,54 @@
+# Post-training Optimization Tool {#pot_README_dev}
+
+Starting with the 2020.1 version, OpenVINO™ toolkit delivers the Post-Training Optimization Tool designed to accelerate the inference of DL models by converting them into a more hardware-friendly representation by applying specific methods that do not require re-training, for example, post-training quantization.
+For more details about the low-precision flow in OpenVINO™, refer to the [Low Precision Optimization Guide](docs/LowPrecisionOptimizationGuide.md).
+
+Post-Training Optimization Tool includes standalone command-line tool and Python* API that provide the following key features:
+
+## Key features:
+
+* Two supported post-training quantization algorithms: fast [DefaultQuantization](openvino/tools/pot/algorithms/quantization/default/README.md) and precise [AccuracyAwareQuantization](openvino/tools/pot/algorithms/quantization/accuracy_aware/README.md), as well as multiple experimental methods.
+* Global optimization of post-training quantization parameters using [Tree-structured Parzen Estimator](openvino/tools/pot/optimization/tpe/README.md).
+* Symmetric and asymmetric quantization schemes. For more details, see the [Quantization](openvino/tools/pot/algorithms/quantization/README.md) section.
+* Per-channel quantization for Convolutional and Fully-Connected layers.
+* Multiple domains: Computer Vision, Recommendation Systems.
+* Ability to implement custom calibration pipeline via supported [API](openvino/tools/pot/api/README.md).
+* Compression for different HW targets such as CPU, GPU, VPU.
+* Post-training sparsity.
+
+## Usage
+
+### System requirements
+- Ubuntu 18.04 or later (64-bit)
+- Python 3.6 or later
+- OpenVINO
+
+### Installation (Temporary)
+1) Clone compression tool repo: `git clone [email protected]:algo/post-training-compression-tool.git`
+2) Download submodules:
+   ```
+   git submodule init
+   git submodule update
+   ```
+3) Clone DLDT repo: `git clone https://gitlab-icv.inn.intel.com/inference-engine/dldt` (Not into the post-training-compression-tool)
+4) Switch dldt to required branch: `feature/low_precision/develop_fp_v10`
+5) Build inference engine (Instruction can be found in dldt repo)
+6) Switch dldt to _mkaglins/poc_ branch (Inference engine is built from _feature/low_precision/develop_fp_v10_ branch to support `FakeQuantize` layers. ModelOptimizer is used from _mkaglins/poc_ branch. So stay on _mkaglins/poc_ branch as you've built IE and don't build it from there again)
+7) Set _PYTHONPATH_ variable: `export PYTHONPATH=<path to DLDT bins>/bin/intel64/Release/lib/python_api/python3.6:<path to DLDT>/dldt/model-optimizer`
+8) Install requirements for accuracy checker:
+    - From POT root: `cd ./thirdparty/open_model_zoo/tools/accuracy_checker`
+    - Call setup script: `python3 setup.py install`
+    - Get back to root POT dir: `cd <PATH_TO_POT_DIR>`
+9) Install requirements for the tool:
+    - Call setup script: `python3 setup.py install`
+
+### Run
+1) Prepare configuration file for the tool based on the examples in the `configs` folder
+2) Navigate to compression tool directory
+3) Launch the tool running the following command:
+    `python3 main.py -c <path to config file> -e`
+
+To test the tool you can use PyTorch Mobilenet_v2 model from `tests/data/models/mobilenetv2/mobilenetv2.onnx`
+
+ - If there're some errors with imports in ModelOptimizer first of all make the following steps:
+    - Checkout _mkaglins/poc_ branch in DLDT (It's important!)

tools/pot/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2020-2021 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

tools/pot/configs/README.md

@@ -0,0 +1,64 @@
+# Configuration File Description {#pot_configs_README}
+
+In the instructions below, the Post-training Optimization Tool directory `<INSTALL_DIR>/deployment_tools/tools/post_training_optimization_toolkit` is referred to as `<POT_DIR>`. `<INSTALL_DIR>` is the directory where Intel&reg; Distribution of OpenVINO&trade; toolkit is installed.
+> **NOTE**: Installation directory is different in the case of PyPI installation and does not contain examples of 
+> configuration files.   
+
+The tool is designed to work with the configuration file where all the parameters required for the optimization are specified. These parameters are organized as a dictionary and stored in
+a JSON file. JSON file allows using comments that are supported by the `jstyleson` Python* package.
+Logically all parameters are divided into three groups:
+- **Model parameters** that are related to the model definition (e.g. model name, model path, etc.)
+- **Engine parameters** that define parameters of the engine which is responsible for the model inference and data preparation used for optimization and evaluation (e.g. preprocessing parameters, dataset path, etc.)
+- **Compression parameters** that are related to the optimization algorithm (e.g. algorithm name and specific parameters)
+
+## Model Parameters
+
+```json
+"model": {
+        "model_name": "model_name",
+        "model": "<MODEL_PATH>",
+        "weights": "<PATH_TO_WEIGHTS>"
+    }
+```
+
+This section contains only three parameters:
+- `"model_name"` - string parameter that defines a model name, e.g. `"MobileNetV2"`
+- `"model"` - string parameter that defines the path to an input model topology (.xml)
+- `"weights"` - string parameter that defines the path to an input model weights (.bin)
+
+## Engine Parameters
+
+```json
+"engine": {
+        "type": "accuracy_checker",
+        "config": "./configs/examples/accuracy_checker/mobilenet_v2.yaml"
+    }
+```
+The main parameter is `"type"` which can take two possible options: `"accuracy_checher"` (default) and `"simplified"`,
+which specify the engine that is used for model inference and validation (if supported):
+- **Simplified mode** engine. This engine can be used only with `DefaultQuantization` algorithm to get fully quantized model 
+using a subset of images. It does not use the Accuracy Checker tool and annotation. To measure accuracy, you should implement 
+your own validation pipeline with OpenVINO API.  
+  - To run the simplified mode, define engine section similar to the example `mobilenetV2_tf_int8_simple_mode.json` file from the `<POT_DIR>/configs/examples/quantization/classification/` directory.
+- **Accuracy Checker** engine. It relies on the [Deep Learning Accuracy Validation Framework](@ref omz_tools_accuracy_checker_README) (Accuracy Checker) when inferencing DL models and working with datasets.
+The benefit of this mode is you can compute accuracy in case you have annotations. It is possible to use accuracy aware
+algorithms family when this mode is selected.
+There are two options to define engine parameters in that mode:
+  - Refer to the existing Accuracy Checker configuration file which is represented by the YAML file. It can be a file used for full-precision model validation. In this case, you should define only the `"config"` parameter containing a path to the AccuracyChecker configuration file.
+  - Define all the [required Accuracy Checker parameters](@ref omz_tools_accuracy_checker_accuracy_checker_launcher_dlsdk_launcher_readme)
+    directly in the JSON file. In this case, POT just passes the corresponding dictionary of parameters to the Accuracy Checker when instantiating it.
+    For more details, refer to the corresponding Accuracy Checker information and examples of configuration files provided with the tool:
+    - For the SSD-MobileNet model:<br>\<POT_DIR\>/configs/examples/quantization/object_detection/ssd_mobilenetv1_int8.json
+
+## Compression Parameters
+
+This section defines optimization algorithms and their parameters. For more details about parameters of the concrete optimization algorithm, please refer to the corresponding
+[documentation](@ref pot_compression_algorithms_quantization_README).
+
+## Examples of the Configuration File
+
+For a quick start, many examples of configuration files are provided and placed to the `<POT_DIR>/configs/examples`
+ folder. There you can find ready-to-use configurations for the models from various domains: Computer Vision (Image 
+ Classification, Object Detection, Segmentation), Natural Language Processing, Recommendation Systems. We basically 
+ put configuration files for the models which require non-default configuration settings in order to get accurate results.
+For details on how to run the Post-Training Optimization Tool with a sample configuration file, see the [example](@ref pot_configs_examples_README).