pypi 1.5.2 release

README.md

@@ -1,22 +1,32 @@
-tf2onnx - convert TensorFlow models to ONNX models.
+tf2onnx - Convert TensorFlow models to ONNX.
 ========
 
-[![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build?definitionId=16&branchName=master)
+| Build Type | OS | Python | Tensorflow | Onnx opset | Status |
+| ---        | ---    | ---    | ---        | ---        | ---    |
+| Unit Test - Basic | Linux, MacOS<sup>\*</sup>, Windows<sup>\*</sup> | 3.5, 3.6 | 1.5-1.14 | 7-10 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=16&branchName=master) |
+| Unit Test - Full | Linux, MacOS, Windows | 3.5, 3.6, 3.7 | 1.5-1.14 | 7-10 | [![Build Status](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_apis/build/status/unit_test-matrix?branchName=master)](https://dev.azure.com/tensorflow-onnx/tensorflow-onnx/_build/latest?definitionId=18&branchName=master) | |
+
+<a name="build_status_footnote">\*</a> Only test on python3.6, TF1.14.
 
 # Supported ONNX version
 tensorflow-onnx will use the ONNX version installed on your system and installs the latest ONNX version if none is found.
 
-We support opset 6 to 10. By default we use opset 7 for the resulting ONNX graph since most runtimes will support opset 7.
+We support opset 6 to 10. By default we use opset 7 for the resulting ONNX graph since most runtimes will support opset 7. Support for future opsets add added as they are released.
 
-If you want the graph to be generated with a newer opset, use ```--opset``` in the command line, for example ```--opset 10```.
+If you want the graph to be generated with a specific opset, use ```--opset``` in the command line, for example ```--opset 10```.
 
 # Status
-We support many TensorFlow models. Support for Fully Connected and Convolutional networks is mature. Dynamic LSTM/GRU/Attention networks should work but the code for this is evolving. 
-A list of models that we use for testing can be found [here](tests/run_pretrained_models.yaml)
+We support many TensorFlow models. Support for Fully Connected, Convolutional and dynamic LSTM networks is mature.
+A list of models that we use for testing can be found [here](tests/run_pretrained_models.yaml).
 
 Supported RNN classes and APIs: LSTMCell, BasicLSTMCell, GRUCell, GRUBlockCell, MultiRNNCell, and user defined RNN cells inheriting rnn_cell_impl.RNNCell, used along with DropoutWrapper, BahdanauAttention, AttentionWrapper.
 Check [tips](examples/rnn_tips.md) when converting RNN models.
 
+You find a list of supported Tensorflow ops and their mapping to ONNX [here](support_status.md).
+
+Tensorflow has broad functionality and occacional mapping it to ONNX creates issues.
+The common issues we run into we try to document here [Troubleshooting Guide](Troubleshooting.md).
+
 # Prerequisites
 
 ## Install TensorFlow
@@ -41,7 +51,7 @@ For pytorch/caffe2, follow the instructions here:
 We tested with pytorch/caffe2 and onnxruntime and unit tests are passing for those.
 
 ## Supported Tensorflow and Python Versions
-We are testing with tensorflow 1.5-1.13 and anaconda **3.5,3.6,3.7**.
+We are testing with tensorflow 1.5-1.14 and anaconda **3.5,3.6,3.7**.
 
 # Installation
 ## From pypi
@@ -64,8 +74,10 @@ python setup.py bdist_wheel
 
 # Usage
 
-To convert a TensorFlow model, tf2onnx prefers a ```frozen TensorFlow graph``` and the user needs to specify inputs and outputs for the graph by passing the input and output
-names with ```--inputs INPUTS``` and ```--outputs OUTPUTS```. 
+You find a end to end tutorial for ssd-mobilenet [here](tutorials/ConvertingSSDMobilenetToONNX.ipynb).
+
+To convert a TensorFlow model, tf2onnx supports ```saved_model```, ```checkpoint``` or ```frozen graph``` formats. We recommend the ```saved_model``` format. If ```checkpoint``` or ```frozen graph``` formats are used, the user needs to specify inputs and outputs for the graph by passing the input and output
+names with ```--inputs INPUTS``` and ```--outputs OUTPUTS```.
 
 ```
 python -m tf2onnx.convert 
@@ -108,7 +120,6 @@ the runtime may support custom ops that are not defined in onnx. A user can aske
 ### --fold_const
 when set, TensorFlow fold_constants transformation will be applied before conversion. This will benefit features including Transpose optimization (e.g. Transpose operations introduced during tf-graph-to-onnx-graph conversion will be removed), and RNN unit conversion (for example LSTM). Older TensorFlow version might run into issues with this option depending on the model.
 
-
 Usage example (run following commands in tensorflow-onnx root directory):
 ```
 python -m tf2onnx.convert\

Troubleshooting.md

@@ -0,0 +1,36 @@
+tf2onnx - common issues when converting models.
+========
+
+## tensorflow op is not supported
+Example:
+
+```ValueError: tensorflow op NonMaxSuppression is not supported``` 
+
+means that the given tensorflow op is not mapped to ONNX. This could have multiple reasons:
+
+(1) we have not gotten to implement it. NonMaxSuppression is such an example: we implemented NonMaxSuppressionV2 and NonMaxSuppressionV3 but not the older NonMaxSuppression op.
+
+To get this fixed you can open an issue or send us a PR with a fix.
+
+(2) There is no direct mapping to ONNX.
+
+Sometimes there is no direct mapping from tensorflow to ONNX. We took care are of the most common cases. But for less frequently used ops there might be a mapping missing. To get this fixed there 2 options:
+
+a) in tf2onnx you can compose the op out of different ops. A good example for this the [Erf op](https://github.com/onnx/tensorflow-onnx/blob/master/tf2onnx/onnx_opset/math.py#L317). Before opset-9 this tf2onnx composes Erf with other ONNX ops.
+
+b) You request the missing op to be added to [ONNX](https://github.com/onnx/onnx). After it is added to ONNX and some runtime implements it we'll add it to tf2onnx. You can see that this happened for the Erf Op. Starting with opset-9, ONNX added it - tf2onnx no longer composes the op and instead passes it to ONNX.
+
+c) The op is too complex to compose and it's to exotic to add to ONNX. In that cases you can use a custom op to implement it. Custom ops are documented in the [README](README.md) and there is an example [here](https://github.com/onnx/tensorflow-onnx/blob/master/examples/custom_op_via_python.py). There are 2 flavors of it:
+- you could compose the functionality by using multiple ONNX ops.
+- you can implement the op in your runtime as custom op (assuming that most runtimes do have such a mechanism) and then map it in tf2onnx as custom op.
+
+## get tensor value: ... must be Const
+
+There is a common group of errors that reports ```get tensor value: ... must be Const```.
+The reason for this is that there is a dynamic input of a tensorflow op but the equivalent ONNX op uses a static attribute. In other words in tensorflow that input is only known at runtime but in ONNX it need to be known at graph creation time.
+
+An example of this is the [ONNX Slice operator before opset-10](https://github.com/onnx/onnx/blob/master/docs/Changelog.md#Slice-1) - the start and end of the slice are static attributes that need to be known at graph creation. In tensorflow the [strided slice op](https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/strided-slice) allows dynamic inputs. tf2onnx will try to find the real value of begin and end of the slice and can find them in most cases. But if those are real dynamic values calculate at runtime it will result in the message ```get tensor value: ... must be Const```.
+
+You can pass the options ```--fold_const``` in the tf2onnx command line that allows tf2onnx to apply more aggressive constant folding which will increase chances to find a constant.
+
+If this doesn't work the model is most likely not to be able to convert to ONNX. We used to see this a lot of issue with the ONNX Slice op and in opset-10 was updated for exactly this reason.

VERSION_NUMBER

@@ -1 +1 @@
-1.5.1
+1.5.2

ci_build/azure_pipelines/coveragerc

@@ -0,0 +1,3 @@
+[paths]
+source =
+  ./

ci_build/azure_pipelines/onnxruntime_nightly_test.yml

@@ -1,14 +1,38 @@
 # Test against latest onnxruntime nightly package
 
-jobs:
-- template: 'templates/job_generator.yml'
-  parameters:
-    tf_versions: ['1.13.1']
-    onnx_opsets: ['']
-    onnx_backends:
-      onnxruntime: ['']
-    job:
-      steps:
-      - template: 'unit_test.yml'
-        parameters:
-            onnx_opsets: ['10', '9', '8', '7']
+stages:
+  - stage:
+    jobs:
+    - template: 'templates/job_generator.yml'
+      parameters:
+        platforms: ['linux', 'windows', 'mac']
+        python_versions: ['3.6', '3.5']
+        tf_versions: ['1.13.1','1.12', '1.11', '1.10', '1.9', '1.8', '1.7', '1.6', '1.5']
+        onnx_opsets: ['']
+        onnx_backends: {onnxruntime: ['nightly']}
+        job:
+          steps:
+          - template: 'unit_test.yml'
+        report_coverage: 'True'
+
+    - template: 'templates/job_generator.yml'
+      parameters:
+        platforms: ['linux', 'windows', 'mac']
+        python_versions: ['3.7', '3.6', '3.5']
+        tf_versions: ['1.14']
+        onnx_opsets: ['']
+        onnx_backends: {onnxruntime: ['nightly']}
+        job:
+          steps:
+          - template: 'unit_test.yml'
+        report_coverage: 'True'
+
+  - template: 'templates/combine_test_coverage.yml'
+
+schedules:
+- cron: "0 10 * * *"
+  displayName: Daily onnxruntime nightly unittest
+  branches:
+    include:
+    - master
+  always: true

ci_build/azure_pipelines/pretrained_model_test-matrix.yml

@@ -5,7 +5,7 @@ jobs:
   parameters:
     platforms: ['linux', 'windows', 'mac']
     python_versions: ['3.6', '3.5']
-    tf_versions: ['1.12', '1.11', '1.10', '1.9', '1.8', '1.7', '1.6', '1.5']
+    tf_versions: ['1.13.1', '1.12', '1.11', '1.10', '1.9', '1.8', '1.7', '1.6', '1.5']
     job:
       steps:
       - template: 'pretrained_model_test.yml'
@@ -14,7 +14,7 @@ jobs:
   parameters:
     platforms: ['linux', 'windows', 'mac']
     python_versions: ['3.7', '3.6', '3.5']
-    tf_versions: ['1.13.1']
+    tf_versions: ['1.14']
     job:
       steps:
       - template: 'pretrained_model_test.yml'

ci_build/azure_pipelines/pretrained_model_test.yml

@@ -4,15 +4,15 @@ jobs:
 - template: 'templates/job_generator.yml'
   parameters:
     python_versions: ['3.7', '3.6', '3.5']
-    tf_versions: ['1.13.1']
+    tf_versions: ['1.14.0']
     job:
       steps:
       - template: 'pretrained_model_test.yml'
 
 - template: 'templates/job_generator.yml'
   parameters:
     platforms: ['windows', 'mac']
-    tf_versions: ['1.13.1']
+    tf_versions: ['1.14.0']
     job:
       steps:
       - template: 'pretrained_model_test.yml'

ci_build/azure_pipelines/pylint.yml

@@ -10,5 +10,5 @@ jobs:
           set -ex
           pip install pylint
           pip freeze
-          pylint --rcfile=tools/pylintrc --ignore=version.py --disable=cyclic-import tf2onnx tests/*.py tools
+          pylint --rcfile=tools/pylintrc --ignore=version.py --disable=cyclic-import tf2onnx tests/*.py tools -j 0
         displayName: 'Pylint'

ci_build/azure_pipelines/templates/combine_test_coverage.yml

@@ -0,0 +1,48 @@
+# combine and report unittest coverage
+
+parameters:
+  artifact_name: 'single_test_coverage'
+
+stages:
+- stage:
+  jobs:
+  - job: 'combine_and_report_coverage'
+    variables:
+      CI_ARTIFACT_NAME: '${{ parameters.artifact_name }}'
+
+    pool:
+      vmImage: 'ubuntu-16.04'
+
+    steps:
+    - task: DownloadBuildArtifacts@0
+      displayName: 'Download Single Test Coverage'
+      inputs:
+        artifactName: '${{ parameters.artifact_name }}'
+        downloadPath: $(System.DefaultWorkingDirectory)
+
+    - task: CondaEnvironment@1
+      inputs:
+        createCustomEnvironment: 'true'
+        environmentName: 'tf2onnx'
+        packageSpecs: 'python=3.6'
+        updateConda: 'false'
+
+    - bash: |
+        pip install -U coverage
+      condition: succeeded()
+      displayName: 'Install Coverage'
+  
+    - bash: |
+        cat ${CI_ARTIFACT_NAME}/.coveragerc_paths* >> ci_build/azure_pipelines/coveragerc
+        coverage combine --rcfile ci_build/azure_pipelines/coveragerc ${CI_ARTIFACT_NAME}
+        coverage report
+        coverage html -d ${BUILD_ARTIFACTSTAGINGDIRECTORY}/coverage_report
+      condition: succeeded()
+      displayName: 'Combine And Report Test Coverage'
+  
+    - task: PublishBuildArtifacts@1
+      condition: succeeded()
+      inputs:
+        pathtoPublish: '$(Build.ArtifactStagingDirectory)'
+        artifactName: 'test_coverage_report'
+      displayName: 'Deploy Test Coverage Report'