Skip to content

Commit

Permalink
[Docs][PyOV] Documentation for Custom Python Operation (openvinotoolk…
Browse files Browse the repository at this point in the history
…it#25615)

### Details:
 - Extend documentation with python examples

### Tickets:
 - CVS-141644
 - CVS-148475

---------

Co-authored-by: Michal Lukaszewski <[email protected]>
Co-authored-by: Karol Blaszczak <[email protected]>
Co-authored-by: Sergey Lyalin <[email protected]>
Co-authored-by: Sebastian Golebiewski <[email protected]>
  • Loading branch information
5 people authored Feb 3, 2025
1 parent 9c5b97c commit 0b3e161
Show file tree
Hide file tree
Showing 4 changed files with 182 additions and 33 deletions.
45 changes: 45 additions & 0 deletions docs/articles_en/assets/snippets/ov_custom_op.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Copyright (C) 2018-2025 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
#

# ! [op:common_include]
from openvino import Op
# ! [op:common_include]



# ! [op:header]
class Identity(Op):
# ! [op:header]

# ! [op:ctor]
def __init__(self, inputs=None, **attrs):
super().__init__(self, inputs)
self._attrs = attrs
# ! [op:ctor]

# ! [op:validate]
def validate_and_infer_types(self):
self.set_output_type(0, self.get_input_element_type(0), self.get_input_partial_shape(0))
# ! [op:validate]

# ! [op:copy]
def clone_with_new_inputs(self, new_inputs):
return Identity(new_inputs)
# ! [op:copy]

# ! [op:evaluate]
def evaluate(self, outputs, inputs):
outputs[0].shape = inputs[0].shape
inputs[0].copy_to(outputs[0])
return True

def has_evaluate(self):
return True
# ! [op:evaluate]

# ! [op:visit_attributes]
def visit_attributes(self, visitor):
visitor.on_attributes(self._attrs)
return True
# ! [op:visit_attributes]
18 changes: 15 additions & 3 deletions docs/articles_en/assets/snippets/ov_extensions.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,30 @@
#

import openvino as ov
from ov_custom_op import Identity

#! [py_frontend_extension_ThresholdedReLU_header]
import openvino.runtime.opset12 as ops
import openvino.runtime.opset14 as ops
from openvino.frontend import ConversionExtension
#! [py_frontend_extension_ThresholdedReLU_header]

#! [add_extension]
# Not implemented
core = ov.Core()

# Use operation type to add operation extension
core.add_extension(Identity)

# or you can add operation extension object which is equivalent form
core.add_extension(ov.OpExtension(Identity))
#! [add_extension]

#! [add_frontend_extension]
# Not implemented
# Register more sophisticated mapping with decomposition
def conversion(node):
input_node = node.get_input(0)
return Identity(input_node).outputs()

core.add_extension(ConversionExtension("Identity", conversion))
#! [add_frontend_extension]

from utils import get_path_to_extension_library
Expand Down
6 changes: 3 additions & 3 deletions docs/articles_en/documentation/openvino-extensibility.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ If such decomposition is not possible or appears too bulky with a large number o

You might prefer implementing a custom operation class if you already have a generic C++ implementation of operation kernel. Otherwise, try to decompose the operation first, as described above. Then, after verifying correctness of inference and resulting performance, you may move on to optional implementation of Bare Metal C++.

Additionally, it is also possible to implement custom operations using Python. OpenVINO provides a Python API that allows you to define and register custom operations. This can be particularly useful for rapid prototyping and testing of new operations.

Mapping from Framework Operation
################################

Expand Down Expand Up @@ -110,9 +112,7 @@ The ``Identity`` is a custom operation class defined in :doc:`Custom Operation G
:language: cpp
:fragment: [add_frontend_extension]

When Python API is used, there is no way to implement a custom OpenVINO operation. Even if custom OpenVINO operation is implemented in C++ and loaded into the runtime by a shared library, there is still no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.

Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used.
If custom OpenVINO operation is implemented in C++ and loaded into the runtime through a shared library, there is no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.

.. _create_a_library_with_extensions:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,62 @@ Custom OpenVINO Operations
custom operations to support models with operations
not supported by OpenVINO.

OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_a_library_with_extensions>` for more details on library creation and usage. The remaining part of this document describes how to implement an operation class.
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. A custom operation might be implemented both in C++ and Python.```

Also it is possible to create a shared library with custom operation implemented in C++ first and load it using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_a_library_with_extensions>` for more details on library creation and usage. The remaining part of this document describes how to implement an operation class using both the C++ API and Python API.

Operation Class
###############

To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
.. tab-set::


.. tab-item:: Python
:sync: py

To add your custom operation, create a new class that extends ``openvino.Op``, which is in turn derived from ``openvino.Node``, the base class for all graph operations in OpenVINO™. To add ``openvino.Op`` you need to import it.


.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:common_include]

Follow the steps below to add a simple custom operation:

1. Define the ``__init__`` method to initialize the class with inputs and attributes.

2. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``openvino.Node``. Set the inferred shape and element type of the output using ``set_output_type``.

3. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard types like ``int``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.

4. Override ``evaluate`` method with the code that will run when this operation is encountered in the model graph during the model inference. It works only for CPU device and enables OpenVINO runtime to run your arbitrary Python code as a part of model inference. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method which returns `True`, this method allows to get information about availability of ``evaluate`` method for the operation.

5. Override the ``clone_with_new_inputs``, which is an optional method that graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.

.. tab-item:: C++
:sync: cpp

To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:

.. doxygensnippet:: src/core/template_extension/identity.hpp
:language: cpp
:fragment: [op:common_include]

Follow the steps below to add a custom operation:
.. doxygensnippet:: src/core/template_extension/identity.hpp
:language: cpp
:fragment: [op:common_include]

Follow the steps below to add a custom operation:

1. Add the ``OPENVINO_OP`` macro which defines a ``NodeTypeInfo`` object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an operation currently consists of a string operation identifier and a string for operation version.
1. Add the ``OPENVINO_OP`` macro. The type info of an operation consists of a string operation identifier and a string for operation version.

2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.

3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.
3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.

4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.
4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.

5. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard C++ types like ``int64_t``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.
5. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard C++ types like ``int64_t``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.

6. Override ``evaluate`` method, which enables fallback of some devices to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method, this method allows to get information about availability of ``evaluate`` method for the operation.

6. Override ``evaluate``, which is an optional method that enables fallback of some devices to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method, this method allows to get information about availability of ``evaluate`` method for the operation.

Based on that, declaration of an operation class can look as follows:

Expand All @@ -43,43 +75,103 @@ OpenVINO™ operation contains two constructors:
* Default constructor, which enables you to create an operation without attributes
* Constructor that creates and validates an operation with specified inputs and attributes

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:ctor]
.. tab-set::

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:ctor]

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:ctor]

``validate_and_infer_types()``
++++++++++++++++++++++++++++++

``ov::Node::validate_and_infer_types`` method validates operation attributes and calculates output shapes using attributes of the operation.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:validate]
.. tab-set::

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:validate]

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:validate]

``clone_with_new_inputs()``
+++++++++++++++++++++++++++

``ov::Node::clone_with_new_inputs`` method creates a copy of the operation with new inputs.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:copy]
.. tab-set::

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:copy]

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:copy]

``visit_attributes()``
++++++++++++++++++++++

``ov::Node::visit_attributes`` method enables you to visit all operation attributes.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:visit_attributes]
.. tab-set::

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:visit_attributes]

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:visit_attributes]

``evaluate() and has_evaluate()``
+++++++++++++++++++++++++++++++++

``ov::Node::evaluate`` method enables you to apply constant folding to an operation.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:evaluate]
.. tab-set::

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:evaluate]

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:evaluate]

0 comments on commit 0b3e161

Please sign in to comment.