Graph

Basic building blocks to describe a computation Graph

• Node (graph.h/.cc)

  • What's the difference between an Op and a Node?

  • What are the common properties for a node?

    • Can a node's name be empty?
    • What's the identifier of a node in a graph? Index or Name?

  • Advance: Function Ops, node with FunctionBody

• Graph (graph.h/.cc)

  • How to traverse from one node to another node?

  • What's the difference between a GraphInput and an Initializer?

  • Look for an example using GetProducerNode() and GetConsumerNodes()

  • What's the purpose of Graph::Resolve()?

    • How is ShapeAndTypeInference invoked?

• NodeArg (node_arg.h)

  • What the relationship between a graph edge and a NodeArg?
  • What's the unique identifier of a NodeArg in a graph?
  • Action: Look for some example using Graph::GetOrCreateNodeArg() (You will need to use this at some point)

Graph Transformers

• Understand the difference between GraphTransformer and RewriteRule

• Understanding the purpose of GraphTransformerManager

  • How to register a set of graph transformers into a session?

• Understanding the two versions of graph_transformer_utils.cc (onnxruntime/orttraining ones)

• Get familiar with graph_utils.cc

• Experiment with onnx.helper to compose a onnx model from the script (see transpose_matmul_gen.py for examples)

• Action: Implement a graph transformer to get hands-on experience

Training Graph

• Understand the workflow of training graph transformation

• Understand GraphAugmenter (graph_augmenter.h/.cc)

• GradientGraphBuilder

  • Understand the purpose/usage of STOP_GRADIENT_EDGES
  • Understand the meaning of x_node_args/y_node_args
  • Advance: Understand the back-propagation process in GradientGraphBuilder::Build()

• Per Op GradientBuilder

  • Understand the Gradient Registry (gradient_builder_registry.cc)

  • Understand the Gradient Builder Declaration (gradient_builder.h)

  • Read a few examples in Gradient Builder Implementation (gradient_builder.cc)

    • Understand the shorthands of I, GI, O, GO (gradient_builder_base.h)

    • Understand how gradient subgraph is composed with existing ops, followings are good examples

      • Easy: GetDropoutGradient, GetSqrtGradient
      • Medium: GetAddSubGradient, GetMulGradient
      • Hard: GetMatMulGradient, GetGemmGradient

    • Understand how broadcasting is handled when building gradient graph GradientBuilderBase::HandleBroadcasting()

    • Action: Implement a gradient definition for an op to get hands-on experience

    • Good reference implementation： https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/array_grad.py