fix graph_net/tools/generate_subgraph_dataset #603
Conversation
|
Thanks for your contribution! |
| function dimension_generalizer(){ | ||
| echo ">>> [1] dimension generalize for samoles under ${model_list}." | ||
| echo ">>>" | ||
| python3 -m graph_net.model_path_handler \ |
There was a problem hiding this comment.
这个脚本是为了给每个样本配置需要的维度变换Pass，目前这些Pass都已经更新到样本的graph_net.json中，因此这里不需要执行。
There was a problem hiding this comment.
这个function包含三个python，如果“这些Pass都已经更新到样本的graph_net.json中”，那么前两个python可以不执行，但是第三个python是生成不同维度的samples的子空间的，参考#566 (comment)
这点需要明确一下我们的要求到底是什么
| fusible_subgraph_list=${DECOMPOSE_WORKSPACE}/fusible_subgraph_sample_list.txt | ||
| deduplicated_fusible_subgraphs_list=${DECOMPOSE_WORKSPACE}/deduplicated_fusible_subgraph_sample_list.txt | ||
| dimension_generalizer_samples_list=dimension_generalizer_sample_list.txt | ||
| model_runnable_predicator=ModelRunnablePredicator |
There was a problem hiding this comment.
该运行配置并没有给多个步骤复用，可直接写到对应的命令里面
There was a problem hiding this comment.
我觉得可以保留着，方便统一格式
| RENAMED_FUSIBLE_SUBGRAPH_DIR=$DECOMPOSE_WORKSPACE/renamed_fusible_subgraphs | ||
| DEDUPLICATED_FUSIBLE_SUBGRAPH_DIR=$DECOMPOSE_WORKSPACE/deduplicated_fusible_subgraphs | ||
| UNITTESTS_OUTPUT_DIR=$DECOMPOSE_WORKSPACE/unittests | ||
| DIMENSION_GENERALIZER_OUTPUT_DIR=/tmp/dimension_generalized_samples |
There was a problem hiding this comment.
统一在DECOMPOSE_WORKSPACE目录下建立子目录，保存每个步骤执行的产物
| EOF | ||
| ) | ||
|
|
||
| python3 -m graph_net.model_path_handler \ |
There was a problem hiding this comment.
这一步的功能是什么？看代码是更新了输入约束文件，需要集成进来吗？
There was a problem hiding this comment.
这一步的功能对应整图维度泛化进度统计，如果确认Pass都已经更新到样本的graph_net.json中，那这一步可以不需要执行
| "handler_config": { | ||
| "output_json_file_name": "cumsum_num_kernels.json", | ||
| "model_path_prefix": "${DEVICE_REWRITED_OUTPUT_DIR}", | ||
| "model_path_prefix": "${GRAPH_NET_ROOT}", |
|
|
||
| generate_op_names 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_op_names_${suffix}.txt | ||
| generate_split_point 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_split_point_${suffix}.txt | ||
| range_decompose 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_range_decompose_${suffix}.txt |
There was a problem hiding this comment.
前序新增了维度泛化步骤，维度泛化可能会修改model.py的内容，因为有些维度值是使用立即数的方式写死在了model.py里面，那range_decompose是不是应该以维度泛化后的样本作为输入？
There was a problem hiding this comment.
这就需要对齐一下了，如果“range_decompose以维度泛化后的样本作为输入”，那么就不符合这个流程图了，需要对齐一下
|
|
||
| group_subgraph_sources 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_group_subgraph_sources_${suffix}.txt | ||
| analyze_subgraph_dependences 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_subgraph_dependencies_${suffix}.txt | ||
| rewrite_subgraph_shapes_per_index 2>&1 | tee ${DECOMPOSE_WORKSPACE}/log_rewrite_subgraph_shapes_${suffix}.txt |
There was a problem hiding this comment.
这3步是为了做一件事，前面2步是为了收集信息，最后1步才是真正应用样本变换，可以合并到1步，并且注意任何生成的json文件、样本等，都统一放到DECOMPOSE_WORKSPACE指定的目录下面。
2 participants
PR Category
other
Description
维度泛化集成