Horizontal fusion without using concats

[MaheshRavishankar]

Current implementation of horizontal fusion of contractions concats the original operands to create a contraction op. This adds a lot of transient memory foot print overhead (and high initialization copy costs). It also prevents horizontal fusion from kicking in some cases. This is a tracking issue for revamping horizontal fusion to address these issues.

https://github.com/MaheshRavishankar/iree/tree/noconcat_horizontal_fusion_e2e is the dev branch for these changes. This is a tracking issue for changes required to make these work in general.

For now these are the examples being used for development purposes

test1.mlir
test2.mlir
test3.mlir
test4.mlir

These can be compiled using https://github.com/MaheshRavishankar/iree/tree/noconcat_horizontal_fusion_e2e

iree-compile <input file> --iree-hal-target-device=hip --iree-hip-target=gfx942 --iree-dispatch-creation-enable-aggressive-fusion=true --iree-dispatch-creation-enable-fuse-horizontal-contractions=true -o <output vmfb>

Here are some tasks that need to be addressed (and tentative assignees)

• Land [DispatchCreation] Changes to dispatch region in preparation for horizontal fusion changes. #19876 - @MaheshRavishankar
• Fix issue reported in Horizontal fusion without using concats #19877 (comment) - @Groverkss
• Verify the generated code for test1.mlir, test2.mlir test3.mlir using vector distribute - @Groverkss
• test4.mlir currently fails in TileAndDistributeToWorkgroupsUsingForAllOp pass - @pashu123
• Verify the generated code for test1.mlir, test2.mlir test3.mlir using tile and fuse - @nirvedhmeshram

[MaheshRavishankar]

I needed to go a bit further on this path to see any actual benefit. So needed to make some changes to handle something of this form

test5.mlir

I got it through till vector distribute pass, but it fails with this input.
@Groverkss could you take a look

iree-opt --pass-pipeline=builtin.module(func.func(iree-llvmgpu-vector-distribute)) vector_distribute_repro.mlir

vector_distribute_repro.mlir:1:1: error: 'func.func' op failed to distribute

func.func @horizontal_fusion_transpose_v_dispatch_0_generic_2x10x4096x64x640_i8xi8xi8xi8xi32xi32xi32() attributes{
    translation_info = #iree_codegen.translation_info<pipeline = LLVMGPUVectorDistribute workgroup_size = [256, 1, 1] subgroup_size = 64, {}>} {              ^