[llvm-mca] Add instruction customization

[r-belenov]

I'd like to get feedback on the idea of customizing properties of individual instruction (using custom private InstrDesc and callback in InstrBuilder::createInstruction). Most obvious use is to model cache misses by manual annotations in the input assembler code, but it also opens the door for "what-if" analysis during CPU design (thus generic customization seems better than just specifying the latency). I have internal implementation; primary idea is here - https://squidex.jugru.team/api/assets/srm/40f77a92-171d-425b-8174-8b306afb22fe/llvm-upside-down-cppconf2025.pdf - on slide 49 (slides in Russian, but the code should be clear). Still need to get approval to contribute the code from the employer, but first would like to know whether it may fit well in MCA code base.

[r-belenov]

@adibiagio

[llvmbot]

@llvm/issue-subscribers-tools-llvm-mca

Author: None (r-belenov)

I'd like to get feedback on the idea of customizing properties of individual instruction (using custom private InstrDesc and callback in InstrBuilder::createInstruction). Most obvious use is to model cache misses by manual annotations in the input assembler code, but it also opens the door for "what-if" analysis during CPU design (thus generic customization seems better than just specifying the latency). I have internal implementation; primary idea is here - https://squidex.jugru.team/api/assets/srm/40f77a92-171d-425b-8174-8b306afb22fe/llvm-upside-down-cppconf2025.pdf - on slide 49 (slides in Russian, but the code should be clear). Still need to get approval to contribute the code from the employer, but first would like to know whether it may fit well in MCA code base.

[mshockwave]

Have you checked out:

• mca::InstrPostProcess: Allow you to modify mca::Instruction after it's created. AMDGPU has an example of this.
• mca::CustomBehavior: Allow you to insert artificial (pipeline) hazard into MCA's simulation. AMDGPU has an example of this.
• mca::InstrumentManager: Allow you to parse custom assembly comments and customize the scheduling class (which effectively customizes mca::InstrDesc because the latter is created from scheduling class) you want to use. RISC-V uses it on vector instructions.

[r-belenov]

Thank you for suggestions! I checked these mechanisms, and it looks they are not sufficient; please correct me if I am wrong.

Let's consider data cache misses modeling. For that we need to change the latency of particular load instruction which is stored in llvm::mca::InstrDesc (MaxLatency for entire instruction and Writes[i].Latency for specific destination register).

• mca::InstrPostProcess modifies mca::Instruction, which has a constant reference to InstrDesc (and descriptions can be reused for similar instructions). It cannot modify mca::Instruction::Desc that keeps the latencies
• mca::CustomBehavior affects instruction dispatch, while we want to affect instruction execution (number of cycles between dispatch to load port and ready state of destination register)
• mca::InstrumentManager can only select from scheduling class IDs already encoded in machine model; we need the ability to change the latency to arbitrary number, basically requiring almost infinite number of possible IDs

Custom modifier of InstrDesc solves these issues and also allows to introduce arbitrary modifications to the instruction, even to model the instructions not present in machine model without recompilation.

[adibiagio]

Have you considered other approaches where users annotate slow load instructions using special code comments in the input assembly?

You could then teach the LSUnit about how to consume that extra knowledge and change how certain load latencies are simulated.

This is obviously just an idea, and it is based on the assumption that using code annotations is enough. In practice, it really depends on how accurate you want to be when simulating memory.

As a side note: LLVM cost models are not particularly good at describing load latencies. On x86, load latency often defaults to what vendors describe as the (optimistic) "load-to-use latency" (which assumes an L1 hit).
Load latency is often manually folded into the normal instruction latency. This is not great for customization.

Here is an idea:

Say that we add extra tablegen descriptors for describing extra L2,L3 latency.
E.g.: (pseudo-code , not real tablegen code)

def MemoryNode<"name>;

def L1Node : MemoryNode<"L1">
{
  uint extraLatency = 0;   // w.r.t. parent
  MemoryNode parent = nullptr;
}

def L2Node : MemoryNode<"L2">
{
  uint extraLatency = 20;   // w.r.t. parent
  MemoryNode parent = L1Node;
}

def L3Node : MemoryNode<"L3">
{
  uint extraLatency = 20;   // w.r.t. parent
  MemoryNode parent = L2Node;
}

We could reuse those memory node names in MCA comments to tag instructions which "mayload".

For example
ldrbw1, [x7, x0] // LLVM-MCA-L2

At that point, LSUnit would know that the load is expected to hit on memory node "L2". So it will add +20 cycles (based on the pseudo-tablegen-code above).

The LSUnit would then change the total latency of that load (after dispatch) by adding the "extraLatency" field from L2Node. In case of unknown memory node names, you could have that, by default, LSUnit simply fall-back to using the normal instruction latency.

That way, you wouldn't need to change/customize InstrDesc. However, users will have to explicitly tag loads using ASM code comments. Another limitation is that the LSUnit would always assume that same latency for every iteration. We don't want to implement a cache simulator after all (although, people could do it in downstream implementations - if they are crazy enough :) ).

Not sure if that makes sense. These are just some ideas, and I don't know whether they would work for you.

As @mshockwave pointed out, there are already a lot of mechanisms in place for customizing things. Adding yet-another mechanism in MCA is not ideal, and we should think about whether extending the cost model is instead the right way to go.
I am not against customizing things in general. However, changing the InstrDesc is a big thing, and it makes me wonder whether that's the right way to go.

I hope it makes sense.

[r-belenov]

Have you considered other approaches where users annotate slow load instructions using special code comments in the input assembly?

That's exactly how I test my approach - mark slow loads with the comments of the form "// LLVM-MCA-LATENCY: xxx", propagate them to the loop that feeds instructions to InstructionBuilder and create appropriate modifier that changes the latency in the InstrDesc. The comment has the actual value in clockticks, though, not the cache level; actual latencies for different levels are kept in the file external to MCA. I think it is better to customize them in runtime rather than encode in tablegen - while L2 latency for specific core model may be constant, L3 usually depends on the uncore and DRAM latency may vary greatly - and latency value in clockticks depends on the core and DRAM frequencies; there are also other memory types like HBM or NVRAM with different latencies. It is better to have the ability to model code sensitivity to different latency values specified in runtime rather than have a choice from several predefined values for L2/L3/DRAM.

Another limitation is that the LSUnit would always assume that same latency for every iteration. We don't want to implement a cache simulator after all

Agree; it's possible to approximate a mix via modeling different accesses separately and taking weighted average of clockticks (not the same as different accesses on different iterations, but it's better than nothing).

Load latency is often manually folded into the normal instruction latency.

I see; I'm focused on ARM64 with dedicated load/store instructions. X86 is more complex, but changing total latency according to accessed memory level may still work.

I can name another thing that can be done via arbitrary InstrDesc modification - compare/branch fusion. We can preprocess instruction sequence, detect compares that can be fused (with the code external to MCA) and remove compare, but modify the InstrDesc for specific branch adding the source operands of compare in order to keep the data dependencies. I don't know how to do it with existing customization approaches.

Adding yet-another mechanism in MCA is not ideal, and we should think about whether extending the cost model is instead the right way to go.

Agree that it's better to keep things to the minimum, but proposed change is pretty small and allows to do different customizations on the user side without other changes of MCA library. And yes, it is mostly targeted to MCA-as-a-library usage, though IMHO latency customization with a comment makes sense for llvm-mca tool.

[mshockwave]

A single InstrDesc is shared among many different instructions, if you change the Latency of a InstrDesc of a load. Then every other loads sharing the same schedclass are also affected. I don't think that is what you want, right?
If you want to attach a unique InstrDesc, have you tried using variant scheduling class, which will create a unique InstrDesc for each instruction.

MCA is a useful tool for validating the performance characteristics of scheduling models -- which are mostly static -- by "augmenting" them with dynamic bits, namely, the whole pipeline simulation scheme. So I can understand your desire to add more dynamics into the scene. However, philosophically, I'm not sure if adding a way to customize latency -- albeit great for design space exploration, that I agree -- is deviating away from the said core mission of MCA, which centralizing the scheduling model, a little too far. Also, although being a small change by itself, making too many things customizable might not be a good long-term solution from a software engineering perspective because we might end up with many small knobs that are too niche to be used more than once.

That said, I actually done a similar thing to utilize MCA to its maximum potential for dynamic analysis in the past: https://youtu.be/ZGEP7JEIKNo?si=eUoVHPi2rNjCCqUl . And we had to either create our own MCA pipeline stage or change some code in Scheduler or LSUnit etc. so I can definitely feel you there. But I didn't feel like many of those changes fit well into upstream MCA for the reasons I outlined earlier, thus I only upstreamed a fraction of the improvements.

The bottom line is: I think you brought up a good idea and by all mean you should try implementing in your own tree. But when it comes to upstream, sometimes there are other aspects we need to take care of beyond the merit of the idea itself.

I can name another thing that can be done via arbitrary InstrDesc modification - compare/branch fusion. We can preprocess instruction sequence, detect compares that can be fused (with the code external to MCA) and remove compare, but modify the InstrDesc for specific branch adding the source operands of compare in order to keep the data dependencies. I don't know how to do it with existing customization approaches.

I think this example can be achieved with swapping scheduling classes of the fused branch (and maybe the to-be-eliminated compare) with InstrumentManager::getSchedClassID. Although one of getSchedClassID' s parameters is a list of mca::Instrument, I believe it works just fine even without any instruments.

[r-belenov]

A single InstrDesc is shared among many different instructions, if you change the Latency of a InstrDesc of a load. Then every other loads sharing the same schedclass are also affected. I don't think that is what you want, right?

Yes, that's why I force creation of private InstrDesc for every modified instruction (please check the slide I mentioned in the original post).

I think you brought up a good idea and by all mean you should try implementing in your own tree.

It's already implemented internally and works for our tasks; I'm considering opening it, but that also requires some work with corporate bureaucracy to formally open the changes from internal repository, not much sense unless there are chances of upstreaming.

I think this example can be achieved with swapping scheduling classes of the fused branch

Does it mean that the scheduling class for the fused branch should be present in machine model? My idea is to achieve maximum flexibility in runtime without recompilation.

And, regardless of the outcome - thanks all for llvm-mca itself and for the discussion, it's what I hoped for - not outright rejection, but careful consideration of pros, cons and possible alternatives.

[r-belenov]

making too many things customizable might not be a good long-term solution from a software engineering perspective because we might end up with many small knobs that are too niche to be used more than once.

That's why I propose not the specific knob to change the latency, but generic InstrDesc customizaiton, so that it can be reused for different purposes without additional modifications of MCA core (since the actual customization logic is located outside of MCA).

[adibiagio]

Have you considered other approaches where users annotate slow load instructions using special code comments in the input assembly?

That's exactly how I test my approach - mark slow loads with the comments of the form "// LLVM-MCA-LATENCY: xxx", propagate them to the loop that feeds instructions to InstructionBuilder and create appropriate modifier that changes the latency in the InstrDesc. The comment has the actual value in clockticks, though, not the cache level; actual latencies for different levels are kept in the file external to MCA. I think it is better to customize them in runtime rather than encode in tablegen - while L2 latency for specific core model may be constant, L3 usually depends on the uncore and DRAM latency may vary greatly - and latency value in clockticks depends on the core and DRAM frequencies; there are also other memory types like HBM or NVRAM with different latencies. It is better to have the ability to model code sensitivity to different latency values specified in runtime rather than have a choice from several predefined values for L2/L3/DRAM.

I agree, it would only make sense to have descriptors for core caches.

I guess - at least in theory - we could pass to llvm-mca a definition file (e.g. a tablegen-like file) containing new definitions and overrides for instruction/memory latencies. This would be a big project though, and I am not suggesting that it is necessarily a good idea. However, it is a possibility.

Another limitation is that the LSUnit would always assume that same latency for every iteration. We don't want to implement a cache simulator after all

Agree; it's possible to approximate a mix via modeling different accesses separately and taking weighted average of clockticks (not the same as different accesses on different iterations, but it's better than nothing).

Load latency is often manually folded into the normal instruction latency.

I see; I'm focused on ARM64 with dedicated load/store instructions. X86 is more complex, but changing total latency according to accessed memory level may still work.

Yep.

I can name another thing that can be done via arbitrary InstrDesc modification - compare/branch fusion. We can preprocess instruction sequence, detect compares that can be fused (with the code external to MCA) and remove compare, but modify the InstrDesc for specific branch adding the source operands of compare in order to keep the data dependencies. I don't know how to do it with existing customization approaches.

As a side note: some llvm subtargets (at least a few Intel ones) already know about branch fusion candidates. However, that knowledge doesn't come from a tablegen file, and it is mainly used by instruction scheduling algorithms to "mutate" the dependency graph for a code region.
See for example https://github.com/llvm-mirror/llvm/blob/master/lib/Target/X86/X86MacroFusion.cpp

This is to say that some knowledge about branch fusion (at least for some subtargets) already exists.
In theory, if that knowledge was also accessible to mca (maybe using new tablegen descriptors for the subtarget), we could teach mca how to treat some instruction pairs as one.

[adibiagio]

It's already implemented internally and works for our tasks; I'm considering opening it, but that also requires some work with corporate bureaucracy to formally open the changes from internal repository, not much sense unless there are chances of upstreaming.

I hope that you will be allowed to send your patch.

I think this example can be achieved with swapping scheduling classes of the fused branch

Does it mean that the scheduling class for the fused branch should be present in machine model? My idea is to achieve maximum flexibility in runtime without recompilation.

And, regardless of the outcome - thanks all for llvm-mca itself and for the discussion, it's what I hoped for - not outright rejection, but careful consideration of pros, cons and possible alternatives.

At this stage, we are still trying to assess pros and cons. I personally don't have a strong opinion about it. So, please, don't feel like your patch would be rejected. On the contrary, I appreciate the fact that you are interested in contributing to mca. I think that ultimately we will have to see your patch before deciding whether it is a good fit or not. So, I do hope that you will be allowed to send it for review.

-Andrea

[r-belenov]

Thanks for the answer! Anyway, load latencies and branch fusion are just two things. My primary point is that custom InstrDesc modification via user-supplied lambda allows more specialized use cases without introducing the details into MCA codebase. It may involve "what if" experiments with microarchitecture properties, and for that tablegen data for existing subtargets would not be sufficient. I agree that it looks like yet another customization mechanism; however, it is generic, powerful and the changes are pretty small. Also agree that all these use cases may look too specialized; if that's the consensus, upstreaming is probably not the right thing.

I'll try to make the patch public so we can continue the discussion with more details.

Regards, Roman