Feature Request: Direct FP8 conversion from convert_hf_to_gguf.py

[bmtwl]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Hello, I've long wanted a way to go straight from FP8 models to a q8 gguf and took a swing at it by modifying prepare_tensors to track tensors and marry up the scale tensor/multiplying it into a float32 tensor: https://github.com/bmtwl/llama.cpp/blob/convert-hf-from-fp8/convert_hf_to_gguf.py
The conversion starts correctly, but fails on writing due to size/shape differences, which is where I'm losing the plot:
RuntimeError: The size of tensor a (18432) must match the size of tensor b (144) at non-singleton dimension 1
I'm hoping the remaining issue is something small. Anyone want to take a look and figure out where I'm going wrong?

Motivation

Right now safetensors need to be converted to BF16 before being quanted into anything else, a step that is theoretically not needed

Possible Implementation

deal with the scale part and force the FP8 tensors into another format before converting

[ggerganov]

This seems to work:

diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index c8bf3c538..7306be4f9 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -256,11 +256,29 @@ class ModelBase:
     def prepare_tensors(self):
         max_name_len = max(len(s) for _, s in self.tensor_map.mapping.values()) + len(".weight,")
 
+        # Collect all tensors into a dictionary to handle FP8 scale tensors
+        all_tensors = {}
         for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
+            all_tensors[name] = data_torch
+
+        for name, data_torch in all_tensors.items():
             # we don't need these
             if name.endswith((".attention.masked_bias", ".attention.bias", ".rotary_emb.inv_freq")):
                 continue
 
+            # Check if this is an FP8 tensor and has a scale tensor
+            if data_torch.dtype == torch.float8_e4m3fn:
+                scale_name = name + '_scale_inv'
+                if scale_name in all_tensors:
+                    # tile the scales to match the data shape
+                    # TODO: more general
+                    scale = all_tensors[scale_name].to(torch.float32)
+                    scale = scale.repeat_interleave(128, 0)
+                    scale = scale.repeat_interleave(128, 1)
+
+                    data_torch = data_torch.to(torch.float32) * scale
+                    self.tensor_names.remove(scale_name)
+
             old_dtype = data_torch.dtype
 
             # convert any unsupported data types to float32
@@ -6851,6 +6869,13 @@ class Exaone4Model(TextModel):
 
                 yield (self.format_tensor_name(gguf.MODEL_TENSOR.ROPE_FREQS), torch.tensor(rope_factors, dtype=torch.float32))
 
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
+        if name.endswith(".weight_scale_inv"):
+            print(f"Skipping {name} ");
+            return []
+
+        return [(self.map_tensor_name(name), data_torch)]
+
 
 @ModelBase.register("GraniteForCausalLM")
 class GraniteModel(LlamaModel):

Tested with https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-1.2B-FP8, though I think there is some issue with the chat template, so can't verify 100% that the model is coherent.

I think it would be useful to add more general support for conversion from sub-FP16 types to ggml types. This is going to become more relevant soon. cc @compilade for guidance how to implement this properly.

[bmtwl]

Thanks @ggerganov, that was exactly the hint I needed to get unstuck!
I'm working on generalizing it for R1/K2. We'll see how far I'm able to get.
@compilade, any help or direction would be greatly appreciated!

[bmtwl]

Here's a more generalized version (although maybe not optimized).
I'm converting Kimi K2 to Q8_0 right now, so I should know in the morning whether the resulting gguf is usable or not.

--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -256,11 +256,42 @@
     def prepare_tensors(self):
         max_name_len = max(len(s) for _, s in self.tensor_map.mapping.values()) + len(".weight,")
 
+        # Collect all tensors into a dictionary to handle FP8 scale tensors
+        all_tensors = {}
         for name, data_torch in chain(self.generate_extra_tensors(), self.get_tensors()):
+            all_tensors[name] = data_torch
+
+        for name, data_torch in all_tensors.items():
             # we don't need these
-            if name.endswith((".attention.masked_bias", ".attention.bias", ".rotary_emb.inv_freq")):
+            if name.endswith((".attention.masked_bias", ".attention.bias", ".rotary_emb.inv_freq", "_scale_inv")):
                 continue
 
+            # Check if this is an FP8 tensor and has a scale tensor
+            if data_torch.dtype == torch.float8_e4m3fn:
+                scale_name = name + '_scale_inv'
+                if scale_name in all_tensors:
+                    # tile the scales to match the data shape
+                    scale = all_tensors[scale_name].to(torch.float32)
+                    # Dynamically compute repeat factors for each dimension
+                    for dim in range(scale.ndim):
+                        data_dim = data_torch.shape[dim]
+                        scale_dim = scale.shape[dim]
+                        repeat_factor = (data_dim + scale_dim - 1) // scale_dim
+                        scale = scale.repeat_interleave(repeat_factor, dim=dim)
+                        # Slice to match data_dim if necessary
+                        if scale.shape[dim] > data_dim:
+                            scale = scale.narrow(dim, 0, data_dim) 
+                    data_torch = data_torch.to(torch.float32) * scale
+                    self.tensor_names.remove(scale_name)
+
+            old_dtype = data_torch.dtype
+
+            # convert any unsupported data types to float32
+            if data_torch.dtype not in (torch.float16, torch.float32):
+                data_torch = data_torch.to(torch.float32)
+                all_tensors[name] = data_torch  # Update the tensor in the dictionary
+
+
             old_dtype = data_torch.dtype
 
             # convert any unsupported data types to float32
@@ -6851,6 +6882,13 @@
 
                 yield (self.format_tensor_name(gguf.MODEL_TENSOR.ROPE_FREQS), torch.tensor(rope_factors, dtype=torch.float32))
 
+    def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iterable[tuple[str, Tensor]]:
+        if name.endswith(".weight_scale_inv"):
+            print(f"Skipping {name} ");
+            return []
+
+        return [(self.map_tensor_name(name), data_torch)]
+
 
 @ModelBase.register("GraniteForCausalLM")
 class GraniteModel(LlamaModel):

[ggerganov]

To make this more general, we should find a way to avoid adding modify_tensors for each model to skip ".weight_scale_inv".

We also would have to handle other types, not just float8_e4m3fn. I think we can make a map type -> function to be able to implement different conversion functions based on the types.

[bmtwl]

@ggerganov
The initial conversion to q8 and subsequent requant to q4 both produce coherent results, so the code itself is good. Now we just need to generalize with something that is maintainable, like the map function you proposed.
@compilade : do you have a specific implementation in mind to maintain code base consistency?

[compilade]

@compilade : do you have a specific implementation in mind to maintain code base consistency?

@bmtwl Yes, kind of;

I'm thinking of changing ModelBase.get_tensors() to support some additional pre-filters on the tensors (like dequantization).

This would use the quantization_config from config.json to determine what modifications should be made (there is more than only fp8; some quantization types pack smaller weights into U8 tensors, and could/should eventually be handled).

All of this can of course technically be done with modify_tensors, but pre-quantization is not specific to a single model arch, and overridden modify_tensors do not necessarily call the parent implementation, so dequantization should probably happen elsewhere.

There are multiple possible approaches, but the first one I'm going to try is splitting the functionality of get_tensors into an intermediate function returning a dict[str, Callable[[], Tensor]] (mapping tensor names to a function producing the corresponding tensor) to allow overriding (in separate passes) how some of the tensors are loaded (this might also help clean up the remote tensor loading override), and to allow removing tensors from the list of tensor names. This kind of feels redundant with modify_tensors, but the filters/modifiers would be called once per model instead of per tensor.

There are likely cleaner approaches, which I encourage you to find, suggest and/or implement if you want/can.

The main thing is to avoid holding on to the tensor data of the model, which is why a dict[str, Tensor] of the whole model should be avoided (or else it will cause massive memory leaks (since the lazy tensors are materialized when written) unless emptied as it's used).

[whatever1983]

Converting FP8->Q8_0 GGUF directly is such a bad idea. While you can prove that Q8_0's PPL is 99.99% of FP8, it is still a format change and imperfection. There is a thread open for more than 8 months right now to go directly FP8->FP8 GGUF, but doesn't get ggerganov's attention. From an engineering perspective: why would you convert 8bpw to 8.5bpw while losing perfection and losing FP8 tensor core acceleration in 40 and 50 series just so the macs run faster with q8_0? This is almost as bad a political decision as kicking ik's IQK and Trellis quants out of mainline. How are you going to deal with quantized or native trained MXFP4 and NVFP4 formats released in safetensors?

Really, if the engineering direction is going like this, GGUF format will be dropped by even client side users. I hope you realize that as soon as possible, because SGlang/TensorRT is capable of running FP4 safetensors directly now. So for client side models less than 30B in size, just buy a 50 series or $3000 GB10(200B FP4 size) and run the FP4 safetensor format directly is probably the better technical solution now.

Take a look at how the python inferencing solutions are reverse running FP4 safetensors with FP8 or BF16 hardware with marlin kernels, that's the way forward: running models perfectly at FP4 without weight modifications. Not some weekly ggml:sync walled solution.

[compilade]

(Whoops sorry posted too soon accidentally and pressed on the wrong thing (which closed the issue (temporarily)). Should be good now, though.)

@whatever1983 Q8_0 actually is better than FP8 in many cases (at least when both are converted from a higher precision). For example, see https://www.reddit.com/r/StableDiffusion/comments/1eso216/comparison_all_quants_we_have_so_far/.

From an engineering perspective: why would you convert 8bpw to 8.5bpw while losing perfection and losing FP8 tensor core acceleration in 40 and 50 series just so the macs run faster with q8_0?

Because FP8 was not yet implemented? (Except in #10055) It's similar to how BF16 was not implemented for a while. All it needs is enough effort and justification for a new format.

How are you going to deal with quantized or native trained MXFP4 and NVFP4 formats released in safetensors?

With quantization_config in config.json (which I assume is set for these types) when converting.

(I didn't mean to close the issue when posting this, it was an accident and I re-opened it)

[whatever1983]

@compilade: what a surprise 8.5bpw is 0.01% better in PPL than 8bpw. You still changed all of the weights right? It is the same argument in the old days between f16 and bf16. Until BF16 merged in, Bartowski didn't know converting to F16 from BF16 safetensors isn't perfect.

I am not pumping for Nvidia here. Eventually all tech companies will be forced to inference with FP4, even including Apple. So do the hard engineering right way. Support FP8/FP6/FP4 natively in GGUF.

[ubergarm]

@bmtwl

Converting from fp8 safetensors directly to bf16 GGUF definitely would definitely save some space (~2TiB) with Kimi-K2-Instruct! Thanks for looking into it more!

There are three existing methods for casting/converting deepseek/kimi-k2 MLA models tofrom native fp8 safetensors to bf16 GGUF that exist from what I know:

1. The usual method using deepseek's fp8_cast_bf16.py triton with CUDA arch >=sm89 (for native fp8e4m3 support) which needs enough VRAM to hold the tensor it is casting. then from the output bf16 safetensors using convert_hf_to_gguf.py to finally get the bf16 GGUFs. This is the usual method which you want to improve.
2. The old eveshiron fork plus triton-cpu method to go in one step omitting fp8_cast_bf16.py and doing what you are describing. This fork is built on fairydreaming's older MLA implementation so ends up with attn_kv_b which mainline doesn't use psure. But might be of interest given it is most similar to the method in which you are interested in implementing. This method was originally outlined in this HF discussion: https://huggingface.co/daydream-org/DeepSeek-R1-GGUF-11446/discussions/1#67a327570051a98a96ded9e6
3. The "triton-cpu" method is the same as the first usual method, except you can uninstall triton and install triton-cpu which will does not require a GPU. Here is a github issue with some notes on installing triton-cpu: Issues compiling (due to OneDNN/DNNL) triton-lang/triton-cpu#237 (comment) . You also need to updated the deepseek fp8_cast_bf16.py script and change a couple device cuda to cpu. I used this method to convert Kimi-K2-Instruct using less than 100GB RAM highwater mark (would be less on deepseek psure). Nice when the rig has no GPU or the GPU is older and doesn't have native fp8e4m3 support etc.

Hope these old references are not a distraction, but just some context and would love to see this baked right into mainline lcpp!

Cheers!

[bmtwl]

@ubergarm thanks for the links. I've used most of those tools in the past.
The diff I posted above does work for one-step conversion of both R1 and K2 (I've quanted and inferred successfully with both) and doesn't require any triton stuff, so I think its already useful for some people, even if it only handles FP8.
The main issue appears to be getting it into a form acceptable to the lcpp project