Can we prepend some header(example: memref) on Tensor data which custom hardware can use to execute Tensor Data

[akapoor3518]

Hi,
My Question at following simple example:
A = B+C where , A, B & C are one dimensional vector, example of size 32 elements with type GGML_TYPE_F32.
hence i created one dimension Tensor for both leaf
I have setup backend buffer,
each leaf & node Tensor will have size = 432 bytes
but i want to prepend header of 64 bytes
Hence each tensor has size = 64 + 432
I can encode stuff on 64bytes which my custom hardware can understand during compute function.
Is this possible, please let me know.
Thanks,
Anoop

[slaren]

You can override the get_alloc_size function of the buffer type so that it allocates 64 extra bytes for the tensor, e.g. make it return ggml_nbytes(t) + 64;. Then you can adjust the set_tensor and get_tensor functions to skip the header, and use the init_tensor function to initialize it.

[akapoor3518]

Thanks,
Following exactly i did but this issue with set_tensor & get_tensor where its throwing GGML_ASSERT
here is example where i am seeing GGML_ASSERT
GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor write out of bounds");

[slaren]

As already mentioned, you need to modify set_tensor and get_tensor to skip the headers. You cannot set or get the header using the standard ggml-backend functions, that's an implementation detail that needs to remain hidden from the clients. If you need that functionality, you will need to add a custom function, which you can expose via header files and the get_proc_address function.

[akapoor3518]

Sorry may be i am missing something but as client i am doing following to skip the header
ggml_backend_tensor_set(model.a, a, sizeof(tensor_data_header), ggml_nbytes(model.a));

but below API generate ASSERT which i am comments to avoid GGML ASSERT, any thought on below standard code where i have commented //GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor write out of bounds");. how to get away with this ASSERT.

void ggml_backend_tensor_set(struct ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer;

GGML_ASSERT(buf != NULL && "tensor buffer not set");
GGML_ASSERT(tensor->data != NULL && "tensor not allocated");
//GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor write out of bounds");

Thanks!

[slaren]

The tensor header needs to be hidden from the abstract ggml-backend interface, that is to say, all the functions to access the tensor data such as get_tensor and set_tensor must behave as if there was no header. Internally, the backend buffer type may allocate enough space for a header and manage it in any way it needs, but this must be hidden from the common ggml-backend function. A simple way to think about it is that all applications must be able to use the backend in the same way as it would use any other backend, with the same calls to ggml_backend_tensor_set and other functions. That's the whole point of having a common interface for all the backends.

A simple way to implement this may be to allocate 64 extra bytes in the alloc_buffer function, and offset the base address of the buffer by the same amount. Then the header of tensors will be located 64 before the tensor->data pointer.

[akapoor3518]

I Think u are missing my point, I have created custom

• .get_alloc_size = */ ggml_backend_companyA_buffer_type_get_alloc_size
  I have also allocated extra buffer from backend buffer my functionally is working perfectly fine no issue. Everything tested no issue

only issue i found that i need to comment below code for ggml_backend_tensor_set & ggml_backend_tensor_get which is not correct i cant commit/push below code at GGML Repo what is solution for below code. Just look at ggml_backend_tensor_set & ggml_backend_tensor_get and le me know what need to done so that we can avoid below GGML_ASSERT(once again everything else perfectly fine, i have defined all backend API to implement my custom Hardware and all functionality is working fine with me)
//GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor write out of bounds");

[akapoor3518]

Please ignore my previous message, I am able to solve with Backend implementation of tensor_set/tensor_get/alloc_func here we are moving the customer header. We can close this discussion. My issue is resolved. Thanks for all help!