[Good First Issue]: Create a GGUF reader

[AlexKoff88]

The idea is to have a functionality that allows reading GGUF format and creating OpenVINO GenAI compatible representation that can be used to instantiate LLMPipeline() from it.
This task includes:

• Parsing GGUF with gguf-tools C++ library: https://github.com/antirez/gguf-tools/
• Using the code similar to MLX to get weights and config. Here is MLX Python API part but we need its C++ functionality: https://github.com/ml-explore/mlx-examples/blob/c117af83b8cbec15523bd0d69e7a57f01237ca89/llms/gguf_llm/models.py#L275
• Creating IR model, OpenVINO tokenizer/detokenizer and config.json on the fly. Here is the POC repository that does this in Python: https://github.com/AlexKoff88/gguf-to-openvino

The initial scope can include support of llama-based LLMs (e.g. llama-3.2 and SmoLMs) and FP16, Q8_0, Q4_0, Q4_1 models.
All the code should be written in C++.

[Geeks-Sid]

Can this be broken down into smaller exact tasks ? This would allow us to pick off tasks one by one and help contributors slowly build something instead of all of at once.

[AlexKoff88]

It can be for sure but the way I see it assumes that these tasks should be executed subsequently. For example:

• One can start by enabling llama-3.2-1b in FP16.
• Parsing and converting tokenizer from GGUF format to OpenVINO (tokenizer/detokenizer models). After that, we will have core functionality in place.
• Then, a few tasks can be executed in parallel:
  • Enable Q8_0 llama
  • Enable Q4_0 and Q4_1 llama
  • Enable and verify other llama-based models such as Llama-3.1-8B, SmolLMs
  • Enable the most popular quantization schemes such as Q4_K_M
  • Enable Qwen model family

...