Skip to content

AKSW/LLM-KG-Bench-v3-0-results

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Results generated with LLM-KG-Bench version 3

This repo contains evaluation results generated with LLM-KG-Bench version 3 for 38* open and proprietary LLMs. This includes the configuration, the model format preference table and the plots.

The raw result files are passwort protected to reduce the chance of test data leakage into LLM training data. Please do not spread them unencrypted. You can unzip them with Leave-Finest-Coats-Serena-Cheats-46

*Note: The LLM "Solar pro preview 22B" skipped several test cases due to context window limitations.

CC BY 4.0 This dataset is licensed under CC BY 4.0.

Navigating Open LLM Semantic Web Technology Support with Capability Compass

Family 0.5B 1B 1.5B 3B/3.8B 7B/8B MoE Active 6-14B 14B 32B/33B 70B/72B
Llama 3.0
--> Meta-Llama-3-8B-Instruct Meta-Llama-3-70B-Instruct
Llama 3.1
--> Llama-3.1-8B-Instruct Llama-3.1-70B-Instruct
Llama 3.2
--> Llama-3.2-1B-Instruct Llama-3.2-3B-Instruct
Llama 3.3
--> Llama-3.3-70B-Instruct
Phi 3.0
--> Phi-3-mini-128k-instruct Phi-3-small-128k-instruct Phi-3-medium-128k-instruct
Phi 3.5
--> Phi-3.5-mini-instruct Phi-3.5-MoE-instruct
Qwen2
--> Qwen2-0.5B-Instruct Qwen2-1.5B-Instruct Qwen2-7B-Instruct Qwen2-57B-A14B-Instruct Qwen2-72B-Instruct
Qwen2.5
--> Qwen2.5-0.5B-Instruct Qwen2.5-1.5B-Instruct Qwen2.5-3B-Instruct Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct Qwen2.5-32B-Instruct Qwen2.5-72B-Instruct
Qwen2.5-Coder
--> Qwen2.5-Coder-32B-Instruct
Infly-OpenCoder
--> OpenCoder-8B-Instruct
Deepseek-coder
--> deepseek-coder-33b-instruct

Model Preference Turtle vs. JSON-LD:

RdfConnectionExplainStatic RdfFriendCount-1 RdfFriendCount-2 RdfSyntaxFixList Sparql2AnswerListOrga Text2AnswerListOrga
Claude 3.5 Haiku JSON TTL TTL - - -
Claude 3.5 Sonnet - - - - - -
Deepseek-Coder-33B - JSON JSON JSON - -
GPT3.5 2024/01 - JSON JSON - - TTL
GPT4o 2024/11 - - - JSON - -
GPT4o-mini 2024/07 - - TTL - - TTL
GPTo1-mini 2024/09 - - - - - -
GPTo1-pre 2024/09 - - - - - -
Gemini 1.5 Flash - TTL - - - -
Gemini 1.5 Pro - - - - - -
Gemini 2.0 Flash Exp - - - - - -
Llama-3.3-70B - JSON JSON JSON - -
Meta-Llama-3-70B - JSON JSON JSON - -
Meta-Llama-3-8B TTL - - - - -
Meta-Llama-3.1-70B - - - JSON TTL -
Meta-Llama-3.1-8B - JSON JSON JSON - -
Meta-Llama-3.2-1B - - - JSON - -
Meta-Llama-3.2-3B JSON TTL TTL JSON - -
OpenCoder-8B - - - - TTL -
Phi-3-medium-128k TTL - JSON - - TTL
Phi-3-mini-128k - - JSON JSON - -
Phi-3-small-128k JSON - - TTL - -
Phi-3.5-mini TTL JSON - JSON TTL -
Qwen2-0.5B - - JSON - - -
Qwen2-1.5B TTL JSON JSON JSON - -
Qwen2-57B-A14B - - - JSON TTL TTL
Qwen2-72B - - - JSON - -
Qwen2-7B JSON - - - - TTL
Qwen2.5-0.5B - - - - - -
Qwen2.5-1.5B - - - JSON - -
Qwen2.5-14B TTL - - - - TTL
Qwen2.5-32B - TTL TTL JSON JSON -
Qwen2.5-3B JSON - - JSON - -
Qwen2.5-72B JSON JSON JSON - - TTL
Qwen2.5-7B JSON - - - TTL TTL
Qwen2.5-Coder-32B - TTL TTL - - -
Solar-pro-preview-22B* - - - JSON - -
Phi_3.5_MoE_Instruct - JSON JSON JSON TTL -

About

Results generated with LLM-KG-Bench version 3

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 3

  •  
  •  
  •