Berkeley Function Calling Leaderboard Updates (v1.3) #1119

ShishirPatil announced in Announcements

ShishirPatil
Jul 17, 2025
Maintainer

Highlights

🏆 Stable release of Berkeley Function Calling Leaderboard V3 with Multi-step and Multi-turn function call evaluation

What's Changed

Gorilla README and repo structure revamp by @CharlieJCJ in Gorilla README and repo structure revamp #799
[BFCL] Fix live_parallel_multiple_9-8-0 copy-paste issue by @pkesseli in [BFCL] Fix live_parallel_multiple_9-8-0 copy-paste issue #865
[BFCL] Fix Typo in multi_turn_base_34 Ground Truth by @HuanzhiMao in [BFCL] Fix Typo in multi_turn_base_34 Ground Truth #876
Adding New Model Haha-7B by @ZydHaha in Adding New Model Haha-7B #858
[BFCL Chore] Implement retry_with_backoff for Amazon Nova Handler by @HuanzhiMao in [BFCL Chore] Implement retry_with_backoff for Amazon Nova Handler #880
[BFCL] Fix live_simple_183-108-0 by @pkesseli in [BFCL] Fix live_simple_183-108-0 #872
[BFCL] Fix live_simple_165-98-0 by @pkesseli in [BFCL] Fix live_simple_165-98-0 #871
[BFCL] Fix live_simple_44-18-0 and live_simple_45-18-1 by @pkesseli in [BFCL] Fix live_simple_44-18-0 and live_simple_45-18-1 #870
[BFCL] Fix Nova Handler for Consecutive User Prompt Issue by @HuanzhiMao in [BFCL] Fix Nova Handler for Consecutive User Prompt Issue #881
Add support for QwQ and Sky-T1-32B-Preview by @SumanthRH in Add support for QwQ and Sky-T1-32B-Preview #888
add handler for Bielik by @dominikabasaj in add handler for Bielik #887
[BFCL Chore] Align Score File id with Result File Test Case IDs by @HuanzhiMao in [BFCL Chore] Align Score File id with Result File Test Case IDs #893
Fix minor typo in default system prompt without func by @canyon289 in Fix minor typo in default system prompt without func #895
Falcon3 support by @kirill-fedyanin in Falcon3 support #894
[BFCL] Update tool construction for Palmyra models by @samjulien in [BFCL] Update tool construction for Palmyra models #897
Added compute_exchange_rate to multi_turn_base entry 180 ground truth by @Raymond112514 in Added compute_exchange_rate to multi_turn_base entry 180 ground truth #892
[BFCL] Add New Model o3-mini-2025-01-31 and o3-mini-2025-01-31-FC by @HuanzhiMao in [BFCL] Add New Model o3-mini-2025-01-31 and o3-mini-2025-01-31-FC #898
Add CALM models by @jgreer013 in Add CALM models #900
[BFCL] Add New Model gemini-2.0-flash-001, gemini-2.0-flash-lite-preview-02-05, gemini-2.0-pro-exp-02-05. by @HuanzhiMao in [BFCL] Add New Model gemini-2.0-flash-001, gemini-2.0-flash-lite-preview-02-05, gemini-2.0-pro-exp-02-05. #902
chore: added snippet for hf datasets compatibility by @alt-glitch in chore: added snippet for hf datasets compatibility #906
Update model_metadata.py by @jgreer013 in Update model_metadata.py #907
Rename CALM to CoALM by @jgreer013 in Rename CALM to CoALM #913
Bitagent 8b submission by @VectorForger in Bitagent 8b submission #917
Bitagent 8b Metadata Change by @VectorForger in Bitagent 8b Metadata Change #919
[BFCL] Add New Model gpt-4.5-preview-2025-02-27, gpt-4.5-preview-2025-02-27-FC by @HuanzhiMao in [BFCL] Add New Model gpt-4.5-preview-2025-02-27, gpt-4.5-preview-2025-02-27-FC #922
[BFCL] fix bug in how score_dir is handled for bfcl evaluate by @liamcli in [BFCL] fix bug in how score_dir is handled for bfcl evaluate #924
[BFCL] Add New Model DeepSeek-R1 by @HuanzhiMao in [BFCL] Add New Model DeepSeek-R1 #901
Make all import paths absolute. by @fvisin in Make all import paths absolute. #935
Move logic to eval a task in a separate function. by @fvisin in Move logic to eval a task in a separate function. #933
Fix Gorilla Paper requirements.txt Location to Remove Global Dependency Confusion by @HuanzhiMao in Fix Gorilla Paper requirements.txt Location to Remove Global Dependency Confusion #937
[BFCL] Add _unused Suffix to Unused Dataset Files in the BFCL Benchmark by @HuanzhiMao in [BFCL] Add _unused Suffix to Unused Dataset Files in the BFCL Benchmark #938
[BFCL] Support Local Inference for deepseek-ai/DeepSeek-R1 by @HuanzhiMao in [BFCL] Support Local Inference for deepseek-ai/DeepSeek-R1 #926
[BFCL] Add Support for Qwen2.5 Models in Function Calling Mode by @HuanzhiMao in [BFCL] Add Support for Qwen2.5 Models in Function Calling Mode #925
[BFCL] Add New Model claude-3-7-sonnet-20250219, claude-3-7-sonnet-20250219-FC by @HuanzhiMao in [BFCL] Add New Model claude-3-7-sonnet-20250219, claude-3-7-sonnet-20250219-FC #923
[BFCL] Add handler and meta info for ToolACE-2-8B by @XuHwang in [BFCL] Add handler and meta info for ToolACE-2-8B #941
[BFCL] Reorganized All constant.py Files to a constants Folder by @catherineruoxiwu in [BFCL] Reorganized All constant.py Files to a constants Folder #944
[BFCL] Add New Models gemini-2.0-flash-lite-001, gemini-2.0-flash-thinking-exp-01-21 by @HuanzhiMao in [BFCL] Add New Models gemini-2.0-flash-lite-001, gemini-2.0-flash-thinking-exp-01-21 #942
[BFCL] Add Google Gemma-3 Series Models by @HuanzhiMao in [BFCL] Add Google Gemma-3 Series Models #939
[BFCL] Move model_metadata.py to constants folder by @catherineruoxiwu in [BFCL] Move model_metadata.py to constants folder #949
Add Cohere Command A by @harry-cohere in Add Cohere Command A #951
Reformatted Supported Model Table by @JasonHuang1103 in Reformatted Supported Model Table #961
[BFCL] Use HTTPS instead of HTTP for OMDB by @hrshtv in [BFCL] Use HTTPS instead of HTTP for OMDB #960
[BFCL] Fix ambiguity in exec_parallel_10 question by @amitojsingh2022 in [BFCL] Fix ambiguity in exec_parallel_10 question #962
[BFCL] Fix API Keys Handling by @catherineruoxiwu in [BFCL] Fix API Keys Handling #959
[BFCL] Fix wrong date in live_simple_205-116-13 by @amitojsingh2022 in [BFCL] Fix wrong date in live_simple_205-116-13 #963
[BFCL] Moved Ground Truths for Executable Tests to ./data/possible_answer Folder by @catherineruoxiwu in [BFCL] Moved Ground Truths for Executable Tests to ./data/possible_answer Folder #953
[BFCL] Reorganizing Codes in ./bfcl/eval_checker/executable_eval/data/ by @catherineruoxiwu in [BFCL] Reorganizing Codes in ./bfcl/eval_checker/executable_eval/data/ #954
[BFCL] Add gemini-2.5-pro to the Leaderboard by @catherineruoxiwu in [BFCL] Add gemini-2.5-pro to the Leaderboard #974
[BFCL] Update Retry Logic for Gemini Models by @HuanzhiMao in [BFCL] Update Retry Logic for Gemini Models #976
[BFCL] Fix Typo in multi_turn_base_166 Ground Truth. by @HuanzhiMao in [BFCL] Fix Typo in multi_turn_base_166 Ground Truth. #979
Add Salesforce xLAM-2 series of model handlers and update vLLM version from 0.6.3 to 0.6.5 by @zuxin666 in Add Salesforce xLAM-2 series of model handlers and update vLLM version from 0.6.3 to 0.6.5 #972
[BFCL] Retire Executable Categories from Leaderboard by @HuanzhiMao in [BFCL] Retire Executable Categories from Leaderboard #943
feat. Add Novita LLM Models API by @novita-viktor in feat. Add Novita LLM Models API #980
[BFCL] Add New Models Llama-4-Scout, Llama-4-Maverick by @HuanzhiMao in [BFCL] Add New Models Llama-4-Scout, Llama-4-Maverick #981
[BFCL] Add Support for Fully Offline Model Inference via --local-model-path by @catherineruoxiwu in [BFCL] Add Support for Fully Offline Model Inference via --local-model-path #985
Fix Typo in Model Name for xLAM-2-8b-fc-r by @HuanzhiMao in Fix Typo in Model Name for xLAM-2-8b-fc-r #992
Add ThinkAgents/ThinkAgent-1B by @0xayman in Add ThinkAgents/ThinkAgent-1B #928
[BFCL] Add Grok 3 Models to the Leaderboard by @catherineruoxiwu in [BFCL] Add Grok 3 Models to the Leaderboard #987
[BFCL] Add mistral-large-2411 and mistral-small-2503 by @pracheeti12 in [BFCL] Add mistral-large-2411 and mistral-small-2503 #988
Add xiaoming-14B by @kevin2016 in Add xiaoming-14B #977
[BFCL] Retire Outdated Models from the Leaderboard by @catherineruoxiwu in [BFCL] Retire Outdated Models from the Leaderboard #997
[BFCL] add support for microsoft/Phi-4-mini-instruct by @RobotSail in [BFCL] add support for microsoft/Phi-4-mini-instruct #967
[BFCL] Add microsoft/phi-4 to the Leaderboard by @catherineruoxiwu in [BFCL] Add microsoft/phi-4 to the Leaderboard #1000
[BFCL] Add GPT 4.1 Series Models to the Leaderboard. by @catherineruoxiwu in [BFCL] Add GPT 4.1 Series Models to the Leaderboard. #1002
Bump writer-sdk Dependency Version by @HuanzhiMao in Bump writer-sdk Dependency Version #1006
[BFCL] add model config by @itea1001 in [BFCL] add model config #999
[BFCL] Add Validation for Model Names by @catherineruoxiwu in [BFCL] Add Validation for Model Names #1008
[BFCL] Update Error Message for New Handler Mappings by @catherineruoxiwu in [BFCL] Update Error Message for New Handler Mappings #1013
[BFCL] fix entry id typo in live_multiple_1052-279-0 by @itea1001 in [BFCL] fix entry id typo in live_multiple_1052-279-0 #1022
Update QwQ-32b api by @CostaliyA in Update QwQ-32b api #1014
Migrate to correct testing API by @emmanuel-ferdman in Migrate to correct testing API #1029
Add gemini-2.5-pro-preview-05-06 Models by @Guangyu-Joshua-Feng in Add gemini-2.5-pro-preview-05-06 Models #1031
[BFCL] Add Qwen 3 Series Models to the Leaderboard by @catherineruoxiwu in [BFCL] Add Qwen 3 Series Models to the Leaderboard #1015
[BFCL] Remove latency data for open source models by @errorfourten in [BFCL] Remove latency data for open source models #1033
fix treesitter setup by @CharlieJCJ in fix treesitter setup #1045
[BFCL] Added support for Mistral Medium 3 by @errorfourten in [BFCL] Added support for Mistral Medium 3 #1040
New colab links for gorilla hosted and openfunctions hosted by @ShishirPatil in New colab links for gorilla hosted and openfunctions hosted #1036
[BFCL] Add version to bfcl CLI by @ShishirPatil in [BFCL] Add version to bfcl CLI #1038
Add DM-Cito-8B by @kevin2016 in Add DM-Cito-8B #1017
fix: loosen openai requirements to be >= 1.76.0 by @TheFloatingString in fix: loosen openai requirements to be >= 1.76.0 #1050
[BFCL] Packagerize for PyPI Distribution by @HuanzhiMao in [BFCL] Packagerize for PyPI Distribution #1054
[BFCL] CI: Add “Publish to PyPI” workflow with CalVer-serial auto-versioning by @HuanzhiMao in [BFCL] CI: Add “Publish to PyPI” workflow with CalVer-serial auto-versioning #1055
[BFCL] Replace Exception with SyntaxError for Java and JavaScript Parsers by @TheFloatingString in [BFCL] Replace Exception with SyntaxError for Java and JavaScript Parsers #1057
[BFCL] Support DashScope API Inference for Qwen3 Series by @HuanzhiMao in [BFCL] Support DashScope API Inference for Qwen3 Series #1061
[BFCL] Add type hinting by @TheFloatingString in [BFCL] Add type hinting #1058
[BFCL] Added support for DeepSeek-R1-0528 and DeepSeek-V3-0324 by @errorfourten in [BFCL] Added support for DeepSeek-R1-0528 and DeepSeek-V3-0324 #1063
[BFCL] Add support for Ling-Lite-V1.5 by @fengzhu1 in [BFCL] Add support for Ling-Lite-V1.5 #1056
[BFCL] Omit Reasoning Content from Chat History for Function-Calling Models by @HuanzhiMao in [BFCL] Omit Reasoning Content from Chat History for Function-Calling Models #1064
Add support for llama-3.1-nemotron-ultra-253b-v1 to BFCL by @AdityaGhai18 in Add support for llama-3.1-nemotron-ultra-253b-v1 to BFCL #1032
_get_item() can not handle the "." directory in path string by @YJ3329 in _get_item() can not handle the "." directory in path string #1060
[BFCL] Multi-turn TravelAPI book_flight() Fix by @amitojsingh2022 in [BFCL] Multi-turn TravelAPI book_flight() Fix #966
[BFCL] Fix prompt concatenation bug in Qwen template by @nehcgs in [BFCL] Fix prompt concatenation bug in Qwen template #1068
Add Qwen handler by @zhangyingerjelly in Add Qwen handler #1072
Add traceback logging to json outputs by @imradawoodani in Add traceback logging to json outputs #1074
[BFCL] Fix is_fc_model config propagation by @HuanzhiMao in [BFCL] Fix is_fc_model config propagation #1082
feature: Added RZN-T to the suite of models. by @KevinDayve in feature: Added RZN-T to the suite of models. #1079
Fix typo in month parameter (Febuary -> February) by @Gnav3852 in Fix typo in month parameter (Febuary -> February) #1084
Update irrelevance_232 question by @Gnav3852 in Update irrelevance_232 question #1085
[BFCL] Fixed missing airport route entries by @amitojsingh2022 in [BFCL] Fixed missing airport route entries #1087
[BFCL] Resolve duplicated 'live-relevance_3-3-0' test entry id by @gumgizoa in [BFCL] Resolve duplicated 'live-relevance_3-3-0' test entry id #1086
Added support for Claude 4 family models to BFCL by @Swordscore in Added support for Claude 4 family models to BFCL #1034
Add DM-Cito-8B-v2 by @kevin2016 in Add DM-Cito-8B-v2 #1088
update ground truth for multi turn base by @Daniel-Mash in update ground truth for multi turn base #956
Restrict GitHub Actions Workflow to Run Only on Source Repository by @HuanzhiMao in Restrict GitHub Actions Workflow to Run Only on Source Repository #1089
[BFCL] Add support for Arch-Agent by @nehcgs in [BFCL] Add support for Arch-Agent #1078
[BFCL] Introduce OpenAI Responses API Handler + o4-mini/o3 models by @errorfourten in [BFCL] Introduce OpenAI Responses API Handler + o4-mini/o3 models #1062
nit(docs): Improve README Clarity on Sample Filename by @HuanzhiMao in nit(docs): Improve README Clarity on Sample Filename #1093
[BFCL] Add support for granite-3.1-8b-instruct and granite-3.2-8b-instruct by @RobotSail in [BFCL] Add support for granite-3.1-8b-instruct and granite-3.2-8b-instruct #1041
[BFCL] Replace system role with developer role for OpenAI models by @errorfourten in [BFCL] Replace system role with developer role for OpenAI models #1090
[BFCL] nit: Print traceback on generation error by @HuanzhiMao in [BFCL] nit: Print traceback on generation error #1100
[BFCL] Migrate Gemini Inference to Google AI Studio by @HuanzhiMao in [BFCL] Migrate Gemini Inference to Google AI Studio #1099
[BFCL] Update Gemini model checkpoints to stable 2.5 releases by @HuanzhiMao in [BFCL] Update Gemini model checkpoints to stable 2.5 releases #1102
[BFCL] Reintroduce latency stats for local models, update cost calculation by @Gnav3852 in [BFCL] Reintroduce latency stats for local models, update cost calculation #1098
BitAgent Bounty Model Submission by @VectorForger in BitAgent Bounty Model Submission #1096
[BFCL] Contact Customer Support Multi Turn & Vehicle Control [BFCL] Multi-turn evaluation problem. #914 by @amitojsingh2022 in [BFCL] Contact Customer Support Multi Turn & Vehicle Control #914 #1110

New Contributors

@pkesseli made their first contribution in [BFCL] Fix live_parallel_multiple_9-8-0 copy-paste issue #865
@ZydHaha made their first contribution in Adding New Model Haha-7B #858
@SumanthRH made their first contribution in Add support for QwQ and Sky-T1-32B-Preview #888
@dominikabasaj made their first contribution in add handler for Bielik #887
@canyon289 made their first contribution in Fix minor typo in default system prompt without func #895
@kirill-fedyanin made their first contribution in Falcon3 support #894
@jgreer013 made their first contribution in Add CALM models #900
@alt-glitch made their first contribution in chore: added snippet for hf datasets compatibility #906
@VectorForger made their first contribution in Bitagent 8b submission #917
@liamcli made their first contribution in [BFCL] fix bug in how score_dir is handled for bfcl evaluate #924
@fvisin made their first contribution in Make all import paths absolute. #935
@catherineruoxiwu made their first contribution in [BFCL] Reorganized All constant.py Files to a constants Folder #944
@JasonHuang1103 made their first contribution in Reformatted Supported Model Table #961
@hrshtv made their first contribution in [BFCL] Use HTTPS instead of HTTP for OMDB #960
@novita-viktor made their first contribution in feat. Add Novita LLM Models API #980
@0xayman made their first contribution in Add ThinkAgents/ThinkAgent-1B #928
@pracheeti12 made their first contribution in [BFCL] Add mistral-large-2411 and mistral-small-2503 #988
@kevin2016 made their first contribution in Add xiaoming-14B #977
@RobotSail made their first contribution in [BFCL] add support for microsoft/Phi-4-mini-instruct #967
@itea1001 made their first contribution in [BFCL] add model config #999
@CostaliyA made their first contribution in Update QwQ-32b api #1014
@emmanuel-ferdman made their first contribution in Migrate to correct testing API #1029
@Guangyu-Joshua-Feng made their first contribution in Add gemini-2.5-pro-preview-05-06 Models #1031
@errorfourten made their first contribution in [BFCL] Remove latency data for open source models #1033
@TheFloatingString made their first contribution in fix: loosen openai requirements to be >= 1.76.0 #1050
@fengzhu1 made their first contribution in [BFCL] Add support for Ling-Lite-V1.5 #1056
@AdityaGhai18 made their first contribution in Add support for llama-3.1-nemotron-ultra-253b-v1 to BFCL #1032
@YJ3329 made their first contribution in _get_item() can not handle the "." directory in path string #1060
@nehcgs made their first contribution in [BFCL] Fix prompt concatenation bug in Qwen template #1068
@zhangyingerjelly made their first contribution in Add Qwen handler #1072
@imradawoodani made their first contribution in Add traceback logging to json outputs #1074
@KevinDayve made their first contribution in feature: Added RZN-T to the suite of models. #1079
@Gnav3852 made their first contribution in Fix typo in month parameter (Febuary -> February) #1084
@gumgizoa made their first contribution in [BFCL] Resolve duplicated 'live-relevance_3-3-0' test entry id #1086
@Swordscore made their first contribution in Added support for Claude 4 family models to BFCL #1034
@Daniel-Mash made their first contribution in update ground truth for multi turn base #956

Full Changelog: v1.2...v1.3

This discussion was created from the release Berkeley Function Calling Leaderboard Updates (v1.2).

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment