-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
33 changed files
with
6,505 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Academic Project Page Template | ||
| This is an academic paper project page template. | ||
|
|
||
|
|
||
| Example project pages built using this template are: | ||
| - https://vision.huji.ac.il/spectral_detuning/ | ||
| - https://vision.huji.ac.il/podd/ | ||
| - https://dreamix-video-editing.github.io | ||
| - https://vision.huji.ac.il/conffusion/ | ||
| - https://vision.huji.ac.il/3d_ads/ | ||
| - https://vision.huji.ac.il/ssrl_ad/ | ||
| - https://vision.huji.ac.il/deepsim/ | ||
|
|
||
|
|
||
|
|
||
| ## Start using the template | ||
| To start using the template click on `Use this Template`. | ||
|
|
||
| The template uses html for controlling the content and css for controlling the style. | ||
| To edit the websites contents edit the `index.html` file. It contains different HTML "building blocks", use whichever ones you need and comment out the rest. | ||
|
|
||
| **IMPORTANT!** Make sure to replace the `favicon.ico` under `static/images/` with one of your own, otherwise your favicon is going to be a dreambooth image of me. | ||
|
|
||
| ## Components | ||
| - Teaser video | ||
| - Images Carousel | ||
| - Youtube embedding | ||
| - Video Carousel | ||
| - PDF Poster | ||
| - Bibtex citation | ||
|
|
||
| ## Tips: | ||
| - The `index.html` file contains comments instructing you what to replace, you should follow these comments. | ||
| - The `meta` tags in the `index.html` file are used to provide metadata about your paper | ||
| (e.g. helping search engine index the website, showing a preview image when sharing the website, etc.) | ||
| - The resolution of images and videos can usually be around 1920-2048, there rarely a need for better resolution that take longer to load. | ||
| - All the images and videos you use should be compressed to allow for fast loading of the website (and thus better indexing by search engines). For images, you can use [TinyPNG](https://tinypng.com), for videos you can need to find the tradeoff between size and quality. | ||
| - When using large video files (larger than 10MB), it's better to use youtube for hosting the video as serving the video from the website can take time. | ||
| - Using a tracker can help you analyze the traffic and see where users came from. [statcounter](https://statcounter.com) is a free, easy to use tracker that takes under 5 minutes to set up. | ||
| - This project page can also be made into a github pages website. | ||
| - Replace the favicon to one of your choosing (the default one is of the Hebrew University). | ||
| - Suggestions, improvements and comments are welcome, simply open an issue or contact me. You can find my contact information at [https://pages.cs.huji.ac.il/eliahu-horwitz/](https://pages.cs.huji.ac.il/eliahu-horwitz/) | ||
|
|
||
| ## Acknowledgments | ||
| Parts of this project page were adopted from the [Nerfies](https://nerfies.github.io/) page. | ||
|
|
||
| ## Website License | ||
| <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| # Academic Project Page Template | ||
| This is an academic paper project page template. | ||
|
|
||
|
|
||
| Example project pages built using this template are: | ||
| - https://vision.huji.ac.il/spectral_detuning/ | ||
| - https://vision.huji.ac.il/podd/ | ||
| - https://dreamix-video-editing.github.io | ||
| - https://vision.huji.ac.il/conffusion/ | ||
| - https://vision.huji.ac.il/3d_ads/ | ||
| - https://vision.huji.ac.il/ssrl_ad/ | ||
| - https://vision.huji.ac.il/deepsim/ | ||
|
|
||
|
|
||
|
|
||
| ## Start using the template | ||
| To start using the template click on `Use this Template`. | ||
|
|
||
| The template uses html for controlling the content and css for controlling the style. | ||
| To edit the websites contents edit the `index.html` file. It contains different HTML "building blocks", use whichever ones you need and comment out the rest. | ||
|
|
||
| **IMPORTANT!** Make sure to replace the `favicon.ico` under `static/images/` with one of your own, otherwise your favicon is going to be a dreambooth image of me. | ||
|
|
||
| ## Components | ||
| - Teaser video | ||
| - Images Carousel | ||
| - Youtube embedding | ||
| - Video Carousel | ||
| - PDF Poster | ||
| - Bibtex citation | ||
|
|
||
| ## Tips: | ||
| - The `index.html` file contains comments instructing you what to replace, you should follow these comments. | ||
| - The `meta` tags in the `index.html` file are used to provide metadata about your paper | ||
| (e.g. helping search engine index the website, showing a preview image when sharing the website, etc.) | ||
| - The resolution of images and videos can usually be around 1920-2048, there rarely a need for better resolution that take longer to load. | ||
| - All the images and videos you use should be compressed to allow for fast loading of the website (and thus better indexing by search engines). For images, you can use [TinyPNG](https://tinypng.com), for videos you can need to find the tradeoff between size and quality. | ||
| - When using large video files (larger than 10MB), it's better to use youtube for hosting the video as serving the video from the website can take time. | ||
| - Using a tracker can help you analyze the traffic and see where users came from. [statcounter](https://statcounter.com) is a free, easy to use tracker that takes under 5 minutes to set up. | ||
| - This project page can also be made into a github pages website. | ||
| - Replace the favicon to one of your choosing (the default one is of the Hebrew University). | ||
| - Suggestions, improvements and comments are welcome, simply open an issue or contact me. You can find my contact information at [https://pages.cs.huji.ac.il/eliahu-horwitz/](https://pages.cs.huji.ac.il/eliahu-horwitz/) | ||
|
|
||
| ## Acknowledgments | ||
| Parts of this project page were adopted from the [Nerfies](https://nerfies.github.io/) page. | ||
|
|
||
| ## Website License | ||
| <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International License</a>. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,198 @@ | ||
| <!DOCTYPE html> | ||
| <html> | ||
| <head> | ||
| <meta charset="utf-8"> | ||
| <!-- Meta tags for social media banners, these should be filled in appropriatly as they are your "business card" --> | ||
| <!-- Replace the content tag with appropriate information --> | ||
| <meta name="description" content="DESCRIPTION META TAG"> | ||
| <meta property="og:title" content="SOCIAL MEDIA TITLE TAG"/> | ||
| <meta property="og:description" content="SOCIAL MEDIA DESCRIPTION TAG TAG"/> | ||
| <meta property="og:url" content="URL OF THE WEBSITE"/> | ||
| <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X630--> | ||
| <meta property="og:image" content="static/image/your_banner_image.png" /> | ||
| <meta property="og:image:width" content="1200"/> | ||
| <meta property="og:image:height" content="630"/> | ||
|
|
||
|
|
||
| <meta name="twitter:title" content="TWITTER BANNER TITLE META TAG"> | ||
| <meta name="twitter:description" content="TWITTER BANNER DESCRIPTION META TAG"> | ||
| <!-- Path to banner image, should be in the path listed below. Optimal dimenssions are 1200X600--> | ||
| <meta name="twitter:image" content="static/images/your_twitter_banner_image.png"> | ||
| <meta name="twitter:card" content="summary_large_image"> | ||
| <!-- Keywords for your paper to be indexed by--> | ||
| <meta name="keywords" content="KEYWORDS SHOULD BE PLACED HERE"> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1"> | ||
|
|
||
|
|
||
| <title>KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models</title> | ||
| <link rel="icon" type="image/x-icon" href="static/images/favicon.ico"> | ||
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" | ||
| rel="stylesheet"> | ||
|
|
||
| <link rel="stylesheet" href="static/css/bulma.min.css"> | ||
| <link rel="stylesheet" href="static/css/bulma-carousel.min.css"> | ||
| <link rel="stylesheet" href="static/css/bulma-slider.min.css"> | ||
| <link rel="stylesheet" href="static/css/fontawesome.all.min.css"> | ||
| <link rel="stylesheet" | ||
| href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> | ||
| <link rel="stylesheet" href="static/css/index.css"> | ||
|
|
||
| <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> | ||
| <script src="https://documentcloud.adobe.com/view-sdk/main.js"></script> | ||
| <script defer src="static/js/fontawesome.all.min.js"></script> | ||
| <script src="static/js/bulma-carousel.min.js"></script> | ||
| <script src="static/js/bulma-slider.min.js"></script> | ||
| <script src="static/js/index.js"></script> | ||
| </head> | ||
| <body> | ||
|
|
||
|
|
||
| <section class="hero"> | ||
| <div class="hero-body"> | ||
| <div class="container is-max-desktop"> | ||
| <div class="columns is-centered"> | ||
| <div class="column has-text-centered"> | ||
| <h1 class="title is-1 publication-title">KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models</h1> | ||
| <div class="is-size-5 publication-authors"> | ||
| <!-- Paper authors --> | ||
| <span class="author-block">Zhuohao Yu<sup>1</sup>,</span> | ||
| <span class="author-block">Chang Gao<sup>1</sup>,</span> | ||
| <span class="author-block">Wenjin Yao<sup>1</sup>,</span> | ||
| <span class="author-block">Yidong Wang<sup>1</sup>,</span><br> | ||
| <span class="author-block">Wei Ye<sup>1*</sup>,</span> | ||
| <span class="author-block">Jindong Wang<sup>2</sup>,</span> | ||
| <span class="author-block">Xing Xie<sup>2</sup>,</span> | ||
| <span class="author-block">Yue Zhang<sup>3</sup>,</span> | ||
| <span class="author-block">Shikun Zhang<sup>1</sup></span> | ||
|
|
||
| </div> | ||
|
|
||
| <div class="is-size-5 publication-authors"> | ||
| <span class="author-block">Peking University<sup>1</sup>, Microsoft Research<sup>2</sup>, Westlake University<sup>3</sup>, <br>ACL 2024</span> | ||
| <span class="eql-cntrb"><small><br><sup>*</sup>Corresponding Author</small></span> | ||
| </div> | ||
|
|
||
| <div class="column has-text-centered"> | ||
| <div class="publication-links"> | ||
| <!-- Arxiv PDF link --> | ||
| <span class="link-block"> | ||
| <a href="https://arxiv.org/pdf/2402.15043.pdf" target="_blank" | ||
| class="external-link button is-normal is-rounded is-dark"> | ||
| <span class="icon"> | ||
| <i class="fas fa-file-pdf"></i> | ||
| </span> | ||
| <span>Paper</span> | ||
| </a> | ||
| </span> | ||
|
|
||
| <!-- Supplementary PDF link --> | ||
| <!-- <span class="link-block"> | ||
| <a href="static/pdfs/supplementary_material.pdf" target="_blank" | ||
| class="external-link button is-normal is-rounded is-dark"> | ||
| <span class="icon"> | ||
| <i class="fas fa-file-pdf"></i> | ||
| </span> | ||
| <span>Supplementary</span> | ||
| </a> --> | ||
| </span> | ||
|
|
||
| <!-- Github link --> | ||
| <span class="link-block"> | ||
| <a href="https://github.com/zhuohaoyu/KIEval" target="_blank" | ||
| class="external-link button is-normal is-rounded is-dark"> | ||
| <span class="icon"> | ||
| <i class="fab fa-github"></i> | ||
| </span> | ||
| <span>Code</span> | ||
| </a> | ||
| </span> | ||
|
|
||
| <!-- ArXiv abstract Link --> | ||
| <!-- <span class="link-block"> | ||
| <a href="https://arxiv.org/abs/2402.15043" target="_blank" | ||
| class="external-link button is-normal is-rounded is-dark"> | ||
| <span class="icon"> | ||
| <i class="ai ai-arxiv"></i> | ||
| </span> | ||
| <span>arXiv</span> | ||
| </a> --> | ||
| </span> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </section> | ||
|
|
||
|
|
||
| <section class="hero teaser"> | ||
| <div class="container is-max-desktop"> | ||
| <div class="hero-body"> | ||
| <h2 class="subtitle has-text-centered"> | ||
| We propose <b>KIEval</b>, an open-source framework, uses dynamic LLM dialogues to interactively evaluate LLMs' genuine understanding across domains, mitigating data contamination that inflates benchmarks scores without improving real-world effectiveness.</b> | ||
|
|
||
| <img src="./static/images/teaser.png" alt="teaser"> | ||
|
|
||
| <!-- Paper abstract --> | ||
| <section class="section hero is-light"> | ||
| <div class="container is-max-desktop"> | ||
| <div class="columns is-centered has-text-centered"> | ||
| <div class="column is-four-fifths"> | ||
| <h2 class="title is-3">Abstract</h2> | ||
| <div class="content has-text-justified"> | ||
| <p> | ||
| Automatic evaluation methods for large language models (LLMs) are hindered by data contamination, leading to inflated assessments of their effectiveness. Existing strategies, which aim to detect contaminated texts, focus on quantifying contamination status instead of accurately gauging model performance. In this paper, we introduce KIEval, a Knowledge-grounded Interactive Evaluation framework, which incorporates an LLM-powered "interactor" role for the first time to accomplish a dynamic contamination-resilient evaluation. Starting with a question in a conventional LLM benchmark involving domain-specific knowledge, KIEval utilizes dynamically generated, multi-round, and knowledge-focused dialogues to determine whether a model's response is merely a recall of benchmark answers or demonstrates a deep comprehension to apply knowledge in more complex conversations. Extensive experiments on seven leading LLMs across five datasets validate KIEval's effectiveness and generalization. We also reveal that data contamination brings no contribution or even negative effect to models' real-world applicability and understanding, and existing contamination detection methods for LLMs can only identify contamination in pre-training but not during supervised fine-tuning. | ||
| </p> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </section> | ||
| <!-- End paper abstract --> | ||
|
|
||
| <!--BibTex citation --> | ||
|
|
||
| <section class="section is-light" id="BibTeX"> | ||
| <div class="container is-max-desktop has-text-left content"> | ||
| <h2 class="title">BibTeX</h2> | ||
| <pre><code>@misc{yu2024kieval, | ||
| title={KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models}, | ||
| author={Zhuohao Yu and Chang Gao and Wenjin Yao and Yidong Wang and Wei Ye and Jindong Wang and Xing Xie and Yue Zhang and Shikun Zhang}, | ||
| year={2024}, | ||
| eprint={2402.15043}, | ||
| archivePrefix={arXiv}, | ||
| primaryClass={cs.CL} | ||
| } | ||
| </code></pre> | ||
| </div> | ||
| </section> | ||
| <!--End BibTex citation --> | ||
|
|
||
|
|
||
| <footer class="footer"> | ||
| <div class="container"> | ||
| <div class="columns is-centered"> | ||
| <div class="column is-8"> | ||
| <div class="content"> | ||
|
|
||
| <p> | ||
| This page was built using the <a href="https://github.com/eliahuhorwitz/Academic-project-page-template" target="_blank">Academic Project Page Template</a> which was adopted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a> project page. | ||
| You are free to borrow the of this website, we just ask that you link back to this page in the footer. <br> This website is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative | ||
| Commons Attribution-ShareAlike 4.0 International License</a>. | ||
| </p> | ||
|
|
||
| </div> | ||
| </div> | ||
| </div> | ||
| </div> | ||
| </footer> | ||
|
|
||
| <!-- Statcounter tracking code --> | ||
|
|
||
| <!-- You can add a tracker to track page visits by creating an account at statcounter.com --> | ||
|
|
||
| <!-- End of Statcounter Code --> | ||
|
|
||
| </body> | ||
| </html> |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.