Add new paper:

[wyzh0912]

Title

Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization

Published Date

2025-01-25

Source

arXiv

Head Name

Retrieval Head

Summary

• Innovation: The paper introduces RHIO, a framework designed to improve contextual faithfulness in retrieval-augmented LLMs by explicitly teaching them to distinguish between faithful and unfaithful outputs using control tokens and augmented unfaithful samples generated by masking retrieval heads.

• Tasks: The study involves augmenting unfaithful samples by masking retrieval heads, employing faithfulness-aware tuning with control tokens to teach LLMs to differentiate between faithful and unfaithful outputs, and using self-induced decoding to enhance faithfulness in long-form question answering tasks.

• Significant Result: RHIO significantly improves the faithfulness of LLMs in long-form question answering tasks, achieving average gains in faithfulness of 12.84% and 12.59% for 7B and 13B models, respectively, even outperforming the state-of-the-art GPT-4o on the GroundBench benchmark.