Add new paper:

[wyzh0912]

Title

MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads

Published Date

2025-02-19

Source

arXiv

Head Name

Retrieval Head

Summary

• Innovation: The paper introduces MuDAF, a method that enhances the retrieval capabilities of attention heads in large language models by using contrastive learning to optimize attention distributions. This aims to improve the focus on relevant information in long-context tasks, particularly in MDQA.

• Tasks: The study involves identifying and improving specific attention heads responsible for information retrieval within long-context question answering tasks. It applies MuDAF to enhance attention heads through contrastive learning, making them more adept at focusing on relevant passages and reducing distractions from irrelevant content.

• Significant Result: MuDAF significantly improves the performance of large language models in long-context question answering tasks, surpassing baseline models and even achieving better results than GPT-4o on certain datasets. The method enhances the retrieval capabilities of attention heads, leading to more focused and accurate information retrieval.