KTO refactoring

[albertvillanova]

Systematically refactor KTOTrainer to align with modern trainer architecture patterns, improving maintainability, reducing complexity, and adding missing features.

Problem

KTOTrainer currently has complex code with several architectural issues.

Refactoring Plan

1: Foundation & Infrastructure

Establish modern patterns without breaking functionality

• Modernize Model Initialization: Refactor KTO [1/N]: Modernize model initialization #4783
  • Use create_model_from_path() helper
  • Simplify dtype and device_map handling
  • Clean up ref_model initialization
• Improve Config Validation: Refactor KTO [2/N]: Improve config validation in KTOConfig #4787

2: Dataset Processing Modernization

Adopt standard trainers' clean preprocessing patterns

• Extract _prepare_dataset method: Refactor KTO [3/N]: Extract dataset processing to _prepare_dataset method #4788

3: Reference Model Handling

Simplify ref model management

4: Loss Computation & Forward Pass

Reduce duplication in loss computation

5: Metrics & Logging

Match standard metrics system

6: Evaluation & Generation

Modern evaluation features

7: Documentation

Complete, clear documentation

Phase 8: Advanced Features

Feature parity with standard trainers

Phase 9: Testing

Production ready

[albertvillanova]

After internal discussion with @qgallouedec, we are coordinating to refactor both DPO and KTO. See related:

• Refactor DPO #3906

WIP: Features to Remove

First, remove some features. Benefits:

• Less code to refactor
• Simpler architecture
• Easier DPO alignment
• Clearer core functionality
• Faster refactoring

A. Remove Encoder-Decoder Support

PR:

• Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support #4792

Why Remove:

• Adds ~200-300 lines of conditional logic
• Complicates every dataset processing step
• Rarely used (most preference learning is causal LM)
• Different tokenization, label handling, model kwargs
• Testing burden (2x test coverage needed)

Already broken, minimal user impact.

B. Separate Prompt/Completion Truncation

PR:

• Refactor KTO coordinated with DPO [b/N]: Simplify truncation logic #4808

Why Remove:

• Adds ~80 lines of complex truncation logic
• Confusing UX (users don't know which to set)
• Most users just want max_length
• truncation_mode is redundant with tokenizer settings

Replace with simple max_length only.

C. Remove ref_model_init_kwargs

PR:

• Refactor KTO coordinated with DPO [c/N]: Remove ref_model_init_kwargs #4837

Why?

• Simplify model instantiation

D. Remove base_model_attribute_name

PR:

• Refactor KTO coordinated with DPO [d/N]: Remove base_model_attribute_name #4862

Why?

• Simplify base model retrieval logic

[albertvillanova]

I am resuming this task: Align KTO with DPO.