Skip to content

Better saving of chrf3 for quality estimation #888

@mshannon-sil

Description

@mshannon-sil

We should save the projected chrf3 scores alongside the usability scores in the verse, chapter, and book usability files. Currently only the verse scores are being saved, and in their own separate file which makes it more difficult to see them side by side with usability.

As part of this, we should consider the best way to obtain projected chrf3 at a chapter and book level. Currently, the verse confidences are used to get a projected chrf3 and then usability at the verse level. And then the verse level usabilities are averaged to get chapter and book usabilities. However, we already have chapter and book confidences being saved during the test step (calculated as the geometric mean of the verse confidences), so a possibly better alternative would be to project chrf3 and usability for those chapter and book confidences directly. I'd like to do some comparison to see how different the resulting usabilities would be using this method compared to how it works right now.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Status

🏗 In progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions