(feat):Enhanced language detection

[sudoskys]

with https://github.com/aboSamoor/pycld2

checkout Mimino666/langdetect#51 (comment)

[sudoskys]

pycld aboSamoor/pycld2#60 cant build wheel

[sudoskys]

After evaluation, this work will no longer be continued.

[neutron-nerve]

Enhanced Language Detection Issue Report

Issue Summary

• Title: (feat): Enhanced language detection
• Created by: @sudoskys
• Created on: 2024-01-26
• Closed by: @sudoskys
• Closed on: 2024-07-06

Description

The issue proposed the integration of enhanced language detection capabilities using the PyCLD2 library, as referenced in the aboSamoor/pycld2 repository. This enhancement was suggested to improve language identification accuracy and speed for the LlmKira/fast-langdetect project.

Key Events & Discussion

1. Initial Proposal:

   • A proposal was made to integrate the PyCLD2 library into the project for enhanced language detection, inspired by a recommendation on the Mimino666/langdetect issue.

2. Build Challenges:

   • An issue related to building the PyCLD2 wheel was highlighted by @sudoskys in the following comment:

     pycld https://github.com/aboSamoor/pycld2/issues/60 cant build wheel
     

3. Evaluation Outcome:

   • After thorough evaluation and attempts to overcome the build challenges, the contributor @sudoskys concluded that the integration of PyCLD2 would not proceed further:

     After evaluation, this work will no longer be continued.
     

Final Outcome

Due to unresolved issues with building the PyCLD2 wheel, and subsequent evaluation determining that further progress was not feasible, the proposed enhancement for integrating PyCLD2 was formally closed.

Conclusion

The initiative to enhance language detection using PyCLD2 has been reviewed and ultimately concluded without implementation due to technical difficulties. The project will continue to seek other viable solutions for improving language identification.

Acknowledgment

We extend our gratitude to @sudoskys for the proposal and diligent evaluation of the potential enhancement.

---

This report consolidates the details of the now closed issue to provide a clear and concise summary for future reference.