[Bug]: Issues with Unicode characters in UTF-8 text content

[mqudsi]

Is there an existing issue for the same bug?

• I have checked the existing issues.

RAGFlow workspace code commit ID

N/A

RAGFlow image version

5fb9136

Other environment information

Actual behavior

After adding a UTF-8 document to the knowledgebase that contains Unicode symbols, I've noticed that ragflow always generates text fragments that corrupt the Unicode symbols (rendering them as latin-1, apparently).

Expected behavior

Steps to reproduce

Upload a document as a .txt file saved as UTF-8 containing the following:

> “sample text,”

Additional information

No response

[KevinHuSh]

What's LLM do you choose?

[mqudsi]

ChatGPT 4o

[mqudsi]

I don't think it's an issue with the llm integration; here is the same problem displayed in the vector search results on the "Search" page rather than the "Chat" page:

[mqudsi]

It seems you are converting to Windows-1252 (a legacy encoding) at some point and then emitting it as if it were UTF-8, because I can correct it with iconv: