Thai Language Formatting Issue

[mahirocoko]

I am experiencing an issue with the Biome-Zed plugin where Thai language text is not being formatted correctly. Specifically, the word "หน้าหลัก" is being changed to "หนาหลก". This is causing readability issues for Thai users.

Steps to Reproduce

• Type "หน้าหลัก".
• After formatting, the text changes to "หนาหลก".

Expected Behavior
The text "หน้าหลัก" should remain as it is and not be altered to "หนาหลก".

Actual Behavior
The text "หน้าหลัก" is incorrectly formatted to "หนาหลก".

Environment
Biome-Zed Version: [0.143.7]
Operating System: [macOS with M1 chip]

My Setting

{
  "format_on_save": "on",
  "lsp": {
    "biome": {
      "settings": {
        "require_config_file": true
      }
    }
  },
  "formatter": {
    "external": {
      "command": "./node_modules/@biomejs/biome/bin/biome",
      "arguments": ["format", "--write", "--stdin-file-path", "{buffer_path}"]
    }
  },
  "code_actions_on_format": {
    "source.fixAll.biome": true,
    "source.organizeImports.biome": true
  }
}

349785754-be501ca7-b15c-480b-91b4-ae9719f4b9a4.mov

[nhedger]

We've had a similar report for the IntelliJ extension and suspect it may be related to output colorization on the CLI.

Would you mind passing --colors=off to your formatting command to see if it fixes the issue for you ?

[mahirocoko]

We've had a similar report for the IntelliJ extension and suspect it may be related to output colorization on the CLI.

Would you mind passing --colors=off to your formatting command to see if it fixes the issue for you ?

Thank you for the suggestion, but unfortunately, turning off colors with --colors=off did not resolve the issue.
According to the example video, did I do it correctly?

Screen.Recording.2567-07-18.at.11.39.37.mov

[nhedger]

Got access to a laptop and was able to reproduce your issue. --colors=off indeed doesn't seem to fix the issue, thanks for trying! We'll have to investigate what is happening.

Similarly to biomejs/biome-intellij#71, this seems to happen only when using STDIN, calling format on the file directly works as expected.

[nhedger]

Welp, it's actually the other way around.

// หน้าหลัก
const a = 1

Forcing colors

cat test.tsx | biome format --write --colors=force --stdin-file-path test.tsx
// หน้าหลัก
const a = 1;

Disabling colors

cat test.tsx| biome format --write --colors=off --stdin-file-path test.tsx
// หนาหลก
const a = 1;

Would you ming trying this too?

--colors=force

[mahirocoko]

cat test.tsx | biome format --write --colors=force --stdin-file-path test.tsx

Thank you for the suggestion. I tried forcing colors with --colors=force. However, neither approach resolved the issue.

When forcing colors:

�[0m'use client'

import type { FC } from 'react'

interface IComponentNameProps {}

const ComponentName: FC<IComponentNameProps> = () => {
  return <div>หน้า</div>
}

export default ComponentName
�[0m

It still produces the incorrect output. However, these options did help with resolving the issue related to the language.

[nhedger]

This is what I believe is happening:

1. When disabling colors using --colors=off or leaving it on auto, Biome disables colors by default when using stdin.
2. This leads Biome to assume that colors are not supported, which makes it go into the following block.

   biome/crates/biome_console/src/write/termcolor.rs

   Lines 167 to 182 in 3aeedd4

   	if cfg!(windows) || !self.writer.supports_color() {
   	let is_ascii = grapheme_is_ascii(grapheme);
   	if !is_ascii {
   	let replacement = unicode_to_ascii(grapheme.chars().nth(0).unwrap());
   	replacement.encode_utf8(&mut buffer);
   	if let Err(err) = self.writer.write_all(&buffer[..replacement.len_utf8()]) {
   	self.error = Err(err);
   	return Err(fmt::Error);
   	}
   	continue;
   	}
   	};

3. Because the character is not ASCII, we attempt to find a replacement character, but unicode_to_ascii only accepts a single char, and in this case we take the first char of the grapheme. since the น้ grapheme is composed of two chars this means that we lose ้, ending up with น.

I'll move this issue to the main repo as I don't believe it's specific to the extension