Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support combining sequences that don't normalize #2616

Open
tig opened this issue May 9, 2023 · 0 comments · May be fixed by #3877
Open

Support combining sequences that don't normalize #2616

tig opened this issue May 9, 2023 · 0 comments · May be fixed by #3877
Labels
Milestone

Comments

@tig
Copy link
Collaborator

tig commented May 9, 2023

See this discussion for more: #2939

What's broken today (v2):

  • Combining Marks that don't normalize are not supported
    • /u0301 at column 0 renders ́ even though /u0301 is a zero-width codepoint.
    • a/u0301 works because ConsoleDriver normalizes it á
    • a/u301/u0301 doesn't work. ConsoleDriver normalizes it to á and ignores the 2nd /u0301.
    • TextField/TextView` strip out CMs on paste.
  • non-BMP characters
    • WindowsDriver
      • Not supported by Windows Terminal when using the WriteConsoleOutput (what WindowsDriver currently does). WindowsDriver currently outputs non-BMP characters as Rune.ReplacementChar
    • CursesDriver
      • Single-width non-BMP chars output as a space
      • Wide non-BMP characters are output as *
    • NetDriver
      • Single-width non-BMP chars correctly
      • Wide non-BMP characters chars output correctly

Note, TextView/TextField and other input views do not handle any of this stuff correctly (mostly).

Related:

@tig tig added the bug label May 9, 2023
@tig tig moved this to 🆕 New in Terminal.Gui V2 Beta Aug 7, 2023
BDisp added a commit to BDisp/Terminal.Gui that referenced this issue Oct 24, 2023
tig added a commit that referenced this issue Oct 29, 2023
…malize (#2932)

* Fixes #2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

---------

Co-authored-by: Tig <[email protected]>
tig added a commit that referenced this issue Jan 5, 2024
…uences (#3094)

* Fixes #2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

* Initial experiment

* Created ANSiDriver. Updated UI Catalog command line handling

* Fixed ForceDriver logic

* Fixed ForceDriver logic

* Updating P/Invoke

* Force16 colors WIP

* Fixed 16 colo mode

* Updated unit tests

* UI catalog tweak

* Added chinese scenario from bdisp

* Disabled AnsiDriver unit tests for now.

* Code cleanup

* Initial commit (fork from v2_fixes_2610_WT_VTS)

* Code cleanup

* Removed nativemethods.txt

* Removed not needed native stuff

* Code cleanup

* Ensures command line handler doesn't eat exceptions

---------

Co-authored-by: BDisp <[email protected]>
BDisp added a commit to BDisp/Terminal.Gui that referenced this issue Jan 5, 2024
…uences (gui-cs#3094)

* Fixes gui-cs#2616. Support combining sequences that don't normalize

* Decouples Application from ConsoleDriver in TestHelpers

* Updates driver tests to match new arch

* Start on making all driver tests test all drivers

* Improves handling if combining marks.

* Fix unit tests fails.

* Fix unit tests fails.

* Handling combining mask.

* Tying to fix this unit test that sometimes fail.

* Add support for combining mask on NetDriver.

* Enable CombiningMarks as List<Rune>.

* Prevents combining marks on invalid runes default and space.

* Formatting for CI tests.

* Fix non-normalized combining mark to add 1 to Col.

* Reformatting for retest the CI.

* Forces non-normalized CMs to be ignored.

* Initial experiment

* Created ANSiDriver. Updated UI Catalog command line handling

* Fixed ForceDriver logic

* Fixed ForceDriver logic

* Updating P/Invoke

* Force16 colors WIP

* Fixed 16 colo mode

* Updated unit tests

* UI catalog tweak

* Added chinese scenario from bdisp

* Disabled AnsiDriver unit tests for now.

* Code cleanup

* Initial commit (fork from v2_fixes_2610_WT_VTS)

* Code cleanup

* Removed nativemethods.txt

* Removed not needed native stuff

* Code cleanup

* Ensures command line handler doesn't eat exceptions

---------

Co-authored-by: BDisp <[email protected]>
@tig tig added this to the v2 Release milestone Jul 11, 2024
@tig tig mentioned this issue Aug 19, 2024
BDisp added a commit to BDisp/Terminal.Gui that referenced this issue Dec 3, 2024
@BDisp BDisp linked a pull request Dec 3, 2024 that will close this issue
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Status: 🆕 Not Triaged
Development

Successfully merging a pull request may close this issue.

1 participant