how do you sort by name?

[HeavySnowJakarta]

Describe the bug

This issue talks about things of how to sort by name (of files, tags, etc).

With English, it's just okay to sort by A-Z order. For languages like Russian I think sort by their names also works (although i don't know how it goes actually). The way to sort files in Chinese looks weird. Usually file managers of different systems sort them by the A-Z order of Pinyin, but on Spacedrive I can't tell how they're sorted. It makes no sence.

Also I wanna ask users who speak Japanese, Korean and other languages about the experience of sorting by file name.

Reproduction

No response

Expected behavior

No response

Platform and versions

Alpha v3.0.1 for windows-x86_64, but users of different platforms may concern

Stack trace

No response

Additional context

No response

[HeavySnowJakarta]

DO NOT TELL ME YOU SORT BY THE UNICODE POINT 🧐

[iLynxcat]

as of current we are simply using SQLite sorting according to @fogodev (haven't looked myself but I believe him lol). worth looking into for future releases! probably not soon given the difficulty of implementing an algorithm to reliably do it for all alphabets. there may be a library but its not a priority for implementation where we're at, but will likely be a must-have for 1.0. creating an issue for this, maybe an open-source contributor will get to it before we do! if not we are eyeballing this issue.

[Rsfinder]

there are many challenges with the current sort order
i have been collecting data for a long time, so I work with files in many different languages,
and Spacedrive is not yet sorted as accurately as Explorer sorting.

The challenge now is.
1. non-alphabetic characters are not sorted correctly
2. lowercase and uppercase alphabets have been separated
3. there are not enough types of sorting in Favorites and Tags
4. inaccurate sorting when there are many files

For example, windows explorer sorts in the following order

1.0 Symbols
1.1. sign (' / - / ゛ / etc)
1.2. space / tab
1.3 General symbols
2.0 Numerals
2.1 Arabic numerals
2.2 Round numbers
2.3 Roman numerals
2.4 Infinity
3.0 Alphabets
3.1 Latin alphabet
3.2 Greek alphabet
3.3 Cyrillic alphabet
3.4 English unit symbols
4.0 Pseudonyms
4.1 Hiragana
4.2 Katakana
4.3 Katakana unit symbols
5.0 General Kanji

for more information, please refer to the following URL...
(sorry I couldn't find a detailed site in English. It's in Japanese, so please use a translator.
https://qiita.com/TETSURO1999/items/a6e954bf1770ef8799e3

in addition to the alphabet, the following major characters are used in asian countries.

Japanese: Hiragana / Katakana / Kanji + Fullwidth / Halfwidth
Chinese: 2 types of Kanji... Traditional / Simplified Chinese
Korean: Hangul
Russian: Cyrillic
Other: Pictograms, special symbols

these characters are very difficult to understand from an english-speaking point of view.
but, if Spacedrive cannot support these characters, the number of Asian users will not increase.
as an Asian user, if you have any issues related to characters, i will be happy to help.

[HeavySnowJakarta]

@Rsfinder I may tell you something about CJK characters as I know.

Japanese: Hiragana / Katakana / Kanji + Fullwidth / Halfwidth
Chinese: 2 types of Kanji... Traditional / Simplified Chinese
Korean: Hangul

Generally speaking: Chinese character (in English) = Hanzi (in Mandarin Chinese) = Kanji (in Japanese) = Hanja (in Korean) = Hantu (in Vienamese although they may do not use this word) = 汉字 = 漢字. As Unicode characters they are just equal but they read different in different languages. As the most widely used logogram there are so many regions for Chinese characters to consider. At least the following things are necessary for now:

• Mainland China: We use simplified Chinese here. Hanzi will be resorted by the A-Z order of pinyin, so you have to get the romanization of the characters first.
• HK&Macau: People in Hong Kong and Macau use traditional Chinese, and they usually speak Cantonese instead of Mandarin, so sort by Pinyin may be not suitable for them. I asked one friend in Macau and he says they are sorted by the number of the strokes of that character. The same as how the olympic teams are sorted during the Beijing Olympics (2008&2022).
• Taiwan: People in Taiwan use traditional Chinese and speak Mandarin so perhaps you should not decide how to sort just according to the language used. I know nobody in Taiwan so actually I don't know whether they use Pinyin and how the characters are sorted. Maybe I have to pick one on the web lol
• Japan: Also I know nobody in Japan. As I know both Kanji and Pseudonyms are most likely to be sorted by the order of Gujuon.
• Korea: Northern (if they could know this toy lol) and southern Korean seldom use Hanja so for now it's not urgent to consider Hanja. Korean characters have to be sorted in the order of Hangul.
• Vietnam: They use Chinese characters even less than Korean people but the current Vietnamese characters will be enough to be a thing for you. I'm even not sure whether there is a Vietnamese translation for the app!
• Other language users: How could I know how Chinese characters are sorted in other languages? The most users just regard these things as gibberish! 蘸呼橘嘚酮峙们拟们郝啊！

[Rsfinder]

Thanks for the insight🙏
i’m Japanese so i’m not familiar with the actual situation of Chinese and Vietnamese so it is very helpful.

to add, at least in my Japanese Explorer

Hiragana（ひらがな） / Katakana（カタカナ） / Kanji（漢字） are all sorted in the order of
“aiueo~n（あいうえお～ん）”, and the on-yomi (Chinese reading) are used for Kanji.
※1 there are exceptions.
※2 Kanji have both Chinese-derived “on'yomi”(Chinese reading) and Japanese-derived “kun'yomi”(Japanese reading)

ex,

・あし a shi
・イカ i ka
・うみ u mi
・安心 an shin｜安:アン an (chinese) / やす yasu (japanese)
・椅子 i su　 ｜椅: i (chinese)

Japanese sorting may not be that complicated since
it can be divided into "symbols", "numbers", "alphabets", "hiragana" & "katakana", and "kanji".

and...there are two other types of Japanese: full-width and half-width,

全角:Full-width (Japanese size): Ａ １　／
半角:Half-width (normal alphabet): A 1 /

but if we can do this, at least Japanese sorting will be perfect.

・Alphabet:A～Z
・Numbers:1～∞
・Hiragana / Katakana / Kanji: “a～n”
・Do not use full-width and half-width characters as the basis for separating them

（i'm relying on a translator to write this sentence, looking at it objectively, Japanese is too difficult.
　the kanji are the most troublesome. it's no wonder you can't sort them correctly...😞）

or rather, before dealing with asian languages....

2. lowercase and uppercase alphabets have been separated
3. there are not enough types of sorting in Favorites and Tags
4. inaccurate sorting when there are many files

i hope these issues get resolved soon as a priority. 2 and 4 are very important.

[HeavySnowJakarta]

Today I've checked if there is available libraries to do the related work. The key is icu (for rust it can be a crate) which provides Collator objects to do operations.

[HeavySnowJakarta]

So @iLynxcat @fogodev here's all to do:

• find out where the sorting of sqlite is used
• import icu::collator
• use Collator::compare to replace how it sorts

[fogodev]

We already did some investigation in this matter. The main issue is that we need to interact with SQLite on a more direct manner to do it, but we're currently many abstraction layers above it, (Prisma, Quaint, Rusqlite, etc). We haven't forgotten this issue, but we're currently busy on other issues, mainly related to sync.

Android for example ships a version of SQLite already with the necessary collation mentioned here http://www.sqlite.org/faq.html#q18
Our current approach will be trying to link with a libsqlite with this extension enable, as the default one has it disabled. But we still need to dig further in this direction, because as I said, we don't depend on sqlite directly.

We can't do it in the Rust side, because we list files by paginating them directly from sqlite. To sort in Rust we would need to load literally your entire file_paths table in memory for the desired location, sort in Rust, discard everything that will not be included in the current pagination and return the results. For each pagination in the UI. Which is a big no-go performance wise. Specially because one of our main audiences are potentially big data hoarders with many TB of files. (I only tested with Jamie on TB scale, managed to index some TBs of media files in a couple of minutes, even in dev mode, with my current implementation of parallel indexer and file_identifier).