Man page should reveal more facts about --punctuation-wrap yes

[jidanni]

Man page says:

--punctuation-wrap Boolean (no if unset)
This option specifies if Tidy should line wrap after some Unicode or Chinese punctuation characters.

Observed effects on rendered HTML:

< 返程則免計程車。 下坡步行至「燥坑」公車站牌。 記得帶公車卡… When leaving no
---
> 返程則免計程車。 下坡步行至 「燥坑」公車站牌。 記得帶公車卡… When leaving no

So not only does --punctuation-wrap yes insert "after some Unicode or
Chinese punctuation characters" ... but it also inserts before

$ unicode 「
U+300C LEFT CORNER BRACKET
UTF-8: e3 80 8c UTF-16BE: 300c Decimal: 「 Octal: \030014
「
Category: Ps (Punctuation, Open); East Asian width: W (wide)
Unicode block: 3000..303F; CJK Symbols and Punctuation
Bidi: ON (Other Neutrals)

Character is mirrored

So please have the man page give the full story. (Seen in 5.6.0.)

[geoffmcl]

@jidanni thanks for the interesting docs issue... my first time experimenting with the option --punctuation-wrap yes... in 5.7.24...

There are lots of comments in the code about this, maybe starting with the table unicode4cat... to check if a utf-8 character, is a wrappable sort... and certainly 「 is one of them... along with some 246 others...

Yes, it seems in my brief tests, the docs should read something like ... should line wrap before/after some Unicode ... is that your suggestion?

And yes, inserting a newline, wrapping the text, before, or after, these so called punctuation characters, can cause a browser rendering change, in that this newline may be rendered as a space, where there was none before...

The src code comment here makes that point clear... and also documents the before special cases, while the balance of the table will be after...

So could consider an additional docs line like, say, Be aware, this option can cause a space, replacing any newline wrap character added, in browser rendering, where there was none previously.... or something...

If you, or others, have better, alternate, more, etc docs changes, then let's hear it... thanks...

And as always, a patch or PR implementing this doc change, very much appreciated... thanks...

[jidanni]

What in the world is the source code doing hardwiring the whole table in when it should be getting it from some standard library etc.

And yes it should say 'before and/or after'... but the user will scratch their head even more, so maybe a new function could be added:
$ tidy --cough-up-line-break-details
that would spit out lists of what characters it is going to break before, and after, etc.

[geoffmcl]

@jidanni thanks for the response...

... some standard library

What standard library is that?

You sort of half agree to some doc change... not sure...

Then launch into there should be a new option, --cough-up-line-break-details... phef! seems no answer...

May be there should/could such a need, desire, ... for such a new help option, but maybe that should/could be a new issue, hopefully opened by someone contributing code... willing to contribute to a solution... write code... present PR...

Meantime, have suggested, a little, beginning solution, to this issue... start to fix the docs...

As always, patch, PR, ... implementing this doc change, very much appreciated... thanks...

[jidanni]

https://developer.gnome.org/glib/stable/glib-Unicode-Manipulation.html Oops, I thought the g of glib was like the g of gcc. Anyway I'm sure there is some proper way of doing this instead of reinventing the wheel. OK just have the man page say it will "possibly add breaks before Unicode classes XX, YY, and after ZZ, AA". Mention: "If it will" depends on how many characters are on a line. But "what character classes it affects" should be explicitly listed.

[geoffmcl]

@jidanni thanks for the feedback...

Looking into the history of this special wrap support, this table of just 264 values, brought forward, or introduced?, when the new pprint.c modules was created, commit b92d7aa, Nov 2011, b92d7aab8#diff-4a4e354609aef4a160784a5caa18e868, and has not really been touched since then... some code cleanup...

Of course the docs should clearly mention, this is only if you have a line of characters greater than the configured wrap width, and you enable this option... dah..

In looking for classes, groups, ... on which codes it is ok to add newline, before or after, found nothing definitive yet... look forward to links...

Meantime can only look at the tidy table... extract the src, added some visual enhancements, thru a perl script... produced utf8-wrap2 ... with some loose grouping, at least for before/after...

In general, seems a sound attempt, at listing, viable code points... where a newline could be allowed... if wrap crossed... etc...

But this is first a question about improving the docs...

If you think some codepoint should be changed, added, deleted, then open a NEW issue for that... with reasons, samples, etc... thanks..

If you think there is a library tidy could use, then again that should be a NEW issue... maybe at least with proof of concept code changes... thanks...

A few suggetions for the docs change have been made...

Look forward to further feedback... particularly patches, PR, ... thanks...