Rich Text

[cooper-joe]

Rich text

Summary

Rich text functionality is needed. Several apps allow users to enter text in different contexts. Some examples:

• interpretations/comments in all analytics apps
• notes in Capture app
• dashboard items in Dashboards
• custom forms in Capture app

These and many more user inputs need to support rich text. Different contexts will require different rich features, so the user interface will need to be flexible.

Discussion points

There are several main discussion points:

• how to store/manipulate rich text? (html, markdown, etc.)
• how to expose rich text controls to the user? (Toolbar, inline special characters)
  • should we use a third-party library? (draft, quill, etc.)
• what rich text features will we include? (formatting, links, images, mentions, etc.)

What are we doing now?

Rich text is already available in Analytics interpretations. Should we use this as a starting point?

The user interface

The user interface could consist of two parts:

• toolbar style, always available buttons that create and adjust formatting when clicked
• inline markdown-style formatting/markup

The type of user interface to use will depend on the context. Dense, information rich UIs intended for tech-comfortable users will benefit from inline markup. A user with low tech literacy will benefit from Word-style formatting controls and WYSIWYG editing.

It is possible to combine these two interfaces and allow them to be used interchangeably via a toggle (as used in JIRA). Whether we do this, and how we do it, is a topic to discuss.

Next steps

To move this discussion forward, @dhis2/front-end please contribute experience, opinions and ideas for the above points. Thanks! :)

I have started work on some initial ideas for the user interface. This discussion will inform those designs, especially 'What rich text features will we include?'.

[cooper-joe]

Rich text functionality

Features required, based on my experience and previous discussion and feature requests, sorted into must-haves and nice-to-haves.

Formatting

Must-haves

• bold
• italic
• underline
• text color (pick from palette of x colors)
• bullet list (multi-level)
• URL

Nice to haves/Maybes

• font selection
• text size (small, regular, large, XL)
• text background color (pick from palette of x colors)
• number list (multi-level)
• image
• quote
• emoji
• code block/inline code
• predefined styles:
  • header level 1
  • header level 2
  • header level 3
  • body

DHIS2 specific

Must-haves

• user mentions (@a_dhis2_user)

Maybes

• analytical object link (e.g. search for a chart via @ and link it without providing the URL)

[HendrikThePendric]

Rich text is already available in Analytics interpretations. Should we use this as a starting point?

I think this would be a great idea, because the current rich text component in d2-ui:

• Is a markdown based editor, like Slack/ Github (this means it uses inline special characters)
• From what I could tell by a quick inspection, the formatting logic is completely decoupled from the UI, so it would work with ui-core and d2-ui components already

To me that approach makes a lot of sense.

[HendrikThePendric]

Also, I think it is important that in the end it is important that we store the formatted text in some sort of markdown format, not HTML (I remember seeing discussions about that in the past but can't provide a reference). And if we do so, I think that the following features would be a bit unconventional:

• text color (pick from palette of x colors) (MUST HAVE)
• font selection
• text size (small, regular, large, XL)
• text background color (pick from palette of x colors)

I guess anything is possible, and I am not an expert in Markdown syntax, but I think it would be convenient if we could do without the features above. @cooper-joe is text-color perhaps something we could do without as well?

[ghost]

Also, I think it is important that in the end it is important that we store the formatted text in some sort of markdown format, not HTML (I remember seeing discussions about that in the past but can't provide a reference). And if we do so, I think that the following features would be a bit unconventional:

* text color (pick from palette of x colors) (MUST HAVE)

* font selection

* text size (small, regular, large, XL)

* text background color (pick from palette of x colors)

I guess anything is possible, and I am not an expert in Markdown syntax, but I think it would be convenient if we could do without the features above. @cooper-joe is text-color perhaps something we could do without as well?

Yeah I've also never seen those features in markdown before. Markdown-based editors is what I've seen pretty much exclusively lately for rich text editing. I think I remember some xml-based ones in the past, but that was quite a while ago. It makes sense to me to stick to what the common variants of markdown allow.

[cooper-joe]

I guess anything is possible, and I am not an expert in Markdown syntax, but I think it would be convenient if we could do without the features above. @cooper-joe is text-color perhaps something we could do without as well?

Text-color and text-size are often requested features, especially for communicating with text on dashboards. This doesn't mean we have to include them, but I think we need good reasoning not to. The reasons you stated above are already convincing.

Markdown would be simpler to implement, store and use. It is limited, of course. I am concerned that choosing markdown would cut off any non-markdown functionality in the future. Many of our users will be used to text-editing via Word, so no 'text size' or 'text color' is a major omission. Using HTML to get access to these few features seems like overkill and opens up rich text editing to 'you-can-do-anything' editing.

What about extending markdown with a few of our own features? Has anyone got any experience with that? (For example, a custom tag like regular text ::size-L:: Large text ::size-L::)

[HendrikThePendric]

Apart from the benefits you mention, using a markdown format would mostly be beneficial from a security perspective.

I don't have any experience with creating custom markdown features, but it will be possible to implement technically. There are many markdown "flavours" and then we'd be creating our own. And by doing so, plus having a WYSIWYG UI, we would make things more familiar for users that are used to working in MS Office etc. However, at the same time, we might make things more complicated for people who are accustomed to working with markdown directly....

[ghost]

Also, it'd be more complicated for us. Which might lead to bugs, slower updates, etc. I'm not sure if there are existing solutions that allow customisation. I'll see if I can find anything.

[ghost]

So there's something like this: https://www.npmjs.com/package/markdown-it. It allows for extending the syntax, so if we want extra syntax this would an option. In my experience markdown parsers are fairly heavy though, that's a downside (lazy loading it would be nice). And the more we stick to the defaults, the easier it'll be to maintain, the more stable, etc. Does it make sense to start out simple for example, and add more if it seems absolutely necessary along the way?

[cooper-joe]

There are many markdown "flavours" and then we'd be creating our own. And by doing so, plus having a WYSIWYG UI, we would make things more familiar for users that are used to working in MS Office etc. However, at the same time, we might make things more complicated for people who are accustomed to working with markdown directly....

In my anecdotal experience, our users are far more comfortable with MS Office than markdown. Also, provided we started at the base markdown-flavor and provided a toggleable 'raw' non-wysiwyg editor, markdown preferring users shouldn't suffer.

Does it make sense to start out simple for example, and add more if it seems absolutely necessary along the way?

Yes, I agree that ordinarily that is the best approach. However, in this case we need understand what the hard requirements are I think. If we start simple (markdown) and try to add text-color at a later date, but it turns out to be too difficult, we would either need to drop text-color or start again with HTML. If text-color is a hard requirement the only option is starting over. So, it looks like to make a decision here we need to find out whether these non-markdown features are absolutely required. I'll ask some questions and see what I can find out.

[HendrikThePendric]

If text-color is a hard requirement the only option is starting over.

I don't think this is a likely outcome, since the lib that @ismay found is extendible/customisable.

On the other hand, if you would be able to assess whether font-size/color is a requirement upfront, that would be very helpful.

[varl]

Rich text is already available in Analytics interpretations. Should we use this as a starting point?

I think this would be a great idea, because the current rich text component in d2-ui:

* Is a markdown based editor, like Slack/ Github (this means it uses inline special characters)

* From what I could tell by a quick inspection, the formatting logic is completely decoupled from the UI, so it would work with ui-core and d2-ui components already

To me that approach makes a lot of sense.

I've ported the d2-ui-rich-text component to ui-widgets a while back: https://github.com/dhis2/ui-widgets/tree/port-rich-text.

The branch probably needs updating though.

Our implementation uses Markdown-It by the way: https://github.com/dhis2/d2-ui/blob/master/packages/rich-text/src/parser/MdParser.js#L1

We already deviate from Markdown in d2-ui-rich-text (slightly), so technically we have a DHIS2-flavored Markdown that we can expand on.

[varl]

Priorities

I disagree that text color should be on the "must have" list. Here's a revised proposal based on the implementation we have now as v1:

Must-haves (v2)

* bold
* italic
* underline
* bullet list (multi-level)
* number list (multi-level)
* headings (level 1-6)

Nice to haves (v3)

* image
* URL
* quote
* preformatted text

Maybe in the future

* font selection
* text size (small, regular, large, XL)
* text color (pick from palette of x colors)
* text background color (pick from palette of x colors)

I think that any font and color customization is out of scope for the foreseeable future, there are a lot of complexities lurking in here related to accessibility and other features that rely on stylesheets (e.g. Dark Mode, High Contrast Mode) to operate, so that is why I would put them in the "maybe later" pile.

A final note is that "emoji" support was considered a "must have" when the current rich text implementation was built, so that could go under "DHIS2 Must haves".

wrt/ Storing in HTML

During the June 2018 release we debated HTML vs. Markdown, and settled on moving away from storing HTML in the database and towards "plain-text storage".

One reason we did so is because we did not want to rely on dangerouslySetInnerHTML as it opened us up to XSS attacks[1].

We should use a sanitizer like https://github.com/cure53/DOMPurify in our rich text component in combination with dangerouslySetInnerHTML. However, I'd much rather sidestep that additional complexity everywhere though. I doubt that even if we re-open this discussion with backend tech leads they will want to go back to storing HTML in the database.

[varl]

So, it looks like to make a decision here we need to find out whether these non-markdown features are absolutely required. I'll ask some questions and see what I can find out.

We can support a hybrid model of HTML-in-Markdown which is commonly used around the Web, so even if we go for Markdown-ish formatting now we aren't locked out of implementing support for text color/size/background, etc.

(Given that we are allowed to store HTML in the database again, which in either case of HTML and Hybrid we would have to do.)

[cooper-joe]

Thanks for the clarfications @varl. Looks like the logical starting point is to build upon the custom Markdown-It markup we use today, following the priorities you outlined above. I'll work from this base point in the design specs.

The complexities of supporting color attributes suggests that this is indeed a maybe in the future feature. The component design will allow for these features, but I won't spec them in the first instance to avoid confusion.

Also, +1 to emoji support being a must-have. The current support in d2-ui-rich-text is a good starting point and I think any expansion can follow that pattern. I think the major difficulty here will be agreeing which emojis to support.

[cooper-joe]

I have another question about rich text functionality:

Looking at other libraries/frameworks, it seems the most common technique is to use contentEditable on an element and allow the editing to take place in there. As I understand it, that means we would not be able to use the TextArea ui-core component when using rich text features? Is there an obvious, proven concept/structure we should follow here?

This relates to the design stage I am currently working through. I'm facing with questions like:

• should a rich text toolbar be standalone, or can it be embedded inside a TextArea?
• would RichTextArea be a new component?
• could a rich text toolbar apply styles inside other components (e.g. a normal InputField?)

[varl]

If we have to render the result as the user types then no, we would not be able to use the TextArea. I strongly recommend a similar solution that e.g. GitHub has, with a "write" mode in plain text and a "preview" mode to see the rendered result.

Using contentEditable is a bad idea because of the complexities it introduces. Keep it simple.

[amcgee]

While I strongly agree with the KISS principle, I don't know that we should follow the GitHub model here. As @cooper-joe has pointed out, our users are familiar with MS Word and not Markdown. The audiences for Slack and GitHub are the opposite. I think asking our largely non-technical users to learn a markup language just to write rich text is wrong and will lead to a lot of frustration. If we wanted to confirm this the right way we could test the different functionalities with some of those users, but my hunch is that they would confirm the preference for an immersive RTE experience.

[ghost]

Wouldn't Viktors proposal still be close to that functionality in a way? You do still have the formatting controls at the top, etc. I can imagine that seeing the markup in the text would be novel to people used to the ms word editing experience, but would it be that jarring?

I can imagine that the technical simplicity of Viktor's proposed solution would contribute to a better user experience as well. Maybe not in providing the exact editing environment they're used to, but in stability and a more bug free experience. Also valuable.

[amcgee]

I think it could be quite jarring or confusing to our users, yes. We're making assumptions either way, so again the only way to really know would be to do a proper test with real users.

It's a trade-off - it definitely would be beneficial from a developer perspective to keep things simple, but I don't think we can assume that the user will recognize the benefit of that trade-off when they see what they might consider a serious regression - removal of the existing in-place rich text editing functionality (100% the same as the Slack editor debacle, just in the opposite direction :-D )

[ghost]

Yeah true. Yeah it would be nice have user feedback on this. Would that be possible?

[amcgee]

I don't know how much precedent there is, but it would be good to create one and it should be possible. CC @cooper-joe

[cooper-joe]

Yeah true. Yeah it would be nice have user feedback on this. Would that be possible?

I don't know how much precedent there is, but it would be good to create one and it should be possible. CC @cooper-joe

There have been several attempts at user testing in different ways. To summarise:

• Sending prototypes to directly the community results in little to no feedback
• In person testing/feedback gives most value, but is most time and resource consuming
• Sending prototypes to specific users (e.g. HISP members) results in good feedback, but the scope of feedback is necessarily narrow (because, for example, HISP members are expert DHIS2 users and are their abilities/understanding are not representative of an average data entry clerk.)
• testing locally with available users that have an approximate level of understanding as our target user. This is difficult to quantify, but has produced useful results.

This topic would likely benefit from a combination of all of the approaches. This would require realistic prototypes.

---

To address the point directly: I can confidently say that a low-tech-literate user will be confused by non-WYSIWYG text editing. Without training materials and in-context guidance, I do not think a user who is used to MS Word style editing will be comfortable seeing _*markup*_ like [this]() when they use the rich text features. This could result in these users simply not using rich text. While user testing would be useful to see the extent to which this is true, I think we can reasonably assume the above.

Depending on the trade-offs we want to make, a simple, non-WYSIWYG may be acceptable/our only choice. That depends on the decisions we reach regarding:

• is it acceptable that only high-tech-literate users use the rich text functionality?
• could we provide enough in-context guidance to help low-tech-literate users use rich text?
• are we open to using libraries to achieve a WYSIWYG solution (Trix, for example)

and, ultimately:

• how much time and resources can we invest in this component/functionality?

[amcgee]

Thanks @cooper-joe!

Depending on the trade-offs we want to make, a simple, non-WYSIWYG may be acceptable/our only choice.

I agree. (harping here) It's also important to keep in mind that non-WYSIWYG is actually a regression in user-land, since there is currently a usable WYSIWYG rich text editor.

Scratch that, current behavior shows markup. I was misremembering, sorry!

[varl]

how much time and resources can we invest in this component/functionality?

It's not just the upfront development cost either, it is also the long-term maintenance cost attached to it.

Using off-the-shelf solutions like Trix comes with a bundle size cost as well (around 500 kb non-gzipped) and requires a CSS loader. The latter can be worked around by porting the CSS to styled-jsx. That also means that we now have to manually port Trix CSS to styled-jsx when we update Trix. Now it has a higher maintenance cost.

We need to be acutely aware of and check our ambitions to our capacity. We are already stretching ourselves thin, and I've seen WYSIWYG editors maintained by an entire team, especially when e.g. considering RTL. We don't know how that is going to play out for us yet, so staying conservative in our technical choices gives us more flexibility to adapt in the future.

Finally, It will be easier for us to upgrade to a WYSIWYG mode than to downgrade from it.

[varl]

Going through JIRA and the CoP, I don't see any reported issues about the current implementation in the sense that the markup itself is posing any problems.

There are bug reports that the shortcuts (ctrl+b) etc. don't work right and that seems to be more important than markup being visible in the editor.

This feedback is valuable because it is unsolicited.

If there had been feedback that the rich-text functionality isn't working at all, then we could investigate if WYSIWYG would help those users.

Being posed the question "do you prefer the Word-like writing experience or the GitHub writing experience" is a question that uses anchoring to create a mental bias. It's like asking if someone prefers this:

Or this:

They both get you from A to B, but if you had the choice you'd probably want B, even if what you need is A (e.g. because you have steady access to petrol but not electricity).

[cooper-joe]

There are bug reports that the shortcuts (ctrl+b) etc. don't work right and that seems to be more important than markup being visible in the editor.

This feedback is valuable because it is unsolicited.

@varl do you have links to these tickets? The only ones I found were reported by a core dev (Jennifer, DHIS2-5819 DHIS2-5773)

Also worth discussing is this ticket requesting rich text support for dashboard items (one of the main use cases for this component). The initial request says that markdown is suitable. The request has been made by BAO employee, I'm unsure of their tech-literacy, but there is no feedback on the ticket saying markdown is not suitable. The ticket has 11 votes. Interestingly, following the comments it seems a solution was built by BAO that used a WYSIWYG editor in the end. I am going to try to find out more information about the requirements and how/why the WYSIWYG editor was used.

Being posed the question "do you prefer the Word-like writing experience or the GitHub writing experience" is a question that uses anchoring to create a mental bias.

True. A better measure, or question, could be: how do you want to add formatted text in DHIS2?

---

I strongly agree with the points @varl raises about the technical challenges building and maintaining a WYSIWYG editor will pose. Judging by the limited feedback we have, what may be more important than how it gets done is that it actually gets done. Discussing WYSIWYG vs. markdown is valid, but we need to be pragmatic.

An attempt at a conclusion:

To move forward, we could say:

• a rich text parsing component/module will be made available in ui-*
• a rich text-friendly textarea variation will be built, including:
  • a toolbar and shortcuts to support auto tag creation (like github)
  • preview mode
  • in-context help text for formatting

Once this is complete, tickets like DHIS2-4809 can be done. Users that need rich text have a way to achieve that. The job of design will be to make the functionality we can provide usable for as many users as possible.

From this point, if there are issues with markdown input that is completely preventing its usage, we can look into making a WYSIWYG component that can connect to the rich text editor. This may prove to be necessary, or unreachable, but at least the rich text functionality would still be available, just via other means.

[varl]

I am OK with moving forward with that conclusion, and prefer working iteratively towards a "full" solution if necessary.

[amcgee]

Being posed the question "do you prefer the Word-like writing experience or the GitHub writing experience" is a question that uses anchoring to create a mental bias.

Agreed, of course, but a lot could be learned by showing one of our non-technical users i.e. the GitHub comment editor for the first time and observing what they think, if/how they learn to use it, and whether they find it uncomfortable or frustrating.