Don't re-layout score after every key stroke when typing fast

[julenComposer]

Your idea

Dear devs,
I propose a feature for a smoother workflow, by setting a stand-by time (200 milliseconds maybe) under which the UI does not update until the user stops typing. If possible (and reasonable), I think this would allow a smoother worflow, especially when dealing with large projects/slow machines.

Problem to be solved

While working on a very large score (550+ bars, 20 instruments), the UI update went slower and slower overtime. This is perfectly normal for any software, but since I am quite experienced with the software by now, many times I don't really need the score to be updated right on the spot when I add new elements to the score.
Maybe by adding a feature to enable freezing of the UI update for some fraction of seconds would help performance and reduce lags and software-not-respondings.

Prior art

No response

Additional context

No response

[cbjeukendrup]

I do think that the UI should update immediately when you type, as in that the letters appear and the cursor moves forward; otherwise it feels like it's lagging. But indeed, it might be an idea to re-layout the score only after a certain delay instead of after every character.

But if the re-layout process can be optimised so much that it doesn't introduce freezing, that's even better of course :)

[julenComposer]

I do think that the UI should update immediately when you type, as in that the letters appear and the cursor moves forward; otherwise it feels like it's lagging. But indeed, it's a good idea to re-layout the score only after a certain delay instead of after every character.

Indeed, thank you Casper, that I think would be a more reasonable option, thank you.

Right now it seems to me at least - I don't have any experitise to understand the code - that the graphical interface tries to render each new element you add to the score immediately after the input. The system is implemented very very well, but as my last project grew bigger it eventually took at least 1.5x the time (I didn't measure it, but I would really say at least this ratio) to compose, because of the graphical rendering.

[MarcSabatella]

I would assume this might apply to note input as well as text. FWIW, I did experiment with this sort of thing back during 3.0 development when we were working on the partial relayout feature (only laying out what changed) and it wasn’t very good yet. Also a version where the main score updated but parts did not. It had potential but I never followed through as eventually the partial relayout feature seemed to be working well enough.

[irwir]

Long ago, when computers were slower and mostly character-oriented, there was a text editor that could combine several user actions and display the final result only.
Something like this: cursor is at the beginning of a file (BOF); for a quick sequence - "go to EOF", "go to BOF", two times "down" - no screen redraws, only cursor jumps to the beginning of the third line.

Possibly, if next character or command arrived while screen update was prepared or performed, next updates would be delayed until fast input sequence ends and there is a pause.

[eiffel31]

General comment, application independent: in many cases, the drawing time might be higher than user new input time. This can come from slow hardware, fast user, heavy loaded system, complex rendering shape...

From my experience, a very efficient way to handle this is:

• on user input event
  • push user modification request in the modification queue
  • set an alarm with 0 ms delay for the main loop to handle update
• in the update function
  • consider all the modifications in the queue
  • make a single draw update

With such principle, the interface is as responsive as it can be. When the user make some input, then the update process is started without delay. If the user is making input quicker than the system can manage, then the system will merge some inputs and the display delay will not be more than the refresh time.

Maybe some similar behavior is already implemented for the real-time play rendering and can be applied to user input. Otherwise, the same principle can be applied for playing: the position change is just an event pushing a pending redraw request.

[cbjeukendrup]

The drawing is indeed handled that way by Qt. The source of latency in the case of MuseScore is the layout calculations, but indeed perhaps a similar mechanism could be applied here. Perhaps we should do these calculations in a background thread, so that the UI thread is not blocked, so that the user can continue triggering events while the calculations for the previous events are being finished. That's difficult to do though, because when a user action modifies the score, that might also influence which next actions are available. So you'd either need to wait for the calculations of the first action to complete before you can determine whether the next action should be available in the UI, or you'd need to predict that.

[eiffel31]

I guess this can be split in 2 parts:

• when user input modifies the data, the structure has to reflect the change and next available actions are defined from the last state
• the structure result in a layout computing and display. This task is usually time consuming

The first part (handling users inputs) should be computable without layout rendering, hence user inputs are managed quickly while the rendering loop updates the screen at a lower refresh rate, when the layout is ready.

[irwir]

That's difficult to do though, because when a user action modifies the score, that might also influence which next actions are available.

Does it make a difference?
Currently if user types ahead non-available or invalid command, it would be ignored or not executed.
In any case, input should be processed sequentially.

• the structure result in a layout computing and display.

For best results layout computing and actual drawing should be separated.
This way it would be possible to drop partially calculated layout if additional input arrived.
If necessary, drawing could be made on an off-screen canvas, then prepared image to be blitted to screen.

[eiffel31]

This way it would be possible to drop partially calculated layout if additional input arrived.

This is the wrong way to do. Each time you start computing a layout, you have to run it to the end and update the display accordingly.
If you drop in progress computing each time a new input arrives, then in a situation where the computing time is longer than the user input delay to the next input, you end up with a frozen interface until the user stop. This is very annoying because on such interface you make a bunch of inputs, then stop to let the program refresh, check the result, and start again.
If you fully process and update the display, then your application display is updated at the higher rate that the system can do, you always have the best available state, that is the state that was available when the compute step started. From the user point of view, the program is processing inputs in real time with a constant display update delay.

Just a remark: you have exactly this behavior when you are playing a pipe organ in some big church where the keyboard is away from the pipes. Even at very moderate speed, each time you press a key, you hear the note from the previous key.

[irwir]

This is the wrong way to do. Each time you start computing a layout, you have to run it to the end and update the display accordingly.

This describes the current program's behaviuour; and it was found unsatisfactory.
There is no gain in seeing an already invalidated screen. Much less, to wait until a number of invalidated screens was being drawn.

If you fully process and update the display, then your application display is updated at the higher rate that the system can do, you always have the best available state, that is the state that was available when the compute step started. From the user point of view, the program is processing inputs in real time with a constant display update delay.

The problem is that all necessary computations could not be performed at a rate required for real time display.
When you know that your typing is faster than drawing could be done - this is essentially non-interactive process; and it should be fine to show only the final result.