fix(datasource/graphene) swap selected layer when activating merge tool

[chrisj]

to allow place line tool to function. Swap back when tool is unloaded. Also swapping back to the merge tool layer's legacy tool

[jbms]

While this won't necessarily cause significant problems, it would seem to be cleaner to instead find a way to allow the merge line placement to work without modifying the "selected layer".

[chrisj]

@jbms I can take a look at using the approach of overriding the "at:shift?+control+mousedown0" event. Maybe I can redirect it to the line tool.

[unidesigner]

I'm running into a perhaps related issue: if I enable the merge tool (or multicut tool) with m on https://spelunker.cave-explorer.org/ - and create a sharing state, the tool is not properly activated when I open the sharing link. The activated tool displayed previously as merge line at the top is not there anymore. It works when I activate an annotation tool from an annotation layer (e.g. annotate point). Just having "tool": "annotateMergeLine" in the graphene segmentation layer does not seem to be enough to properly register the tool. If I activate the segmentation layer with right click, the tool does not show up and I have to reactivate it again.

The loss of the tool can also be seen by clicking on the Edit JSON state button.

[chrisj]

@unidesigner
I think this is a separate issue but There is no state representation of an active toggleable tool.

#354
Jeremy wrote:

Another question is whether there should now be a way to encode in the JSON state which tool is activated.

I don't see a reason why it would be difficult to implement.

[unidesigner]

@chrisj
I tried to look into the neuroglancer code, but it's not straightforward to understand what is happening. I see the annotateMergeLine tool property in the graphene segmentation layer, so there seems to be information about the current active tool in the state, but the ui and keybindings are not restored. They are restored properly with the annotation layer tools. By properly restored I mean when refreshing the page e.g. the annotate point header is still shown.

Do you have any hints how I could start to address this?

[chrisj]

@unidesigner

The annotation tool types are considered legacy tools in the codebase. Those are part of the state, but the new tool has no state representation as being active.

The global tool binder only supports one active tool I believe so it might be simple. I'm not familiar with the local tool binder and if that needs to be considered.

[unidesigner]

In my use case, the user wants to quickly switch between split/merge operations in a graphene layer, and performing annotations operations (adding, removing lines etc.). So keeping the e.g. merge tool active in the segmentation layer would allow to toggle between the tools by activating either the segmentation or annotation layer. One would not be able to support this if there were only a single global active tool.

The question would be how a global active tool (like the merge tool, encoded in the global state) would then interact with the legacy annotation tools.

@jbms Would you be able to advice on how to move forward with this?

[jbms]

The add point/line/box/ellipsoid annotation tools are what is now called "legacy tools" in that there is at most a single active annotation tool per layer, and the layer needs to be "selected" in order to use the tool.

The new tools are ones to which you you can bind shift+letter in order to activate the tool. These are global key bindings that don't depend on the "selected layer".

Originally tools only remained active while holding down shift+letter, but Chris added the concept of a toggle/modal tool that remains active even after releasing shift+letter. Pressing shift+letter again deactivates the tool, and additionally if another non-modal tool is activated by pressing shift+letter, the "modal" tool becomes temporarily deactivated but is activated again once the non-modal tool is deactivated.

I'd like to migrate the "legacy" annotation tools to be "new tools" instead and eliminate the "legacy tool" concept entirely --- we just need to think through any usability challenges that would create.

In any case I don't think a mechanism to activate a modal tool via the JSON state would be difficult to problematic.

[unidesigner]

I think the approach to have global key bindings to select a tool that does not depend on a selected layer is useful. However, I see an issue when you have multiple annotation layer (e.g. one to annotate points, another to annotate lines) and you want to make sure that the tool you select creates annotations in the appropriate layer. This would not work without a selected layer concept. How would you deal with this?

I was thinking that perhaps the selection widget could also be used to show a list of available tools (e.g. all the tools that are available in the current workspace) and you can click the one you want to select. Annotation tools (and probably also other tools such as the split merge tools) would also have an associated layer in this overview, and could be serialized/deserialized in the JSON state.

The mechanism to switch between the tools as introduced by Chris makes a lot of sense to me for various proofreading workflows.

[chrisj]

@unidesigner the new tools are associated with a specific layer. I added "layer independent tool keybinds" in this PR: #464.

An issue I see when migrating annotations to the new format would be how could we keep ctrl+click being able to add point in annotation layer 1, line in layer 2, etc. One way would be that active annotation type is a property on the layer, and the generic annotation tool would just look up the type on the layer. You could also make a tool that had a defined annotation type and ignored the layer active type.

I think people will not want to lose ctrl+click to add annotations on the active layer. Ctrl could be bound to the generic annotation tool.

[jbms]

Right now the "selected layer" only applies to two things:

1. which layer is shown in the default layer side panel (but other layer side panels can be shown separately)
2. which layer the ctrl+click "annotate" command applies to

It seems to me that a "selected layer" concept is most natural if there is a set of common tools, e.g. T1, T2, and T3, that the user may want to apply to e.g. both layer L1 and layer L2. Then it would be convenient to have a binding for layer-independent tools T1, T2, and T3, and the user can switch the "selected layer" between L1 and L2. This is particularly convenient if the user will commonly want to perform more than one action on the same layer before switching to another layer.

However, I'm not sure how common this sort of workflow is. It seems to me that in most cases the user only wants to apply a given tool to a single layer, e.g. apply tool T1 to L1, T2 to L1, and T3 to L2.

As far as I understand, the layer-independent tool bindings that chris introduces in #464 are to address the issue of wanting to be able to create fixed tool bindings for e.g. the graphene actions, for which there would likely be just a single layer in the viewer at a time anyway. However, maybe graphene is a case where you want to have a common set of layer-independent tools and can then easily switch which layer they apply to.

I agree that being able to use ctrl+click to create an annotation, or a similar short binding, is important, and that having to use shift+letter+click or similar would be unfortunate. One possible solution is to allow e.g. the "annotate line" tool to be togglable/modal --- when it is activated, ctrl+click does the annotate line action for a given layer. So you could bind shift+X to "annotate line on layer L1", and then press shift+X once to activate "annotate line" on layer L1, and then use ctrl+click to place lines. Then shift+Y could activate "annotate point on layer L2".

One question is whether it would be important to allow more than one modal tool to be active at a time, and if so, how to avoid overlapping key bindings. It is common in drawing programs to support both a "primary" and "secondary" tool that use different mouse buttons. Potentially we could add that concept to neuroglancer, and allow tools to specify a separate input event binding map for when they are the primary vs secondary tool.

[fcollman]

To help ground this discussion, I might suggest one workflow that we have in mind is that users are using an annotation layer to move their attention from cell to cell, synapse to synapse and either do some annotation work, or note that they have completed some task at this point in space. Simultaneously with this, they might need to update a segmentation layer. As a more specific example, a workflow might produce a set of locations and associated segment IDs that might have an error. The user wants to iterate through the annotations across these locations, inspect the resulting segmentation and make edits to it if necessary. Perhaps they want to have another annotation layer where they can lay down points and commentary about the state the segmentation there, so they can collaborate with someone else on what they are seeing at each location.

I'm sure there are other workflows and use cases that others might have in mind. Stephan it might be useful if you outlined them here so we can make sure that a solution covers them.

[unidesigner]

@jbms I also think that the second workflow, with a given tool to only a single layer, would be the most common.

I'm not sure though how common the use case is where one wants to use graphene actions to multiple (graphene) layers, so a fixed tool binding could make sense. Maybe @chrisj could clarify.

I would also keep ctrl-click for annotation creation not to complicate things.

@fcollman The use case I want to support is very similar or the same to what you describe. A user wants to add/remove lines for synapse annotations in an annotation layer, but sometimes needs to fix a graphene cell segmentation layer by performing split/merge operations intermittently, and then switch back to synapse annotation mode.

In my setup, I also update the neuroglancer state (e.g. update the position) externally, which causes the merge tool to unregister inproperly (the div at the bottom still shows, but the tool is not active anymore and cannot be used). The user has then to always activate the merge tool again to make it function properly again. All I need at this point is basically a fix that the merge tool stays remains active on the graphene layer after selecting an annotation layer, a url/hash update and the selection back to the graphene layer (right-click on the layer).