Skip to content

image and heatmap traces jump/flash on Firefox and Safari #4377

@jonmmease

Description

@jonmmease
Contributor

The new image trace seems to have a flashing/jumping behavior when zoomed in and panning on Firefox and Safari. (Firefox version 69.0, Safari version 13.0.3)

Example from https://plot.ly/python/imshow/

image_jump

This jumping does not happen on Chrome but I do see it on Firefox/Safari. Not sure if it's related, but in all three browsers, the tooltip also seems to jump to another location when the pan is completed.

Activity

antoinerg

antoinerg commented on Nov 21, 2019

@antoinerg
Contributor

@jonmmease I think the issue also occurs on heatmap (https://plot.ly/python/heatmap/). Could you rename the issue to reflect that?

The problem stems from reinjecting an image when pan operation is completed. I thought this pattern was safe since it's been used for a long time in heatmap. Maybe this issue only affects newer versions of Firefox 🤔 ?

changed the title [-]image trace jumps/flashes on firefox[/-] [+]image and heatmap traces jump/flash on Firefox and Safari[/+] on Nov 21, 2019
jonmmease

jonmmease commented on Nov 21, 2019

@jonmmease
ContributorAuthor

Could you rename the issue to reflect that?

Done, thanks!

etpinard

etpinard commented on Nov 28, 2019

@etpinard
Contributor

Hmm. I can't replicate in FF 70

jonmmease

jonmmease commented on Nov 30, 2019

@jonmmease
ContributorAuthor

Hmm, I am seeing it in FF 70 on Ubuntu. @antoinerg were you able to reproduce?

antoinerg

antoinerg commented on Dec 1, 2019

@antoinerg
Contributor

Hmm, I am seeing it in FF 70 on Ubuntu. @antoinerg were you able to reproduce?

I can reproduce it in FF 70.0.1 (64-bit) on Linux (NixOS distro).

firasm

firasm commented on Dec 10, 2019

@firasm

I can also reproduce in Safari 13.0.3 on macOS 10.15.1.

On the same system, the problem does not appear in Chrome Version 78.0.3904.108.

added this to the v1.53.0 milestone on Dec 12, 2019
etpinard

etpinard commented on Jan 7, 2020

@etpinard
Contributor

@antoinerg do you think there's an "easy" way to fix this bug?

Or do you think our canvas -> pixels -> img inside the _paper SVG is fundamentally flawed and FF does behaves as expected?

antoinerg

antoinerg commented on Jan 8, 2020

@antoinerg
Contributor

Or do you think our canvas -> pixels -> img inside the _paper SVG is fundamentally flawed and FF does behaves as expected?

It's hard to say which one behaves right and I suspect the specification may not be clear in this regard.

do you think there's an "easy" way to fix this bug?

Idea for an easy fix: have two images (A and B) and alternate between the two. Once you pan, image B content is updated and moved in front of image A (so no flash in principle). Then if you pan again, image A content is updated and moved in front and so on. Hopefully, this approach works.

If we only target a fix for FF: leverage image-rendering: pixelated, draw the image only once and things will fly 🐎.

alexcjohnson

alexcjohnson commented on Jan 8, 2020

@alexcjohnson
Collaborator

So we're reverting the drag, then redrawing with the new view... and it seems like even if we do this synchronously, some browsers will do two repaints (if that's it, there will be async edge cases in other browsers too). I wonder if we can just not revert the drag, and tweak the redraw process if necessary so it still works right. I think it's this call that's doing it:

function dragTail() {
// put the subplot viewboxes back to default (Because we're going to)
// be repositioning the data in the relayout. But DON'T call
// ticksAndAnnotations again - it's unnecessary and would overwrite `updates`
updateSubplots([0, 0, pw, ph]);

updateSubplots seems to modify some internal state as well as adjusting viewboxes, we still may need the internal state even if we don't change anything visible.

self-assigned this
on Jan 9, 2020
antoinerg

antoinerg commented on Jan 21, 2020

@antoinerg
Contributor

3 remaining items

antoinerg

antoinerg commented on Feb 18, 2020

@antoinerg
Contributor

@etpinard at the moment, when an axis has a scaleanchor attribute (the default for image), the calc step is called on every pan operation. For the planned improvement to the image trace type, this is somewhat undesirable. We'd like to produce the image in calc once and then simply move/scale it in the plot phase when a pan operation is completed. The goal is to save ourselves from the sometimes expensive operation of looping over all the pixels in view (and then some). Do you see potential roadblocks or flaws in trying to achieve what I just described? How would you go about skipping the calc phase when it's being triggered by a pan?

etpinard

etpinard commented on Feb 19, 2020

@etpinard
Contributor

Here's the relevant logic that makes zoom interactions go through the 'calc' pipeline:

// figure out if we need to recalculate axis constraints
var constraints = fullLayout._axisConstraintGroups || [];
for(axId in rangesAltered) {
for(i = 0; i < constraints.length; i++) {
var group = constraints[i];
if(group[axId]) {
// Always recalc if we're changing constrained ranges.
// Otherwise it's possible to violate the constraints by
// specifying arbitrary ranges for all axes in the group.
// this way some ranges may expand beyond what's specified,
// as they do at first draw, to satisfy the constraints.
flags.calc = true;
for(var groupAxId in group) {
if(!rangesAltered[groupAxId]) {
Axes.getFromId(gd, groupAxId)._constraintShrinkable = true;
}
}
}
}
}

By the looks of it, all the range "constraints" operations are taken care of in

exports.enforce = function enforce(gd) {
var fullLayout = gd._fullLayout;
var constraintGroups = fullLayout._axisConstraintGroups || [];
var i, j, axisID, ax, normScale, mode, factor;
for(i = 0; i < constraintGroups.length; i++) {
var group = constraintGroups[i];
var axisIDs = Object.keys(group);
var minScale = Infinity;
var maxScale = 0;
// mostly matchScale will be the same as minScale
// ie we expand axis ranges to encompass *everything*
// that's currently in any of their ranges, but during
// autorange of a subset of axes we will ignore other
// axes for this purpose.
var matchScale = Infinity;
var normScales = {};
var axes = {};
var hasAnyDomainConstraint = false;
// find the (normalized) scale of each axis in the group
for(j = 0; j < axisIDs.length; j++) {
axisID = axisIDs[j];
axes[axisID] = ax = fullLayout[id2name(axisID)];
if(ax._inputDomain) ax.domain = ax._inputDomain.slice();
else ax._inputDomain = ax.domain.slice();
if(!ax._inputRange) ax._inputRange = ax.range.slice();
// set axis scale here so we can use _m rather than
// having to calculate it from length and range
ax.setScale();
// abs: inverted scales still satisfy the constraint
normScales[axisID] = normScale = Math.abs(ax._m) / group[axisID];
minScale = Math.min(minScale, normScale);
if(ax.constrain === 'domain' || !ax._constraintShrinkable) {
matchScale = Math.min(matchScale, normScale);
}
// this has served its purpose, so remove it
delete ax._constraintShrinkable;
maxScale = Math.max(maxScale, normScale);
if(ax.constrain === 'domain') hasAnyDomainConstraint = true;
}
// Do we have a constraint mismatch? Give a small buffer for rounding errors
if(minScale > ALMOST_EQUAL * maxScale && !hasAnyDomainConstraint) continue;
// now increase any ranges we need to until all normalized scales are equal
for(j = 0; j < axisIDs.length; j++) {
axisID = axisIDs[j];
normScale = normScales[axisID];
ax = axes[axisID];
mode = ax.constrain;
// even if the scale didn't change, if we're shrinking domain
// we need to recalculate in case `constraintoward` changed
if(normScale !== matchScale || mode === 'domain') {
factor = normScale / matchScale;
if(mode === 'range') {
scaleZoom(ax, factor);
} else {
// mode === 'domain'
var inputDomain = ax._inputDomain;
var domainShrunk = (ax.domain[1] - ax.domain[0]) /
(inputDomain[1] - inputDomain[0]);
var rangeShrunk = (ax.r2l(ax.range[1]) - ax.r2l(ax.range[0])) /
(ax.r2l(ax._inputRange[1]) - ax.r2l(ax._inputRange[0]));
factor /= domainShrunk;
if(factor * rangeShrunk < 1) {
// we've asked to magnify the axis more than we can just by
// enlarging the domain - so we need to constrict range
ax.domain = ax._input.domain = inputDomain.slice();
scaleZoom(ax, factor);
continue;
}
if(rangeShrunk < 1) {
// the range has previously been constricted by ^^, but we've
// switched to the domain-constricted regime, so reset range
ax.range = ax._input.range = ax._inputRange.slice();
factor *= rangeShrunk;
}
if(ax.autorange) {
/*
* range & factor may need to change because range was
* calculated for the larger scaling, so some pixel
* paddings may get cut off when we reduce the domain.
*
* This is easier than the regular autorange calculation
* because we already know the scaling `m`, but we still
* need to cut out impossible constraints (like
* annotations with super-long arrows). That's what
* outerMin/Max are for - if the expansion was going to
* go beyond the original domain, it must be impossible
*/
var rl0 = ax.r2l(ax.range[0]);
var rl1 = ax.r2l(ax.range[1]);
var rangeCenter = (rl0 + rl1) / 2;
var rangeMin = rangeCenter;
var rangeMax = rangeCenter;
var halfRange = Math.abs(rl1 - rangeCenter);
// extra tiny bit for rounding errors, in case we actually
// *are* expanding to the full domain
var outerMin = rangeCenter - halfRange * factor * 1.0001;
var outerMax = rangeCenter + halfRange * factor * 1.0001;
var getPad = makePadFn(ax);
updateDomain(ax, factor);
var m = Math.abs(ax._m);
var extremes = concatExtremes(gd, ax);
var minArray = extremes.min;
var maxArray = extremes.max;
var newVal;
var k;
for(k = 0; k < minArray.length; k++) {
newVal = minArray[k].val - getPad(minArray[k]) / m;
if(newVal > outerMin && newVal < rangeMin) {
rangeMin = newVal;
}
}
for(k = 0; k < maxArray.length; k++) {
newVal = maxArray[k].val + getPad(maxArray[k]) / m;
if(newVal < outerMax && newVal > rangeMax) {
rangeMax = newVal;
}
}
var domainExpand = (rangeMax - rangeMin) / (2 * halfRange);
factor /= domainExpand;
rangeMin = ax.l2r(rangeMin);
rangeMax = ax.l2r(rangeMax);
ax.range = ax._input.range = (rl0 < rl1) ?
[rangeMin, rangeMax] : [rangeMax, rangeMin];
}
updateDomain(ax, factor);
}
}
}
}
};

which is called during

exports.doAutoRangeAndConstraints = function(gd) {
var fullLayout = gd._fullLayout;
var axList = Axes.list(gd, '', true);
var matchGroups = fullLayout._axisMatchGroups || [];
var axLookup = {};
var ax;
var axRng;
for(var i = 0; i < axList.length; i++) {
ax = axList[i];
cleanAxisConstraints(gd, ax);
doAutoRange(gd, ax);
axLookup[ax._id] = 1;
}
enforceAxisConstraints(gd);
groupLoop:
for(var j = 0; j < matchGroups.length; j++) {
var group = matchGroups[j];
var rng = null;
var id;
for(id in group) {
ax = Axes.getFromId(gd, id);
// skip over 'missing' axes which do not pass through doAutoRange
if(!axLookup[ax._id]) continue;
// if one axis has autorange false, we're done
if(ax.autorange === false) continue groupLoop;
axRng = Lib.simpleMap(ax.range, ax.r2l);
if(rng) {
if(rng[0] < rng[1]) {
rng[0] = Math.min(rng[0], axRng[0]);
rng[1] = Math.max(rng[1], axRng[1]);
} else {
rng[0] = Math.max(rng[0], axRng[0]);
rng[1] = Math.min(rng[1], axRng[1]);
}
} else {
rng = axRng;
}
}
for(id in group) {
ax = Axes.getFromId(gd, id);
ax.range = Lib.simpleMap(rng, ax.l2r);
ax._input.range = ax.range.slice();
ax.setScale();
}
}
};

from a few places in plot_api.js including in the "fast" 'axrange' edit pipeline:

seq.push(
clearSelect,
subroutines.doAutoRangeAndConstraints,
drawAxes,
subroutines.drawData,
subroutines.finalDraw
);


So perhaps zoom/pan interaction just work already under the fast 'axrange' edit pipeline? @antoinerg have you tried commenting the above block in _relayout to see in any tests fail?

We'll have to be extra careful though. I wouldn't want to trust 100% our test coverage for constraints.

But yeah, to answer your question, making zoom/pan on constrained axes not have to go through the 'calc' pipeline would be a huge win for image traces. Maybe 'axrange' works already, maybe we'll need a new flavour of that 'axrange' pipeline, but yeah we can do better than 'calc'!

etpinard

etpinard commented on Feb 19, 2020

@etpinard
Contributor

(Ping, I updated my comment above)

jackparmer

jackparmer commented on Sep 10, 2020

@jackparmer
Contributor

This issue has been tagged with NEEDS SPON$OR

A community PR for this feature would certainly be welcome, but our experience is deeper features like this are difficult to complete without the Plotly maintainers leading the effort.

Sponsorship range: $10k-$15k

What Sponsorship includes:

  • Completion of this feature to the Sponsor's satisfaction, in a manner coherent with the rest of the Plotly.js library and API
  • Tests for this feature
  • Long-term support (continued support of this feature in the latest version of Plotly.js)
  • Documentation at plotly.com/javascript
  • Possibility of integrating this feature with Plotly Graphing Libraries (Python, R, F#, Julia, MATLAB, etc)
  • Possibility of integrating this feature with Dash
  • Feature announcement on community.plotly.com with shout out to Sponsor (or can remain anonymous)
  • Gratification of advancing the world's most downloaded, interactive scientific graphing libraries (>50M downloads across supported languages)

Please include the link to this issue when contacting us to discuss.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3backlogbugsomething brokenperformancesomething is slow

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @nicolaskruchten@antoinerg@gvwilson@jackparmer@firasm

        Issue actions

          image and heatmap traces jump/flash on Firefox and Safari · Issue #4377 · plotly/plotly.js