report: Add notebook mode.

[daavoo]

Renders existing HTML report inside an IFrame and updates it on each next_step.

Closes #309

---

Grabacion.de.pantalla.2023-02-07.a.las.11.58.34.mp4

[daavoo]

I need to add tests and manually try the different frameworks but I think it is ready to play around @dberenbaum

It kind of highlights the need for iterative/dvc-render#21
Also unsure about the values to hardcode the size of the IFrame

[shcheklein]

Added a recording to P.R. description.

🙏

Q: do we want this to be enabled by default? Feels a bit too intrusive (and it'll overlap with VS Code, Studio monitoring, etc). I think it would be great to log options when you run an experiment, .e.g:

If you want to see report you can do one this:

- embed into Notebooks ...
- VS Code ...
- Studio ....

wdyt?

[dberenbaum]

Thanks @daavoo! Seems to work for me on other repos. Here's a more realistic demo (using https://github.com/iterative/lstm_seq2seq with tweaked params to run faster) to see how it interacts with other notebook outputs:

Screen.Recording.2023-01-31.at.12.48.53.PM.mov

At the end, I saw some repeated "blinking." Could make_report be called many times at the end for some reason?

Screen.Recording.2023-01-31.at.12.50.44.PM.mov

[daavoo]

to see how it interacts with other notebook outputs:

um. looks like the instantiation of the callback (and thus Live object) is being delayed there.
That is unfortunate, as the placeholder IPython.display doesn't do the trick until Live gets instantiated.

To verify, could you try manually instantiating Live object outside and passing it to the callback?

[daavoo]

At the end, I saw some repeated "blinking." Could make_report be called many times at the end for some reason?

Will take a look using the repo

[dberenbaum]

Q: do we want this to be enabled by default? Feels a bit too intrusive (and it'll overlap with VS Code, Studio monitoring, etc). I think it would be great to log options when you run an experiment, .e.g:

🤔 @shcheklein Can you explain more what you would expect all the options to say? Could you turn the message off? Is it specific to this notebook mode, or should it apply to any mode? What do you see as the default onboarding workflow for both VS Code and non-VS Code users to see live metrics?

[daavoo]

Q: do we want this to be enabled by default? Feels a bit too intrusive (and it'll overlap with VS Code, Studio monitoring, etc).

This is enabled in auto mode (no explicit report value was provided) and it only activates if inside notebook.

For those overlaps, we could choose to not use the notebook display if there are signals that we are overlapping.

For Studio monitoring, it should be doable because DVCLive has the info available.
For VSCode I am not sure if there is an easy way for DVCLive to detect that the DVC extension is enabled

[dberenbaum]

@daavoo Can we clean up the table? If we want to keep any kind of HTML output, it seems like this would go a long way towards making it feel more polished.

[dberenbaum]

um. looks like the instantiation of the callback (and thus Live object) is being delayed there. That is unfortunate, as the placeholder IPython.display doesn't do the trick until Live gets instantiated.

It's delayed because it is set up to only log every 50 steps.

To verify, could you try manually instantiating Live object outside and passing it to the callback?

Sure. It does generate the placeholder immediately. Not sure if that looks any better?

Screen.Recording.2023-01-31.at.1.47.26.PM.mov

[daavoo]

It's delayed because it is set up to only log every 50 steps.

I mean that in some frameworks the Live object is instantiated when the callback is instantiated but lightning uses that experiment property that appears to be delayed until needed.
I think we could change this in our logger, though .

Sure. It does generate the placeholder immediately. Not sure if that looks any better?

Yeah I know it doesn't matter right now 😅

However, the point is that the placeholder could be actually a not so ugly HTML of equal size to the future IFrame so the space is clearly reserved in the output, and it doesn't interfere with progressbars or whatever output the framework might generate

[shcheklein]

🤔 @shcheklein Can you explain more what you would expect all the options to say?

List of things you can do to see actual live updates:

• If you are in nobebook ... adding an option to live object (e.g. "render=True")...
• use Studio (a link to docs)
• install VS Code extension (link to the marketplace)

Could you turn the message off?

Yes.

Is it specific to this notebook mode, or should it apply to any mode?

Any mode

What do you see as the default onboarding workflow for both VS Code and non-VS Code users to see live metrics?

Same as today + a message that would help navigating.

---

To me rendering an HTML like this inside notebook by default feels a bit too intrusive. It overlaps potentially with VS Code in some cases (ppl can get confused). Even if we go that path we should at least also show how to disable this in the HTML itself.

[dberenbaum]

At the end, I saw some repeated "blinking." Could make_report be called many times at the end for some reason?

Looks like it's specific to lightning -- it calculates the validation metrics for each step at the end of the epoch instead of during each step.

[daavoo]

To me rendering an HTML like this inside notebook by default feels a bit too intrusive.

Is your concern about how it is implemented (IFrame), about the space that it takes, or both?

It overlaps potentially with VS Code in some cases (ppl can get confused).

If I understand correctly, the scenario overlapping would be someone with the VSCode editor and the DVC extension installed, running the notebook inside a Git+DVC Repo (dvc init has been run), right?

I might be wrong but I see that scenario as far less likely than any others like Google Colab, Kaggle Notebooks, Jupyterlab servers, or just people running the notebook without meeting the above conditions (running local notebooks outside a Git Repo is also common)

I think for the overlapping scenario, the best would be to figure out how to get a signal that the VSCode and DVC extension are up and running (dvc extension could expose env var maybe?).

[shcheklein]

Is your concern about how it is implemented (IFrame), about the space that it takes, or both?

Neither :).

It just doesn't feel right (by default). Fels a bit too much of a side effect from running the code by default. Are there any other examples that come to your mind that would by default do similar things?

E.g. W&B could have potentially embedded an IFrame into the notebook by default as well.

My expectation when I run my code is to see the regular output that is usually expected and produced by the framework itself. Our additional output should be interfering with it much I think.

[dberenbaum]

E.g. W&B could have potentially embedded an IFrame into the notebook by default as well.

For context, here's what W&B does (note that I can't get the example repo to work): https://docs.wandb.ai/guides/track/jupyter

Anyway, let's discuss auto behavior as a follow-up and not block the rest of the PR? We can merge without making it part of auto since that will be the less intrusive option while we discuss it.

[shcheklein]

So, for W&B you need to:

... start a new cell with %%wandb to see live graphs in the notebook. 

Anyway, let's discuss auto behavior as a follow-up and not block the rest of the PR? We can merge without making it part of auto since that will be the less intrusive option while we discuss it.

sounds good to me!

[daavoo]

Updated the P.R.

@shcheklein Behavior is only enabled if report=notebook. Let's discuss auto and messaging about options in a follow-up

@dberenbaum Updated the init placeholder to take the same space as the subsequent IFrame so the space gets "reserved" on init and it doesn't mess up with existing logs because of the resizing that was happening before.

Let's discuss improvements to appearance as a follow-up in iterative/dvc-render#21 unless we consider it a blocker to merge this.