Export plot

[hoclun-rigsep]

Hello.

Like many, I've been amazed by how quickly I can do what I want to do with Visidata. For instance I took the infant weight data from the CDC and very quickly plotted the various percentile tracks.

But what if I want to get the plot in some other format than terminal display? Is there an 'export to (svg/png/etc)' feature that I'm unaware of, or some sensible pipeline?

[frosencrantz]

I don't believe there are any plot exporters at this time. Though that sounds like a wonderful idea if you wanted to create a plugin.

For now you need to save your filtered data to a file and then use something like R or gnuplot to generate the graph you wanted. I would really like to see the ability to create more types of plots, and the ability to save to a format that could be used by a web browser.

Maybe someone else has a better suggestion.

[hoclun-rigsep]

Right. I'm not ruling out writing a plugin for this. Said plugin would output a vega-lite spec and/or a gnuplot script, with the data inline, so you could pipe it or save and tweak it. However:

1. I have no idea when I would be able to undertake this or how long it would take me
2. It's very possible I'm not the right person for the job.

Therefore my next post will be an attempt to inspire others.

[saulpw]

Hi @hoclun-rigsep, there is now an svg saver for graphs in VisiData Plus (just go to a VisiData graph and save as .svg). It's pretty simple and uses matplotlib to generate the svg.

I'm sure it could be improved in lots of ways, and/or provide a framework for similar plugins. Let the games begin!

[ghost]

Considering that VisiData is already Python based, I wonder if Altair would be a good starting point. (Depends on whether you want to keep the plugin dependency free)

[hoclun-rigsep]

@DAVIDWALES Agreed. I believe I would use Altair. There is a method to output the json of the spec.

[hoclun-rigsep]

My vision for this is a plugin that produces a Vega-Lite spec and/or a gnuplot script with the data inline, which can be piped to the appropriate utility perhaps after some customization. We're all terminal lovers here and both of those plotting programs are widely available/portable and good at what they do.

My quick inspiration story goes like this: I have a three-month-old daughter and I saw the doctor plotting her weight curve along with various percentile curves. I found the baby weight data here, https://www.cdc.gov/growthcharts/html_charts/wtageinf.htm and after something like 22 keystrokes in VisiData, I had exactly what I had seen in the doctor's office.

This is just amazing and everyone involved in achieving this result should be very proud. But it was a bit like going 1000 mph down a dead end. The decisions VisiData made regarding the plot were all great: axis labels, et cetera. But, what if I wanted to add my daughter's data and then publish the result? I'd be starting from scratch in gnuplot or Vega-Lite and it would take a lot longer to get the same plot.

Thoughts?

[hoclun-rigsep]

I find this exchange very motivating. I think I can do what you ask. Could you describe the shape of the hardcoded input data that my bit of code would operate on?

[saulpw]

It's probably just a few columns, one with the x-axis, one with categorical data to determine line style/color, and then the associated y-axis columns. If you take your example and share the 22 keystrokes (a clean .vd commandlog file would be preferred), and the corresponding vegalite/gnuplot code that generates an equivalent graph, I think that would be sufficient.

[hoclun-rigsep]

Okay, first thing's first. 13 keystrokes!
infant_weight_keystrokes.tsv.txt

[saulpw]

Nice job! Here's a 10-keystroke version :)

infant.vd.txt

[hoclun-rigsep]

Here are two vega-lite specs. In one we melt the data using visidata (so that gets us up to 12 or 13 keystrokes in visidata); in the other we use vega-lite's fold operator. The problem with the fold operator is, you need to know the names of the fields ahead of time. My next step is to see if that can be overcome by using Altair.

I also rendered the plot once as a line and once as points.

a.json.txt

b.json.txt

I started looking at the plugin docs and I guess this would basically be implemented as a saver? *.vega-lite.json?

[hoclun-rigsep]

@saulpw I'm having a lot of fun with this thing so: more gratitude to you and the other contributors and maintainers. Since you have made vdplus public I've based my work on that, the idea being to use Altair to output a valid vega-lite spec that you can then render with

npx -p canvas -p vega -p vega-lite vl2{vg,png,svg,pdf} unnamed_graph.json > a.{vg,png,svg,pdf}

[hoclun-rigsep]

Latest improvement: detects independent variables of type date and tells Vega-Lite that this is a "temporal" field rather than a quantitative one.

[saulpw]

Very cool, @hoclun-rigsep! Do you have this code available anywhere I could look at?

[tomer-ben-david]

How to use this? is this a plugin? do you have an example of how to use the vega lite with visidata?

[hoclun-rigsep]

Code is rough. Here's a sneak peak. I intend to run it against more cases in the sample data directory and see where sensible defaults can be provided.

One UI issue is, you have to type a.vegalite when you save the file, but the result is named a.vegalite.json because 1) it is json and 2) iirc, the vl2* commands expect it to be named this way.

Another problem that just occurred to me now that I should probably work out would be performance/OOM on large data sets.

import sys
import collections

from visidata import VisiData, Canvas, vd, Progress
from visidata._types import date

vd.option('vegalite_mark', 'point', 'point | line')
vd.option('vegalite_dependent_label', None, 'Label for y-axis')

@Canvas.api
def plot_vegalite(self):
    import altair
    nerrors = 0
    nplotted = 0

    self.reset()

    vd.status('loading data points')
    catcols = [c for c in self.xcols if not vd.isNumeric(c)]
    numcols = [c for c in self.xcols if vd.isNumeric(c)]
    dependent_label = vd.options.get("vegalite_dependent_label")
    if not dependent_label:
        if len(self.ycols) == 1:
            dependent_label = self.ycols[0].name
        else:
            dependent_label = ""
    independent_label = numcols[0].name
    independent_type = 'quantitative' if numcols[0].type != date else 'temporal'
    data = []
    for ycol in self.ycols[:]:
        for rownum, row in enumerate(Progress(self.sourceRows, 'plotting')):  # rows being plotted from source
            try:
                for c in catcols or [None]:
                    record = {
                        'x': numcols[0].getValue(row),
                        'y': ycol.getValue(row),
                        'label': c.getValue(row) if c else ycol.name,
                        }
                    data.append(record)

                nplotted += 1
            except Exception:
                nerrors += 1
                if vd.options.debug:
                    raise
    
    return altair.Chart(
        data=altair.Data(values=data),
        mark=vd.options.get('vegalite_mark') and 'line',
        encoding={
            'x': altair.X('x', type=independent_type,
                axis=altair.Axis(title=independent_label)),
            'y': altair.Y('y', type='quantitative',
                axis=altair.Axis(title=dependent_label)),
            'color': altair.Color('label', type='nominal'),
            }
    )

@VisiData.api
def save_vegalite(vd, p, *sheets):
    import altair
    output_format = vd.options.get('vegalite_output_fmt')
    if len(sheets) > 1:
        vd.warning("We're only going to handle the first sheet")
    for vs in [sheets[0]]: # when is len(sheets) going to be > 1 ?
        if not isinstance(vs, Canvas):
            vd.warning(f'{vs.name} not a Canvas')
            continue
        chart = sheets[0].plot_vegalite()
        chart.save(str(p) + '.json', 'json')

[hoclun-rigsep]

How to use this? is this a plugin? do you have an example of how to use the vega lite with visidata?

@tomer-ben-david Right now you can take the code quoted above, put it in .visidata/plugins/vd_vl.py, add import plugins.vd_vl to your .visidatarc, and then ctrl-s a plot in VisiData and save it as a.vegalite. Then run

npx -p canvas -p vega -p vega-lite vl2{vg,png,svg,pdf} a.vegalite.json > a.{vg,png,svg,pdf}

for output to {vega spec, PNG, SVG, PDF}.

[tomer-ben-david]

+1 for gnuplot integation.

[daviewales]

@hoclun-rigsep I just found this after working on my own version...
I made a plugin which lets you build a vega-lite spec from a regular VisiData Sheet.
You just press ` and enter the encoding channels for each column.
It automatically determines the vega-lite type based on the visidata column type.

See:
#1895