Is there a way to run certain operations across identically formatted files (either on-open or after-the-fact)?

[i-jey]

Say I have a set of static files, data1.csv, data2.csv, etc., all of which adhere to the same format (i.e have the same header):

• img_id: float,
• timestamp: float,
• name: str
• etc

I have two questions:

1. If I open all these files up in visidata (maybe with some bash magic: find . -type f -name "data*.csv" -exec vd {} +), is there a way to apply operations across all sheets at the same time (e.g setting column types for example).

2. Alternatively, is there a way to define a set of common operations on a set of known files (i.e any file which has the name data*.csv, will automatically open in visidata with certain presets). For example:

• Set the img_id column to int, the timestamp to float, etc.
• Add certain aggregators
• etc.

The most similar discussions seem to be

• Feature request: Allow defining automatic actions when opening a sheet #1681 (comment), however that one seemed too general (i.e setting common operations across filetypes automatically)
• Best way to create, re-use and share column transformation functions? #1246, but that discussion seems to have gone stale in Dec 2021

If I've missed an obvious discussion, apologies! Please let me know and I'll close this one. Cheers!

[saulpw]

Hi @i-jey, good question! I'd say the easiest way would be to make a custom type in your .visidatarc:

@VisiData.api
def open_jey(vd, path):
    vs = CsvSheet(path.name, source=path)
    vs.options.whatever = True    
    return vs

# to use a particular set of columns, make a custom sheet (substitute JeySheet for CsvSheet in the above)
class JeySheet(CsvSheet):
    columns = [
        ItemColumn('timestamp', type=float, aggstr='min max'),
        ItemColumn('name', type=str),
    ]

and then you would use vd -f jey *.csv.

[i-jey]

That's sweet, thank you for the speedy reply! (and more generally, for this awesome tool).

Cheers!

[i-jey]

@saulpw As a quick follow-up, unfortunately the above didn't quite work for me. When I load a sheet with the above in my .visidatarc, for a fraction of a second I see the specified header (with the expected types), but then it appears to be 'overwritten' by the default header (with everything defaulting to string).

I took a look through some of the source code but it wasn't apparent to me when/why this happens - maybe because of the @asyncthread in SequenceSheet's reload() call? I've tried on both vd2.12dev and v2.11.1

Some additional mucking about that may/may not help debugging efforts:

• Commenting out self.setCols() in SequenceSheet, (sheets.py, line 904) causes the expected headers (and their types) to show up, but none of the rows are loaded

[saulpw]

@i-jey, ah, yeah, the SequenceSheet (which is what CsvSheet inherits from) replaces the columns entirely. We could make it smarter, to preserve attributes of columns across a reload, or maybe it would be easier to add an option that does the same (like reload_preserve_cols or something).

[saulpw]

As of aba4efc (per #1488), I added a Sheet.knownCols property, to specify properties of known columns, in visidatarc:

Sheet.knownCols.timestamp.type = date
Sheet.knownCols.timestamp.fmtstr = '%H:%M:%S'

which will set the type and fmtstr of any column named timestamp after the sheet finishes loading.

This manual configuration in visidastarc seems like the simplest solution for your request. Saving metadata presets for a whole sheet, to be applied to other data sources, would probably be better handled with .vds metadata-only files, as suggested in #892. Let's give this a try for now and see how far it gets us. Maybe we can add some "AI" that notices when you've applied the same metadata multiple times to the same-named column, and then automatically adds it to the knownCols in your .visidatarc.