Support for bibtex files

[Cs137]

Hi and many thanks for this awesome piece of software! 😍

I would like to use vd to manipulate my bibtex library. I can load a bibtex file into vd, but it would be very convenient if the data would be tabulated, like e.g. json data. The bibtex-key + all existing tags could be presented as column headers with rows for the individual entries. Is there any way to achieve this currently already? Glad for any advice

[saulpw]

Hi, @Cs137, thanks and you're welcome! Tell me, how would you extract the data in Python? If you can get a list of dicts (or other Python object), we can make a loader for it really easily!

[Cs137]

Dear @saulpw,

I continued looking into the bibtex integration, but am struggling with basics trying to set up the loader.

I created a ~/visidata/config.py file with the content suggested by you, but have the impression the iterload method is not called at all. I added some print statements and opened a bib file with with vd using the --debug flag, the last print statement I get is the assignment of the columns:

COL_ENTRYTYPE = 'type'
COL_CITATIONKEY = 'citationkey'
COL_FIELDS = 'fields'

@VisiData.api
def open_bibtex(vd, p):
    print("Started 'open_bibtex' function...")         # <-- This is not printed
    return BibtexSheet(p.name, source=p)


class BibtexSheet(Sheet):
    print("Initialised 'BibtexSheet' instance...")     # <-- This is printed
    columns = [
        ItemColumn(COL_ENTRYTYPE),
        ItemColumn(COL_CITATIONKEY),
        ItemColumn(COL_FIELDS),
        ItemColumn(COL_FIELDS + '.author'),
    ]
    print("Columns: ", columns)                        # <-- This is printed

    def iterload(self):
        print("Started 'iterload' method...")          # <-- This is not printed
        current_entry = None
        for line in self.source.open_text():
            line = line.strip()
            if not line or line.startswith("%"):
                continue
            if line.startswith("@"):
                if current_entry:
                    entries.append(current_entry)
                parts = line.split("{")
                current_entry = {
                    COL_ENTRYTYPE: parts[0][1:].lower(),
                    COL_CITATIONKEY: parts[1][:-1],
                    COL_FIELDS: {},
                }
            elif line.startswith("}"):
                if current_entry:
                    entries.append(current_entry)
                current_entry = None
            else:
                parts = line.split("=")
                if len(parts) == 2:
                    key = parts[0].strip().lower()
                    value = parts[1].strip()[:-1].strip('{}"')
                    if current_entry:
                        current_entry[COL_FIELDS][key] = value
            if current_entry is not None:
                if ' and ' in current_entry[COL_FIELDS]['author']:
                    authors = current_entry[COL_FIELDS]['author'].split(' and ')
                else:
                    authors = [current_entry[COL_FIELDS]['author']]
                current_entry[COL_FIELDS]['author'] = authors

                yield current_entry

I created in the meantime also a small python package, which does what I had in mind to do with vd; but out of curiosity I'd like to get it implemented in vd as well.

The parser of my package handles the author assignment better than the example above and tries to get the first and last names for the individual authors from the string, which are added to a list of dicts. The latter replaces the content of the author field.

The resulting entries list can also be used to initialise a pandas df, maybe returning that df as Sheet might be a better approach for an integration of bibtex into vd. But I have to admit that I am a bit lost in how to adjust the loader, even though I looked into your api docs. Would you be so kind to give me another hint?

Best regards,
Christian

[saulpw]

Hi @Cs137, your print statements are running unconditionally at import time, and not at runtime; only the ones in open_bibtex and iterload will actually run because it's trying to open a .bibtex file. You should use vd.status() for these anyway.

The code looks fine at a first glance. I'm not actually sure why it's importing the file at ~/visidata/config.py though. The actual file you should put these in, until you're more familiar with how Python imports work in general, is ~/.visidatarc.

Please include a test bibtex file, and the actual command-line you use to open it, so we can try to reproduce your issue, and figure out what's going on.

[Cs137]

Hi @saulpw, thanks for your fast response and sorry for my mistake. The file's path is ~/.config/visidata/config.py. I was first unsure whether it gets imported at all, that's why I started adding the print statements. The bib file I am trying to open is the one linked above, I added a wget command to obtain the file and the way how I try opening it underneath:

wget http://tug.ctan.org/tex-archive/macros/latex/contrib/aguplus/sample.bib
vd --debug --open_bibtex sample.bib

I added the --open_bibtex just as a test. It is also not parsed without it. I think an assignment that the bib file should be handled by the open_bibtex function is missing somewhere.

[Cs137]

Sorry for my unknowledge, vd --debug -f bibtex sample.bib does the job and shows that there is some work on the parser left. I will pick it up tomorrow and will find my way to add the loader to vd based on your docs. Thanks for your help!

[saulpw]

Ah yes, and if .bib is the common extension for these files, just name the function open_bib.