Parse integers in other bases

[hoclun-rigsep]

Would it be cool if VisiData dealt smoothly with columns of integers expressed as hex, octal, or bin, when you press #?

[saulpw]

Well you can use =int(colname, 16) etc, which doesn't seem too onerous. Would you want auto-detection of base, or how would it work?

[hoclun-rigsep]

I agree, that is not too onerous, and feel sheepish for not having thought of it right away.

Yet autodetection of base according to a small number of conventional string representations seems like it would be easy to implement. Catch the ValueError, attempt to match one of those conventions, re-raise if you can't. Only happens when user deliberately types a column so no issue with parsing "beef" as an integer.

Common ways of representing 255, case not withstanding:

255
ff
0xff
0377
0o377
0b11111111

Let me know if you want me to pursue.

[saulpw]

It's trickier than you would think; see the other cases where we autodetect. Which and how many rows do we try? What if we get a conflict? Should 0377 be autodetected as 255 or 377?

Possibly I could see an "autodetect column types" feature/plugin, but I think it would need a lot of heuristics, and I would want it to be a separate deliberate command, and not bound to the standard # type. And handle all manner of type detection: ints of different bases, floats in different locales, dates in different formats, etc.

[hoclun-rigsep]

I think it's wise to be wary of an "autodetect column types" feature; I also think that's separate from a "fancy version of int() Column.type"

To my (hasty) evaluation it looks like if I press # on a column, VisiData calls getTypedValue() and therefore int() on every row, regardless of how many there are. What if instead of calling int() it called this?

def better_int(x):
    try:
        return int(x)
    except ValueError:
        if x.startswith('0b'):
            base = 2
        elif x.startswith('0o'):
            base = 8
        elif (x.startswith('0x')
               or set(x) >= set('abcdef')
               or set(x) >= set('ABCDEF')):
            base = 16
        else:
            raise
        return int(x, base=base)

I think (but haven't proved) that those are very performant checks, that only happen as a direct result of someone pressing #, and degrade into existing behavior.

I don't want it to seem like I'm pushing this—I don't even think it's very important in light of =int(colname, 16)—but it seems like low-hanging fruit.

("0377" is tricky; one could see someone showing up with unvalidated data that for some reason has leading zeroes, and I wouldn't want to cross the line into comparing rows. Probably the least surprising policy is to treat it like a decimal.)

(For all I know it would be more performant to just call int(x, base) with each radix until one works or they all fail...)

[saulpw]

You can try it out yourself! Add the code above to your visidatarc, along with:

vd.addType(better_int, '#')
TableSheet.addCommand('#', 'type-better-int', 'cursorCol.type=better_int')

(This won't handle everything, as isNumeric won't recognize it as a numeric type. But it should get pretty close).

For new features like this, my first step is usually to try it out myself like this for a few weeks. I'm often surprised by features that I think sound like good ideas, only to find that in practice they cause more trouble than they're worth. It's tricky making a general-purpose data utility used by thousands of people :)

[hoclun-rigsep]

I can believe that!