Recycling IDs in LMDB Store

[hmottestad]

I was wondering if we really need to recycle IDs in the LMDB Store?

We use long for IDs, with max long being 2^63-1.

If we allocate 1 000 000 IDs per second we can allocate new IDs for 292 471 years.

So I'm wondering if we can't just always mint new ones and simplify the logic a bit by not needing to keep track of unused IDs?

Calculation: ((2^63)-1)/1000000/60/60/24/365

[hmottestad]

@kenwenzel What do you think?

[kenwenzel]

I see the main advantage that smaller IDs map to shorter varints. Especially for predicate IDs this is very helpful.

[hmottestad]

That's true. It's a lot faster to only read/write a single byte.

Reason I'm asking is because I'm working on merge join support and I kinda feel that recycling IDs is problematic, even though I can't quite put my finger on why. Maybe I'm wrong.

[kenwenzel]

We can simply disable it if required.
I'm also working on this extensible ID scheme that allows to encode values into IDs. I hope that this is unproblematic?