Index + Limit

[JoshBurrell]

Hey everyone,

Looking to see if it is either currently possible or a possibility in the future to have a table that uses both index & limit together.
Eg index='security_id' & limit=86400' so for every unique security_id I can have up to 24hrs of top of book 1 second data.

I know currently I could guesstimate my number of securities I would have and then make my limit be 86400 * expected securities but say I only update when the top of book changes some very liquid securities could could drown out less liquid securities as the table reaches the limit.

Is this approach not recommended compared to a more structured fixed number of securities and everything gets the same number of updates.

Cheers

[texodus]

From the docs on this feature:

limit cannot be used in conjunction with index.

Even if they could be, it would not work as you describe - index is a primary key and adding a limit on top would just limit the number of unique primary keys the set could contain, it would not just allow Table to store duplicate indices.

There is a remove() method for selecting indices to clear, but you'd need to first query and/or track them yourself. Instead, you may try implementing a periodic (daily or whatever) try using a View + filter to move just the recent records, something like:

const cutoff = new Date();
cutoff.setDate(cutoff.getDate() - 1)
const ticks = await table.view({filter: [["lastUpdate", ">", cutoff]]);
const arrow = await ticks.to_arrow();
await table.replace(ticks);

[JoshBurrell]

Got it, thanks