Datastore garbage collection across replicas

[nate-kis]

Hi, just looking for what the best practices are for having multiple replicas for SpiceDb and specifically around garbage colleciton.

I notice that we get log entries stating datastore garbage collection completed successfully and often these are run at the same time across multiple of our pods.

I don't know if it's related, but we sometimes get a timed out error on our grpc connections and it always seems to correspond to when there is garbage collection running, so wondering if there is potentially an issue that garbage collection blocks connections to the grpc, or if there is a way that only one of the replicas needs to run garbage collection ,or if it is fine as is.

Thanks!

[tstirrat15]

How are you running your SpiceDB instances?

The coincidence of timeouts and garbage collection may have something to do with the max number of datastore connections that each pod will open, which is configured through --datastore-conn-pool-read-max-open and --datastore-conn-pool-write-max-open. If those numbers are too low and you have sufficient traffic, you can end up in a situation where SpiceDB is waiting for a DB connection to become available from the pool.

[vroldanbet]

There is also an issue that was addressed recently, which makes GC way faster: #1859

[nate-kis]

Running it in kubernetes, and have the connections set as 20 for read, and 10 for write - so it could make sense it could be limited on the connections.

Is there anything on best practices for these settings?

We don't have a lot of traffic going to spicedb yet as we are doing a transition to it, but about to hit it with more load.

[vroldanbet]

There are no specific settings - it depends on your workload. You could test disabling GC temporarily to see if that's the problem. If the problem was contention on connection pools, we could look into making GC happen outside of the connection pools, just like Watch API does for CockroachDB datastore.

SpiceDB offers metrics around connection pools, so you should be able to see if that's where the problem lies.