Scylla on low memory footpring (512M) returning Unprepared(0x2500) on executing prepared statement after successfully replying to opPrepare(0x9) with resultKindPrepared(0x4)

[dkropachev]

There is a test in scylladb/gocql called TestBatchObserve, it creates batch of 100 queries and executes it.

The way driver dials with statements within the batch is it prepares every query, remembering it's prepared statement in the cache and then runs the query.
If it gets an Unprepared(0x2500) it invalidates the prepared statement cache record and rerun the query.

What happens when this test is ran on 5.2.0 and newer, with memory limited to 512M is that it spins indefinitely because it always gets Unprepared(0x2500), while if you increase memory limit to 1G it works great.

My understanding is that scylla have some buffer reserved for prepared statements that operates with on LRU-like logic.
I.e. when memory limit on the buffer is reached it remove oldest record and inject new one.

As result driver can't be sure that statement it's just prepared is actually remembered by scylla.

Related scylladb/gocql issue - scylladb/gocql#126

• I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 5.2.0, 5.2.1, 5.2.2, 5.2.3
Cluster size: 1
OS (RHEL/CentOS/Ubuntu/AWS AMI): docker image

[avelanarius]

Refs #10440 - a similar issue from the past. That issue was fixed by changing the eviction behavior of loading_cache (note that the fix should be improved: #10674).

One possible improvement to prevent infinite spining:

The way it works now (looking at Rust Driver, but others are similar): let's suppose that there is a batch statement of N prepared statements and the driver sends this statement to Scylla. Scylla might respond that some statement in that batch is not prepared (not present in prepared statements cache). The driver will then re-prepare all statements in that batch. However, if the cache is too small, it won't be possible for all statements to be in the cache at the same time.

Improvement:

1. The driver(s) should not infinite loop, instead impose some limit of how many times it will try to re-prepare (some draft PR from the past: Re-prepare statement throttle java-driver#143)
2. When Scylla receives a batch of N prepared statements, it should check if the prepared statements cache is large enough and return some error if it's not.

[dkropachev]

2. When Scylla receives a batch of N prepared statements, it should check if the prepared statements cache is large enough and return some error if it's not.

Even if cache is large enough it does not prevent going into the state when scylla can't completed concurent batches.
Say you have buffer for 20 records, 10 parallel clients are trying execute batches of 10 unique statements in parallel, at the time when given client completed preparing statements in the batch, other clients already pushed ~`9*10 prepared statements, evicting whatever was prepared by the client, despite the fact that buffer is larger than batch size.

[avikivity]

The minimum memory requirement is 1GB/shard.