Understanding Neon Via Architecture Diagram

[phoenix24]

I've been able to sketch a high-level diagram, from my understanding so far by reviewing various readme's and rfc's., however I do have a few questions I can use help with -

questions,

• does the above high-level architecture diagram reflect neon db's architecture to a reasonable accuracy?
• a page-server is multi-tenant, ie. one page-server can serve multiple tenant; but what does it mean inverse, can a tenant by served by multiple page-servers?
• similarly to a page-server, is a safekeeper multi-tenant too?
• similarly to a page-server, is compute-node multi-tenant too; if so what does it mean for MVCC, and if not then what does it mean for proxy (read-traffic vs write traffic) ?
• proxy is stateless, is it correct to assume it can be horizontally scaled?
• can compute node be setup in read-only mode, thereby allowing to scale read-throughput?
• would aws s3 be the only blob store supported or are there plans to support other vendors in the future?

[hlinnaka]

• does the above high-level architecture diagram reflect neon db's architecture to a reasonable accuracy?

Yep! I'm not sure what exactly the boxes and different lines mean; e.g. does an arrow mean the flow of WAL or network connections in general, or what. I'd suggest adding a legend. But yes, it looks correct at a high level.

• a page-server is multi-tenant, ie. one page-server can serve multiple tenant; but what does it mean inverse, can a tenant by served by multiple page-servers?

Currently no, but we have plans for it in the future. One future feature is sharding, so that if you have a large database, it could be sharded across multiple pageservers. Another is that if you need high availability, or need more capacity for the reads, the same data could be held in multiple pageservers so that the compute node can send GetPage request to any one of them.

• similarly to a page-server, is a safekeeper multi-tenant too?

Yes

• similarly to a page-server, is compute-node multi-tenant too; if so what does it mean for MVCC, and if not then what does it mean for proxy (read-traffic vs write traffic) ?

No, each compute node serves a single tenant.

• proxy is stateless, is it correct to assume it can be horizontally scaled?

Yeah, it's stateless, in the sense that it doesn't have any local disk, and it can be horizontally scaled easily.

It does have state in the sense that it proxies all the connections through it, though. So if a proxy dies, all the connections through it will be disconnected.

• can compute node be setup in read-only mode, thereby allowing to scale read-throughput?

Yes.

• would aws s3 be the only blob store supported or are there plans to support other vendors in the future?

It should work with any S3-compatible cloud storage, we use it as a dumb blob store, we don't rely on any advanced features. That said, we've only tested it with S3. Oh, and we do use the "moto" python mock S3 server in the tests.

[js00070]

For multiple compute-nodes, I guess the primary compute-node should keep sending redo logs to the read-only compute-nodes, and the read-only compute nodes should keep applying redo logs to the pages in buffer pool?
Or maybe primary compute-node can just keep sending the latest redo log LSN which have written to the safekeepers and read-only nodes can read redo logs from safekeeper?

[hlinnaka]

Yep. @MMeent actually wrote a prototype for that a while ago already, see #850. We put that on hold for now, as we need to figure out the user interface for it, but it's basically ready. We could optimize that in many ways, e.g. by being smarter by not fetching pages from the pageserver into the standby's buffer pool just for WAL redo, or by not replaying the WAL in the standby but evicting the pages from buffer pool instead, or by sending an abbreviated version of the WAL stream to the standby with just the page numbers of the modified blocks. But the first basic version is there.

PITR read-only nodes that don't follow the primary but are anchored at a particular LSN, already work on 'main'.

[js00070]

Yep. @MMeent actually wrote a prototype for that a while ago already, see #850. We put that on hold for now, as we need to figure out the user interface for it, but it's basically ready. We could optimize that in many ways, e.g. by being smarter by not fetching pages from the pageserver into the standby's buffer pool just for WAL redo, or by not replaying the WAL in the standby but evicting the pages from buffer pool instead, or by sending an abbreviated version of the WAL stream to the standby with just the page numbers of the modified blocks. But the first basic version is there.

PITR read-only nodes that don't follow the primary but are anchored at a particular LSN, already work on 'main'.

Sounds good! And I can think of some following issues.

1. I guess each of the read querys on the read-only node may need keeping a consistent read-view to achieve repeatable-read, which may require primary node to keep sending information about it's current active transactions?
2. The proxy can split read-write querys and read-only querys. A user may find he can't read a record immediately after he commits the record insert transaction if the replication to read-only nodes is asynchronous.
3. If the WAL replay procedure on read-only node fall behind the primary node, the pageserver may need to keep a lot of old versions of all pages because read-only node may still need to read these old versions.

[phoenix24]

No, each compute node serves a single tenant.

Does it mean that proxy does tenant aware query routing?

[phoenix24]

I'd suggest adding a legend. But yes, it looks correct at a high level.

Thank you, Yes - I'll add an updated diagram.