Self hosting

[karlbateman]

Hi everyone, this is such an exciting and cool project. I've got a quick question regarding privately hosting Neon:

From the documentation one can install and run Neon locally, is this something which specifically targets local development or can Neon be self hosted? If the latter, is this something that you'll be providing dedicated documentation and/or tooling to support?

[SomeoneToIgnore]

Hi.

is this something which specifically targets local development

One can run neon on the local machine without any servers, we have a special set of "e2e" tests doing exactly this: https://github.com/neondatabase/neon/tree/main/test_runner

There's a set of tools you need to run together to have a fully working system locally:

• a safekeeper component (at least one, 3 for something "regular")
• a pageserver component
• a compute component
• etcd

You don't have to start them by hand though, there are two ways to manage those:

• neon_local CLI tool for regular development usage, fully described in the root README: https://github.com/neondatabase/neon/blob/main/README.md#running-local-installation
• zenith_fixtures.py to use in some sort of automation like test set up. Check the tests around to see how it's used.
  NOTE: this component now partially uses neon_local and has an "old" name, it will get renamed and refactored heavily later.

can Neon be self hosted

Pretty much the same way as the dev env could be started, the self hosted one could be.
One key difference that might help here is the Dockerfile that we have, we build images out of it and compute:

• https://registry.hub.docker.com/r/neondatabase/neon/tags
• https://registry.hub.docker.com/r/neondatabase/compute-node/tags

The images' maintenance will become more structured with time, for now we only have latest x86 Docker images pushed for every code change in main.

[edude03]

Not sure if I should start a new discussion - but I'm working on getting neon setup in kuberentes just to kick the tires, and although I can get each component to start, configuring them seems to be ... undocumented?

For example, you need a cluster spec for the compute_node. The only one I found was in the tests folder. Using that one, it specifies a tenant and timeline, which I figured out you need to create I guess using the HTTP api on pageserver. But once I do that, I noticed that the wal_source_connstr is wrong (it's 0.0.0.0:5454, which can't work?) so more digging is required to make it work.

[edude03]

I actually managed to fix this issue - the listen address in safekeeper needs to be set to a resolvable address - it's mostly working now except it breaks if two compute instances run at the same time. Point remains, there are clearly some operational things I'll need to figure out :)

[mingfang]

I actually managed to fix this issue - the listen address in safekeeper needs to be set to a resolvable address - it's mostly working now except it breaks if two compute instances run at the same time. Point remains, there are clearly some operational things I'll need to figure out :)

@edude03 thanks, setting the safekeeper listen address was key.

[mingfang]

My setup was working fine for a while but all the sudden started to fail.
The safekeepers are constantly printing out this error

2023-01-10T06:17:58.457307Z  INFO {tid=126}: got query "IDENTIFY_SYSTEM" in timeline Some(80ed3be9cbb1a738d90ff39e7e4ccaaa)                                                                                                                          
2023-01-10T06:17:58.457741Z  INFO {tid=126}: got query "START_REPLICATION PHYSICAL 0/2494A30" in timeline Some(80ed3be9cbb1a738d90ff39e7e4ccaaa)                                                                                                     
2023-01-10T06:17:58.457823Z  INFO {tid=126}:WAL sender{ttid=ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa}: Start replication from 0/2494A30 till None                                                                           
2023-01-10T06:17:58.457884Z  INFO {tid=126}:WAL sender{ttid=ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa}: segment download about to start from remote path RemotePath("ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39
2023-01-10T06:17:58.545201Z ERROR {tid=126}: query handler for 'START_REPLICATION PHYSICAL 0/2494A30' failed: Failed to process query for timeline ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa                                 
                                                                                                                                                                                                                                                     
Caused by:                                                                                                                                                                                                                                           
    0: Failed to open WAL segment download stream for remote path RemotePath("ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa/000000010000000000000002")                                                                           
    1: No file found for the remote object id given                                                                                                                                                                                                  
2023-01-10T06:17:58.545288Z ERROR connection handler exited: reader taken                                                                                                                                                                            

I checked S3 and this file ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa/000000010000000000000002 does not exist.
But the file before it,ca5fbf3053bc9ea10eb2f32949f78c91/80ed3be9cbb1a738d90ff39e7e4ccaaa/000000010000000000000001, does exist.
Any ideas?