Add Postgres DB backend for the indexer.

[deuszx]

Motivation

Turns out our SQLite file for Testnet is quite big already – over 4 GB. We also learned that it might be useful to have a read-only access to the DB remotely, which is not possible with SQLite.

Proposal

Add support for Postgres backend. Explorer frontend was also updated. So were docker scripts.

Test Plan

Tested locally with all 3 options:

1. --memory
2. --sqlite <path_to_file>
3. --postgres <local_instance>

Integration tests were added for postgres.

Release Plan

• Nothing to do / These changes follow the usual release cycle.

Links

• reviewer checklist

[ndr-ds]

Looks good overall. Back to you, as I have a few questions that could be blocking.
Just FYI, the file is actually currently over 18GB already 😅

[ndr-ds]

Why are we building this in the indexer image? We have the main Dockerfile with this already.

[deuszx]

I think it started differently – this image was slightly smaller than Dockerfile 🤔 I'll see if I can do it again.

[deuszx]

Actually, Dockerfile doesn't have linera-storage-server and linera-exporter – so half of the binaries I need here. Since it doesn't need them (docker-based validators don't run storage-service and that's needed for the exporter) I would prefer to NOT add it there. This makes these changes necessary.

[ndr-ds]

Then add just the storage-server one here? The server and proxy binaries are huge. Please let's not add it here

[deuszx]

That's why I don't want to add them there – they're not needed in the Dockerfile that is used externally and ARE needed here for testing the exporter-indexer stack. If you're using it in other deployments then there should exist a different Dockerfile – this one is for testing locally, not deployments.

[ndr-ds]

I know for a fact that building linera-proxy and linera-server for this image (which is used by both the exporter and the indexer) is not needed, as that's the current state in production. So since this is a "test" image, we need a production image that doesn't unnecessarily links these binaries and pushes them to the Google's Artifact registry, for example.

[ndr-ds]

Yes I get it that you want this to be a test image with everything. I'm just asking you to create a production one then, in this case

[deuszx]

I don't know what's needed in production. If you tell me exactly how it should be configured I can do it.

[deuszx]

I know for a fact that building linera-proxy and linera-server for this image (which is used by both the exporter and the indexer) is not needed, as that's the current state in production. So since this is a "test" image, we need a production image that doesn't unnecessarily links these binaries and pushes them to the Google's Artifact registry, for example.

Maybe it's not running but it's needed when building – otherwise cargo complains that it's missing dependencies.

[ndr-ds]

It's just whatever this file had before your changes, with the web directory fix, I guess

[ndr-ds]

Could we call this just once in the bottom?

[deuszx]

Actually, we can't. IndexerGrpcServer::new expects D type – it doesn't expect D: DatabaseTrait – so all of these if branches return different type (IndexerDatabase<Sqlite>, IndexerDatabase<Postgres>).

[ndr-ds]

Maybe a helper function then? Something like this:

async fn serve_with_db<D: IndexerDatabase + 'static>(
    database: D, 
    port: u16
) -> Result<(), IndexerError> 
where D::Error: Into<ProcessingError> {
    let grpc_server = IndexerGrpcServer::new(database);
    grpc_server.serve(port).await.map_err(IndexerError::Other)
}

[deuszx]

I prefer to leave as is.

[ndr-ds]

Why? Generally we don't want to duplicate code, if we can

[ndr-ds]

Could we specify to include just db::memory here instead?

[deuszx]

This is about skipping those tests (ie these are not arguments which DB use).

[deuszx]

But we don't need to skip db::sqlite as they're fairly fast since they run in memory.

[deuszx]

Done in Fix indexer.yml.

[ndr-ds]

I know it's skipping... I'm saying that being explicit about what you're including is generally better than specifying what you're excluding. Because a test can be added, and you'll just start running it automatically, even if it doesn't need to be run, if the person forgets to skip it here.
Can't you do something like:

cargo test --locked -p linera-indexer --lib -- db::memory

?

[deuszx]

This version is better for the future as I don't want to add explicitly what needs to run every time I add new tests. I just know I don't want to run postgres-based ones.

[ndr-ds]

Back to you for my replies