Reduce Duckdb memory consumption

[dsmilkov]

In the short term, we should have a single global db connection that can make unique view names for each dataset, and read multiple database files using ATTACH: https://duckdb.org/docs/sql/statements/attach.html

This will hopefully help with minimizing any caching that duckdb does for each different connection.

[dsmilkov]

it should also speedup opening new datasets

[brilee]

https://duckdb.org/2024/01/26/multi-database-support-in-duckdb.html as per this recent blog post, a solution may be to construct a postgres cache table to take advantage of postgres's stronger on-disk index support, instead of relying on duckdb's indexing