Discussion on deploying (pre-)build static libraries

[wearpants]

While I understand statically linked duckdb .a libraries are not included in upstream packages by default & building them is slow, including binary blobs in the source tree smells bad at best and is a supply chain attack vector (by targeting whoever is building them) at worst. Also seems to be the source of a number of other problems, see: #336, #323, #54, #38, #120, #279, #330

It should be possible to build Go bindings entirely from source and/or publicly released artifacts from other projects.

[ayuhito]

If I understand correctly, go-duckdb uses an amalgamation file generated from the C++ source which introduces a mountain of complexity and made Windows blocked for a good year or two.

Looking at the new Node Neo client, I wonder if it makes sense to rely on the C API directly (which should also play nicer with CGO).

To use DuckDB’s C API, on the other hand, one only needs to depend on released binaries. This significantly simplifies the maintenance required, speeds up builds, and minimizes the cost of updates. It removes the uncertainty and risk of rebuilding DuckDB.

It's definitely a monumental undertaking to switch bindings, but I think it would significantly help in the long term. Using the same API as Neo would also mean we have a direct shared point of reference and the C API would get a lot of priority for bug fixes etc.

[taniabogatsch]

It's definitely a monumental undertaking to switch bindings, but I think it would significantly help in the long term.

I agree. Both with the expected effort and the necessity...

We are currently using the C API function calls, but we need to change how we use them in the (not too far) future. I'd love to have go-duckdb run on top of some bindings-only package. I've been experimenting with our recent C API changes here... I also saw this new discussion here: #342. Although I have no experience with purego, and I am not sure how active that project is.

[wearpants]

Is there a smaller than "rewrite the whole thing" solution that's possible here? Such as:

• get upstream to build & publish .a's
• building them ourselves in this project
• or an (unofficial) duckdb-static-libs project
• something else I haven't thought of

[taniabogatsch]

get upstream to build & publish .a's

If duckdb were to provide these, how would we then pull them into the project with go get / build / import ...go-duckdb? To keep the experience as seamless as possible.

[wearpants]

I'm not a go expert, but we seem to have hit the limit of what's possible with plain go get

• Including the .a in this repo's source tree is compromising functionality (b/c of GH's file size limits) as well as compromising security/reproducibility (by having the .a's hand built).
• that seems to necessitate some external step or tooling, whether manually or Makefile downloading the artifact from a published GH release (in this project or upstream).
• A compromise option might be using git-lfs with a published, reproducible artifact included, but that still requires users to set up git-lfs.

But given the various conflicting restrictions in play, I don't see how to do this just with go get.

A good first step would be getting the build recipe and artifacts for the .a's available somewhere, along with necessary instructions/tooling to use that in builds of this library, so that users can customize & reproducibly build the .a. Basically separate the "build .a" and "go get distribution" issues for the moment.

[taniabogatsch]

A good first step would be getting the build recipe and artifacts for the .a's available

I agree. I've opened a first PR for OSX arm + amd builds here: duckdb/duckdb#15682.
I'll try to get these into our deploy pipelines (also for other OS + arch combinations), as duckdb publishing some pre-build static libraries is generally a good idea, independent of this project.

Having both

1. some pre-build static libraries available through duckdb
2. and instructions on how to build your own .as

Sounds like a good way forward to me.

We should also discuss how people would best go get this project and pull in their necessary library if they aren't building it (you mentioned manually or Makefile). I'd love some more input from people using this project about this... :)

[wearpants]

We should also discuss how people would best go get this project and pull in their necessary library if they aren't building it (you mentioned manually or Makefile).

mage I guess?

go get just doesn't support anything outside of git sources... Spoke to an old friend from Google, he said that's an artifact of Go evolving in the context of google's larger build system (bazel)

[rymurr]

Hi All,
Thanks for starting this thread @wearpants its very timely for me. I thought I would share what we have been working on as I think it has some relevance.

Our use case is to query duckdb from go with a number of extensions pre-loaded. We don't want to download/cache extensions and some of the extensions we are loading are under active development so we don't want to use the official duckdb repo.

We have found that:

1. go get is a must. Anything else is not ergonomic for our developers
2. git-lfs is a bit of a pain in association with go get (and surprisingly expensive on github!) so we have resigned ourselves to having a large repo full of binary files
3. static libraries are challenging once we add in extensions and the vcpkg dependencies. We ended up with 60+ static libraries and combining them in such a way that files were under the github limit and compile order was simple was very difficult (we never actually managed it on macOS)

Our solution in the end was to build everything as a dynamic library and use rpath in the #cgo LDFLAGS directive. The resulting binary doesn't need any extra LD_LIBRARY_PATH envs to find the shared libraries with rpath and there are much fewer files to deal with and distribute. The main downside for us is that we have to ensure we always set -tags duckdb_use_lib to make sure our dynamic libraries get used instead of the static one in go-duckdb. You can see our work here: https://github.com/sundeck-io/duckdb-go-extensions

Not sure if this is helpful at all, though we have found it works for us.

[taniabogatsch]

Closing this as we've just moved this project to import the duckdb-go-bindings package.
Any further improvements to the pre-built libraries in that module should be discussed there. :)

#386