Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions docs/supported-sources/cratedb.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@
massive amounts of data in near real-time, even with complex queries. It is
PostgreSQL-compatible, and based on Lucene.

ingestr supports CrateDB as a source and destination database.
ingestr supports CrateDB as a source and destination database, using different
adapters and protocols (HTTP vs. PostgreSQL, see below).

## Source

For connecting to CrateDB as a database source, ingestr uses its SQLAlchemy
dialect package [sqlalchemy-cratedb].
dialect package [sqlalchemy-cratedb], effectively talking HTTP to port 4200
by default.

### URI format

Expand Down Expand Up @@ -56,11 +58,17 @@ duckdb cratedb.duckdb 'SELECT * FROM dest.summits LIMIT 5'

<img alt="CrateDB_img" src="../media/cratedb-source.png" />

Note that, as of February 2026, for compatibility and harmonization reasons,
the CrateDB source adapter can also be addressed using the `cratedb://`-style
URI format like outlined below. However, please note that both are still
different adapters using different protocols.

## Destination

For connecting to CrateDB as a database destination, ingestr uses the
[dlt cratedb adapter], which is based on the [dlt postgres adapter],
in turn using the [psycopg2] package.
in turn using the [psycopg2] package, effectively speaking the PostgreSQL
wire protocol on port 5432 by default.

### URI format

Expand Down
1 change: 1 addition & 0 deletions ingestr/src/factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@
SQL_SOURCE_SCHEMES = [
"bigquery",
"crate",
"cratedb",
"duckdb",
"mssql",
"mssql+pyodbc",
Expand Down
22 changes: 22 additions & 0 deletions ingestr/src/sources.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,28 @@ def eng_callback(engine):
return engine.execution_options(read_only=True)

engine_adapter_callback = eng_callback

if uri.startswith("cratedb://"):
# CrateDB's SQLAlchemy dialect uses the `crate://` protocol scheme,
# but we would like to harmonize on `cratedb://` across the board.
# https://github.com/bruin-data/bruin/issues/1640

# Rewrite parameter `sslmode` to `ssl`.
parsed_uri = urlparse(uri)
query_params = parse_qs(parsed_uri.query)
if "sslmode" in query_params:
sslmode = query_params.get("sslmode")[0]
del query_params["sslmode"]
ssl = "false"
if sslmode.lower() in ["require", "verify-ca", "verify-full"]:
ssl = "true"
query_params["ssl"] = [ssl]

# Rebuild URI
uri = parsed_uri._replace(
scheme="crate", query=urlencode(query_params, doseq=True)
).geturl()

from dlt.common.libs.sql_alchemy import (
Engine,
MetaData,
Expand Down
Loading