-
Notifications
You must be signed in to change notification settings - Fork 63
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add fivetran destination connector docs. (#28285)
Added docs for the destination connector, as well as cross links about the source connector. GitOrigin-RevId: 78d84fd65588ffa974da2c082e297c9e1160e959
- Loading branch information
1 parent
450e6e2
commit 4e0e27a
Showing
4 changed files
with
199 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
--- | ||
name: Convex | ||
title: Convex destination connector for Fivetran | ||
description: Documentation and setup guide for the Convex destination connector for Fivetran | ||
--- | ||
|
||
# Convex {% typeBadge connector="convex_destination" /%} {% availabilityBadge connector="convex_destination" /%} | ||
|
||
[Convex](https://convex.dev) is the open-source backend for application builders. | ||
An all-in-one platform with thoughtful, product-centric APIs. | ||
|
||
Note that Convex can also be set up as a [source](/docs/databases/convex) | ||
|
||
--- | ||
|
||
## Setup guide | ||
|
||
Follow our [step-by-step Convex setup guide](/docs/destinations/convex_destination/setup-guide) to connect Convex as a destination with Fivetran. | ||
|
||
--- | ||
|
||
## Sync overview | ||
|
||
Once Fivetran is connected to your Convex destination, the connector will attempt to sync your data. | ||
It may ask you to update your `convex/schema.ts` in your deployment to match the format of your source. | ||
Once the `convex/schema.ts` matches, data will continu to sync. | ||
|
||
--- | ||
|
||
## Configuration | ||
|
||
You will need your deployment URL and deploy key in order to configure the Convex Connector for Fivetran. You can find both on your project's [Production Deployment Settings page](https://docs.convex.dev/dashboard/deployments/deployment-settings). | ||
|
||
--- | ||
|
||
## Schema information | ||
|
||
Fivetran tries to replicate the database and columns from your configured source to your destination Convex according to Fivetran's [standard database update strategies](/docs/databases#transformationandmappingoverview). | ||
|
||
### Type transformations and mapping | ||
|
||
As the connector extracts your data from your source, it matches the supported Fivetran types to [Convex data types](https://docs.convex.dev/database/types). | ||
|
||
The following table illustrates how the connector transforms the Fivetran data types into Convex data types. | ||
|
||
| Fivetran Type | Convex Type | Equivalence | | ||
| ------------- | ----------- | ----------- | | ||
| BOOLEAN | Boolean | Exact | | ||
| SHORT | Float64 | Inexact | | ||
| INT | Float64 | Inexact | | ||
| LONG | Int64 | Exact | | ||
| DECIMAL | String | Inexact | | ||
| FLOAT | Float64 | Inexact | | ||
| DOUBLE | Float64 | Exact | | ||
| NAIVEDATE | String | Inexact | | ||
| NAIVEDATETIME | String | Inexact | | ||
| UTCDATETIME | Float64 | Inexact | | ||
| BINARY | Bytes | Exact | | ||
| STRING | String | Exact | | ||
| NULL | Null | Exact | | ||
| JSON | Object | Inexact | | ||
|
||
> NOTE: Short/Int are converted to float64 for ease of use in javascript (as `number`). There is no data loss as Number.MAX_SAFE_INTEGER = 2^53 - 1. | ||
> NOTE: Decimal is converted to string to ensure no data loss. Eg "1234.5678" | ||
> NOTE: Naive date uses standard string representation of YYYY-MM-DD. | ||
> NOTE: Naive datetime uses standard string representation of YYYY-MM-DD HH:MM:SS. | ||
> NOTE: UTC datetime uses milliseconds since UNIX epoch | ||
### Fivetran-generated data | ||
|
||
Fivetran adds a single `fivetran` column containing a Convex object to the source data. | ||
Some of the fields (`synced`, `deleted`) are for internal purposes. | ||
|
||
### Table and column name transformations | ||
|
||
If your source table is `default.cars`, then your convex table will be named `default_cars`. | ||
Convex deployments do not have a concept of namespaced tables, so it uses this notation to preserve | ||
the namespace information. | ||
|
||
Columns names that begin with `_` are not supported in Convex. Instead, those columns are synced to the | ||
destination nested within the `fivetran.columns` column. For example, a column named `_line` would end up in the `fivetran.columns.line` nested column. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,106 @@ | ||
--- | ||
name: Convex Destination Setup Guide | ||
title: Convex Destination Connector Setup Guide | ||
description: Read step-by-step instructions on how to connect your Convex deployment as a destination using Fivetran. | ||
menuPosition: 0 | ||
--- | ||
|
||
| ||
|
||
# Convex Setup Guide {% typeBadge connector="convex" /%} {% availabilityBadge connector="convex" /%} | ||
|
||
Follow our setup guide to connect Fivetran to Convex as a destination. | ||
|
||
Note that Convex can also be set up as a [source](/docs/databases/convex) | ||
|
||
--- | ||
|
||
| ||
|
||
## Prerequisites | ||
|
||
To connect your Convex deployment to Fivetran, you need the following: | ||
- A [Convex account](https://dashboard.convex.dev) | ||
- A Convex deployment. See [Convex's documentation](https://docs.convex.dev/) to get started. | ||
- Your Convex deployment's URL (e.g., `https://jaded-raven-991.convex.cloud`) | ||
- Your Convex deployment's deploy key. You can find both the deployment URL and deploy key on the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings) page. | ||
|
||
--- | ||
|
||
| ||
|
||
## Setup instructions | ||
|
||
| ||
|
||
### <span class="step-item">Locate your Deployment Credentials</span> | ||
|
||
1. Navigate to your deployment on the [Convex Dashboard](https://dashboard.convex.dev/). | ||
2. Navigate to the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings). | ||
3. Find your deployment URL and deploy key and make a note of them. You will need them to configure Fivetran. | ||
|
||
### <span class="step-item">Setup Fivetran Destination</span> | ||
|
||
1. Log into your fivetran account | ||
2. Go to the destinations page and click "Add Destination". | ||
3. Select Convex as the destination type | ||
4. Enter your deployment credentials. | ||
5. Click **Save & Test**. Fivetran tests and validates connection to your Convex deployment. | ||
|
||
### Configuring the `fivetran_metadata` connector | ||
|
||
### Creating a new connector | ||
|
||
1. From fivetran dashboard, click Connectors -> Add Connector | ||
2. Select and configure a data source of your choice. | ||
3. Select your Convex destination that you created above. | ||
4. Start your initial sync by clicking "Sync Now". | ||
5. The initial sync may fail with a user action to update [convex/schema.ts](https://docs.convex.dev/database/schemas). Follow the instructions in the error message to update your `schema.ts`. The error message will give you code to copy-paste into your `schema.ts`. It may take around 20 minutes for the error to appear. | ||
6. Deploy your `schema.ts` with `npx convex deploy`. | ||
7. Retry your initial sync by clicking "Sync Now" | ||
|
||
A sample schema.ts for a source with a single table `email.cars` - may look like this. | ||
```typescript | ||
import { defineSchema, defineTable } from "convex/server"; | ||
import { v } from "convex/values"; | ||
|
||
const fivetranTables = { | ||
email_cars: defineTable({ | ||
description: v.union(v.string(), v.null()), | ||
fivetran: v.object({ | ||
columns: v.object({ | ||
directory: v.union(v.string(), v.null()), | ||
file: v.union(v.string(), v.null()), | ||
line: v.union(v.int64(), v.null()), | ||
modified: v.union(v.float64(), v.null()), | ||
}), | ||
synced: v.float64(), | ||
}), | ||
make: v.union(v.string(), v.null()), | ||
model: v.union(v.string(), v.null()), | ||
price: v.union(v.float64(), v.null()), | ||
year: v.union(v.int64(), v.null()), | ||
}) | ||
.index("by_fivetran_synced", ["fivetran.synced"]) | ||
.index("by_primary_key", [ | ||
"fivetran.columns.directory", | ||
"fivetran.columns.file", | ||
"fivetran.columns.line", | ||
"fivetran.columns.modified", | ||
]), | ||
}; | ||
|
||
export default defineSchema({ | ||
...fivetranTables, | ||
}); | ||
``` | ||
|
||
--- | ||
|
||
## Related articles | ||
|
||
| ||
[<i aria-hidden="true" class="material-icons">description</i> Destination Connector Overview](/docs/destinations/convex_destination) | ||
[<i aria-hidden="true" class="material-icons">description</i> Source Connector Overview](/docs/databases/convex) | ||
<b> </b> | ||
[<i aria-hidden="true" class="material-icons">home</i> Documentation Home](/docs/getting-started) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters