From 4e0e27aedeaccb3f65928fde906c31f2616e4b39 Mon Sep 17 00:00:00 2001 From: Nipunn Koorapati Date: Wed, 24 Jul 2024 15:34:12 -0700 Subject: [PATCH] Add fivetran destination connector docs. (#28285) Added docs for the destination connector, as well as cross links about the source connector. GitOrigin-RevId: 78d84fd65588ffa974da2c082e297c9e1160e959 --- crates/fivetran_destination/docs/overview.md | 81 +++++++++++++ .../fivetran_destination/docs/setup-guide.md | 106 ++++++++++++++++++ crates/fivetran_source/docs/overview.md | 10 +- crates/fivetran_source/docs/setup-guide.md | 9 +- 4 files changed, 199 insertions(+), 7 deletions(-) create mode 100644 crates/fivetran_destination/docs/overview.md create mode 100644 crates/fivetran_destination/docs/setup-guide.md diff --git a/crates/fivetran_destination/docs/overview.md b/crates/fivetran_destination/docs/overview.md new file mode 100644 index 00000000..3ee2ce65 --- /dev/null +++ b/crates/fivetran_destination/docs/overview.md @@ -0,0 +1,81 @@ +--- +name: Convex +title: Convex destination connector for Fivetran +description: Documentation and setup guide for the Convex destination connector for Fivetran +--- + +# Convex {% typeBadge connector="convex_destination" /%} {% availabilityBadge connector="convex_destination" /%} + +[Convex](https://convex.dev) is the open-source backend for application builders. +An all-in-one platform with thoughtful, product-centric APIs. + +Note that Convex can also be set up as a [source](/docs/databases/convex) + +--- + +## Setup guide + +Follow our [step-by-step Convex setup guide](/docs/destinations/convex_destination/setup-guide) to connect Convex as a destination with Fivetran. + +--- + +## Sync overview + +Once Fivetran is connected to your Convex destination, the connector will attempt to sync your data. +It may ask you to update your `convex/schema.ts` in your deployment to match the format of your source. +Once the `convex/schema.ts` matches, data will continu to sync. + +--- + +## Configuration + +You will need your deployment URL and deploy key in order to configure the Convex Connector for Fivetran. You can find both on your project's [Production Deployment Settings page](https://docs.convex.dev/dashboard/deployments/deployment-settings). + +--- + +## Schema information + +Fivetran tries to replicate the database and columns from your configured source to your destination Convex according to Fivetran's [standard database update strategies](/docs/databases#transformationandmappingoverview). + +### Type transformations and mapping + +As the connector extracts your data from your source, it matches the supported Fivetran types to [Convex data types](https://docs.convex.dev/database/types). + +The following table illustrates how the connector transforms the Fivetran data types into Convex data types. + +| Fivetran Type | Convex Type | Equivalence | +| ------------- | ----------- | ----------- | +| BOOLEAN | Boolean | Exact | +| SHORT | Float64 | Inexact | +| INT | Float64 | Inexact | +| LONG | Int64 | Exact | +| DECIMAL | String | Inexact | +| FLOAT | Float64 | Inexact | +| DOUBLE | Float64 | Exact | +| NAIVEDATE | String | Inexact | +| NAIVEDATETIME | String | Inexact | +| UTCDATETIME | Float64 | Inexact | +| BINARY | Bytes | Exact | +| STRING | String | Exact | +| NULL | Null | Exact | +| JSON | Object | Inexact | + +> NOTE: Short/Int are converted to float64 for ease of use in javascript (as `number`). There is no data loss as Number.MAX_SAFE_INTEGER = 2^53 - 1. +> NOTE: Decimal is converted to string to ensure no data loss. Eg "1234.5678" +> NOTE: Naive date uses standard string representation of YYYY-MM-DD. +> NOTE: Naive datetime uses standard string representation of YYYY-MM-DD HH:MM:SS. +> NOTE: UTC datetime uses milliseconds since UNIX epoch + +### Fivetran-generated data + +Fivetran adds a single `fivetran` column containing a Convex object to the source data. +Some of the fields (`synced`, `deleted`) are for internal purposes. + +### Table and column name transformations + +If your source table is `default.cars`, then your convex table will be named `default_cars`. +Convex deployments do not have a concept of namespaced tables, so it uses this notation to preserve +the namespace information. + +Columns names that begin with `_` are not supported in Convex. Instead, those columns are synced to the +destination nested within the `fivetran.columns` column. For example, a column named `_line` would end up in the `fivetran.columns.line` nested column. diff --git a/crates/fivetran_destination/docs/setup-guide.md b/crates/fivetran_destination/docs/setup-guide.md new file mode 100644 index 00000000..d50cf3d5 --- /dev/null +++ b/crates/fivetran_destination/docs/setup-guide.md @@ -0,0 +1,106 @@ +--- +name: Convex Destination Setup Guide +title: Convex Destination Connector Setup Guide +description: Read step-by-step instructions on how to connect your Convex deployment as a destination using Fivetran. +menuPosition: 0 +--- + +​ + +# Convex Setup Guide {% typeBadge connector="convex" /%} {% availabilityBadge connector="convex" /%} + +​ Follow our setup guide to connect Fivetran to Convex as a destination. ​ + +Note that Convex can also be set up as a [source](/docs/databases/convex) + +--- + +​ + +## Prerequisites + +​To connect your Convex deployment to Fivetran, you need the following: +- A [Convex account](https://dashboard.convex.dev) +- A Convex deployment. See [Convex's documentation](https://docs.convex.dev/) to get started. +- Your Convex deployment's URL (e.g., `https://jaded-raven-991.convex.cloud`) +- Your Convex deployment's deploy key. You can find both the deployment URL and deploy key on the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings) page. ​ + +--- + +​ + +## Setup instructions + +​ + +### Locate your Deployment Credentials + +1. Navigate to your deployment on the [Convex Dashboard](https://dashboard.convex.dev/).​ +2. Navigate to the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings). +3. Find your deployment URL and deploy key and make a note of them. You will need them to configure Fivetran. + +### Setup Fivetran Destination + +1. Log into your fivetran account +2. Go to the destinations page and click "Add Destination". +3. Select Convex as the destination type +4. Enter your deployment credentials. +5. Click **Save & Test**. Fivetran tests and validates connection to your Convex deployment. ​ + +### Configuring the `fivetran_metadata` connector + +### Creating a new connector + +1. From fivetran dashboard, click Connectors -> Add Connector +2. Select and configure a data source of your choice. +3. Select your Convex destination that you created above. +4. Start your initial sync by clicking "Sync Now". +5. The initial sync may fail with a user action to update [convex/schema.ts](https://docs.convex.dev/database/schemas). Follow the instructions in the error message to update your `schema.ts`. The error message will give you code to copy-paste into your `schema.ts`. It may take around 20 minutes for the error to appear. +6. Deploy your `schema.ts` with `npx convex deploy`. +7. Retry your initial sync by clicking "Sync Now" + +A sample schema.ts for a source with a single table `email.cars` - may look like this. +```typescript +import { defineSchema, defineTable } from "convex/server"; +import { v } from "convex/values"; + +const fivetranTables = { + email_cars: defineTable({ + description: v.union(v.string(), v.null()), + fivetran: v.object({ + columns: v.object({ + directory: v.union(v.string(), v.null()), + file: v.union(v.string(), v.null()), + line: v.union(v.int64(), v.null()), + modified: v.union(v.float64(), v.null()), + }), + synced: v.float64(), + }), + make: v.union(v.string(), v.null()), + model: v.union(v.string(), v.null()), + price: v.union(v.float64(), v.null()), + year: v.union(v.int64(), v.null()), + }) + .index("by_fivetran_synced", ["fivetran.synced"]) + .index("by_primary_key", [ + "fivetran.columns.directory", + "fivetran.columns.file", + "fivetran.columns.line", + "fivetran.columns.modified", + ]), +}; + +export default defineSchema({ + ...fivetranTables, +}); +``` + +--- + +## Related articles + +​ +[ Destination Connector Overview](/docs/destinations/convex_destination) +[ Source Connector Overview](/docs/databases/convex) +​ ​ +[ Documentation Home](/docs/getting-started) diff --git a/crates/fivetran_source/docs/overview.md b/crates/fivetran_source/docs/overview.md index f82ad65f..417a6d50 100644 --- a/crates/fivetran_source/docs/overview.md +++ b/crates/fivetran_source/docs/overview.md @@ -1,13 +1,15 @@ --- name: Convex -title: Convex connector for Fivetran -description: Documentation and setup guide for the Convex connector for Fivetran +title: Convex source connector for Fivetran +description: Documentation and setup guide for the Convex source connector for Fivetran --- # Convex {% typeBadge connector="convex" /%} {% availabilityBadge connector="convex" /%} -[Convex](https://convex.dev) is a full-stack TypeScript development platform. -Replace your database, server functions, and glue code. +[Convex](https://convex.dev) is the open-source backend for application builders. +An all-in-one platform with thoughtful, product-centric APIs. + +Note that Convex can also be set up as a [destination](/docs/destinations/convex_destination) --- diff --git a/crates/fivetran_source/docs/setup-guide.md b/crates/fivetran_source/docs/setup-guide.md index 02191fd9..a9a937d2 100644 --- a/crates/fivetran_source/docs/setup-guide.md +++ b/crates/fivetran_source/docs/setup-guide.md @@ -1,6 +1,6 @@ --- -name: Convex Setup Guide -title: Convex Connector Setup Guide +name: Convex Database Connector Setup Guide +title: Convex Database Connector Setup Guide description: Read step-by-step instructions on how to connect your Convex deployment with your destination using Fivetran connectors. menuPosition: 0 --- @@ -11,6 +11,8 @@ menuPosition: 0 ​ Follow our setup guide to connect Convex to Fivetran. ​ +Note that Convex can also be set up as a [destination](/docs/destinations/convex) + --- ​ @@ -58,6 +60,7 @@ Fivetran performs the following tests to ensure that we can connect to your Conv ## Related articles ​ -[ Connector Overview](/docs/databases/convex) +[ Source Connector Overview](/docs/databases/convex) +[ Destination Connector Overview](/docs/destinations/convex_destination) ​ ​ [ Documentation Home](/docs/getting-started)