Skip to content

Commit

Permalink
Add fivetran destination connector docs. (#28285)
Browse files Browse the repository at this point in the history
Added docs for the destination connector, as well as cross links about the source connector.

GitOrigin-RevId: 78d84fd65588ffa974da2c082e297c9e1160e959
  • Loading branch information
nipunn1313 authored and Convex, Inc. committed Jul 24, 2024
1 parent 450e6e2 commit 4e0e27a
Show file tree
Hide file tree
Showing 4 changed files with 199 additions and 7 deletions.
81 changes: 81 additions & 0 deletions crates/fivetran_destination/docs/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
name: Convex
title: Convex destination connector for Fivetran
description: Documentation and setup guide for the Convex destination connector for Fivetran
---

# Convex {% typeBadge connector="convex_destination" /%} {% availabilityBadge connector="convex_destination" /%}

[Convex](https://convex.dev) is the open-source backend for application builders.
An all-in-one platform with thoughtful, product-centric APIs.

Note that Convex can also be set up as a [source](/docs/databases/convex)

---

## Setup guide

Follow our [step-by-step Convex setup guide](/docs/destinations/convex_destination/setup-guide) to connect Convex as a destination with Fivetran.

---

## Sync overview

Once Fivetran is connected to your Convex destination, the connector will attempt to sync your data.
It may ask you to update your `convex/schema.ts` in your deployment to match the format of your source.
Once the `convex/schema.ts` matches, data will continu to sync.

---

## Configuration

You will need your deployment URL and deploy key in order to configure the Convex Connector for Fivetran. You can find both on your project's [Production Deployment Settings page](https://docs.convex.dev/dashboard/deployments/deployment-settings).

---

## Schema information

Fivetran tries to replicate the database and columns from your configured source to your destination Convex according to Fivetran's [standard database update strategies](/docs/databases#transformationandmappingoverview).

### Type transformations and mapping

As the connector extracts your data from your source, it matches the supported Fivetran types to [Convex data types](https://docs.convex.dev/database/types).

The following table illustrates how the connector transforms the Fivetran data types into Convex data types.

| Fivetran Type | Convex Type | Equivalence |
| ------------- | ----------- | ----------- |
| BOOLEAN | Boolean | Exact |
| SHORT | Float64 | Inexact |
| INT | Float64 | Inexact |
| LONG | Int64 | Exact |
| DECIMAL | String | Inexact |
| FLOAT | Float64 | Inexact |
| DOUBLE | Float64 | Exact |
| NAIVEDATE | String | Inexact |
| NAIVEDATETIME | String | Inexact |
| UTCDATETIME | Float64 | Inexact |
| BINARY | Bytes | Exact |
| STRING | String | Exact |
| NULL | Null | Exact |
| JSON | Object | Inexact |

> NOTE: Short/Int are converted to float64 for ease of use in javascript (as `number`). There is no data loss as Number.MAX_SAFE_INTEGER = 2^53 - 1.
> NOTE: Decimal is converted to string to ensure no data loss. Eg "1234.5678"
> NOTE: Naive date uses standard string representation of YYYY-MM-DD.
> NOTE: Naive datetime uses standard string representation of YYYY-MM-DD HH:MM:SS.
> NOTE: UTC datetime uses milliseconds since UNIX epoch
### Fivetran-generated data

Fivetran adds a single `fivetran` column containing a Convex object to the source data.
Some of the fields (`synced`, `deleted`) are for internal purposes.

### Table and column name transformations

If your source table is `default.cars`, then your convex table will be named `default_cars`.
Convex deployments do not have a concept of namespaced tables, so it uses this notation to preserve
the namespace information.

Columns names that begin with `_` are not supported in Convex. Instead, those columns are synced to the
destination nested within the `fivetran.columns` column. For example, a column named `_line` would end up in the `fivetran.columns.line` nested column.
106 changes: 106 additions & 0 deletions crates/fivetran_destination/docs/setup-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
---
name: Convex Destination Setup Guide
title: Convex Destination Connector Setup Guide
description: Read step-by-step instructions on how to connect your Convex deployment as a destination using Fivetran.
menuPosition: 0
---


# Convex Setup Guide {% typeBadge connector="convex" /%} {% availabilityBadge connector="convex" /%}

​ Follow our setup guide to connect Fivetran to Convex as a destination. ​

Note that Convex can also be set up as a [source](/docs/databases/convex)

---


## Prerequisites

​To connect your Convex deployment to Fivetran, you need the following:
- A [Convex account](https://dashboard.convex.dev)
- A Convex deployment. See [Convex's documentation](https://docs.convex.dev/) to get started.
- Your Convex deployment's URL (e.g., `https://jaded-raven-991.convex.cloud`)
- Your Convex deployment's deploy key. You can find both the deployment URL and deploy key on the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings) page. ​

---


## Setup instructions


### <span class="step-item">Locate your Deployment Credentials</span>

1. Navigate to your deployment on the [Convex Dashboard](https://dashboard.convex.dev/).​
2. Navigate to the [Production Deployment Settings](https://docs.convex.dev/dashboard/deployments/deployment-settings).
3. Find your deployment URL and deploy key and make a note of them. You will need them to configure Fivetran.

### <span class="step-item">Setup Fivetran Destination</span>

1. Log into your fivetran account
2. Go to the destinations page and click "Add Destination".
3. Select Convex as the destination type
4. Enter your deployment credentials.
5. Click **Save & Test**. Fivetran tests and validates connection to your Convex deployment. ​

### Configuring the `fivetran_metadata` connector

### Creating a new connector

1. From fivetran dashboard, click Connectors -> Add Connector
2. Select and configure a data source of your choice.
3. Select your Convex destination that you created above.
4. Start your initial sync by clicking "Sync Now".
5. The initial sync may fail with a user action to update [convex/schema.ts](https://docs.convex.dev/database/schemas). Follow the instructions in the error message to update your `schema.ts`. The error message will give you code to copy-paste into your `schema.ts`. It may take around 20 minutes for the error to appear.
6. Deploy your `schema.ts` with `npx convex deploy`.
7. Retry your initial sync by clicking "Sync Now"

A sample schema.ts for a source with a single table `email.cars` - may look like this.
```typescript
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

const fivetranTables = {
email_cars: defineTable({
description: v.union(v.string(), v.null()),
fivetran: v.object({
columns: v.object({
directory: v.union(v.string(), v.null()),
file: v.union(v.string(), v.null()),
line: v.union(v.int64(), v.null()),
modified: v.union(v.float64(), v.null()),
}),
synced: v.float64(),
}),
make: v.union(v.string(), v.null()),
model: v.union(v.string(), v.null()),
price: v.union(v.float64(), v.null()),
year: v.union(v.int64(), v.null()),
})
.index("by_fivetran_synced", ["fivetran.synced"])
.index("by_primary_key", [
"fivetran.columns.directory",
"fivetran.columns.file",
"fivetran.columns.line",
"fivetran.columns.modified",
]),
};

export default defineSchema({
...fivetranTables,
});
```

---

## Related articles

[<i aria-hidden="true" class="material-icons">description</i> Destination Connector Overview](/docs/destinations/convex_destination)
[<i aria-hidden="true" class="material-icons">description</i> Source Connector Overview](/docs/databases/convex)
​ <b> </b> ​
[<i aria-hidden="true" class="material-icons">home</i> Documentation Home](/docs/getting-started)
10 changes: 6 additions & 4 deletions crates/fivetran_source/docs/overview.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
---
name: Convex
title: Convex connector for Fivetran
description: Documentation and setup guide for the Convex connector for Fivetran
title: Convex source connector for Fivetran
description: Documentation and setup guide for the Convex source connector for Fivetran
---

# Convex {% typeBadge connector="convex" /%} {% availabilityBadge connector="convex" /%}

[Convex](https://convex.dev) is a full-stack TypeScript development platform.
Replace your database, server functions, and glue code.
[Convex](https://convex.dev) is the open-source backend for application builders.
An all-in-one platform with thoughtful, product-centric APIs.

Note that Convex can also be set up as a [destination](/docs/destinations/convex_destination)

---

Expand Down
9 changes: 6 additions & 3 deletions crates/fivetran_source/docs/setup-guide.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: Convex Setup Guide
title: Convex Connector Setup Guide
name: Convex Database Connector Setup Guide
title: Convex Database Connector Setup Guide
description: Read step-by-step instructions on how to connect your Convex deployment with your destination using Fivetran connectors.
menuPosition: 0
---
Expand All @@ -11,6 +11,8 @@ menuPosition: 0

​ Follow our setup guide to connect Convex to Fivetran. ​

Note that Convex can also be set up as a [destination](/docs/destinations/convex)

---

Expand Down Expand Up @@ -58,6 +60,7 @@ Fivetran performs the following tests to ensure that we can connect to your Conv
## Related articles

[<i aria-hidden="true" class="material-icons">description</i> Connector Overview](/docs/databases/convex)
[<i aria-hidden="true" class="material-icons">description</i> Source Connector Overview](/docs/databases/convex)
[<i aria-hidden="true" class="material-icons">description</i> Destination Connector Overview](/docs/destinations/convex_destination)
​ <b> </b> ​
[<i aria-hidden="true" class="material-icons">home</i> Documentation Home](/docs/getting-started)

0 comments on commit 4e0e27a

Please sign in to comment.