Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/_snippets/clickpipes/bigquery/_permissions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import cp_iam from '@site/static/images/integrations/data-ingestion/clickpipes/bigquery/cp_iam.png';
import Image from '@theme/IdealImage';

ClickPipes authenticates to your Google Cloud project using a [service account key](https://docs.cloud.google.com/iam/docs/keys-create-delete). We recommend creating a dedicated service account with the minimum required set of [permissions](#permissions) to allow ClickPipes to export data from BigQuery, load it into the staging GCS bucket, and read it into ClickHouse.

To create a service account:

1. Select **IAM and admin** from the navigation menu in the Google Cloud console
2. Select **Service accounts**
3. Click **Create service account**

<Image img={cp_iam} alt="Creating a service account key with BigQuery and Cloud Storage permissions" size="lg" border/>

### Service account permissions {#permissions}

The following service account permissions are required:

#### BigQuery {#bigquery}

The service account must have the following BigQuery roles:

* [`roles/bigquery.dataViewer`](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer)
* [`roles/bigquery.jobUser`](https://docs.cloud.google.com/bigquery/docs/access-control#bigquery.jobUser)

To further scope access, we recommend using [IAM conditions](https://docs.cloud.google.com/bigquery/docs/conditions) to restrict the resources the role has access to. For example, you can restrict the `dataViewer` role to the specific dataset containing the tables you want to sync:

```plaintext
resource.name.startsWith("projects/<PROJECT_ID>/datasets/<DATASET_NAME>")
```

#### Cloud Storage {#cloud-storage}

The service account must have the following Cloud Storage roles:

* [`roles/storage.objectAdmin`](https://docs.cloud.google.com/storage/docs/access-control/iam-roles#storage.objectAdmin)
* [`roles/storage.bucketViewer`](https://docs.cloud.google.com/storage/docs/access-control/iam-roles#storage.bucketViewer)

To further scope access, we recommend using [IAM conditions](https://docs.cloud.google.com/bigquery/docs/conditions) to restrict the resources the role has access to. For example, you can restrict the `objectAdmin` and `bucketViewer` roles to the dedicated bucket created for ClickPipes syncs.

```plaintext
resource.name.startsWith("projects/_/buckets/<BUCKET_NAME>")
```
11 changes: 11 additions & 0 deletions docs/_snippets/clickpipes/bigquery/_select-data-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
import cp_step0 from '@site/static/images/integrations/data-ingestion/clickpipes/cp_step0.png';
import cp_step1 from '@site/static/images/integrations/data-ingestion/clickpipes/bigquery/cp_step1.png';
import Image from '@theme/IdealImage';

**1.** In ClickHouse Cloud, select **Data sources** in the main navigation menu and click **Create ClickPipe**.

<Image img={cp_step0} alt="Select imports" size="lg" border/>

**2.** Click the **BigQuery** tile.

<Image img={cp_step1} alt="Select BigQuery tile" size="lg" border/>
6 changes: 6 additions & 0 deletions docs/_snippets/clickpipes/bigquery/_service-account-key.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Depending on the migration approach you plan to use, you will need to create one of the following credentials:

| Migration approach | Credential type | Documentation | Notes |
|------------------------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| ClickPipes for CDC | Service key | [Create a service key](https://docs.cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) | Make sure to select **JSON** rather than **P12** as the **Key type**. |
| GCS bulk-load | `HMAC` key | [Create an `HMAC` key](https://docs.cloud.google.com/storage/docs/authentication/managing-hmackeys) | - |
14 changes: 14 additions & 0 deletions docs/_snippets/clickpipes/bigquery/_setup-connection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import cp_step2 from '@site/static/images/integrations/data-ingestion/clickpipes/bigquery/cp_step2.png';
import Image from '@theme/IdealImage';

To set up a new ClickPipe, you must provide details on how to connect to and authenticate with your BigQuery data warehouse, and a staging GCS bucket.

**1.** Upload the `.json` key for the service account you created for ClickPipes. Ensure the service account has the minimum required set of [permissions](/integrations/clickpipes/bigquery/overview#permissions).

<Image img={cp_step2} alt="Upload service account key" size="lg" border/>

**2.** Select the **Replication method**. In Private Preview, the only supported option is [**Initial load only**](/integrations/clickpipes/bigquery/overview#initial-load).

**3.** Provide the path to the GCS bucket for staging data during the initial load.

**4.** Click **Next** to validate.
Loading
Loading