Source versioning: Postgres, MySQL and Load generator #647

bobbyiliev · 2024-09-03T09:40:17Z

Initial implementation for the source versioning refactor as per #646

The main changes to consider:

Marking the table attribute as optional and deprecated for both the MySQL and Postgres sources
Introduced a new all_tables bool attribute for the MySQL and the Loadgen sources, as in the past this was defaulting always using FOR ALL TABLES in the load gen sources (auction, marketing, tpch) and in the MySQL case whenever no table blocks were defined, we defaulted to FOR ALL TABLES. This all_tables bool attribute allows us to create sources without any tables defined as per the source versioning work
Introducing the new materialize_source_table_{mysql|postgres|load_generator} resource which allows us to do CREATE TABLE ... FROM SOURCE ...

Things that are still pending: #646

morsapaes · 2024-09-11T15:21:15Z

docs/resources/source_table.md

@@ -29,7 +29,7 @@ description: |-
 - `ownership_role` (String) The owernship role of the object.
 - `region` (String) The region to use for the resource connection. If not set, the default region is used.
 - `schema_name` (String) The identifier for the table schema in Materialize. Defaults to `public`.
- `text_columns` (List of String) Columns to be decoded as text.
+- `text_columns` (List of String) Columns to be decoded as text. Not supported for the load generator sources, if the source is a load generator, the attribute will be ignored.


Similar to sources, we might want a source table resource per source type? The existing source-level options will basically shift to source table-level.

Yes indeed, I was just thinking about this. With the MySQL and Postgres sources, it is probably fine, but as soon as we add Kafka and Webhook sources, the logic will get out of hand.

Will refactor this to have a separate source table resource per source!

rjobanp

nice work!

rjobanp · 2024-09-16T13:07:37Z

docs/resources/source_kafka.md

+- `start_offset` (List of Number, Deprecated) Read partitions from the specified offset. Deprecated: Use the new materialize_source_table_kafka resource instead.
+- `start_timestamp` (Number, Deprecated) Use the specified value to set `START OFFSET` based on the Kafka timestamp. Deprecated: Use the new materialize_source_table_kafka resource instead.


these two options are still currently only possible on the top-level CREATE SOURCE statement for kafka sources -- not yet on a per-table basis. It will require a non-trivial amount more refactoring to allow them on a per-table basis so I'm unsure if we will do that work until it's requested by a customer

Ah yes! Good catch! Thank you!

rjobanp · 2024-09-16T13:08:26Z

docs/resources/source_mysql.md

@@ -53,12 +53,12 @@ resource "materialize_source_mysql" "test" {
 - `comment` (String) **Public Preview** Comment on an object in the database.
 - `database_name` (String) The identifier for the source database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
 - `expose_progress` (Block List, Max: 1) The name of the progress collection for the source. If this is not specified, the collection will be named `<src_name>_progress`. (see [below for nested schema](#nestedblock--expose_progress))
- `ignore_columns` (List of String, Deprecated) Ignore specific columns when reading data from MySQL. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table resource instead.
+- `ignore_columns` (List of String, Deprecated) Ignore specific columns when reading data from MySQL. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_mysql resource instead.


fyi this option is also being renamed MaterializeInc/materialize#29438 but the old name will be aliased to the new one, so this shouldn't break

Sounds good! I will go ahead and use the exclude columns for the new table source resource!

rjobanp · 2024-09-16T13:09:12Z

docs/resources/source_table_kafka.md

+- `start_offset` (List of Number) Read partitions from the specified offset.
+- `start_timestamp` (Number) Use the specified value to set `START OFFSET` based on the Kafka timestamp.


these aren't currently available on a per-table basis for kafka sources

rjobanp · 2024-09-16T13:09:34Z

docs/resources/source_table_kafka.md

+- `schema_name` (String) The identifier for the source schema in Materialize. Defaults to `public`.
+- `start_offset` (List of Number) Read partitions from the specified offset.
+- `start_timestamp` (Number) Use the specified value to set `START OFFSET` based on the Kafka timestamp.
+- `upstream_schema_name` (String) The schema of the table in the upstream database.


what does this refer to for kafka sources? we might just want to omit it since the upstream reference should just be the kafka topic name

Good catch, this was an overlook on my end in the schema for the Kafka source table resource.

rjobanp · 2024-09-16T13:10:49Z

pkg/materialize/source_table_kafka.go

+	startOffset      []int
+	startTimestamp   int


these two aren't used below and also aren't possible on the statement

rjobanp · 2024-09-16T13:13:55Z

docs/guides/materialize_source_table.md

+
+This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source}` resource.
+
+For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process.


Suggested change

For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process.

For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create at least one `materialize_source_table_kafka` table to hold data for the kafka topic.

@morsapaes might have better wording for this but I think we should be clear that this migration needs to happen for sources that previously didn't have subsources too (e.g. kafka)

rjobanp · 2024-09-16T13:15:49Z

docs/guides/materialize_source_table.md

+
+The same approach can be used for other source types such as Postgres, eg. `materialize_source_table_postgres`.
+
+## Automated Migration Process (TBD)


nice - this is great! We will probably want to figure out how to tell them that they will be able to coordinate the 'automated' migration process with their field-engineer representative if they go down this path

rjobanp · 2024-09-17T14:16:09Z

@morsapaes @bobbyiliev let's discuss this PR at the sources & sinks meeting this week - we should decide when it makes sense to merge this - my thinking is we should do so whenever we move into private preview for the source versioning feature. But if we want to merge sooner and just have a disclaimer that the things mentioned as 'deprecated' here are not actually yet deprecated, that could work too

bobbyiliev · 2024-09-30T07:19:16Z

One thing that we can consider here as per this old tracking issue: #391 is take the chance and decide if we still want to rename some of the attributes in the new source table resources:

Lists For attributes that use list, we have more cases of singular than plural.

Attribute	Resource	Type	Plural	Comment
start_offset	materialize_source_kafka	List of Strings		In Materialize the attribute is singular `START OFFSET` even though it is a list of strings
header	materialize_source_kafka	List of Strings		In Materialize the attribute is singular `HEADER` even though it is a list of strings
text_columns	materialize_source_postgres	List of Strings	X

Blocks We had decided should be singular. There are some blocks that use plural so this could be a good chance to rename those attributes in the new source table load gen resource:

Attribute	Resource	Type	Plural
auction_options	materialize_source_load_generator	Block	X
counter_options	materialize_source_load_generator	Block	X
marketing_options	materialize_source_load_generator	Block	X
tpch_options	materialize_source_load_generator	Block	X
check_options	materialize_webhook	Block	X

arusahni

Don't let the ~51 comments scare you off -- they're mostly nits. Great job on this!!!

arusahni · 2024-10-11T14:48:25Z

pkg/materialize/source_table_kafka.go

+				options = append(options, fmt.Sprintf(`FORMAT AVRO USING CONFLUENT SCHEMA REGISTRY CONNECTION %s`, QualifiedName(b.format.Avro.SchemaRegistryConnection.DatabaseName, b.format.Avro.SchemaRegistryConnection.SchemaName, b.format.Avro.SchemaRegistryConnection.Name)))
+			}
+			if b.format.Avro.KeyStrategy != "" {
+				options = append(options, fmt.Sprintf(`KEY STRATEGY %s`, b.format.Avro.KeyStrategy))


Should we be quoting/escaping this?

Here we would not need to escape this is a keyword. If we were to escape it it would result in the following error on the Materialize side:

Expected one of ID or LATEST or INLINE, found string literal "INLINE"

We do however have validation in place on the terraform schema side already, so a user will only be able to specify one of the following values:

terraform-provider-materialize/pkg/resources/config.go

Lines 58 to 62 in 6831919

var strategy = []string{

"INLINE",

"ID",

"LATEST",

}

arusahni · 2024-10-11T14:48:34Z

pkg/materialize/source_table_kafka.go

+				options = append(options, fmt.Sprintf(`KEY STRATEGY %s`, b.format.Avro.KeyStrategy))
+			}
+			if b.format.Avro.ValueStrategy != "" {
+				options = append(options, fmt.Sprintf(`VALUE STRATEGY %s`, b.format.Avro.ValueStrategy))


Should we be quoting/escaping this?

Same as above:

Here we would not need to escape this is a keyword. If we were to escape it it would result in the following error on the Materialize side:

Expected one of ID or LATEST or INLINE, found string literal "INLINE"

We do however have validation in place on the terraform schema side already, so a user will only be able to specify one of the following values:

terraform-provider-materialize/pkg/resources/config.go

Lines 58 to 62 in 6831919

var strategy = []string{

"INLINE",

"ID",

"LATEST",

}

arusahni · 2024-10-11T14:50:34Z

pkg/materialize/source_table_kafka.go

+
+		if b.format.Protobuf != nil {
+			if b.format.Protobuf.SchemaRegistryConnection.Name != "" && b.format.Protobuf.MessageName != "" {
+				options = append(options, fmt.Sprintf(`FORMAT PROTOBUF MESSAGE '%s' USING CONFLUENT SCHEMA REGISTRY CONNECTION %s`, b.format.Protobuf.MessageName, QualifiedName(b.format.Protobuf.SchemaRegistryConnection.DatabaseName, b.format.Protobuf.SchemaRegistryConnection.SchemaName, b.format.Protobuf.SchemaRegistryConnection.Name)))


We should probably quote the MessageName

arusahni · 2024-10-11T14:52:51Z

pkg/materialize/source_table_kafka.go

+				options = append(options, fmt.Sprintf(`FORMAT CSV WITH %d COLUMNS`, b.format.Csv.Columns))
+			}
+			if b.format.Csv.Header != nil {
+				options = append(options, fmt.Sprintf(`FORMAT CSV WITH HEADER ( %s )`, strings.Join(b.format.Csv.Header, ", ")))
+			}
+			if b.format.Csv.DelimitedBy != "" {
+				options = append(options, fmt.Sprintf(`DELIMITER '%s'`, b.format.Csv.DelimitedBy))


Should we be quoting/escaping these?

the format.Csv.Columns is of TypeInt so no need to quote it, but good catch for the other ones! 🙇

arusahni · 2024-10-11T15:00:16Z

pkg/materialize/source_table_kafka.go

+
+		if b.keyFormat.Protobuf != nil {
+			if b.keyFormat.Protobuf.SchemaRegistryConnection.Name != "" && b.keyFormat.Protobuf.MessageName != "" {
+				options = append(options, fmt.Sprintf(`KEY FORMAT PROTOBUF MESSAGE '%s' USING CONFLUENT SCHEMA REGISTRY CONNECTION %s`, b.keyFormat.Protobuf.MessageName, QualifiedName(b.keyFormat.Protobuf.SchemaRegistryConnection.DatabaseName, b.keyFormat.Protobuf.SchemaRegistryConnection.SchemaName, b.keyFormat.Protobuf.SchemaRegistryConnection.Name)))


We should quote the MessageName

arusahni · 2024-10-11T20:36:40Z

pkg/resources/resource_source_mysql.go

+		Description: "Specify the tables to be included in the source. Deprecated: Use the new materialize_source_table_mysql resource instead.",
+		Deprecated:  "Use the new materialize_source_table_mysql resource instead.",


Suggested change

Description: "Specify the tables to be included in the source. Deprecated: Use the new materialize_source_table_mysql resource instead.",

Deprecated: "Use the new materialize_source_table_mysql resource instead.",

Description: "Specify the tables to be included in the source. Deprecated: Use the new `materialize_source_table_mysql` resource instead.",

Deprecated: "Use the new `materialize_source_table_mysql` resource instead.",

arusahni · 2024-10-11T20:37:06Z

pkg/resources/resource_source_mysql.go

@@ -76,6 +79,13 @@ var sourceMySQLSchema = map[string]*schema.Schema{
 			},
 		},
 	},
+	"all_tables": {
+		Description: "Include all tables in the source. If `table` is specified, this will be ignored.",
+		Deprecated:  "Use the new materialize_source_table_mysql resource instead.",


Suggested change

Deprecated: "Use the new materialize_source_table_mysql resource instead.",

Deprecated: "Use the new `materialize_source_table_mysql` resource instead.",

arusahni · 2024-10-11T20:37:38Z

pkg/resources/resource_source_postgres.go

+		Description: "Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_postgres resource instead.",
+		Deprecated:  "Use the new materialize_source_table_postgres resource instead.",


Suggested change

Description: "Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_postgres resource instead.",

Deprecated: "Use the new materialize_source_table_postgres resource instead.",

Description: "Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new `materialize_source_table_postgres` resource instead.",

Deprecated: "Use the new `materialize_source_table_postgres` resource instead.",

arusahni · 2024-10-11T20:37:56Z

pkg/resources/resource_source_postgres.go

+		Description: "Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new materialize_source_table_postgres resource instead.",
+		Deprecated:  "Use the new materialize_source_table_postgres resource instead.",


Suggested change

Description: "Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new materialize_source_table_postgres resource instead.",

Deprecated: "Use the new materialize_source_table_postgres resource instead.",

Description: "Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new `materialize_source_table_postgres` resource instead.",

Deprecated: "Use the new `materialize_source_table_postgres` resource instead.",

arusahni · 2024-10-11T20:39:25Z

pkg/resources/resource_source_table_mysql.go

+		ForceNew:    true,
+	},
+	"exclude_columns": {
+		Description: "Exclude specific columns when reading data from MySQL. The option used to be called `ignore_columns`.",


Suggested change

Description: "Exclude specific columns when reading data from MySQL. The option used to be called `ignore_columns`.",

Description: "Exclude specific columns when reading data from MySQL. This option used to be called `ignore_columns`.",

rjobanp

looks solid! Just a few minor comments on some things that have been updated, and one about the migration guide

rjobanp · 2024-10-18T15:21:16Z

docs/data-sources/source_reference.md

+# generated by https://github.com/hashicorp/terraform-plugin-docs
+page_title: "materialize_source_reference Data Source - terraform-provider-materialize"
+subcategory: ""
+description: |-


Can we populate this to explain that these are 'available' source references? Such that these expose all the possible upstream references that this source can create a table for, not necessarily all the references it is already ingesting

rjobanp · 2024-10-18T15:26:10Z

docs/guides/materialize_source_table.md

+
+This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source_type}` resource.
+
+For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source_type}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create at least one `materialize_source_table_kafka` table to hold data for the kafka topic.


Suggested change

For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source_type}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create at least one `materialize_source_table_kafka` table to hold data for the kafka topic.

For each MySQL and Postgres source, you will need to create a new `materialize_source_table_{source_type}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create a `materialize_source_table_kafka` table with the same name as the kafka source to contain the data for the kafka topic.

rjobanp · 2024-10-18T15:31:54Z

docs/guides/materialize_source_table.md

+
+In previous versions of the Materialize Terraform provider, source tables were defined within the source resource itself and were considered subsources of the source rather than separate entities.
+
+This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source_type}` resource.


I'm confused on whether this guide is meant to explain how to migrate using just the manual process (which is outlined below and looks correct), or also to explain how to reconcile your terraform configuration with the results of the 'automatic' migration process that we would do for a customer with a catalog migration. Specifically, the paragraph below sounds like it is explaining what to do to handle the automatic migration process, but it doesn't explain that the manual process would then be unnecessary.

It might be simpler if this guide just explained the manual process and we had a separate one to explain the 'automatic' process. Or we could just assume any terraform user would only do the manual process. @morsapaes thoughts?

Good point! I've removed some of the confusing text so that this should focus on the manual migration!

Happy to work on a follow up migration guide for the automated migration if we decide to do that!

rjobanp · 2024-10-18T15:32:24Z

docs/guides/materialize_source_table.md

+
+## Future Improvements
+
+The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them.


Suggested change

The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them.

Webhooks sources have not yet been migrated to the new model. Once this changes, the migration process will be updated to include them.

rjobanp · 2024-10-18T15:35:31Z

docs/resources/source_table_kafka.md

+
+- `name` (String) The identifier for the source table.
+- `source` (Block List, Min: 1, Max: 1) The source this table is created from. (see [below for nested schema](#nestedblock--source))
+- `topic` (String) The name of the Kafka topic in the Kafka cluster.


this is actually no longer necessary to include for Kafka sources -- if the (REFERENCE ..) option in the statement is omitted it will still work since there is only one possible kafka topic that the table can reference

rjobanp · 2024-10-18T15:38:31Z

pkg/materialize/source_reference.go

+	p := map[string]string{
+		"sr.source_id": sourceId,
+	}
+	q := sourceReferenceQuery.QueryPredicate(p)


this would be a bonus, but there is a PR to implement a REFRESH SOURCE REFERENCES <source> statement here MaterializeInc/materialize#29923 and we could automatically run that before listing the references which would mean we always get the most 'up to date' view of them

Just implemented this!

terraform-provider-materialize/pkg/materialize/source_reference.go

Lines 65 to 69 in 147606a

func refreshSourceReferences(conn *sqlx.DB, sourceName, schemaName, databaseName string) error {

query := fmt.Sprintf(`ALTER SOURCE %s REFRESH REFERENCES`, QualifiedName(databaseName, schemaName, sourceName))

_, err := conn.Exec(query)

return err

}

rjobanp · 2024-10-18T15:39:16Z

pkg/materialize/source_table.go

+	q.WriteString(` (REFERENCE `)
+
+	if b.upstreamSchemaName != "" {
+		q.WriteString(fmt.Sprintf(`%s.`, QuoteIdentifier(b.upstreamSchemaName)))
+	}
+	q.WriteString(QuoteIdentifier(b.upstreamName))
+
+	q.WriteString(")")


mentioned above that this can be made optional for kafka and single-output load generator sources

rjobanp · 2024-10-18T15:42:21Z

pkg/materialize/source_table_kafka.go

+		mz_sources.name AS source_name,
+		source_schemas.name AS source_schema_name,
+		source_databases.name AS source_database_name,
+		mz_kafka_source_tables.topic AS upstream_table_name,


I have a PR open now to add the envelope, key_format and value_format columns to this table MaterializeInc/materialize#30076

Neat! I created a tracking issue for this for the moment: #665

Should be ok to handle in a follow up PR later on!

ParkMyCar

Really just skimmed the entire PR but I think it all looks good! Biggest feedback would be following up on quoting identifiers that Aru commented on, but if there aren't any concerns there I would say we're good to go!

ParkMyCar · 2024-11-13T22:43:10Z

pkg/provider/acceptance_source_table_kafka_test.go

+					resource.TestCheckResourceAttr("materialize_source_table_kafka.test_kafka", "name", nameSpace+"_table_kafka"),
+					resource.TestCheckResourceAttr("materialize_source_table_kafka.test_kafka", "database_name", "materialize"),
+					resource.TestCheckResourceAttr("materialize_source_table_kafka.test_kafka", "schema_name", "public"),
+					// resource.TestCheckResourceAttr("materialize_source_table_kafka.test_kafka", "qualified_sql_name", fmt.Sprintf(`"materialize"."public"."%s_table_kafka"`, nameSpace)),


left behind?

…orking

bobbyiliev changed the title ~~Source versioning~~ [WIP] Source versioning Sep 3, 2024

bobbyiliev mentioned this pull request Sep 3, 2024

Add source versioning design doc #645

Merged

bobbyiliev force-pushed the source-versioning branch 3 times, most recently from c4a61c8 to e202851 Compare September 9, 2024 13:33

morsapaes reviewed Sep 11, 2024

View reviewed changes

bobbyiliev changed the title ~~[WIP] Source versioning~~ Source versioning: Postgres, MySQL and Load generator Sep 13, 2024

bobbyiliev marked this pull request as ready for review September 13, 2024 12:29

bobbyiliev requested a review from a team as a code owner September 13, 2024 12:29

bobbyiliev requested review from arusahni and rjobanp and removed request for a team September 13, 2024 12:29

rjobanp reviewed Sep 16, 2024

View reviewed changes

bobbyiliev force-pushed the source-versioning branch from 0b63a29 to 7870d48 Compare September 17, 2024 13:14

bobbyiliev force-pushed the source-versioning branch 3 times, most recently from 61cd55c to 9b96939 Compare September 24, 2024 09:43

bobbyiliev force-pushed the source-versioning branch from a467a3e to b0ee9ba Compare October 6, 2024 11:22

arusahni approved these changes Oct 11, 2024

View reviewed changes

bobbyiliev force-pushed the source-versioning branch from af8d738 to 232787a Compare October 15, 2024 11:12

rjobanp approved these changes Oct 18, 2024

View reviewed changes

bobbyiliev force-pushed the source-versioning branch 2 times, most recently from 36cd7d1 to 147606a Compare October 28, 2024 19:50

ParkMyCar approved these changes Nov 13, 2024

View reviewed changes

bobbyiliev added 4 commits November 18, 2024 10:37

Source versioning initial implementation

386208b

Use source table instead of table from source

cff207b

MySQL source: separate for tables and all tables

0da3d90

Loadgen source: add all tables bool attr

d4e0179

bobbyiliev added 28 commits November 18, 2024 10:37

Review updates

217b00c

Update guide migration guide

d58981e

Update guide migration guide

95f929f

Add import examples for Kafka source tables

74f4872

Add upstream mysql and postgres table names

9c83cf2

Fix unit tests

586a33f

Add Kafka upstream references

b3bfb63

Add integration tests

845c630

Fix failing test

6a761da

Extend data source to include upstream names

7dcfbc4

Small updates

b5eef55

Switch back to latest image

e1b0799

FromAsCasing: 'as' and 'FROM' keywords' casing do not match

c5b9832

Add source reference data source

d4dddf2

Add source reference data source example

362316e

First round of the initial PR change requests

7a05395

Fix failing tests

7515e5e

Fix failing tests

963c1ce

Second round of the initial PR change requests

6ec2839

Add unit tests to data source source reference

155765c

PR change requests

9c24845

Fix failing tests

0c2b229

Fix failing tests

d59cb8e

Add alter source refresh to data source

c66b7d3

Add new kafk acolumns from mz_kafka_source_tables

d06f029

Fix failing MockSourceTableKafkaScan test

697caaf

Remove confusing line from migration guide

e9c56f1

Remove a left behind comment

b411b09

bobbyiliev force-pushed the source-versioning branch from 8d433e5 to b411b09 Compare November 18, 2024 08:37

explicitly enable create table from source as --all-features is not w…

99d66f5

…orking

		- `start_offset` (List of Number, Deprecated) Read partitions from the specified offset. Deprecated: Use the new materialize_source_table_kafka resource instead.
		- `start_timestamp` (Number, Deprecated) Use the specified value to set `START OFFSET` based on the Kafka timestamp. Deprecated: Use the new materialize_source_table_kafka resource instead.

		- `start_offset` (List of Number) Read partitions from the specified offset.
		- `start_timestamp` (Number) Use the specified value to set `START OFFSET` based on the Kafka timestamp.


		This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source}` resource.

		For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process.


		The same approach can be used for other source types such as Postgres, eg. `materialize_source_table_postgres`.

		## Automated Migration Process (TBD)

		Description: "Specify the tables to be included in the source. Deprecated: Use the new materialize_source_table_mysql resource instead.",
		Deprecated: "Use the new materialize_source_table_mysql resource instead.",

	Deprecated: "Use the new materialize_source_table_mysql resource instead.",
	Deprecated: "Use the new `materialize_source_table_mysql` resource instead.",

		Description: "Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_postgres resource instead.",
		Deprecated: "Use the new materialize_source_table_postgres resource instead.",

		Description: "Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new materialize_source_table_postgres resource instead.",
		Deprecated: "Use the new materialize_source_table_postgres resource instead.",

	Description: "Exclude specific columns when reading data from MySQL. The option used to be called `ignore_columns`.",
	Description: "Exclude specific columns when reading data from MySQL. This option used to be called `ignore_columns`.",


		In previous versions of the Materialize Terraform provider, source tables were defined within the source resource itself and were considered subsources of the source rather than separate entities.

		This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source_type}` resource.


		## Future Improvements

		The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them.

	The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them.
	Webhooks sources have not yet been migrated to the new model. Once this changes, the migration process will be updated to include them.

	func refreshSourceReferences(conn *sqlx.DB, sourceName, schemaName, databaseName string) error {
	query := fmt.Sprintf(`ALTER SOURCE %s REFRESH REFERENCES`, QualifiedName(databaseName, schemaName, sourceName))
	_, err := conn.Exec(query)
	return err
	}

Source versioning: Postgres, MySQL and Load generator #647

Are you sure you want to change the base?

Source versioning: Postgres, MySQL and Load generator #647

Conversation

bobbyiliev commented Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rjobanp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rjobanp commented Sep 17, 2024 • edited Loading

bobbyiliev commented Sep 30, 2024

arusahni left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rjobanp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ParkMyCar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobbyiliev commented Sep 3, 2024 •

edited

Loading

rjobanp commented Sep 17, 2024 •

edited

Loading