v4.6.4

Clickhouse & substreams-sink-sql from-proto

• An clickhouse_table_options option order_by_fields is required now when using substreams-sink-sql from-proto, for example you should go from:

  message Transfer {
    option (schema.table) = {
      name: "tranfers"
    };
  
    string id = 1;
    string from = 2;
    string to = 3;
  }

  To:

  message Transfer {
   option (schema.table) = {
     name: "tranfers",
     clickhouse_table_options: {
       order_by_fields: [
         { name: "id" }
       ]
     }
   };
  
   string id = 1;
   string from = 2;
   string to = 3;
  }

  The selected column will be used to as the ORDER BY <column> clause when generating the table schema using ReplacingMergeTree.

• Fixed generated schema column _blocks_.number to be of type Uint64.

• Fixed generated schema column <table>._block_number_ on all tables to be of type UInt64.

v4.6.3

Breaking Changes

• Some system fields likeblock_number and block_timestamp have been rename with prefix and suffix '_' to prevent collision with user field.

Fixed

• Replaced hardcoded field names with dialect constants in PostgreSQL inserters for better consistency
  • Updated accumulator_inserter.go to use DialectFieldBlockNumber and DialectFieldBlockTimestamp constants
  • Updated row_inserter.go to use DialectFieldBlockNumber and DialectFieldBlockTimestamp constants
  • Ensures all proto-related code uses centralized field name constants instead of hardcoded strings

v4.6.2

Added

• Added flush retry mechanism with configurable retry count and incremental backoff delay
  • New --flush-retry-count flag (default: 3) to set number of retry attempts for flush operations
  • New --flush-retry-delay flag (default: 1s) to set base delay for incremental retry backoff
  • Automatic retry on flush failures with exponential delay (1s, 2s, 3s, etc.)
  • Improved error handling and logging for flush operations

v4.6.1

• Added support for --clickhouse-cluster on all CREATE statements.
• Added support for specifying the output-module on the command-line in 'from-proto' mode, without requiring a "sink config" in the package.
  ex: substreams-sink-sql from-proto psql://.. substreams.yaml map_my_data
• Fixed a bug causing a panic when the protobuf file containing the outputType is not the last file

KNOWN ISSUES:

• from-proto execution mode does not support 'repeated fields for native types' in protobuf definition,
  ex: repeated string or repeated bytes
  Only repeated MyStruct is supported for now.

v4.6.0

• Added support for Upsert operation in Postgres, Clickhouse isn't supported right now.

• Improved batch block flush logic to flush after a certain number of blocks instead of taking by modulo.

• Fixed cursor loading on Clickhouse.

• Fixed a bug with not flushing rows on stream completion.

• Added --clickhouse-cluster flag. If non-empty the sink will apply a ON CLUSTER <cluster> clause when setting up tables and replace table engines with their Replicated counterparts.

• Added flushed block number and time drift metrics.

v4.5.0

• Added more flags to configure flushing intervals.

  Available flags are batch-block-flush-interval, batch-row-flush-interval and live-block-flush-interval, check substreams-sink-sql run --help for full documentation about the flags.

v4.4.0

• Added support for the Clickhouse Date type.

• Fixed handling of the Clickhouse Array type.

• Removed the check for duplicate primary keys on the Clickhouse dialect. This allows inserting multiple rows with the same primary key.

v4.3.0

• Added a check for non-existent columns in Clickhouse
• Added support for Nullable types in Clickhouse.
• Bump substreams to v0.11.1
• Bump substreams-sink to v0.5.0

v4.2.2

• Fix major bug when receiving empty MapOutput

v4.2.1

DO NOT USE THIS VERSION USE ANYTHING >= v.4.2.2

• Bump substreams to v1.10.3 to support new manifest data like protobuf:excludePaths