Skip to content

opensearch output: action does not support "create" #4515

Description

@chenraoCR

Description

The output opensearch action does not support create, as a result can not sink messages into index of datastream type in opensearch

Location

The supportted action is in internal/impl/opensearch/output.go

Test

Preparation

Opensearch: 3.6.0
Kafka: 4.0.0
Redpandadata/connect: 4.90.3

Create events as a datastream in opensearch

Configure connector output

output_resources:
  - label: o_kafka_os
    broker:
      pattern: greedy
      copies: 5
      outputs:
        - drop_on:
            error_patterns:
              - illegal_argument_exception
              - Compressor detection
              - invalid json
            output:
              opensearch:
                urls: [ "${OS_URI}" ]
                action: index
                index: 'events'
                id: '${! uuid_v4() }'

Result

Input data in kafka, and then redpanda-connect will has following error in log:

time="2026-06-15T02:44:20Z" level=error msg="Failed to send message to opensearch: illegal_argument_exception: only write ops with an op_type of create are allowed in data streams" @service=redpanda-connect  label="" path=root.output_resources.broker.outputs.0.drop_on.output
time="2026-06-15T02:44:20Z" level=warning msg="Message dropped due to error matching pattern 0: illegal_argument_exception: only write ops with an op_type of create are allowed in data streams" @service=redpanda-connect label="" path=root.output_resources.broker.outputs.0

Proposal

Add the following code into internal/impl/opensearch/output.go, just like elasticsearch_v9

	case "index":
		r = &opensearchutil.BulkIndexerItem{
			Index:  p.Index,
			Action: "index",
			Body:   bytes.NewReader(p.Payload),
		}
		if p.ID != "" {
			r.DocumentID = p.ID
		}
		if p.Routing != "" {
			r.Routing = &p.Routing
		}
+	case "create":
+		r = &opensearchutil.BulkIndexerItem{
+			Index:  p.Index,
+			Action: "create",
+			Body:   bytes.NewReader(p.Payload),
+		}
+		if p.ID != "" {
+			r.DocumentID = p.ID
+		}
+		if p.Routing != "" {
+			r.Routing = &p.Routing
+		}

Then can set the action to "create", then can write data into a index of datastream in opensearch

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions