Lightweight data provider (2.17.0)

Overview

This component serves as a data provider for th2-data-services. It will connect to the cassandra database via cradle api and expose the data stored in there as REST resources. This component is similar to rpt-data-provider but the last one contains additional GUI-specific logic.

API

You can download OpenAPI schema from here: http://<provider_address>:<port>/openapi

You can view the endpoints documentation from here: http://<provider_address>:<port>/redoc

You can see the Swagger UI from here: http://<provider_address>:<port>/swagger

REST

http://localhost:8080/event/{id} - returns a single event with the specified id.

Example: http://localhost:8080/event/book:scope:20221031130000000000000:eventId

Example with batch: http://localhost:8080/event/book:scope:20221031130000000000000:batchId>book:scope:20221031130000000000000:eventId

http://localhost:8080/message/{id} - returns a single message with the specified id

Example: http://localhost:8080/message/book:session_alias:<direction>:20221031130000000000000:1

direction - 1 - first, 2 - second

Timestamp

Timestamp in HTTP request might be specified either in milliseconds or in nanoseconds. The value will be interpreted as milliseconds if it is less than 1_000_000_000 ^ 2. Otherwise, it will be interpreted as nanoseconds.

SSE requests API

http://localhost:8080/search/sse/events - create a sse channel of event metadata that matches the filter. Accepts following query parameters:

• startTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search starting point. Required
• endTimestamp - number, unix timestamp in milliseconds or nanos - Sets the timestamp to which the search will be performed, starting with startTimestamp. Required.
• parentEvent - text - parent event id of expected child-events.
• searchDirection - the direction for search (next,previous). Default, next
• resultCountLimit - limit the result responses
• bookId - book ID for requested events (*required)
• scope - scope for requested events (*required)
• filters - set of filters to apply. Example, filters=name,type

Supported filters:

• type - filter by events type
• name - filter by events name

Filter parameters:

• <filter_name>-value|<filter_name>-values filter values to apply. Repeatable
• <filter_name>-negative - inverts the filter. <filter_name>-negative=true
• <filter_name>-conjunct - if true the actual value should match all expected values. <filter_name>-conjunct=true
• <filter_name>-operator - matching operator [EQUAL, CONTAIN, START_WITH, END_WITH]. <filter_name>-operation=EQUAL

http://localhost:8080/search/sse/messages - create a sse channel of messages that matches the filter. Accepts following query parameters:

• startTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search starting point. One of the 'startTimestamp' or 'messageId' must not be null

• messageId - text, accepts multiple values. List of message IDs to restore search. Defaults to null. One of the 'startTimestamp' or 'messageId' must not be null

• stream - text, accepts multiple values - Sets the stream ids to search in. Case-sensitive. Required. Example: alias - requests all direction for alias; alias:<direction> - requests specified direction for alias.

• searchDirection - next/previous - Sets the lookup direction. Can be used for pagination. Defaults to next.

• resultCountLimit - number - Sets the maximum amount of messages to return. Defaults to null (unlimited).

• endTimestamp - number, unix timestamp in milliseconds or nanos - Sets the timestamp to which the search will be performed, starting with startTimestamp. When searchDirection is previous, endTimestamp must be less then startTimestamp. Defaults to null (search can be stopped after reaching resultCountLimit).

• responseFormat - text, accepts multiple values - sets response formats. Possible values: BASE_64, PROTO_PARSED, JSON_PARSED. default value - BASE_64 & PROTO_PARSED.

• keepOpen - keeps pulling for updates until have not found any message outside the requested interval. Disabled by default

• bookId - book ID for requested messages (*required)

http://localhost:8080/search/sse/messages/group - creates an SSE channel of messages that matches the requested group for the requested time period

• startTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search starting point. Must not be null
• endTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search ending point. Must not be null
• group - the repeatable parameter java.time.Instantwith group names to request. At least one must be specified
• keepOpen - keeps pulling for updates until have not found any message outside the requested interval. Disabled by default
• bookId - book ID for requested messages (*required)
• responseFormat - text, accepts multiple values - sets response formats. Possible values: BASE_64, PROTO_PARSED, JSON_PARSED. default value - BASE_64 & PROTO_PARSED. Example: http://localhost:8080/search/sse/messages/group?group=A&group=B&startTimestamp=15600000&endTimestamp=15700000

http://localhost:8080/search/sse/page-infos - creates an SSE channel of page infos that matches the requested book id for the requested time period

• startTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search starting point. Must not be null
• endTimestamp - number, unix timestamp in milliseconds or nanos - Sets the search ending point. Must not be null
• bookId - book ID for requested messages (*required)
• resultCountLimit - number - Sets the maximum amount of page events to return. Defaults to null (unlimited). Example: http://localhost:8080/search/sse/page-infos?startTimestamp=15600000&endTimestamp=15700000&bookId=book1

Elements in channel match the format sse:

event: 'event' / 'message' | 'close' | 'error' | 'keep_alive' | 'page_info'
data: 'Event metadata object' / 'message' | 'Empty data' | 'HTTP Error code' | 'Empty data' | 'page info'
id: event / message id | null | null | null | page id

Configuration

schema component description example (lw-data-provider.yml):

CR V1

apiVersion: th2.exactpro.com/v1
kind: Th2CoreBox
metadata:
  name: lw-data-provider
spec:
  image-name: ghcr.io/th2-net/th2-lw-data-provider
  image-version: 1.0.3
  type: th2-rpt-data-provider
  custom-config:
    hostname: 0.0.0.0 # IP to listen to requests. 
    port: 8080 
    
#   keepAliveTimeout: 5000 # timeout in milliseconds. keep_alive sending frequency
#   maxBufferDecodeQueue: 10000 # buffer size for messages that sent to decode but anwers hasn't been received 
#   decodingTimeout: 60000 # timeout expecting answers from codec. 
#   batchSizeBytes: 256KB # the max size of the batch in bytes. You can use 'MB,KB' suffixes or a plain int value
#   codecUsePinAttributes: true # send raw message to specified codec (true) or send to all codecs (false) 
#   responseFormats: string list # resolve data for selected formats only. (allowed values: BASE_64, PARSED)
#   responseBufferSize: 8192 # output buffer size for download operations
#   flushSseAfter: 0 # number of SSE emitted before flushing data to the output stream. 0 means flush after each event
#   gzipCompressionLevel: -1 # integer value of gzip compression level. This option is used when user requests data via HTTP with enabled commpression. 
#      * -1: default compression level
#      * 0: no compression
#      * 1: best speed
#      * 9: best compression

  pins: # pins are used to communicate with codec components to parse message data
    - name: to_codec
      connection-type: mq
      attributes:
        - to_codec
        - raw
        - publish
    - name: from_codec
      connection-type: mq
      attributes:
        - from_codec
        - parsed
        - subscribe
  extended-settings:    
    service:
      enabled: true
      type: NodePort
      endpoints:
        - name: 'grpc'
          targetPort: 8080
      ingress: 
        urlPaths: 
          - '/lw-dataprovider/(.*)'
    envVariables:
      JAVA_TOOL_OPTIONS: "-XX:+ExitOnOutOfMemoryError -Ddatastax-java-driver.advanced.connection.init-query-timeout=\"5000 milliseconds\""
    resources:
      limits:
        memory: 2000Mi
        cpu: 600m
      requests:
        memory: 300Mi
        cpu: 50m

CR V2

apiVersion: th2.exactpro.com/v2
kind: Th2CoreBox
metadata:
  name: lw-data-provider
spec:
  imageName: ghcr.io/th2-net/th2-lw-data-provider
  imageVersion: 1.0.3
  type: th2-rpt-data-provider
  customConfig:
    hostname: 0.0.0.0 # IP to listen to requests. 
    port: 8080 
    
#   keepAliveTimeout: 5000 # timeout in milliseconds. keep_alive sending frequency
#   maxBufferDecodeQueue: 10000 # buffer size for messages that sent to decode but answers hasn't been received 
#   decodingTimeout: 60000 # timeout expecting answers from codec. 
#   batchSizeBytes: 256KB # the max size of the batch in bytes. You can use 'MB,KB' suffixes or a plain int value
#   validateCradleData: false # validate data loaded from cradle. NOTE: Enabled validation affect performance 
#   codecUsePinAttributes: true # send raw message to specified codec (true) or send to all codecs (false) 
#   responseFormats: string list # resolve data for selected formats only. (allowed values: BASE_64, PARSED)
    

  # pins are used to communicate with codec components to parse message data
  pins:
    mq:
      publishers:
      - name: to_codec
        attributes:
          - to_codec
          - raw
          - publish
      subscribers:
      - name: from_codec
        attributes:
          - from_codec
          - parsed
          - subscribe
        linkTo:
          - box: codec
            pin: from_codec_decode_general
  extendedSettings:
    service:
      enabled: true
      nodePort:  # Required if you use ingress url path
        - name: 'connect'
          targetPort: 8080
#          exposedPort: 30042 # if you need a constant port to be exposed
      ingress: 
        urlPaths: 
          - '/lw-dataprovider/'
    envVariables:
      JAVA_TOOL_OPTIONS: "-XX:+ExitOnOutOfMemoryError -Ddatastax-java-driver.advanced.connection.init-query-timeout=\"5000 milliseconds\""
    resources:
      limits:
        memory: 2000Mi
        cpu: 600m
      requests:
        memory: 300Mi
        cpu: 50m

Release notes:

2.17.0

API

• [GH-183] Provided operator option to filter for download event REST API and gRPC calls.
• Provided GRPC_HTTP / HTTP_GRPC modes (CR configuration will be supported by infra in future)

Update:

• th2 gradle plugin to 0.3.10
• kotlin to 2.2.21
• openapi to 6.7.0-2
• common to 5.16.1-dev
• grpc-common to 4.7.2
• micrometer to 1.15.5

2.16.0

Serialization

• Serialize cradle event instead of prepared internal structure.
• Refactored serialization logic of ProviderMessage53Transport class

Prometheus

• Reused metric child to improve Prometheus measurement performance.
• Provided new metrics: await_next_sse_event, write_sse_event metrics
• Used Summary instead of Histogram for measurement

API

• Provided rootOnly flag for download event REST API and gRPC calls.
• Event filter works with cradle event instead of prepared internal structure.

Configuration

• Added responseBufferSize option with default value 0.
• Changed default responseQueueSize from 1000 to 100. Internal ByteBuffer pool size used for download operations dependents on this parameter

Update:

• th2 gradle plugin to 0.3.7
• common to 5.16.0-dev
• common-utils to 2.4.0-dev
• grpc-common to 4.7.1
• cradle to 5.7.0-dev
• openapi to 6.7.0-1
• kotlin to 2.2.10
• kotlinx-serialization to 1.9.0
• kotlin-logging to 7.0.13
• micrometer to 1.15.4

2.15.0

Update:

• Update kotlin to 2.2.0
• Update javalin to 6.7.0
• Update openapi to 6.7.0
• Update kotlin-logging to 7.0.7
• Update cradle API to 5.5.1-dev
• Update micrometer to 1.15.1

2.14.0

Update:

• Update Javalin to 6.5.0
• Update kotlin to 2.1.20
• Update kotlin-logging to 7.0.6
• Update micrometer to 1.14.5
• Update th2 gradle plugin to 0.2.4
• Update th2 common to 5.15.0-dev
• Update cradle API to 5.5.0-dev
• Use Java 21 as build-time and runtime JVM

2.13.3

Fix:

• If two requests have been made using same parameters but one is after another the later one can get a codec timeout when the first request gets codec timeout, but it should not because it was made later.

2.13.2

Fix:

• Not all control characters were escaped during custom serialization.

Update:

• Update th2-gradle-plugin to 0.2.2

2.13.1

Fix:

• Possible deadlock when task API is used with fail-fast strategy and a codec error happens when the response buffer is full

2.13.0

Updates:

• th2 gradle plugin 0.1.6
• kotlin: 1.9.23
• kotlin-logging: 6.0.9
• javalin: 6.3.0
• openapi: 6.3.0
• micrometer: 1.14.1

Migrated:

• from: net.jpountz.lz4:lz4:1.3.0 to: org.lz4:lz4-java:1.8.0

2.12.0

• Conversion to JSON in HTTP handlers is executed in the separate executor. The executor has convThreadPoolSize number of threads
• Added /download/events endpoint to download events as file in JSONL format
• Added EVENTS resource option for /download task endpoint

2.11.1

• Updated:
  • th2 gradle plugin: 0.1.3
  • cradle api: 5.4.4-dev

2.11.0

• Updated:
  • th2 gradle plugin 0.1.1
  • common: 5.14.0-dev
  • cradle api: 5.4.2-dev
  • kotlin-logging: 5.1.4
  • micrometer-bom: 1.13.3

2.10.0

• Updated:
  • th2 gradle plugin 0.0.8
  • common: 5.12.0-dev

2.9.0

• Migrated to th2 gradle plugin 0.0.6
• Added swagger library
• Updated:
  • bom: 4.6.1-dev
  • cradle api: 5.3.0-dev
  • common: 5.11.0-dev
  • micrometer-bom: 1.12.5
  • jetty-bom: 11.0.20
  • javalin: 5.6.5
  • javalin-openapi: 5.6.4

2.7.0

• Updated cradle api: 5.2.0-dev
• Updated common: 5.8.0-dev

2.6.0

• Add download task endpoints:
  • POST /download - register task
  • GET /download/{taskID} - execute task
  • GET /download/{taskID}/status - get task status
  • DELETE /download/{taskID} - remove task
• Add parameter downloadTaskTTL to clean up the completed or not started task after the specified time in milliseconds. 1 hour by default.
• Add EXTERNAL_CONTEXT_PATH env variable to inform provider about external context that is used in requests

2.5.2

• Fix possible deadlock in case the response queue is filled up with keep-alive events and there is no enough space to accumulate the batch for codec

2.5.1

• Enabled Cassandra driver metrics

2.5.0

Updated:

• common: 5.7.1-dev
• grpc-lw-data-provider: 2.3.0

2.4.1

Fixed:

• custom JSON serialization did not do proper escaping for strings

2.4.0

• Added batchSizeBytes parameter to limit batch size by size in bytes rather than count of messages.
• Added validateCradleData parameter to enable/disable validation logic for data loaded from cradle. Currently, managed validation logic includes check for message sequence and timestamp inside a group batch.
• Parameters batchSize and groupRequestBuffer removed. The maximum batch size in messages is computed based on bufferPerQuery or maxBufferDecodeQueue if previous parameter is not set.

2.3.1

Fixed:

• check order of messages in the whole group batch instead of according to streams (session alias + direction).

2.3.0

Feature

• Add gzipCompressionLevel parameter into configuration

2.2.1

• Migrated to the cradle version with fixed load pages where removed field is null problem.
• Updated cradle: 5.1.4-dev

2.2.0

• Add support for requesting message groups in reversed order
• Add filter by stream to gRPC API for group search request

Fixed:

• error reporting when executing group search (if error occurs during a call to the Cradle API stream were closed before the error was reported)

2.8.0

• Updated bom: 4.6.0-dev
• Updated common: 5.10.0-dev (recover subscription of channels on failure)

2.1.0

• Updated bom: 4.5.0-dev
• Updated common: 5.4.0-dev
• Updated kotlin: 1.8.22

2.0.1

• Removed extra fields field from JSON_PARSED message format.
  • Old: {"fields":{"a":{"fields":{"b":"1"}}}}
  • New: {"fields":{"a":{"b":"1"}}}