Principles

Reactivity stores and exchange data in the JSON format. One reason why JSON is used is the flexibility it offers in the data model.

This documentation describes the mandatory fields that must be found in a JSON document and which path can be freely structured. The documentation also pays attention to the data model which is persisted to the database comparing to the informations dedicated to event types. Moreover, since everything is stream in Reactivity, different fields of the data model are flagged to understand when they are defined and where they transit inside the system.

Events stream

Data are always wrapped to an event. This allows Reactivity to understand what to do with the data. Therefore, any producer must use a particular data structure in their JSON document to wrap their data:

{
  "version" : "The module version",
  "event" : "The event type here",
  "data" : "The data here",
  "timestamp" : "The event timestamp",
  "id" : "The event ID",
  "tag" : "A random value that looks like an UID"
}

version

The version is a value indicating the version of the module that produces the message. This version can be used by the consumer to know what kind of assertion can be made on the data. By default, new version must be backward compatible which means that a consumer should be able to deal with an event flagged with an older version.

Version format follows the semver protocol.

event

The event describes the nature of the message. The possible values are an enumeration specific to the type of data contained inside the message.

No field in the message indicates the type of data because it corresponds to the topic they are published to. In the core module, messages can be published to the following topic:

user: the topic where events related to Users are published
organization: the topic where events related to Organization are published
artifact: the topic where events related to Artifacts are published

This statement is true only when the client produce and consumes data directly from/to the topic (through Kafka). However, there is the particular case of a WEB application subscribing to a source of events comming from a suspended SSE connection from the server.

         +--------------+-------------+
+------->|     Topic    |             |
|        +--------------+             |       +-----------------+
|        |     user     | Broadcaster |  SSE  |                 |
|        | organization |             |------>| Web Application |
|        |   artifact   |             |       |                 |
|        +--------------+-------------+       +-----------------+
|
|        +---------------------------+
|        |                           |
+--------+           KAFKA           |
         |                           |
         +---------------------------+

In this simple diagram we can see that the Broadcaster is consuming different topics from Kafka. When a message is received, it’s pushed to the web application that established a SSE connection. The only chance for the web application to understand the type of data represented by a received message is to find topic attribute in the JSON.

{
  "topic" : "organization|user|artifact"
  "version" : "...",
  "event" : "...",
  "data" : "...",
  "timestamp" : "...",
  "id" : "...",
  "tag" : "..."
}

data

The data contains the payload of the event provided by the producer. The structure of the data depends on the topic where is has been published, the event and version fields values. The consumer will use those informations to know what structure is expected in the data value.

timestamp and id

timestamp and id are not defined by the producer but are assigned by the streaming platform. They allow to identify the message inside Reactivity system and to know when it has been published by the consumer.

Therefore, a producer can generate a message that simply have this structure:

{
  "version" : "...",
  "event" : "...",
  "data" : "...",
  "tag" : "..."
}

tag

Since the id is not generated by the producer, it’s not possible for him to recognize a produced message when it’s received through another channel. The tag allows him to address this issue. The producer can associate a random value that looks like an unique UID to the tag attribute, allowing him to implement a kind of acknowledgment for the produced message. This is particularly handy for web applications that receive a message previously produced. In fact, the web application acts as a producer who expects to receive its own messages sent from the broadcaster.

    +----------------------------+ SSE - Consume message A  +-----------------+
+-->|     Topic ---> Broadcaster |------------------------->| Web Application |
|   +----------------------------+                          +-----------------+
|                                                                    |
|   +---------------------------+                                    |
|   |                           |         Produce message A          |
+---+           KAFKA           |<-----------------------------------+
    |                           |
    +---------------------------+

In this figure, we see that the message A produced by the web application is received from the broadcaster. With the tag attribute whose value won’t have been changed, the web application will be able to recognized it and to treat it differently comparing to the messages produced by other instances. For example, the web application can perform an optimistic change on the UI as soon as the message is sent, giving a better feeling of performance to the user. When the message is received, the web application can simply ignore it. The web application can also raise an error if the message is never received (timeout), which means that the message has possibly been discarded for any reason.

Extensions

Additional data types definition

Extensions are allowed to define their own type of data stored in dedicated buckets (the document space in Couchbase) and sent through specific topics (in Kafka). It’s strongly recommended for an extension to use the same message structure described in previous sections.

To avoid clashes with build-in features of Reactivity, any extension topic and bucket must follow the pattern [extension-id]/[topic-name]. The extension-id is an unique identifier for the extension in Reactivity. This value must be used when the extension creates its own topic. The topic-name is a value each extension is free to define. This allows the extensions to define as much as data type they want and store their associated documents independently to the database.

Built-in data types extension

When an extension extends an existing data type (Artifacts, Users or Organizations), the mechanism is different. The extension is allowed to modify the document created by the built-in features of Reactivity. In that scenario, the extension is integrated to a stream processing module that intercepts messages comming from a targetted topic (user, artifact or organization). The extension can validate the message payload and discard it in case of rule violation, modify the payload or trigger any specific action before the message is persisted.

The extension is free to manage specific properties at any level of the message. For instance:

{
    "version" : "The module version",
    "event" : "The event type here",
    "data" : {
        "my-extension-some-property" : "some value"
    },
    "timestamp" : "The event timestamp",
    "id" : "The event ID",
    "my-extension-some-property" : "some value"
}

It’s strongly recommended that an extension does not add specific properties in a different place than what is defined in the above example. Making assertion on the data field structure to add properties in a deeper path is possible but it would strongly couple the core with the extension, which would potentially lead to lots of regressions after a new release.

Built-in data types and associated events

The following section describes the possible values for data and associated event attribute in a message provided by the core of Reactivity.

User

All messages related to the User data type are published to the user topic.

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

The CREATE, READ and UPDATE events for a User has a data attribute associated to a value that looks like this:

{
    "email" : "The user email address",
    "firstname" : "The user first name",
    "lastname" : "The user last name",
    "picture" : "The picture of user profile"
}

READ and UPDATE events are sent for a given User only to the consumers connected to an Organization that registers this User as a member.

email

A string corresponding to the user email. Mandatory attribute.

firstname

A string corresponding to the user first name. Optional attribute.

lastname

A string corresponding to the user last name. Optional attribute.

picture

A base64 string corresponding to the user picture. Optional attribute.

event = 'DELETE'

The DELETE event for a User has a data attribute associated to an empty value. In fact, the event simply indicates that a User identified by the event id has been removed. Therefore, no more information than the id is required to identify the removed view.

This event is sent for a given User only to the consumers connected to an Organization that registers this User as a member.

Organization

All messages related to the Organization data type are published to the organization topic.

event = 'CREATE'

The CREATE event for an Organization has a data attribute associated to a value that looks like this:

{
    "name" : "The name of the organization",
    "picture" : "The picture of the organization",
    "members" : [{
            "id" : "The ID of the organization member",
            "role" : "The role of the member inside the organization"
    }]
}

This event is sent through this topic by producer when a new Organization is created.

name

A string corresponding to the Organization name.
Mandatory attribute.
Must be unique.

picture

A base64 string corresponding to the Organization picture.
Optional attribute.

members

An array containing complex objects with two mandatory fields:

id: the User ID corresponding to the member.
role: the role of the member inside the Organization.

event = 'READ' OR event = 'UPDATE'

The READ and UPDATE events for an Organization have a data attribute associated to a value that looks like this:

{
    "name" : "The name of the organization",
    "picture" : "The picture of the organization"
}

This events are sent when a consumer is reading the Organization associated to the current User.

name

A string corresponding to the Organization name. Mandatory attribute.

picture

A base64 string corresponding to the Organization picture. Optional attribute.

event = 'DELETE'

The DELETE event for an Organization has a data attribute associated to an empty value. In fact, the event simply indicates that an Organization identified by the event id has been deleted. Therefore, no more information than the id is required to identify the removed Organization.

event = 'ADD_CATEGORY' OR event = 'READ_CATEGORY' OR event = 'UPDATE_CATEGORY'

The ADD_CATEGORY, READ_CATEGORY and UPDATE_CATEGORY events for an Organization have a data attribute associated to a value that looks like this:

{
    "organization": "The organization ID"
    "name" : "The name of the category"
    "picture" : "The picture of item category"
}

Reactivity provides User and Status categories out of the box.

organization

The id of the Organization associated to the category.
Mandatory attribute.

name

A string representing the name of the category.
Mandatory attribute.

picture

A base64 string corresponding to the category.
Optional attribute.

event = 'REMOVE_CATEGORY'

The REMOVE_CATEGORY event for an Organization has a data attribute associated to an empty value. In fact, the event simply indicates that a category identified by the event id has been removed. Therefore, no more information than the id is required to identify the removed category.

event = 'ADD_CATEGORY_ITEM' OR event = 'READ_CATEGORY_ITEM' OR event = 'UPDATE_CATEGORY_ITEM'

The ADD_CATEGORY_ITEM, READ_CATEGORY_CATEGORY and UPDATE_CATEGORY_ITEM events for an Organization have a data attribute associated to a value that looks like this:

   {
"category": "The category ID"
       "name" : "The name of the item"
       "picture" : "The picture of the item"
   }

Reactivity provides the following values for the built-in categories:

User: all the current Organization members with their id as name and the picture if any
Status: TODO, WIP and DONE values with no picture

category

The id of the category associated to the item.
Mandatory attribute.

name

A string attribute the name of the item.
Mandatory attribute.

picture

A base64 string corresponding to the item.
Optional attribute.

event = 'REMOVE_CATEGORY_ITEM'

The REMOVE_CATEGORY_ITEM event for an Organization has a data attribute associated to an empty value. In fact, the event simply indicates that an item identified by the event id has been removed. Therefore, no more information than the id is required to identify the removed item.

event = 'ADD_MEMBER' OR event = 'REMOVE_MEMBER' OR event = 'UPDATE_MEMBER'

The ADD_MEMBER, REMOVE_MEMBER, UPDATE_MEMBER events for an Organization have a data attribute associated to a value that looks like this:

{
    "organization" : "The organization ID",
    "members" : [{
        "id" : "The ID of the organization member",
        "role" : "The role of the member inside the organization (only when adding or updating a member)"
    }]
}

Those events are sent when a consumer is connected to an Organization. Only members of that Organization are sent.

name

A string corresponding to the Organization name. Defined only if value has changed.

members

An array defined only if members have changed and containing complex objects with two mandatory fields:

id: the User ID corresponding to the member.
role: the role of the member inside the Organization. Only defined when the event is ADD_MEMBER or UPDATE_MEMBER.

event = 'ADD_VIEW' OR event = 'READ_VIEW'

The ADD_VIEW and READ_VIEW events for an Organization have a data attribute associated to a value that looks like this:

{
    "organization" : "The organization identifier",
    "name" : "The name of the view",
    "period" {
        "from" : "From when the artifacts are displayed",
        "to" : "Moment until artifacts are displayed",
        "category" : "The category id providing the timestamp to use"
    },
    "filters" : [{
        "category" : "Filter the artifact with the specified category",
        "value" : "The value that must equals to the specified category in the filtered artifact"
    }],
    "type" : "Type fo the view"
}

ADD_VIEW event is sent by a producer when a view is created. READ_VIEW event is sent to a consumer when this consumer is connected to an Organization. Only views related to the Organization are sent.

organization

The organization id owning the view.

name

The name of the view is an unique string inside the Organization. The value is displayed as a summary of the view.

period

A complex object describing the period of time covering the displayed informations. This object contains the following attributes:

from: The min date of Artifact. Must be a timestamp in milliseconds. The value is mandatory.
to: The max date of Artifact. Must be a timestamp in milliseconds if defined. The value can be undefined, null or false. In this case no max date is applied.
category: The category id that will be used to read the timestamp from the artifact. Must be a timestamp in milliseconds if defined. The value can be undefined, null or false. In this case the event timestamp will be used.

filters

Optional filters in addition to the period that are applied on the Artifact when selecting the data. filters value is an array of complex object with two attributes:

category: the category id that must be used to filter the Artifact. The value is mandatory.
value: the particular category label that must be associated to the Artifact. The value can be undefined, null or false. In this case any value is accepted since the category exist in the Artifact.

type

The type of view describes a way to display and store the data. The possible values are enumerated by the system. Reactivity supports out of the box three types:

list: display artifacts in a list with one line per item
table: display artifacts in a table with one or two dimensions
timeseries: display the artifacts distributed over a periode of time inside a chart

Additional attributes when type = 'table'.

{
    "columns" : "Category id enumerating the columns",
    "rows" : "Category id enumerating the rows"
}

columns: a category id containing the values displayed in column titles. Artifact will be organized vertically according to the value of this category. This value is mandatory.
rows: a category id containing the values displayed in row titles. Artifact will be organized horizontally according to the value of this category. The value can be undefined, null or false. In this case the table has only one dimension.

Additional attributes when type = 'timeseries'.

{
    "unit" : "The time unit defining the group level of the time series",
    "category" : "The category id used to count artifacts"
    "preferences" : {
        "colors" : [{
        "label" : "The category label that should be displayed with a particular color",
            "value" : "The color to apply for the specified category label"
        }],
        "chart" : "The chat that displays the time series"
    }
}

unit: the time unit that corresponds to the group level of the time series from year to minute. This field is mandatory. Possible values are year, year/month, year/month/day, year/month/day/hour, year/month/day/hour/minute.
category: the category id defining how Artifact must be grouped. This field is mandatory.
preferences.color: an array where each object describes the preferred color to represent a category. Each object has a label attribute for the category label and a value for the color to apply.
prefereces.chart: a string indicated the chart displaying the data. The possible values are managed by the user interface.

event = 'REMOVE_VIEW'

The REMOVE_VIEW event for an Organization has a data attribute associated to an empty value. In fact, the event simply indicates that a view identified by the event id has been removed. Therefore, no more information than the id is required to identify the removed view.

Artifact

All messages related to the Artifact data type are published to the artifact topic.

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

The CREATE, READ and UPDATE events for an Artifact have a data attribute associated to a value that looks like this:

{
    "views" : "The id of the different views this artifacts belongs to",
    "categories" : {
        "A free category key" : "A free category value"
    }
}

views: an array containing the id of the different views containing this Artifact.
categories: a free set of key/value that describes the information defined by the user interface. Reactivity expects the following categories by default:
- A key called user associated to an id allowing to associate a User to an Artifact
- A key called status associated to TODO, WIP or DONE (the default items for the Status category) allowing to give a status to an Artifact

READ and UPDATE events are sent for a given Artifact only to the consumers connected to an Organization that registers the views containing this Artifact.

When the Artifact belongs to a view of type timeseries the categories are actually used to specify time series information:

A key corresponding to the grouping category id associated to a value indicating the count result
A key called timestamp corresponding to a time in milliseconds of the newest counted Artifact

event = 'DELETE'

The DELETE event for an Artifact has a data attribute associated to an empty value. In fact, the event simply indicates that an Artifact identified by the event id has been removed. Therefore, no more information than the id is required to identify the removed Artifact.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

data-model.adoc

data-model.adoc

Principles

Events stream

version

event

data

timestamp and id

tag

Extensions

Additional data types definition

Built-in data types extension

Built-in data types and associated events

User

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

event = 'DELETE'

Organization

event = 'CREATE'

event = 'READ' OR event = 'UPDATE'

event = 'DELETE'

event = 'ADD_CATEGORY' OR event = 'READ_CATEGORY' OR event = 'UPDATE_CATEGORY'

event = 'REMOVE_CATEGORY'

event = 'ADD_CATEGORY_ITEM' OR event = 'READ_CATEGORY_ITEM' OR event = 'UPDATE_CATEGORY_ITEM'

event = 'REMOVE_CATEGORY_ITEM'

event = 'ADD_MEMBER' OR event = 'REMOVE_MEMBER' OR event = 'UPDATE_MEMBER'

event = 'ADD_VIEW' OR event = 'READ_VIEW'

event = 'REMOVE_VIEW'

Artifact

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

event = 'DELETE'

Files

data-model.adoc

Latest commit

History

data-model.adoc

File metadata and controls

Principles

Events stream

version

event

data

timestamp and id

tag

Extensions

Additional data types definition

Built-in data types extension

Built-in data types and associated events

User

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

event = 'DELETE'

Organization

event = 'CREATE'

event = 'READ' OR event = 'UPDATE'

event = 'DELETE'

event = 'ADD_CATEGORY' OR event = 'READ_CATEGORY' OR event = 'UPDATE_CATEGORY'

event = 'REMOVE_CATEGORY'

event = 'ADD_CATEGORY_ITEM' OR event = 'READ_CATEGORY_ITEM' OR event = 'UPDATE_CATEGORY_ITEM'

event = 'REMOVE_CATEGORY_ITEM'

event = 'ADD_MEMBER' OR event = 'REMOVE_MEMBER' OR event = 'UPDATE_MEMBER'

event = 'ADD_VIEW' OR event = 'READ_VIEW'

event = 'REMOVE_VIEW'

Artifact

event = 'CREATE' OR event = 'READ' OR event = 'UPDATE'

event = 'DELETE'