Data Format

Design / Architecture

Data model

Turbine deals with data in the form of json payloads. Turbine is data agnostic and simply views the json blob as a map of key -> value pairs.

{a:1, b:2, c:'string', d:true}

Aggregation dimensions

The only binding Turbine has with the data is the aggregation dimension specified using name and type. Json payloads received from individual instances matching the same aggregation key are combined together. e.g

{type:'weather-data-temp', name:'New York', temp:74}
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:76}

are combined to give

{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:75}

Sub-streams within streams

Name and type are just data classifiers and either one could suffice as the aggregation key. However, the reason for having a composite aggregation key is to be able to represent multiple sub streams over the same connection. This is what type is used for.

One can send data payloads of multiple types over the same connection for efficiency, hence multiplexing multiple streams over the same persistent Turbine connection. Consider this example.

{type:'weather-data-temp', name:'New York', temp:74}
{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:76}

{type:'weather-data-wind-velocity', name:'New York', temp:12}
{type:'weather-data-wind-velocity', name:'Los Angeles', temp:10}

are combined to give

{type:'weather-data-temp', name:'Los Angeles', temp:85}
{type:'weather-data-temp', name:'New York', temp:75}
{type:'weather-data-wind-velocity', name:'New York', temp:12}
{type:'weather-data-wind-velocity', name:'Los Angeles', temp:10}