Incremental / Reactive / DataFlow DataFrames

[sirinath]

Say you have DF a defined as:

A = B + C

Is it possible to provide the functionality with perhaps some special DFs implementation:

1. A is computed lazily when accessed using the latest values in B and C (Pull)
2. When B or C's values changes A gets automatically updated (Push)

[ctn]

Suminda, we do have intention to maintain some kind of clean/dirty flag on
DDFs, for some other purposes.

In this case, can you describe the use cases for which you need the
so-called push/pull behavior?

Sent while mobile. Please excuse typos etc.
On Dec 12, 2014 11:21 PM, "Suminda Dharmasena" [email protected]
wrote:

Say you have DF a defined as:

A = B + C

Is it possible to provide the functionality with perhaps some special DFs
implementation:

1. A is computed lazily when accessed using the latest values in B and
   C (Pull)
2. When B or C's values changes A gets automatically updated (Push)

—
Reply to this email directly or view it on GitHub
#56.

[sirinath]

They are basically:

• Incremental Computing Mode, and
• Reactive Computing Mode

For a non DDF reactive case have a look at: https://github.com/lihaoyi/scala.rx

But in this case it needs to be more fine grain as when individual items change the changes propagate.

[ctn]

I know, as a former professor of computer engineering :-). I'm asking about
your actual use cases, which affects prioritization. Thanks.

Sent while mobile. Please excuse typos etc.
On Dec 13, 2014 10:32 AM, "Suminda Dharmasena" [email protected]
wrote:

They are basically:

• Incremental Computing Mode, and
• Reactive Computing Mode

For a non DDF reactive case have a look at:
https://github.com/lihaoyi/scala.rx

But in this case it needs to be more fine grain as when individual items
change the changes propagate.

—
Reply to this email directly or view it on GitHub
#56 (comment).

[sirinath]

Use cases of what I can think about (biased towards by domain):

• If you want a signal / technical indicator computed as and when new price data becomes available
• Compute minimum variance portfolio as and when economic indicators are updated
• Compute firm wide real-time PnL as and when prices move and new trades are executed
• etc.

Best is to get feedback from others also in other domains.

It is more convenient to all computations as dataflows for realtime computing use cases.

[sirinath]

Following is an interesting product in this space: http://www.ankhor.com/. This is not a library thought but might help you get an idea how best to plan this functionality in DDF.

If you can use DDF to programmatically create this type of flows this would be the ultimate DDF can achieve as I see it.