Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get the latest two events on a specific window using Flint? #85

Open
kant777 opened this issue Jun 27, 2020 · 1 comment
Open

Comments

@kant777
Copy link

kant777 commented Jun 27, 2020

How to get the latest two events using this library?

Say I have the following

time    | price
---------------
1000L | 40
2000L | 20
3000L | 80
4000L | 10
5000L | 60
6000L | 30

I want to do operations like

  1. Get the last 2 events or last 4 events in the past 6 hours
  2. Difference in price for the last two events in 6 hours.

How to do this using Flint? I can write my own udf and solve this but I am wondering if there is any inbuilt function that is already available?

@kant777 kant777 changed the title How to get the latest two events on a specific window using this library? How to get the latest two events on a specific window using Flint? Jun 27, 2020
@dgrnbrg
Copy link

dgrnbrg commented Nov 5, 2020

I've done this by leftJoining the data frame to itself shifted forward w/ a tolerance, but that's hideously slow. I think that you could use addPastWindows for 6 hours, and then use a UDF on that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants