General first impressions after reading the tutorial

[astrojuanlu]

Hi! Juan Luis from Orchest here 👋🏽 I'm writing a blog post on Vaex, and I'm collecting some notes about what I find. Here is a recollection of some intriguing things or errors I found while following the 11 minute intro:

• I'm wondering what's the difference between df.x and df.col.x? I force myself to type df["x"] for consistency, but df.col.{colname} looks like a nice way of namespacing columns as properties. Unsure if that's the original intent though.
• Initially I didn't notice that Vaex Expressions are lazy because it's so fast 🔥 that helmi-dezeeuw-2000-FeH-v2-10percent.hdf5 becomes just too small! (Not a criticism, quite the opposite)
• df.evaluate is mentioned right after df.select and used further down in the visualization section, but the docstring says "Note that this is not how vaex should be used, since it means a copy of the data needs to fit in memory". Wondering if including it in the quickstart should be reconsidered?
• Speaking of which, the quickstart does df.evaluate(..., selection=True), but when trying to understand what possible values does selection receive, the docstring is not very helpful: :param selection: selection to apply - the visualization section gives a hint with selection=[None, df.x < df.y, df.x < -10], but it's still unclear how it works.
• In general, most docstrings are a bit terse.
• Just out of curiosity I tried df("x") and got a vaex.legacy.SubspaceLocal object. If it's legacy, maybe this could emit a DeprecationWarning or similar? I then tried df("x").mean() and got TypeError: __init__() got multiple values for argument 'info', so I stopped pursuing the matter.
• I'm still wrapping my head around df.count(binby="x"), and it seems to work with arbitrary expressions too (df.count(binby="sin(x)") ❓)
• I tried something silly like df.mean("*", binby="x") and got TypeError: mean() missing 1 required positional argument: 'expression', which is not very informative.
• Working with nyctaxi was a bit rough though, I tried nyctaxi["trip_distance"].mean() and after some time the kernel just died. Missing the kind of progress feedback one gets with Dask, although I acknowledge that this might be out of scope.

About some of the questions, I'll probably figure out as soon as I dive deeper in the documentation, but I thought it would be useful to gather these here since they were a bit surprising or unexpected. Hope it helps!

[JovanVeljanoski]

Hey hey!

Thank you for having a careful look at vaex and the tutorials / docs. Much appreciated. Let me try to reply to all the points in the order in which you've raised them. A general comment: we are a small team, and we know there are many low hanging fruits to fix, such as inconsistency in the docs, or things that can be improved there.. so we are hoping maybe someone in the community will want to contribute (easy way to get into vaex perhaps). Othewise we are happy to do it, but at expense of features and a matter of priorities..

Anyway:

• I believe at the moment there is no practical difference. This comes from a time (long ago..) when the API of vaex was a bit diffrerent and less dataframe-y. Nowadays you can use it to disambiguate column names (say you have a column name mean, then df.mean would call the method but df.col.mean would refer to the expression. You can also use df["maen"]).
• Thanks :) You can download or generate a bit larger datasets, e.g. vaex.datasets.iris_1e8() or vaex.datasets.helmi_simulation_data(full=True). The latter is the full version of the df.example(), but still only ~3 million samples.
• There is nothing wrong with using evaluate in principle, but it is more (nowadays) used for internal stuff. It can be handy if you wanna use selections for example. In most cases it is kind of like using .values. You eagarly put the underlying data (if on disk) to memory. So if you are working with 1TB of data you should be careful.
• We have a standardized way to explain what the selection argument can accept, i see now we've missed to update the docstring in evaluate. Just check out other methods that support it, like df.mean. It is the same (mostly. I say mostly because certain cases can accept a list of selections and some a single one only. This should be reflected in the docs).
• Pointers to what should be improved would be helpful! PRs also welcome :)
• A dataframe object is not meant to be callable, so I am surprised that even does something. But if legacy that might explain it. Legacy does not mean we will remove it tho (at least not yet). In general we try to avoid removing features, and keep old code working for as long as possible (i don't necessarily agree with this always, but @maartenbreddels is adamant :) ). Old/legacy code may be removed from docs etc.. and is not meant to be used. We try to hide it.
• That is a nice feature. Kind of like groupby in a way.. so you bin along an axis (or an expression), then you do the operation, in this case count. So df.count(binby="x") is basically creating a histogram of x. You count how many values are in each bin in x. You can specify the number of bins via tha shape param. df.count(binby="sin(x)") is the same, but the binning is done on sin(x). Cool right?
• I honestly don't think you can get any better error message than that :). What would you suggest? It tells you exactly what is wrong, (missing a required argument called 'expression').
• Very surprised by this. How did you get the data? If it something you can (semi)consistently reproduce, can you please open a separate issue on it? Btw, many method in vaex support the progress argument which will give you a progress bar. Try (True, 'widget', 'rich') for different options. Rich is the most informative i think. For more info on a fancier application of progress bars see this guide.

Thanks this was very informative. Please feel free to offer suggestions / PRs / general feedback to things. Especially for easy fixes like bad/old docstrings and stuff like that.

We are meant to update/rework the tutorial for a while now.. but yet to get to it.

Thanks!

@maartenbreddels please check in case I got something wrong..

[astrojuanlu]

For the record, I am trying now nyctaxi.export("nyctaxi.parquet", chunk_size=10_000, progress="rich") to save it to disk before proceeding, and it's making (slow) progress 👍🏽

[JovanVeljanoski]

• About the error of df.mean("*", binby="x"): you are calculating a mean of something, so you need to specify that something explicitly, e.g. About the error of df.mean("y", binby="x"). @maartenbreddels maybe in this case ("*") this should mean "use all columns?". I am not sure tho...
• Can you check that you downloaded the full data? Vaex will stream the data as needed, so if the connection drops or something, the error you saw can be expected. You can try something like df.nop('x') and it will download all the data for column x.

[astrojuanlu]

Doing nyctaxi.nop("passenger_count", progress="rich") keeps the progress at 0 % for a few seconds and then "The kernel for main.ipynb appears to have died. It will restart automatically.":

⠋ vaex                                          ━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 49.62s    
└── ⠋ map reduce: nop                           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% 50.56s    

[JovanVeljanoski]

Do you have enough disk space for it? That will require around.. 16 (?) GB in your home folder.

You can try for example:

import vaex
df = vaex.open('s3://vaex/taxi/yellow_taxi_2009_2015_f32.hdf5?anon=true')
df[:100_000].nop('passenger_count', progress='rich')

The progress bar will remain 0% i think, it does not measure the download progress, it measures the evaluation of the expression, so that is normal.

[astrojuanlu]

Indeed, that was it! I freed some space and now the full thing works. All problems solved :)

[maartenbreddels]

Hi Juan,

thansk for the feedback, this is appreciated.

Jovan answer most, let me add a bit to that

• Just out of curiosity I tried df("x") and got a vaex.legacy.SubspaceLocal object.

Yes, this must be one of the most legacy things in Vaex, me and @JovanVeljanoski joke about this regularly. I think it's time to remove it.

• I tried something silly like df.mean("*", binby="x") and got TypeError: mean() missing 1 required positional argument: 'expression', which is not very informative.

"" only works for count, that mean accepts it is actually a bug. It's there because previously Vaex was a GUI tool (build on qt), that needed a string input, it was easier to support "" as an argument, vs an optional argument. But, like in SQL, it only works with count.

Regarding the crash. That is really annoying, and has to do with using memory mapping for writing to a file. We cannot have an exception handler handle that (it leads to a SIGBUS signal). For that reason alone I've been working on refactoring that part to use regular file IO.

Regards,

Maarten

[astrojuanlu]

Thanks for the extra insight Maarten!