Is vaex.from_dict memory efficient when possible?

[yohplala]

Hi,
Possibly, this should be in feature request.

When vaex create a DataFrame from a dict, and this dict only contains generator, can it 'generate in memory' the data only when it needs to?

I am aware that one cannot know the length of a generator in advance (or so I think) so maybe an additional parameter to specify the length of the DataFrame is required.

Something like:

def zero():
    yield 0.

def one():
    yield 1.

# Data by generators.
n_rows = 5
default_values = {'temperature': zero(), 'pistachios': one(), 'idx': range(n_rows)}
vdf = vaex.from_dict(default_values, size=n_rows)

My use case (I should have started here) consists in loading data from files and joining it with vaex before recording it again this time in a single file.
I know in advance the number of columns, column names, number of rows and so on that are loaded.
When the file is not existing, then I have to use default values like the ones in above example (either constant values, or generated by a range() function).

My current method is to create the array with pandas, then transform it in vaex DataFrame.

But vaex could help make smart use of memory here as well probably?

Last but not least,
If considered, single values in the dict could be interpreted as constant values for the full column, hence allowing the user to write the shorter code:

# Data by generators.
n_rows = 5
default_values = {'temperature': 0., 'pistachios': 1., 'idx': range(n_rows)}
vdf = vaex.from_dict(default_values, size=n_rows)

Should I put this ticket as a feature request?

Thanks for your feedback, bests!

[JovanVeljanoski]

I don't really see vaex supporting generators in the way you describe (although you never know..).

If your goal is to have columns with constant values then indeed we should have a dedicated method for that, and make those columns not use any memory.

There are already a couple of tricks of how you can do this for both numerical and string columns scattered around the issues.

Btw, where is already a vaex.range(..) what will create a virtual column (no memory used) as you would expect.

I hope this helps a bit.

[yohplala]

If your goal is to have columns with constant values then indeed we should have a dedicated method for that, and make those columns not use any memory.

Thanks for the feedback @JovanVeljanoski
Yes, this is my goal. I am aware of some tricks to add a virtual column to an existing vaex Dataframe.
My case here is that the vaex Dataframe is not existing and I have to create with a full 'virtual content'.

I tried to call vaex.range() but I get a AttributeError: module 'vaex' has no attribute 'range'.
I could not find it either in the API doc. Please, is it an addition in the develop branch?

Thanks again for your help.
Have a good day, bests

[JovanVeljanoski]

Ah yeah, i guess the method if vaex.vrange (v for virtual).

Edit:
So something like this should work:

import vaex

df = vaex.from_arrays(x=vaex.vrange(0, 10))

No idea why the vrange does not show up in the API docs.. something to take a look at..

[yohplala]

So something like this should work:

import vaex
df = vaex.from_arrays(x=vaex.vrange(0, 10))

It does the trick, thanks!