Why result different between groupby and count?

[tommyhj217]

Hi

Thanks for make surprise lib.

I tested count() function and compare with group by result.

I used below data.
taxi_path = 's3://vaex/taxi/yellow_taxi_2012_zones.hdf5?anon=true'

The other values are all the same, but the last value of each of the last values is output as zero.
Is there a way to solve this problem?

Thanks
Hyunjun

[maartenbreddels]

This is expected, since binby will use regular bins between a min and max, with half open intervals, e.g. [0, 1), ... [5, 6).
Then the bins is calculated as int(data / (max - min) * shape) (e.g. 6 => int(6 / 6 * 7) = int(7.0) = 7, which falls out of the last bin, and 5 => int(5/7*7) = int(5/6*8) = int(5.8333) = 5. So no data falls in bin 6.

If you pass in the limits, e.g.:

df.count('*', binby=['pickup_hour', 'pickup_day'], shape=[24,7], limits=([-0.5, 23.5], [-0.5, 6.5]))

You get:

Does that make sense?

[tommyhj217]

Thanks for reply my question.
It seems to be one of the best tools available on a single computer.