-
Notifications
You must be signed in to change notification settings - Fork 713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High memory footprint for big dataframes in CausalForest model #883
Comments
One simple comparison that would be useful is how the memory consumption of standard sklearn RandomForests compares on dataframes of the same size, since much of the EconML tree code was forked from sklearn (version 0.24, I believe). Although 180GB does seem excessive, I don't think it is really exponential - if your input has 40M floating point values, the raw data for that alone is 320MB, so this is ~560 times the size of your dataset. Certainly if we can easily optimize things to bring this down we should, but it's not even quadratic in the number of elements. You mention that memory is high for both fit and effect: do you mean that while running those methods memory usage spikes but then comes back down to a more reasonable amount when the method calls complete? |
Yes, memory usage spikes, but then comes back down. I'm trying to investigate better inside fit, but in predict_point_and_var, I identified that the spike of memory comes after the second Parallel call inside var condition, so I think memory spike is probably origined on these rows: EconML/econml/grf/_base_grf.py Lines 703 to 763 in db1e254
|
Another important detail. I was using a treatment dataframe with featurizer, making me have 6 columns in T. I was inspecting code, and, in many steps, they use a cross product of T over T. I think this is contributing for this memory spike too. |
Hi!
I'm facing multiple memory issues when using CausalForestDML class with big dataframes.
In both the fit method and the effect / effect inference methods, larger dataframes (even not so large, ex: 2M rows / 20 columns) tend to have an exponential memory occupation (~180GB ram).
We could start an investigation inside the code to try to minimize the memory footprint / optimize both methods below:
EconML/econml/grf/_base_grf.py
Line 650 in db1e254
EconML/econml/grf/_base_grf.py
Line 192 in db1e254
My first investigations are tending to indicate that problem is originating from these methods above.
Any clues? @kbattocchi
The text was updated successfully, but these errors were encountered: