Skip current model generation when memory limit threshold hit

[emobs]

As mentioned in the tutorials, most of the time AutoTS crashes are caused by memory peaks where not enough memory can be allocated while fitting data and generating models. Sometimes these memory peaks are only occasionally occurring, while other model generations with the same amount of data run fine. Instead of manually eliminating the (current) model on which the script crashed and having to start all over again, wouldn't it be possible to add a functionality to skip generation of a certain model (using certain transformers which may cause memory issues in particular) if the free memory is below a certain threshold (e.g. 10% of the total system memory to be set as a model parameter)? It would help (me myself at least) a lot if such models would just be skipped and the script moved on to the next model generation instead of crashing after which it has to be started again excluding that specific model.

Curious about your opinion on such functionality as a new AutoTS feature.

[winedarksea]

It depends on what you mean by skipping.
It is very difficult to intercede if the memory consumption gets too large in a current model. Maybe something could be done where two processes are created and the first process monitors the second and kills it if it gets too large. But that seems like it might be problematic and unreliable.

Right now you can filter by transformer_list and only specify transformers you trust. The "scalable" list alias is meant to be only safe transformers, and I can modify that as needed. The current_model_file can be specified to save models. If it crashes, view the model saved there, the last one attempted, and post it to me so I can correct it/remove it.

Having a selective memory filtering, where it allows some params if free memory is low but prevents if it is high is possible, but I don't see how that would help very much - usually when it crashes, it crashes because it is trying to allocated a very large (> 2x the system capacity) amount, and starting from 10% vs 60% used memory wouldn't have saved it.

I very much want to solve this problem but it doesn't have any clear solutions. My official current solution is to stick to the "scalable" model_list and transformer_list, and make sure those don't fail on test scripts

[emobs]

Thanks for the extensive reply Colin. I should have known this solution would be too easy and I understand why it isn't a proper one, thanks for explaining that. I'm already working with scalable models and transformers lists, but still have some memory peaks from time to time, especially with large training datasets. Will keep track of the models causing these peaks and send them to you. The most recent crash caused by memory outage was on this model: ``` {"model_number": 759, "model_name": "WindowRegression", "model_param_dict": {"window_size": 20, "input_dim": "univariate", "output_dim": "forecast_length", "normalize_window": false, "max_windows": 5000000, "regression_type": null, "regression_model": {"model": "KNN", "model_params": {"n_neighbors": 10, "weights": "uniform", "p": 2, "leaf_size": 30}}}, "model_transform_dict": {"fillna": "mean", "transformations": {"0": "QuantileTransformer", "1": "RobustScaler"}, "transformation_params": {"0": {"output_distribution": "uniform", "n_quantiles": 100}, "1": {}}}} ``` My (wide) data have dimensions 20000 x 34. Would you expect a lower value setting for the `transformer_max_depth` parameter (currently 6) to help reduce memory usage (apart from less records, e.g. 5000x34)? By the way, I'm sending a copy of this message to your email (bcc) because your Github account seems to be down (it's unreachable from my end, your main account page as well as all the AutoTS pages). Op do 20 jun 2024 om 01:58 schreef Colin Catlin ***@***.***>:

…

It depends on what you mean by skipping. It is very difficult to intercede if the memory consumption gets too large in a current model. Maybe something could be done where two processes are created and the first process monitors the second and kills it if it gets too large. But that seems like it might be problematic and unreliable. Right now you can filter by transformer_list and only specify transformers you trust. The "scalable" list alias is meant to be only safe transformers, and I can modify that as needed. The current_model_file can be specified to save models. If it crashes, view the model saved there, the last one attempted, and post it to me so I can correct it/remove it. Having a selective memory filtering, where it allows some params if free memory is low but prevents if it is high is possible, but I don't see how that would help very much - usually when it crashes, it crashes because it is trying to allocated a very large (> 2x the system capacity) amount, and starting from 10% vs 60% used memory wouldn't have saved it. I very much want to *solve* this problem but it doesn't have any clear solutions. My official current solution is to stick to the "scalable" model_list and transformer_list, and make sure those don't fail on test scripts — Reply to this email directly, view it on GitHub <#246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFI7ZL6BEIXD6OJW7MOURK3ZIILKZAVCNFSM6AAAAABJSBQZFKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TQMRSG42TI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[winedarksea]

Thanks for the heads up on my account being "shadow banned". That was annoying. Thankfully they fixed it within a few hours of me opening a support request (apparently it was an accident caused by an false positive in an automated abuse detection system).

No, I don't think the transformer_max_depth will likely be the issue. Some transformers (LocalLinearTrend, KalmanSmoothing) are known to get a bit memory hungry. But something like your Robust Scaler there in should be fine being used 100 times on the same model without any serious memory issues.

I suspect the issue is with the KNN model there. Those tend to be memory hungry. max_windows can also be made smaller to reduce memory consumption, but that's not really something you can control right now from the search. You can somewhat control the search space on ~Regression models regression_model by using the models_mode arg passing "gradient_boosting", "neuralnets", or "trees", for example, all of which would prevent KNN from being tested.

My main 'large' test is more like 1000 * 30000, I don't really test 20,000 history points, rather 20,000 series, so that could be an issue with my current testing.