tipping_probabilities: Kills Kernel when basin array is large

[mdepitta]

Describe the bug
Upon invoking prob = tipping_probabilities(basins_before,basins_after), Julia kernel dies. I did some testing. Could it depend on the array size? If the array size of the provided basins before/after is large, could it cause Julia Kernel to die (possibly by memory exhaustion)? I am asking since I am new to Julia and do not know how to debug any "Killed" kernel message. In my case, basins_before and basins_after are 4-D arrays of 100x100x100x100 elements, reaching up to 200 MB, but just with three deterministic attractors. The formula for the computation of the tipping probabilities is straightforward. tried with much smaller arrays and `tipping_probabilities' works. I am not sure what else could cause the killing.

Package versions
Status ~/.julia/environments/v1.8/Manifest.toml
[608a59af] ChaosTools v2.9.0
⌅ [5732040d] DelayEmbeddings v2.4.1
[61744808] DynamicalSystems v2.3.2
⌅ [6e36e845] DynamicalSystemsBase v2.8.0
⌅ [ed8fcbec] Entropies v1.1.2
[639c3291] RecurrenceAnalysis v1.8.1
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use status --outdated -m

[mdepitta]

It seems indeed the case. I am reading the system log:

Out of memory: Killed process 13567 (julia) total-vm:33453780kB, anon-rss:29431324kB, file-rss:4kB, shmem-rss:0kB, UID:1000 pgtables:58940kB oom_score_adj:0

It is pretty strange, however, since htop did not show any memory exhaustion. Is there a workaround for this, rather than re-computing basins of attraction in a lower-resolution grid?

[awage]

It seems complicated without using a coarser grid. You need the two basins to compute the tipping probabilities. I don't know if there is a way to compute these probabilities iteratively. I am not familiar with the technique.

[awage]

Looking at the code of tipping_probabilities I think you can use an iterative method to compute the tipping probs. You can submit a PR if you wish and I can help you with the code.

[mdepitta]

Thanks, @awage. I opened the ~/.julia/packages/ChaosTools/src/tipping.jl, and the computation of the probabilities is a matter of two nested for cycles. The main issue, I guess, is when you have very large arrays, the two cycles have to run over all combinations in the provided basins of attraction before and after..., and that makes essentially, in my case, 10^8 x 10^8, which is hardly achievable on a laptop, even if geared towards high-performance. Probably the issue is solvable on my workstation. Eventually, it narrows down to how I provide my basins (the other thread). Hence, let me work on that and update you accordingly.

Thanks in the meantime!

[Datseris]

Notice that Attractors.jl is a separate module now that has several updates and modifications from the version of basins/tipping/etc. currently present in the feature frozen DynamicalSystems.jl documentation. I've transferred the issue to the appropriate repo. I'd recommend using Attractors.jl directly for now.