Memory leak in ADVI

[sajaysurya]

Memory leak while using ADVI with np.ma.array for observed values

A minimal, self-contained, and reproducible example.

import numpy as np
import pymc3 as pm
# generate dataset
x_obs = np.random.normal(loc=37, scale=1, size=100000)
x_obs = np.ma.array(x_obs, mask=x_obs>37)
# define model
with pm.Model() as model:
    z = pm.Normal('z', mu=0, sd=10)
    x = pm.Normal('x', mu=z, sd=1, observed=x_obs)
# fit model
with model:
    approx = pm.fit(n=10000, method='advi', obj_optimizer=pm.adam(learning_rate=0.1), obj_n_mc=2)

Warnings while defining the model

/usr/local/miniconda3/envs/doh/lib/python3.7/site-packages/pymc3/model.py:1266: UserWarning: Data in x contains missing values and will be automatically imputed from the sampling distribution.
  warnings.warn(impute_message, UserWarning)

Additional information
There is no memory leak if either one of the following happens

• obj_n_mc=1
• np.array is used instead of np.ma.array for observed values.

Memory leak while using ADVI without np.ma.array for observed values

A minimal, self-contained, and reproducible example.

import numpy as np
import pymc3 as pm
import theano
# generate dataset
x_obs = np.random.normal(loc=37, scale=1, size=100000)
dim_num = 1000
idx_list = np.random.choice(dim_num, size=20000)
# define model
with pm.Model() as model:
    z = pm.Normal('z', mu=0, sd=1, shape=dim_num)
    x = pm.Normal('x', mu=theano.tensor.sum(z[idx_list]), sd=1, observed=x_obs)
# fit model
with model:
    approx = pm.fit(n=10000, method='advi', obj_optimizer=pm.adam(learning_rate=0.1), obj_n_mc=2)

Additional information
Again, there is no memory leak if obj_n_mc=1. But it leaks (approximately) to the same extent when obj_n_mc=2 and when obj_n_mc=25.

Versions and main components

• PyMC3 Version: 3.6
• Theano Version: 1.0.3
• Python Version: 3.7.3
• Operating system: macOS
• How did you install PyMC3: conda

[junpenglao]

np.array is used instead of np.ma.array for observed values.

np.ma.array observed will make pymc3 model add a free variable to the model with the length equal to the number of masked elements, and it will likely cause the memory leak.

[ferrine]

Increasing obj_n_mc also cause memory consumption, you have obj_n_mc times more calculations for gradient.

[sajaysurya]

Please refer to the edits in the original issue. The memory leak happens gradually with the fitting steps. After filling up most of the RAM, it comfortably fills up the swapspace without slowing down. I also tested it in ubuntu linux (pymc3 installed via pip) on GCP.

[ferrine]

I can have a look after weekends

…

On Fri, Jul 26, 2019, 13:45 Sajaysurya Ganesh ***@***.***> wrote: Please refer to the edits in the original issue. The memory leak happens gradually with the fitting steps. After filling up most of the RAM, it comfortably fills up the swapspace without slowing down. I also tested it in ubuntu linux (pymc3 installed via pip) on GCP. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3566>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACZJX3TG722V7SBVLEEJW53QBLIT3ANCNFSM4IGZBQPQ> .

[ricardoV94]

Closing due to lack of activity

[bravoila]

Is it solved? I also observed memory leakage for large dataset (10m params to estimate)

[fonnesbeck]

This is a very old issue. Please open a new one if you are seeing this now -- are you sure its leakage and not simply exhausting your memory? That's a lot of parameters!