Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError: Can't pickle local object 'IRT_MMLE_2PL.__update_theta_distr.<locals>.update' #22

Open
nix1 opened this issue Jun 30, 2021 · 1 comment

Comments

@nix1
Copy link

nix1 commented Jun 30, 2021

I'm trying to use PyIRT, however it seems that the library has some issues with how it uses multiprocessing. I guess it used to be fine some time ago, I'm currently using Python 3.9.1.

It seems that essentially PyIRT passes a local update function as a target to Process. In effect, when multiprocessing tries to fork and spawn a new process, it tries to dump a pickle of the function, which fails with AttributeError: Can't pickle local object 'IRT_MMLE_2PL.__update_theta_distr.<locals>.update'.

Does anyone have a solution or a workaround to this issue?

@nix1
Copy link
Author

nix1 commented Jun 30, 2021

For now, I just removed creating processes in pyirt/solver/model.py. It probably became slower, but it worked.

So for instance:

        for i in range(num_chunk):
            p = mp.Process(target=update, args=(user_ell, user_cnt, chunk_list[i][0], chunk_list[i][1],))
            procs.append(p)
        if num_chunk > 1:
            procs = procs_operator(procs, 1200, self.check_interval)
        else:
            procs = procs_operator(procs, 7200, 0.1)

becomes:

        for i in range(num_chunk):
            update(user_ell, user_cnt, chunk_list[i][0], chunk_list[i][1])

A similar situation happens around 4 times in this file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant