Skip to content

BUG FIX: Shut down relaunched cluster workers when the plan is stopped - #839

Open
Arthur031221 wants to merge 1 commit into
futureverse:developfrom
Arthur031221:fix-relaunched-worker-leak
Open

Arthur031221 wants to merge 1 commit into
futureverse:developfrom
Arthur031221:fix-relaunched-worker-leak

Conversation

@Arthur031221

Copy link
Copy Markdown

A user of plan(multisession), or of plan(cluster, interrupts = TRUE), who cancels a future and then starts another one gets a relaunched worker that plan(sequential) never shuts down; its process and connection stay open until garbage collection, which also produces the "closing unused connection" warning. The same happens with plan(cluster) when a worker died by other means and was relaunched.

When a terminated worker is relaunched (in requestNode() and handleInterruptedFuture()), the new node is stored in the backend but not in clusterRegistry. clusterRegistry$stopCluster() then closes the old, already closed, node. This change adds clusterRegistry$replaceNode() and calls it at both relaunch sites, when the backend's workers are the cluster the registry started. A cluster passed in by the user, for example plan(cluster, workers = cl), is not owned by the registry, so nothing changes for it.

Reproduction, before the change (Linux, one worker, the backend object held so garbage collection cannot hide it):

library(future)
plan(multisession, workers = I(1))
f <- future(Sys.sleep(30)); Sys.sleep(1); cancel(f)
f2 <- future(42); value(f2)
pid <- plan("backend")$workers[[1]]$session_info$process$pid
b <- plan("backend")
plan(sequential); Sys.sleep(1)
tools::pskill(pid, 0L)   # TRUE: worker still running
nrow(showConnections())  # 1

After the change the worker is gone and there are 0 connections.

The new test test-cancel-relaunch-cluster.R counts open connections around this sequence. On the parent commit it fails (before = 0, after = 1, then Error: n <= n0 is not TRUE); with the change it passes (before = 0, after = 0). R CMD check --as-cran on the change ran 112 test scripts; 111 pass, and test-multisession-libpaths.R fails the same way on the parent commit in my environment. The existing cancel, cluster,worker-termination, interrupts-from-worker-itself, value-error-cancels-set, cluster-connection-clashes, plan and reset tests are among those that pass.

This also touches the symptom in #830. The loop from #820 (100 iterations of 10 erroring futures on 2 multisession workers, options(warn = 1), counting the warning lines) printed "closing unused connection" between 7 and 9 times per run on the parent commit across five runs, and 0 times in every run with the change. The leak is intermittent, so I only claim the cancel and relaunch case shown above. I did not check whether other causes of that warning remain.

When a cluster worker is terminated (for example by cancel()) and then
relaunched, the new node was stored in the backend but not in the
cluster registry. stopCluster() then closed the old node and left the
relaunched worker process and its connection open until garbage
collection.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant