Skip to content

Disable async quorum for the first quorum sync #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Disable async quorum for the first quorum sync #112

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion torchft/manager.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ def __init__(
self._quorum_timeout = quorum_timeout
self._connect_timeout = connect_timeout
self._world_size_mode = world_size_mode
self._first_quorum = True

store_addr = store_addr or os.environ["MASTER_ADDR"]
store_port = store_port or int(os.environ["MASTER_PORT"])
Expand Down Expand Up @@ -404,8 +405,11 @@ def start_quorum(
shrink_only=shrink_only,
quorum_timeout=timeout or self._quorum_timeout,
)
if not self._use_async_quorum:
# If this is the first quorum sync, we need to wait for the result.
# Otherwise, we may mistakenly perform the forward with incorrect weights.
if not self._use_async_quorum or self._first_quorum:
self.wait_quorum()
self._first_quorum = False

if self._healing:
# eagerly apply pending state_dict so we can run the forwards pass
Expand Down
Loading