Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

dist.gather_object does not correctly compute rank in coordinator check #118337

New issue
Copy link
New issue
Copy link
Closed
Closed
dist.gather_object does not correctly compute rank in coordinator check#118337
Copy link
Assignees
wconstab
Labels
high priorityoncall: distributedAdd this issue/PR to distributed oncall triage queueAdd this issue/PR to distributed oncall triage queuetriage review
@mvpatel2000

Description

@mvpatel2000
mvpatel2000
opened on Jan 26, 2024
Issue body actions

🐛 Describe the bug

pytorch/torch/distributed/distributed_c10d.py

Line 2440 in 8c167f9

my_rank = get_rank()

This line should pass in the group so that it gets the rank in the group instead of the global rank. Otherwise, check's for is_coordinator in sharded checkpoint loading will fail.

Versions

Nightly

cc @ezyang @gchanan @zou3519 @kadeng @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @LucasLLC

Metadata

Metadata

Assignees

Labels

high priorityoncall: distributedAdd this issue/PR to distributed oncall triage queueAdd this issue/PR to distributed oncall triage queuetriage review

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions
    Morty Proxy This is a proxified and sanitized view of the page, visit original site.