Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Memory and Throughput Profiling of Train Steps #360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
le1nux merged 21 commits into from
May 26, 2025
Merged

Memory and Throughput Profiling of Train Steps #360

le1nux merged 21 commits into from
May 26, 2025

Conversation

@le1nux
Copy link
Member

@le1nux le1nux commented Apr 24, 2025

What does this PR do?

This PR ..

General Changes

  • ..

Breaking Changes

  • ..

Checklist before submitting final PR

  • My PR is minimal and addresses one issue in isolation
  • I have merged the latest version of the target branch into this feature branch
  • I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
  • I have run a sample config for model training
  • I have checked that all tests run through (python tests/tests.py)
  • I have updated the internal changelog (CHANGELOG_DEV.md)

le1nux added 19 commits April 24, 2025 19:13
@le1nux
refactor: moved Main from __main__ to main
c346f0f
@le1nux
feat: added batch generator util
90c4bbe
@le1nux
refactor: split experiment_id syncing into multiple utity functions
c87688a
@le1nux
feat: implemented grid search setup for profiling
ebaf13e
@le1nux
refactor: added OOM error handling in CudaEnv
bcc0e7b
@le1nux
feat: added torchrun script for distributed profiling
899401a
@le1nux
feat: added profiling README
87296d1
@le1nux
feat: added profiler implementation
d6dddc0
@le1nux
feat: drafted profile logs analyzer
011e41b
@le1nux
chore: minor renamings
e6da67f
@le1nux
refactor: making sure that each compil
2d4a9ad
@le1nux
feat: added torchrun launcher
d990420
@le1nux
refactor: wrapped up the profiler_starter
f4a5b48
@le1nux
feat: added activation checkpoint profiling example
b2aa30b
@le1nux
feat: setup forward pass profiling
94025af
@le1nux
feat: ops in selective op activation checkpointing are now configurable
3ea7cf5
@le1nux
feat: added profliing logs analysis notebook
0b99db2
@le1nux
refactor: adapted the configs and evaluation code for selective op AC
1889776
@le1nux
chore: added profiling experiments to gitignore
c718182
@le1nux le1nux marked this pull request as ready for review May 26, 2025 16:37
@le1nux le1nux merged commit ed2772c into fsdp2_activation_checkpointing May 26, 2025
3 checks passed
@le1nux le1nux deleted the profiling_feature branch May 26, 2025 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@le1nux
Morty Proxy This is a proxified and sanitized view of the page, visit original site.