Add ResNet50 dendritic optimization example#97
Open
PioGodwin-M wants to merge 8 commits into
Open
Add ResNet50 dendritic optimization example#97PioGodwin-M wants to merge 8 commits into
PioGodwin-M wants to merge 8 commits into
Conversation
Contributor
|
Thanks for the submission!
Thanks for the submission! |
Author
|
"Thanks for the feedback! Sorry for the late reply. I’ve fixed the optimizer issue (re-initializing it now prevents the zombie dendrites) and bumped the switch epoch to 10. I also moved the files to the correct folder. I couldn't upload the full PAI folder due to the file size limit, but I've attached the resulting PAI.png graph which shows the learning and drive link of PAI folder : https://drive.google.com/drive/folders/1iFlAc9J2Q7Onsi9eACR5WuoPj_Dw4qLh?usp=drive_link." |
Contributor
|
Cool, looks better, but still looks like dendrites are being added early. You want the network to train until plateau before dendrites are added. I'd raise it to at least 25. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ResNet50 Dendritic Optimization Case Study
Project Overview
This project implements a dendritic optimization strategy for the ResNet50 architecture trained on CIFAR-10/100. utilizing the PerforatedAI framework. It demonstrates how dynamic parameter allocation (artificial dendrites) can improve model efficiency and overcome training plateaus by adding capacity only where needed.
Key Features
Dynamic Growth: Starts with a baseline ResNet50 and automatically grows new connections (dendrites) based on validation performance metrics.
PerforatedAI Integration: Seamlessly wraps PyTorch modules to manage dendritic candidates and structural adaptation.
Optimized Training Loop: Custom training pipeline with automated optimizer resets and learning rate scheduling upon structural changes.
Experiment Tracking: Integrated with Weights & Biases (W&B) for real-time visualization of loss, accuracy, and parameter growth.
Results
Demonstrated autonomous parameter growth (e.g., from ~23.7M to 24.3M parameters) in response to training needs.
Successfully applied dendritic connections to both convolutional and fully connected layers.
Repository Structure
src/train.py: Core training script with PerforatedAI hooks.
src/model.py: ResNet50 model definition.
PAI/: Optimization logs and growth visualization graphs.
This i my PAI.png. kindly review the project and let me know if any changes to be made sir.