Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ADD] Support KFAC if the network has inplace activations #59

Merged
merged 3 commits into from
Nov 8, 2023

Conversation

f-dangel
Copy link
Owner

@f-dangel f-dangel commented Nov 8, 2023

Resolves #58.

@f-dangel f-dangel requested a review from runame November 8, 2023 15:10
@coveralls
Copy link

coveralls commented Nov 8, 2023

Pull Request Test Coverage Report for Build 6802882104

  • 6 of 6 (100.0%) changed or added relevant lines in 1 file are covered.
  • 40 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-4.7%) to 87.82%

Files with Coverage Reduction New Missed Lines %
curvlinops/examples/utils.py 6 41.67%
curvlinops/fisher.py 34 28.85%
Totals Coverage Status
Change from base Build 6801631981: -4.7%
Covered Lines: 721
Relevant Lines: 821

💛 - Coveralls

Copy link
Collaborator

@runame runame left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we also apply the same refactor as in SINGD, where we just use a single forward hook that computes the input covariance and installs the tensor hook on the module output?

test/test_kfac.py Show resolved Hide resolved
@f-dangel
Copy link
Owner Author

f-dangel commented Nov 8, 2023

One point against merging these hooks is that we may want to re-cycle the hook for computing the activation variance (e.g. FOOF).

@runame
Copy link
Collaborator

runame commented Nov 8, 2023

Ok let's stick to separate hooks for now then.

@f-dangel f-dangel merged commit 10180ce into development Nov 8, 2023
12 checks passed
@f-dangel f-dangel deleted the kfac-with-inplace-activations branch November 8, 2023 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants