Separate forward and backwad compilation and support higher order derivatives for aot_function #856
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 7, 2022
ghstack-source-id: 3197eae34a9003563e0b56235540c17a0ba0617c Pull Request resolved: #856
anjali411
commented
Jun 7, 2022
…tion" [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 7, 2022
ghstack-source-id: 4de63f2aff78e0575fc342e13688308c542aa62f Pull Request resolved: #856
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 7, 2022
ghstack-source-id: 4bc7082b2b8cfc1f980b303689ab193e17ccb6c7 Pull Request resolved: #856
anjali411
commented
Jun 7, 2022
functorch/_src/aot_autograd.py
Outdated
| @@ -140,12 +140,15 @@ def create_aot_autograd_function( | |||
| compiled_fw = None | |||
| compiled_bw = None | |||
| num_outs = None | |||
| joint_inputs = None | |||
There was a problem hiding this comment.
need to save these tensors in the context
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 9, 2022
ghstack-source-id: c24ee1b8c252d9aebe99b0beb9139dd3eb223dd4 Pull Request resolved: #856
zou3519
reviewed
Jun 9, 2022
functorch/_src/aot_autograd.py
Outdated
Comment on lines
217
to
220
| func_code = bw_module.code.split('self, ') | ||
| # print(func_code[0] + func_code[1]) | ||
| exec(func_code[0] + func_code[1], globals()) | ||
| f = create_aot_autograd_function(forward, bw_compiler, bw_compiler, partition_fn, aot_decompositions, grad_state) |
There was a problem hiding this comment.
Two questions:
- Why are we passing
forwardto create_aot_autograd_function? I would have expected us to pass bw_module.code without the self argument - What is the exec for? Are you trying to test this without the
create_aot_autograd_functionline?
There was a problem hiding this comment.
- forward is the name of the function generated by running
bw_module.code - exec executes the bw_module.code to create a backward function which is the forward for the next pass
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 14, 2022
ghstack-source-id: 248de2f577fe3d61d4f2c40dc04570978dcc1543 Pull Request resolved: #856
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 21, 2022
ghstack-source-id: 056ba4b15716a5e5fca0481de1d9a02c08415a63 Pull Request resolved: #856
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jun 21, 2022
ghstack-source-id: 9f2c62614c3b7864aa58105a1934f4c4dc0fea60 Pull Request resolved: #856
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jul 12, 2022
ghstack-source-id: 0ce1d4ab26357b8614c57d539cdb61f1ae90a25e Pull Request resolved: #856
…tion" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jul 13, 2022
ghstack-source-id: f154dd1cbab518acd5890090ca081db1ec7fa20a Pull Request resolved: #856
anjali411
changed the title
anjali411
changed the title
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jul 13, 2022
ghstack-source-id: 6575ea077b6baddee3ea60a41ca8f5d75ab5238a Pull Request resolved: #856
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jul 13, 2022
ghstack-source-id: a21556926f1fc1f91f3bcd6e3075d77e99868c29 Pull Request resolved: #856
…r order derivatives for aot_function" Test Plan: Existing tests should pass [ghstack-poisoned]
anjali411
added a commit
that referenced
this pull request
Jul 13, 2022
ghstack-source-id: 0b78895219a89ec3841cf8b0804e1be69bfeed8a Pull Request resolved: #856
anjali411
commented
Jul 15, 2022
| def fake_fn(primals, tangents): | ||
| return fx_g_b(primals, tangents) | ||
| fx_g_b = make_fx(functionalize(fake_fn))(flat_tensor_args, inp_grad_outs) | ||
| saved_value_nodes = _get_saved_values(fx_g_b, saved_value_names) |
There was a problem hiding this comment.
Unfortunately this approach doesn't always work because the newly generated fx graph may not have the same nodes as the previous graph. We need an alternate way to select nodes of interest in this new graph!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Test Plan: Existing tests should pass