Find Mode TrustVerify #3604
Find Mode TrustVerify #3604
Conversation
cderb
requested review from
BrianHarrisonAMD,
BradPepersAMD,
adickin-amd and
JonathanLichtnerAMD
as code owners
include/miopen/miopen.h
Outdated
| miopenConvolutionFindModeTrustVerify = | ||
| 6, |
There was a problem hiding this comment.
Needs docs like the others.
The PR description is probably pretty close to what is needed!
|
The performance of find mode TrustVerify is between the performance of find mode DynamicHybrid and find enforce SearchDbUpdate.
First run with TrustVerify is generally slower than DynamicHybrid. |
| @@ -29,7 +29,6 @@ Enable this feature using these commands: | |||
|
|
|||
| .. code:: bash | |||
|
|
|||
| export MIOPEN_FIND_MODE=3 | |||
There was a problem hiding this comment.
Found that setting find enforce disables find mode inputs. So removing these unnecessary lines.
Co-authored-by: Jeffrey Novotny <[email protected]>
| auto ufdb_sols = miopen::GetSolutions<UserFindDb>(ctx, problem, 1, &invoke_ctx); | ||
| if(ufdb_sols.empty()) |
There was a problem hiding this comment.
Looks like we need to read the DB twice.
I don't think it's a huge deal since it'll be cached, but if we knew which DB the result came from we could avoid the extra read.
There was a problem hiding this comment.
Correct, we need to know if the entry is from the system or the user db, as there are different actions depending. There doesn't appear to be a return from the GetSolutions function presently which would indicate that.
| false); | ||
|
|
||
| const float eval_time = eval_sols.front().GetTime(); | ||
| constexpr float VERIFY_TOLERANCE = 1.10f; |
There was a problem hiding this comment.
10% is pretty tight, but maybe we are okay with that?
There was a problem hiding this comment.
It's ok to loosen this tolerance. I've also seen much higher degrees of variation even when the selected solver and solver parameters are correct.
|
Can we get a list of current issues this should fix or improve? As well as any situations that this may cause drops compared to existing results? In particular, I'm thinking of cases on distributed runs where they may be starting with a blank user db on every node. |
|
This find mode option will function like MIOPEN_FIND_ENFORCE=3 in the worst case. Worst case is when either there is no entry or the system entry is much slower than advertised. I had wanted to split the option for tuning individual solvers to MIOPEN_FIND_ENFORCE, but this env overrides MIOPEN_FIND_MODE. So presently TRUST_VERIFY also effectively enforces MIOPEN_FIND_ENFORCE=SEARCH_DB_UPDATE, which can take quite some time. Having another option that forgoes individual solver tuning may give better results for shorter running applications and would cap the runtime closer to find mode DYNAMIC_HYBRID. |
Additional MIOPEN_FIND_MODE = 6 (TrustVerify)
This mode extends DynamicHybrid
Running with TrustVerify will first attempt to load tuning results from system resources
If no solution is returned tuning will be triggered
If a solution is retuned the user find db will be checked for the solution
If solution is from the user find db it will be used
If a solution is from the system (db or model), the solution will be evaluated and the new time will be compared to the time reported by the solution
If the evaluated / reported time is less than the tolerance threshold then the system result is added to the user db and the solution is returned
If the evaluated / reported time exceeds the tolerance, then tuning will be triggered
This find mode will ensure that all configurations are tuned for the deployment system. Solutions from system resources are verified once and tuned if markedly different from expectation. Results from user dbs are considered reliable and used without further verification.