feat(web): provide lexicon probabilities directly on the search path 📚 #11868
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keymanapp-test-bot
bot
added
the
user-test-missing
User Test ResultsTest specification and instructions User tests are not required Test Artifacts
|
keymanapp-test-bot
bot
removed
the
user-test-missing
This was referenced Jun 25, 2024
Base automatically changed from
refactor/common/models/priority-queue-as-util
to
master
July 3, 2024 01:13
mcdurdin
approved these changes
Jul 5, 2024
| } | ||
| }; | ||
| return; | ||
| } | ||
| } | ||
|
|
||
| get entries(): string[] { | ||
| get entries() { | ||
| const totalWeight = this.totalWeight; |
| @@ -35,6 +35,10 @@ describe('Trie traversal abstractions', function() { | |||
| it('traversal with simple internal nodes', function() { | |||
| var model = new TrieModel(jsonFixture('tries/english-1000')); | |||
|
|
|||
| // Prob: entry weight / total weight | |||
| // "the" is the highest-weighted word in the fixture. | |||
| const PROB_OF_THE = 1000 / 500500; | |||
There was a problem hiding this comment.
Can we move this and PROB_OF_TROUBLE outside the functions so we can use them throughout?
|
Changes in this pull request will be available for download in Keyman version 18.0.70-alpha |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was originally part of #10973.
In order to efficiently traverse a full lexical efficiently for dictionary-based wordbreaking, it's best to directly provide relevant probability data as efficiently as possible. Fortunately, it's easily possible to make this O(1) on the lexical model's internal iterator - the
LexiconTraversaltype. It would take O(log(N)) time to recompute it via the model's.predictmethod instead.Note that this provides two different probability value types:
There are uses for this outside of dictionary-based wordbreaking, too. The latter 'probability' listed above can be useful for optimizing the correction-search - if a path only produces low-frequency words, we should consider other paths that could yield higher-frequency words first.
There's also notable potential for being able to merge / blend two different models together via their
LexiconTraversaliterators in this manner. Noting our upcoming push toward #11872, this would facilitate a fantastic way to achieve that goal - to create a stand-in model for the OS's dictionary and blend that with the loaded lexical-model via traversals.@keymanapp-test-bot skip