[graph-work-side-branch]: temp side branch for graph work #9692
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Important Review skippedAuto reviews are limited to specific labels. 🏷️ Labels to auto review (1)
Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Pull reviewers statsStats of the last 30 days for lnd:
|
|
^ purely a rebase on master |
|
^ rebase on master for inclusion of #9789 |
|
^ rebase to include #9798 |
We want `context.TODO()` to be high signal in the code-base. It should signal clearly that work is required to thread parent context through to the call-site. So to keep the signal-to-noise ratio high, we remove any context.TODO() calls from tests since these will never need to be replace by a parent context.
We want `context.TODO()` to be high signal in the code-base. It should signal clearly that work is required to thread parent context through to the call-site. So to keep the signal-to-noise ratio high, we remove any context.TODO() calls from tests since these will never need to be replace by a parent context. After this commit, there is only a single context.TODO() left in the code-base.
In preparation for starting to thread a single parent context through LND, we update the main `server.Start` method to take a context so that it can later pass it to any subsytem's Start method it calls. We also pass the context to `newServer` since it makes some calls that will eventually reach the DB (for example the graph db).
Pass the parent LND context to the gossiper, let it derive a child context that gets cancelled on Stop. Pass the context through to any methods that will eventually thread it through to any graph DB calls. One `context.TODO()` is added here - this will be removed in the next commit. NOTE: for any internal methods that the context gets passed to, if those methods already listen on the gossiper's `quit` channel, then then don't need to also listen on the passed context's Done() channel because the quit channel is closed at the same time that the context is cancelled.
And remove a context.TODO() that was added in the previous commit.
The `GossiperSyncer` makes various calls to the `ChannelGraphTimeSeries` interface which threads through to the graph DB. So in preparation for threading context through to all the methods on that interface, we update the GossipSyncer accordingly by passing contexts through. Two `context.TODO()`s are added in this commit. They will be removed in the upcoming commits.
Here, we remove one context.TODO() by threading a context through to the SyncManager.
With this, we move a context.TODO() out of the gossiper and into the brontide package - this will be removed in a future PR which focuses on threading contexts through that code.
Since the ChannelGraphBootstrapper implementation makes a call to the graph DB.
For any method that takes a context that has a select that listens on the systems quit channel, we should also listen on the ctx since we should not need to worry about if this context is derived internally or externally.
This commit cleans up the graph test code by removing unused kvdb type parameters from the `createTextVertex` and `createLightningNode` helper methods. We also pass in the testing parameter now so that we dont need to check the error each time we call `createTestVertex`.
Remove the kvdb.Backend parameter from the `createChannelEdge` helper. This is all in preparation for having the unit tests run against any DB backend.
Remove unused kvdb.Backend param from `randEdgePolicy` and `newEdgePolicy` test helpers.
Replace all tests calls to the private `forEachNode` method on the `KVStore` with the exported ForEachNode method. This is in preparation for having the tests run against an abstract DB backend.
The `GraphSource` interface in the `autopilot` package is directly implemented by the `graphdb.KVStore` and so we will eventually thread contexts through to this interface. So in this commit, we start updating the autopilot system to thread contexts through in preparation for passing the context through to any calls made to the GraphSource. Two context.TODOs are added here which will be addressed in follow up commits.
Remove one context.TODO and add one more.
Continue threading context through the autopilot system and remove the remaining context.TODOs.
Later on we will create an interface for the persisted graph data. We want this interface to be as small and as neat as possible. In preparation for this, we remove this unused `Wipe` method.
In preparation for creating a clean interface for the graph store, we want to hide anything that is DB specific from the exposed methods on the interface. Currently the `ForEachNodeChannel` and the `FetchOtherNode` methods of the `KVStore` expose a `kvdb.RTx` parameter which is bbolt specific. There is only one call-site of `ForEachNodeChannel` actually makes use of the passed `kvdb.RTx` parameter, and that is in the `establishPersistentConnections` method of the `server` which then passes the tx parameter to `FetchOtherNode`. So to clean-up the interface such that the `kvdb.RTx` is no longer exposed: we instead create one new method called `ForEachSourceNodeChannel` which can be used to replace the above mentioned call-site. So as of this commit, all the remaining call-site of `ForEachNodeChannel` pass in a nil param for `kvdb.RTx` - meaning we can remove the parameter in a future commit.
Replace all calls to bbolt specific methods on the KVStore to instead use exported methods on the KVStore that are more db-agnostic.
Since we have not removed all call-sites that make use of this parameter, we can remove it. This helps hide DB-specific details from the interface we will introduce for the graph store.
Later on we will store the Alias as a Text field in our sql impls of the graph db. For Postgres, this field then MUST be a valid UTF-8 string. This is also the case in general for the alias according to [bolt 7](https://github.com/lightning/bolts/blob/e1fa25cf00446f3a6a6abbbc9a617cae5b75e39f/07-routing-gossip.md?plain=1#L313)
- Let it do a proper comparison of the full structs passed in. - Pass in a testing parameter so we can remove the returned error. - Make sure the callers are passing in the expected and result parameters in the correct order. - Fix a bug: the compareNodes was not comparing the Features field of the LightningNode structs. Now that it does, one test needed to be updated to properly set the expected Features fields.
Remove the kvdb.Backend return type of the `makeTestGraph` helper. This is in preparation for the helper being used to create a test graph backed by a DB other than bbolt.
The channeldb no longer depends on the graph. So remove the use of MakeTestGraph from tests.
Currently, a few of the graph KVStore methods take the `batch.SchedulerOptions` param. This is only used to set the LazyAdd option. A SchedulerOption is a functional option that takes a `batch.Request` which has bolt-specific fields in it. This commit restructures things a bit such that the `batch.Request` type is no longer part of the `batch.SchedulerOptions` - this will make it easier to implement the graph store with a different DB backend.
In this commit, we introduce the `V1Store` interface which the existing `graphdb.KVStore` implements today. The idea is to eventually create a SQL DB backed implementation of this interface.
Use the new `V1Store` interface as a field in the `ChannelGraph` rather than a pointer to the `KVStore` implementation in preparation for allowing a SQL implementation of `V1Store` to be used by the `ChannelGraph`. Note that two tests are adjusted in this commit to be skipped if the V1Store is not the `KVStore` implementation since the tests are very bbolt specific.
So that we can pass in the abstract V1Store in preparation for adding a SQL implementation of the KVStore.
Later when we introduce our SQL version of the graph store, we will normalise the persistence of the ExtraOpaqueData using the fact that it is always made up of TLV entries. So we update our tests here to ensure that they use valid TLV streams as examples.
In this commit, we check that the extra bytes appended to gossip messages contain valid TLV streams. We do this here for: - channel_announcement - channel_announcement_2 - channel_update - channel_update_2 - node_announcement This is in preparation for the SQL version of the graph store which will normalise TLV streams at persistence time.
Expand this existing test so that it also tests that a node's addresses and feature are fetched correctly after insertion. With this, we ensure that the `FetchNodeFeatures` and `AddrsForNodes` methods of the `V1Store` interface are properly covered by unit tests.
In preparation for a different store impl which may wrap errors, we use `require.ErrorIs` for error assertions rather than direct "=" comparisons in tests.
So we can use`require.Equal` for the ChannelEdgeInfo type.
In preparation for our SQL Graph store which wont explicitly store the chain hash but will instead obtain it from the runtime config, we replace the test chainhash value with that of the mainnet genesis hash.
Instead of returning an error and needing to call `require.NoError` for each call to `MakeTestGraph`, rather just used the available testing variable to require no error within the function itself.
|
^ rebase to include #9804 |
To clearly separate the mission control DB store (which for now will remain kvdb only) from the graph store, we rename the `graphBackend` variable to `mcBackend` in the `testGraphInstance` struct.
Currently none of the calls to MakeTestGraph make use of the KVStoreOptionModifier options but later on we will want to make use of the `WithUseGraphCache` ChannelGraphOption and so we take this opportunity to switch out the functional parameters that the helper function takes.
In this commit, we unify how all unit tests that make use of the graph create their test ChannelGraph instance. This will make it easier to ensure that once we plug in different graph DB implementations, that all unit tests are run against all variants of the graph DB. With this commit, `NewChannelGraph` is mainly only called via `MakeTestGraph` for all tests _except_ for `TestGraphLoading` which needs to be able to reload a ChannelGraph with the same backend. This will be addressed in a follow-up commit once more helpers are defined. Note that in all previous packages where we created a test graph using `kvdb.GetBoltBackend`, we now need to add a `TestMain` function with a call to `kvdb.RunTest` since the `MakeTestGraph` helper uses `GetTestBackend` instead of `kvdb.GetBoltBackend` which requires an embedded postgres instance to be running.
In this commit, we add a `test_kvdb.go` file with a single definition of the `NewTestDB` function. A new version of `MakeTestGraph` (called `MakeTestGraphNew` is added which makes use of this `NewTestDB` function to create the backing `V1Store` passed to the `ChannelGraph` for tests. Later on, we will add new implementations of this method backed by sqlite and postgres. When those are added, then build flags will control which version of `NewTestDB` is called. With this change, the only test call-site of `NewKVStore` is the new `test_kvdb.go` file.
Update the test methods to take a `testing.TB` param instead of a `*testing.T` so that the test functions can be used for all the graph's tests including benchmark tests. We also add a temporary replace so that we can make use of these changes and also since we will soon be adding graph related schemas and queries in this submodule that we'll want to have access to.
In this commit, we implement the postgres and sqlite versions of the NewTestDB function. We add the various build flags so that only one of the three versions of this function can be active at a time. We also introduce the SQLStore struct which is the SQL implementation of the V1Store interface. NOTE: it currently temporarily embeds the KVStore struct so that we can implement the V1Store interface incrementally. For any method not implemented, things will fall back to the KVStore. This is ONLY the case for the time being while this struct is purely used in unit tests only. Once all the methods have been implemented, the KVStore field will be removed from the SQLStore struct.
Add `migrations_dev.go` and `migrations_prod.go` files which each define a `migrationAdditions` slice to be appended to the `migrationConfig` slice. The `migrations_dev.go` file is only built if either the `test_db_postgres` or `test_db_sqlite` build flags are used. This slice will be used to add any migrations that are still under development that we want access to via unit tests and itests but do not want to expose to our release build.
multi: various test preparations for different graph store impl
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR shows all the work merged into the
elle-graphside branch. This will be a mixture of the remaining work from the Graph Abstraction project along with some of the incoming native SQL graph migration work (mainly clean-up/refactor prep work).We can merge this into master once v0.19 has dropped. I'll leave it in draft until then.
This PR has the following PRs merged in:
context.TODO#9681kvdbparameters from exported methods #9695Startmethods #9704