-
Notifications
You must be signed in to change notification settings - Fork 279
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wallet TX Count and time indexing #888
Draft
nodech
wants to merge
15
commits into
handshake-org:master
Choose a base branch
from
nodech:wallet-pagination
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5a9ef88
to
7200b63
Compare
nodech
added a commit
to nodech/hsd
that referenced
this pull request
Mar 20, 2024
HSD: handshake-org#888 BCOIN: bcoin-org/bcoin#605 Co-authored-by: Braydon Fuller <[email protected]>
030aa42
to
0fda018
Compare
TODO
|
26e2f02
to
d4ac794
Compare
This was referenced May 23, 2024
This requires full wdb block entry wipe and rescan. That is handled by PR handshake-org#889. `layout.h` is looked up by height, so only missing data was time. Now we can implement walletdb only median time past calculation.
HSD: handshake-org#888 BCOIN: bcoin-org/bcoin#605 Co-authored-by: Braydon Fuller <[email protected]>
Co-authored-by: Braydon Fuller <[email protected]>
test: Add tests for the wallet.zap.
pending. hsw-cli: - hsw-cli: `history` now accepts new args on top of `--account`: `--reverse`, `--limit`, `--after`, `--after`. - hsw-cli: `pending` now accepts new args, same as above. wallet-http: - Deprecate and remove: `GET /wallet/:id/tx/range` - Deprecate and remove: `GET /wallet/:id/tx/last`
wallet-rpc: The following new methods have been added: - `listhistory` - List history with a limit and in reverse order. - `listhistoryafter` - List history after a txid _(subsequent pages)_. - `listhistorybytime` - List history by giving a timestamp in epoch seconds _(block median time past)_. - `listunconfirmed` - List unconfirmed transactions with a limit and in reverse order. - `listunconfirmedafter` - List unconfirmed transactions after a txid _(subsequent pages)_. - `listunconfirmedbytime` - List unconfirmed transactions by time they where added. wallet-rpc: The following methods have been deprecated: - `listtransactions` - Use `listhistory` and the related methods and the `after` argument for results that do not shift when new blocks arrive. wallet: Remove getHistory and related methods form wallet and txdb.
test: fix NodeContext usage and update tests.
d4ac794
to
6aeceed
Compare
Rebased on current master (5294be7) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
advanced
review difficulty - advanced
breaking-major
Backwards incompatible - Release version
wallet-db
part of the codebase
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a port of the bcoin-org/bcoin#605. It includes fixes mentioned in the PR.
Backport:
Issues:
Related:
WDB Migration:
Minor changes
get median time
to the node http and hs-client node.get entries
to the node http and hs-client node.Wallet changes
Update wdb version to v3. Migration is handled by #889.
getMedianTime
- to get median time past for the height, where height is already stored in the db.getMedianTimeTip
- to get median time past for the yet to be committed block.listUnconfirmed(acc, { limit, reverse })
- Get first or lastlimit
unconfirmed transactions.listUnconfirmedAfter(acc, { hash, limit, reverse })
- Get first or lastlimit
unconfirmed transactions after/before tx with hash:hash
.listUnconfirmedFrom(acc, { hash, limit, reverse })
- Get first or lastlimit
unconfirmed transactions after/before tx with hashhash
, inclusive.listUnconfirmedByTime(acc, { time, limit, reverse })
- Get first or lastlimit
unconfirmed transactions after/beforetime
, inclusive.listHistory(acc, { limit, reverse })
- Get first or lastlimit
unconfirmed/confirmed transactions.listHistoryAfter(acc, { hash, limit, reverse })
- Get first or lastlimit
unconfirmed/confirmed transactions after/before tx with hashhash
.listHistoryFrom(acc, { hash, limit, reverse })
- Get first or lastlimit
confirmed/unconfirmed transactions after/before tx with hashhash
, inclusive.listUnconfirmedByTime(acc, { time, limit, reverse })
- Get first or lastlimit
confirmed/unconfirmed transactions after/beforetime
, inclusive.Median time past is used by TX Pagination for time indexes. See details below.
Wallet HTTP
GET /wallet/:id/tx/history
- The params are nowtime
,after
,limit
, andreverse
.GET /wallet/:id/tx/unconfirmed
- The params are are same as above.Deprecated and removed:
GET /wallet/:id/tx/range
- Instead use thetime
param for the history and unconfirmed endpoints.GET /wallet/:id/tx/last
- Instead usereverse
param for the history and unconfirmed endpoints.Wallet HTTP Client
getHistory
andWallet.getHistory
no longer acceptaccount
, instead accepts object with properties:account
,time
,after
,limit
, andreverse
.getPending
andWallet.getPending
have the same changes asgetHistory
above.Deprecate and remove:
getLast
andWallet.getLast
, see Wallet HTTP note.getRange
andWallet.getRange
, see Wallet HTTP note.Examples
By using
after=<txid>
we can anchor pages so that results will not shiftwhen new blocks and transactions arrive. With
reverse=true
we can changethe order the transactions are returned as latest to genesis. The
limit=<number>
specifies the maximum number of transactions to returnin the result.
The param
time
is in epoch seconds and indexed based on median-time-past(MTP) and
date
is ISO 8601 format. Because multiple transactions can sharethe same time, this can function as an initial query, and then switch to the
above
after
format for the following pages.The same will apply to unconfirmed transactions. The
time
is in epochseconds and indexed based on when the transaction was added to the wallet.
Wallet RPC
The following new methods have been added:
listhistory
- List history with a limit and in reverse order.listhistoryafter
- List history after a txid (subsequent pages).listhistorybytime
- List history by giving a timestamp in epoch seconds (block median time past).listunconfirmed
- List unconfirmed transactions with a limit and in reverse order.listunconfirmedafter
- List unconfirmed transactions after a txid (subsequent pages).listunconfirmedbytime
- List unconfirmed transactions by time they where added.The following methods have been deprecated:
listtransactions
- Uselisthistory
and the related methods and theafter
argument for results that do not shift when new blocks arrive.Wallet CLI (hsw-cli)
history
now accepts new args on top of--account
:--reverse
,--limit
,--after
,--after
.pending
now accepts new args, same as above.TXDB Change
layout.h
- will now also store time for the block, instead of just block hash. This allows us to calculatemedian time past
on the wallet side.layout.I
- Latest Unconfirmed indexed. Makes sure every transactions gets assigned new index.z[height][index] -> tx hash (tx by count)
- index/query transactions by height and txindex (in a received array). TXIndex for unconfirmed transactions is ever increasing entry inlayout.I
.Z[account][height][index]
-> tx hash (tx by count + account) - same as above, just indexed separately for each account.y[hash] -> count (count for tx)
-> look up Count for tx using hash.x[hash] -> undo count (unconfirmed count for tx)
- Stores unconfirm index for confirmed transactions in case they get unconfirmed. (Unconfirmed transactions recover their old index and time)g[time][height][index][hash] -> dummy (tx by time)
- Query confirmed transactions by time, ensures they replicate the same sorting as the count index.G[account][time][height][index][hash] -> dummy (tx by time + account)
- Same as above, just indexed separately for each account.w[time][count][hash] -> dummy (tx by time)
- Stores unconfirmed transactions by time.W[account][time][count][hash] -> dummy (tx by time + account)
- Same as above, just indexed separately for each account.e[hash] -> undo time (unconfirmed time for tx)
- Time when transaction was first seen, unconfirmed/confirmed. Confirmed transactions will recover time index using this.layout.m
andlayout.M
are no longer used.TX Pagination
Pagination introduces sorted indexes in the database, that is consistently sorted thoroughout the tx history, whether it's confirmed or unconfirmed. It uses Count indexes to achieve the sorted behaviour. Note this is not strictly based on the tx creation. Instead confirmed transactions use: height + index in the block. Even if one tx was created before another, they will be sorted based on their position in the block and height.
Time index for confirmed transactions is based on the "median time past". It ensures, that time increases monotonically with each block. They give us pointer to the Count index.
There's also "next" page, which gives us next results based on the last hash of the previous response. This is another pointer to the Count index, indexed separately from time.
Count index
This can look like this, if we have 3 transactions in block 200100:
z[200100][0]
- tx1z[200100][1]
- tx2z[200100][2]
- tx3This means, our wallet has 3 transactions in block 200100 and their index relatively is 0, 1, 2. If we were to reorg and another transactions gets introduced between tx3 and tx2 in the new block, all old entries will get removed and new state will look like this.
z[200100][0]
- tx1z[200100][1]
- tx2z[200100][2]
- new tx3z[200100][3]
- old tx3If you factor in block heights for our transactions, you can see sorted list emerge across different blocks:
z[200100][0]
- tx1z[200100][1]
- tx2z[200101][0]
- tx3 (new block)z[200101][1]
- tx4 (new block)LevelDB allows us to query them with ranges(as it sorts by keys), they will be sorted and also reverse and limits.
Unconfirmed transactions also follow the same scheme. But block height is set to maximum possible value for the height:
0xffffffff
. On top unconfirmed transactions have counter for the number of unconfirmed transactions have occured, and that's used for new unconfirmed transactions. This count never decreases and allows previous indexes to be filled, if confirmed transactions get disconnected. Confirmed transactions will store their previous value, to properly recover them in the count index history.This can look like:
z[0xffffffff][0]
- first ever unconfirmed tx.z[0xffffffff][..]
- ...z[0xffffffff][10000]
- 10kth tx.Once they are confirmed, they move to proper block height count index. Disconnected transactions will fill the gaps.
layout.Z
is used for account based count index.