multi: Add UtxoCache.

UtxoCache is an unspent transaction output cache that sits on top of the
utxo set database and provides significant runtime performance benefits
at the cost of some additional memory usage.  It drastically reduces the
amount of reading and writing to disk, especially during initial block
download when a very large number of blocks are being processed in quick
succession.

The UtxoCache is a read-through cache.  All utxo reads go through the
cache.  When there is a cache miss, the cache loads the missing data
from the database, caches it, and returns it to the caller.

The UtxoCache is a write-back cache.  Writes to the cache are
acknowledged by the cache immediately but are only periodically flushed
to the database.  This allows intermediate steps to effectively be
skipped.  For example, a utxo that is created and then spent in between
flushes never needs to be written to the utxo set in the database.

Due to the write-back nature of the cache, at any given time the
database may not be in sync with the cache, and therefore all utxo reads
and writes MUST go through the cache, and never read or write to the
database directly.

An overview of the changes is as follows:

- Add UtxoCache and UtxoCacheConfig struct types and NewUtxoCache method
  - Update server to create the utxo cache with the configured max size
    and pass to the block chain instance that is created
  - Update all test block chains to create a utxo cache
- Add FetchEntry to UtxoCache
  - FetchEntry returns the specified transaction output from the utxo
    set
  - If the output exists in the cache, it is returned immediately.
    Otherwise, it uses an existing database transaction to fetch the
    output from the database, caches it, and returns it to the caller.
- Add AddEntry to UtxoCache
  - AddEntry adds the specified output to the cache
- Add SpendEntry to UtxoCache
  - SpendEntry marks the specified output as spent
  - Remove entries that are marked as fresh and then subsequently spent.
    This is an optimization to skip writing to the database for outputs
    that are added and spent in between flushes to the database.
- Update UtxoViewpoint to hold the UtxoCache
  - Update fetching entries from the database to fetch entries from the
    cache instead
- Add Commit to UtxoCache
  - Commit updates all entries in the cache based on the state of each
    entry in the provided view
  - All entries in the provided view that are marked as modified and
    spent are removed from the view
  - Additionally, all entries that are added to the cache are removed
    from the provided view
- Add MaybeFlush to UtxoCache
  - MaybeFlush conditionally flushes the cache to the database
  - If the maximum size of the cache has been reached, or if the
    periodic flush duration has been reached, then a flush is required
  - A flush can be forced by setting the force flush parameter
  - Flushing commits all modified entries to the database and
    conditionally evicts entries
  - Entries that are nil or spent are always evicted since they are
    unlikely to be accessed again.  Additionally, if the cache has
    reached its maximum size, entries are evicted based on the height of
    the block that they are contained in.
- Update connect block and disconnect block to commit to the cache and
  conditionally flush to the database
  - Rather than writing to the utxo set in the database every time that
    a block is connected or disconnected, commit the updated view to the
    cache and call MaybeFlush on the cache to conditionally flush it to
    the database
- Add InitUtxoCache to UtxoCache
  - InitUtxoCache initializes the utxo cache by ensuring that the utxo
    set is caught up to the tip of the best chain
  - Since the cache is only flushed to the database periodically, the
    utxo set may not be caught up to the tip of the best chain
  - InitUtxoCache catches the utxo set up by replaying all blocks from
    the block after the block that was last flushed to the tip block
    through the cache
- Add ShutdownUtxoCache to BlockChain
  - ShutdownUtxoCache flushes the utxo cache to the database on
    shutdown.  Since the cache is flushed periodically during initial
    block download and flushed after every block is connected after
    initial block download is complete, this flush that occurs during
    shutdown should finish relatively quickly
  - Note that if an unclean shutdown occurs, the cache will still be
    initialized properly when restarted as during initialization it will
    replay blocks to catch up to the tip block if it was not fully
    flushed before shutting down.  However, it is still preferred to
    flush when shutting down versus always recovering on startup since
    it is faster
- Track the hit ratio of UtxoCache
  - Track the number of hits and misses when accessing the cache in
    order to calculate the overall hit ratio of the cache to gauge its
    performance

blockchain/chain.go

@@ -143,6 +143,7 @@ type BlockChain struct {
 	sigCache            *txscript.SigCache
 	indexManager        indexers.IndexManager
 	interrupt           <-chan struct{}
+	utxoCache           *UtxoCache
 
 	// subsidyCache is the cache that provides quick lookup of subsidy
 	// values.
@@ -619,14 +620,6 @@ func (b *BlockChain) connectBlock(node *blockNode, block, parent *dcrutil.Block,
 			return err
 		}
 
-		// Update the utxo set using the state of the utxo view.  This
-		// entails removing all of the utxos spent and adding the new
-		// ones created by the block.
-		err = dbPutUtxoView(dbTx, view)
-		if err != nil {
-			return err
-		}
-
 		// Update the transaction spend journal by adding a record for
 		// the block that contains all txos spent by it.
 		err = dbPutSpendJournalEntry(dbTx, block.Hash(), stxos)
@@ -676,9 +669,26 @@ func (b *BlockChain) connectBlock(node *blockNode, block, parent *dcrutil.Block,
 		return err
 	}
 
-	// Prune fully spent entries and mark all entries in the view unmodified
-	// now that the modifications have been committed to the database.
-	view.commit()
+	// Commit all entries in the view to the utxo cache.  All entries in the view
+	// that are marked as modified and spent are removed from the view.
+	// Additionally, all entries that are added to the cache are removed from the
+	// view.
+	err = b.utxoCache.Commit(view)
+	if err != nil {
+		return err
+	}
+
+	// Conditionally flush the utxo cache to the database.  Force a flush if the
+	// chain believes it is current since blocks are connected infrequently at
+	// that point.  Only log the flush when the chain is not current as it is
+	// mostly useful to see the flush details when many blocks are being connected
+	// (and subsequently flushed) in quick succession.
+	isCurrent := b.isCurrent(node)
+	err = b.utxoCache.MaybeFlush(&node.hash, uint32(node.height), isCurrent,
+		!isCurrent)
+	if err != nil {
+		return err
+	}
 
 	// This node is now the end of the best chain.
 	b.bestChain.SetTip(node)
@@ -814,14 +824,6 @@ func (b *BlockChain) disconnectBlock(node *blockNode, block, parent *dcrutil.Blo
 			return err
 		}
 
-		// Update the utxo set using the state of the utxo view.  This
-		// entails restoring all of the utxos spent and removing the new
-		// ones created by the block.
-		err = dbPutUtxoView(dbTx, view)
-		if err != nil {
-			return err
-		}
-
 		// Update the transaction spend journal by removing the record
 		// that contains all txos spent by the block.
 		err = dbRemoveSpendJournalEntry(dbTx, block.Hash())
@@ -856,9 +858,21 @@ func (b *BlockChain) disconnectBlock(node *blockNode, block, parent *dcrutil.Blo
 		return err
 	}
 
-	// Prune fully spent entries and mark all entries in the view unmodified
-	// now that the modifications have been committed to the database.
-	view.commit()
+	// Commit all entries in the view to the utxo cache.  All entries in the view
+	// that are marked as modified and spent are removed from the view.
+	// Additionally, all entries that are added to the cache are removed from the
+	// view.
+	err = b.utxoCache.Commit(view)
+	if err != nil {
+		return err
+	}
+
+	// Conditionally flush the utxo cache to the database.
+	err = b.utxoCache.MaybeFlush(&node.parent.hash, uint32(node.parent.height),
+		false, false)
+	if err != nil {
+		return err
+	}
 
 	// This node's parent is now the end of the best chain.
 	b.bestChain.SetTip(node.parent)
@@ -993,7 +1007,7 @@ func (b *BlockChain) reorganizeChainInternal(target *blockNode) error {
 	// using that information to unspend all of the spent txos and remove the
 	// utxos created by the blocks.  In addition, if a block votes against its
 	// parent, the regular transactions are reconnected.
-	view := NewUtxoViewpoint()
+	view := NewUtxoViewpoint(b.utxoCache)
 	view.SetBestHash(&tip.hash)
 	var nextBlockToDetach *dcrutil.Block
 	for tip != nil && tip != fork {
@@ -1049,7 +1063,7 @@ func (b *BlockChain) reorganizeChainInternal(target *blockNode) error {
 		// Update the view to unspend all of the spent txos and remove the utxos
 		// created by the block.  Also, if the block votes against its parent,
 		// reconnect all of the regular transactions.
-		err = view.disconnectBlock(b.db, block, parent, stxos, isTreasuryEnabled)
+		err = view.disconnectBlock(block, parent, stxos, isTreasuryEnabled)
 		if err != nil {
 			return err
 		}
@@ -2125,6 +2139,13 @@ type Config struct {
 	// This field can be nil if the caller does not wish to make use of an
 	// index manager.
 	IndexManager indexers.IndexManager
+
+	// UtxoCache defines a utxo cache that sits on top of the utxo set database.
+	// All utxo reads and writes go through the cache, and never read or write to
+	// the database directly.
+	//
+	// This field is required.
+	UtxoCache *UtxoCache
 }
 
 // New returns a BlockChain instance using the provided configuration details.
@@ -2190,6 +2211,7 @@ func New(ctx context.Context, config *Config) (*BlockChain, error) {
 		calcPriorStakeVersionCache:    make(map[[chainhash.HashSize]byte]uint32),
 		calcVoterVersionIntervalCache: make(map[[chainhash.HashSize]byte]uint32),
 		calcStakeVersionCache:         make(map[[chainhash.HashSize]byte]uint32),
+		utxoCache:                     config.UtxoCache,
 	}
 	b.pruner = newChainPruner(&b)
 

blockchain/chainio.go

@@ -1117,43 +1117,42 @@ func dbFetchUtxoStats(dbTx database.Tx) (*UtxoStats, error) {
 	return &stats, nil
 }
 
-// dbPutUtxoView uses an existing database transaction to update the utxo set in
-// the database based on the provided utxo view contents and state.  In
-// particular, only the entries that have been marked as modified are written to
-// the database.
-func dbPutUtxoView(dbTx database.Tx, view *UtxoViewpoint) error {
-	utxoBucket := dbTx.Metadata().Bucket(utxoSetBucketName)
-	for outpoint, entry := range view.entries {
-		// No need to update the database if the entry was not modified.
-		if entry == nil || !entry.isModified() {
-			continue
-		}
-
-		// Remove the utxo entry if it is spent.
-		if entry.IsSpent() {
-			key := outpointKey(outpoint)
-			err := utxoBucket.Delete(*key)
-			recycleOutpointKey(key)
-			if err != nil {
-				return err
-			}
-
-			continue
-		}
+// dbPutUtxoEntry uses an existing database transaction to update the utxo
+// entry for the given outpoint based on the provided utxo entry state.  In
+// particular, the entry is only written to the database if it is marked as
+// modified, and if the entry is marked as spent it is removed from the
+// database.
+func dbPutUtxoEntry(dbTx database.Tx, outpoint wire.OutPoint, entry *UtxoEntry) error {
+	// No need to update the database if the entry was not modified.
+	if entry == nil || !entry.isModified() {
+		return nil
+	}
 
-		// Serialize and store the utxo entry.
-		serialized, err := serializeUtxoEntry(entry)
-		if err != nil {
-			return err
-		}
+	// Remove the utxo entry if it is spent.
+	utxoBucket := dbTx.Metadata().Bucket(utxoSetBucketName)
+	if entry.IsSpent() {
 		key := outpointKey(outpoint)
-		err = utxoBucket.Put(*key, serialized)
-		// NOTE: The key is intentionally not recycled here since the database
-		// interface contract prohibits modifications.  It will be garbage collected
-		// normally when the database is done with it.
+		err := utxoBucket.Delete(*key)
+		recycleOutpointKey(key)
 		if err != nil {
 			return err
 		}
+
+		return nil
+	}
+
+	// Serialize and store the utxo entry.
+	serialized, err := serializeUtxoEntry(entry)
+	if err != nil {
+		return err
+	}
+	key := outpointKey(outpoint)
+	err = utxoBucket.Put(*key, serialized)
+	// NOTE: The key is intentionally not recycled here since the database
+	// interface contract prohibits modifications.  It will be garbage collected
+	// normally when the database is done with it.
+	if err != nil {
+		return err
 	}
 
 	return nil
@@ -1831,6 +1830,7 @@ func (b *BlockChain) initChainState(ctx context.Context) error {
 	}
 
 	// Attempt to load the chain state from the database.
+	var tip *blockNode
 	err = b.db.View(func(dbTx database.Tx) error {
 		// Fetch the stored best chain state from the database.
 		state, err := dbFetchBestState(dbTx)
@@ -1849,7 +1849,7 @@ func (b *BlockChain) initChainState(ctx context.Context) error {
 		}
 
 		// Set the best chain to the stored best state.
-		tip := b.index.lookupNode(&state.hash)
+		tip = b.index.lookupNode(&state.hash)
 		if tip == nil {
 			return AssertError(fmt.Sprintf("initChainState: cannot find "+
 				"chain tip %s in block index", state.hash))
@@ -1924,7 +1924,14 @@ func (b *BlockChain) initChainState(ctx context.Context) error {
 	}
 
 	// Upgrade the database post block index load as needed.
-	return upgradeDBPostBlockIndexLoad(ctx, b)
+	err = upgradeDBPostBlockIndexLoad(ctx, b)
+	if err != nil {
+		return err
+	}
+
+	// Initialize the utxo cache to ensure that the state of the utxo set is
+	// caught up to the tip of the best chain.
+	return b.InitUtxoCache(tip)
 }
 
 // dbFetchBlockByNode uses an existing database transaction to retrieve the raw

blockchain/common_test.go

@@ -1,5 +1,5 @@
 // Copyright (c) 2013-2016 The btcsuite developers
-// Copyright (c) 2015-2020 The Decred developers
+// Copyright (c) 2015-2021 The Decred developers
 // Use of this source code is governed by an ISC
 // license that can be found in the LICENSE file.
 
@@ -117,6 +117,10 @@ func chainSetup(dbName string, params *chaincfg.Params) (*BlockChain, func(), er
 			ChainParams: &paramsCopy,
 			TimeSource:  NewMedianTime(),
 			SigCache:    sigCache,
+			UtxoCache: NewUtxoCache(&UtxoCacheConfig{
+				DB:      db,
+				MaxSize: 100 * 1024 * 1024, // 100 MiB
+			}),
 		})
 
 	if err != nil {

blockchain/example_test.go

@@ -1,5 +1,5 @@
 // Copyright (c) 2014-2016 The btcsuite developers
-// Copyright (c) 2015-2020 The Decred developers
+// Copyright (c) 2015-2021 The Decred developers
 // Use of this source code is governed by an ISC
 // license that can be found in the LICENSE file.
 
@@ -52,6 +52,10 @@ func ExampleBlockChain_ProcessBlock() {
 			DB:          db,
 			ChainParams: mainNetParams,
 			TimeSource:  blockchain.NewMedianTime(),
+			UtxoCache: blockchain.NewUtxoCache(&blockchain.UtxoCacheConfig{
+				DB:      db,
+				MaxSize: 100 * 1024 * 1024, // 100 MiB
+			}),
 		})
 	if err != nil {
 		fmt.Printf("Failed to create chain instance: %v\n", err)

blockchain/fullblocks_test.go

@@ -1,5 +1,5 @@
 // Copyright (c) 2016 The btcsuite developers
-// Copyright (c) 2016-2020 The Decred developers
+// Copyright (c) 2016-2021 The Decred developers
 // Use of this source code is governed by an ISC
 // license that can be found in the LICENSE file.
 
@@ -111,6 +111,10 @@ func chainSetup(dbName string, params *chaincfg.Params) (*blockchain.BlockChain,
 			ChainParams: &paramsCopy,
 			TimeSource:  blockchain.NewMedianTime(),
 			SigCache:    sigCache,
+			UtxoCache: blockchain.NewUtxoCache(&blockchain.UtxoCacheConfig{
+				DB:      db,
+				MaxSize: 100 * 1024 * 1024, // 100 MiB
+			}),
 		})
 
 	if err != nil {

blockchain/headercmt.go

@@ -74,7 +74,7 @@ func (b *BlockChain) FetchUtxoViewParentTemplate(block *wire.MsgBlock) (*UtxoVie
 	// Since the block template is building on the parent of the current tip,
 	// undo the transactions and spend information for the tip block to reach
 	// the point of view of the block template.
-	view := NewUtxoViewpoint()
+	view := NewUtxoViewpoint(b.utxoCache)
 	view.SetBestHash(&tip.hash)
 	tipBlock, err := b.fetchMainChainBlockByNode(tip)
 	if err != nil {
@@ -103,7 +103,7 @@ func (b *BlockChain) FetchUtxoViewParentTemplate(block *wire.MsgBlock) (*UtxoVie
 	// Update the view to unspend all of the spent txos and remove the utxos
 	// created by the tip block.  Also, if the block votes against its parent,
 	// reconnect all of the regular transactions.
-	err = view.disconnectBlock(b.db, tipBlock, parent, stxos, isTreasuryEnabled)
+	err = view.disconnectBlock(tipBlock, parent, stxos, isTreasuryEnabled)
 	if err != nil {
 		return nil, err
 	}

blockchain/sequencelock_test.go

@@ -55,7 +55,7 @@ func TestCalcSequenceLock(t *testing.T) {
 			PkScript: nil,
 		}},
 	})
-	view := NewUtxoViewpoint()
+	view := NewUtxoViewpoint(nil)
 	view.AddTxOuts(targetTx, int64(numBlocks)-4, 0, noTreasury)
 	view.SetBestHash(&node.hash)
 

blockchain/stakeext.go

@@ -1,5 +1,5 @@
 // Copyright (c) 2013-2014 The btcsuite developers
-// Copyright (c) 2015-2020 The Decred developers
+// Copyright (c) 2015-2021 The Decred developers
 // Use of this source code is governed by an ISC
 // license that can be found in the LICENSE file.
 
@@ -112,7 +112,7 @@ func (b *BlockChain) TicketsWithAddress(address dcrutil.Address, isTreasuryEnabl
 	err := b.db.View(func(dbTx database.Tx) error {
 		for _, hash := range tickets {
 			outpoint := wire.OutPoint{Hash: hash, Index: 0, Tree: wire.TxTreeStake}
-			utxo, err := dbFetchUtxoEntry(dbTx, outpoint)
+			utxo, err := b.utxoCache.FetchEntry(dbTx, outpoint)
 			if err != nil {
 				return err
 			}
@@ -224,7 +224,7 @@ func (b *BlockChain) TicketPoolValue() (dcrutil.Amount, error) {
 	err := b.db.View(func(dbTx database.Tx) error {
 		for _, hash := range sn.LiveTickets() {
 			outpoint := wire.OutPoint{Hash: hash, Index: 0, Tree: wire.TxTreeStake}
-			utxo, err := dbFetchUtxoEntry(dbTx, outpoint)
+			utxo, err := b.utxoCache.FetchEntry(dbTx, outpoint)
 			if err != nil {
 				return err
 			}