Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: fix dbcrash and fee_estimation extended tests #1298

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

delta1
Copy link
Member

@delta1 delta1 commented Dec 1, 2023

It seems these tests have never been run for Elements, since the extended tests are not run in the CI process.

This PR fixes both of them, however there may be a further issue with feature_dbcrash.py running for an extremely long time which we are still in the process of testing.

#1297
#1296

cc @whitslack

@delta1
Copy link
Member Author

delta1 commented Dec 8, 2023

Force pushed to reduce the number of iterations in the dbcrash test, such that it still passes but in a "reasonable" amount of time (about 6 hours in my testing).

I have tested these 2 tests manually and they are now passing.

2023-12-08T19:50:34.715000Z TestFramework (INFO): Tests successful

real    350m18.127s
user    348m12.078s
sys     1m56.746s

@jamesdorfman
Copy link
Contributor

jamesdorfman commented Apr 4, 2024

@delta1 am I running the correct test?
It's not passing for me -- I see the following:

$ ./test/functional/feature_dbcrash.py 
2024-04-03T20:32:29.724000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_ikctzo2y
2024-04-03T21:35:24.776000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-03T21:36:00.556000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [0, 0, 0]
2024-04-03T23:10:26.156000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [0, 0, 0]
2024-04-04T00:45:22.052000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [1, 0, 0]
2024-04-04T02:20:44.177000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-04T02:20:45.540000Z TestFramework (INFO): Restarted nodes: [1, 0, 0]; crashes on restart: 0
2024-04-04T02:20:45.540000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/home/james/elements/test/functional/test_framework/test_framework.py", line 132, in main
    self.run_test()
  File "/home/james/elements/./test/functional/feature_dbcrash.py", line 285, in run_test
    assert self.crashed_on_restart > 0
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2024-04-04T02:20:45.600000Z TestFramework (INFO): Stopping nodes
2024-04-04T02:20:45.856000Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_ikctzo2y
2024-04-04T02:20:45.856000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_ikctzo2y/test_framework.log
2024-04-04T02:20:45.856000Z TestFramework (ERROR): 
2024-04-04T02:20:45.857000Z TestFramework (ERROR): Hint: Call /home/james/byron-elements-clone/elements/test/functional/combine_logs.py '/tmp/bitcoin_func_test_ikctzo2y' to consolidate all logs
2024-04-04T02:20:45.857000Z TestFramework (ERROR): 
2024-04-04T02:20:45.857000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-04T02:20:45.857000Z TestFramework (ERROR): https://github.com/ElementsProject/elements/issues
2024-04-04T02:20:45.857000Z TestFramework (ERROR): 
$ git log
commit 5af2426c14fe62f7b60b84586221cc531251eac8 (HEAD -> issues-1297, origin/issues-1297)
Author: Byron Hambly <[email protected]>
Date:   Fri Dec 1 09:19:34 2023 +0200

    test: fix dbcrash and fee_estimation extended tests
    
    reduces the number of iterations for the dbcrash test so that it "only"
    takes 6 hours

@delta1
Copy link
Member Author

delta1 commented Apr 4, 2024

@jamesdorfman yes you ran the right test. It seems I've reduced the values too much for it to fail (correctly) every time.

Bumped the number of iterations from 3 to 4, please test again. It will take closer to 8 hours now.

@delta1
Copy link
Member Author

delta1 commented Apr 4, 2024

latest run:

> test/functional/feature_dbcrash.py
2024-04-04T09:04:35.877000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_dybbv4yh
2024-04-04T09:48:20.887000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-04T09:48:47.443000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [0, 0, 0]
2024-04-04T10:54:39.535000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [0, 0, 0]
2024-04-04T12:00:57.244000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [1, 0, 0]
2024-04-04T13:07:27.698000Z TestFramework (INFO): Iteration 3, generating 2500 transactions [1, 0, 0]
2024-04-04T14:14:32.395000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-04T14:14:32.482000Z TestFramework (INFO): Restarted nodes: [2, 0, 0]; crashes on restart: 1
2024-04-04T14:14:32.482000Z TestFramework (WARNING): Node 1 never crashed during utxo flush!
2024-04-04T14:14:32.482000Z TestFramework (WARNING): Node 2 never crashed during utxo flush!
2024-04-04T14:14:32.536000Z TestFramework (INFO): Stopping nodes
2024-04-04T14:14:32.840000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_dybbv4yh on exit
2024-04-04T14:14:32.840000Z TestFramework (INFO): Tests successful

approx 6 hours

@jamesdorfman
Copy link
Contributor

jamesdorfman commented Apr 5, 2024

Hmm... seems to still be happening though:

[james@elements-workspace elements]$ ./test/functional/feature_dbcrash.py 
2024-04-04T20:07:55.335000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_e8dp9mz5
2024-04-04T21:11:27.390000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-04T21:12:03.369000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [1, 0, 0]
2024-04-04T22:47:36.411000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [1, 0, 0]
2024-04-05T00:22:26.239000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [2, 0, 0]
2024-04-05T01:57:35.443000Z TestFramework (INFO): Iteration 3, generating 2500 transactions [2, 0, 0]
2024-04-05T03:33:23.923000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-05T03:33:24.848000Z TestFramework (INFO): Restarted nodes: [3, 0, 0]; crashes on restart: 0
2024-04-05T03:33:24.848000Z TestFramework (ERROR): Assertion failed
Traceback (most recent call last):
  File "/home/james/elements/test/functional/test_framework/test_framework.py", line 132, in main
    self.run_test()
  File "/home/james/elements/./test/functional/feature_dbcrash.py", line 285, in run_test
    assert self.crashed_on_restart > 0
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2024-04-05T03:33:24.913000Z TestFramework (INFO): Stopping nodes
2024-04-05T03:33:25.220000Z TestFramework (WARNING): Not cleaning up dir /tmp/bitcoin_func_test_e8dp9mz5
2024-04-05T03:33:25.220000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/bitcoin_func_test_e8dp9mz5/test_framework.log
2024-04-05T03:33:25.220000Z TestFramework (ERROR): 
2024-04-05T03:33:25.221000Z TestFramework (ERROR): Hint: Call /home/james/elements/test/functional/combine_logs.py '/tmp/bitcoin_func_test_e8dp9mz5' to consolidate all logs
2024-04-05T03:33:25.221000Z TestFramework (ERROR): 
2024-04-05T03:33:25.221000Z TestFramework (ERROR): If this failure happened unexpectedly or intermittently, please file a bug and provide a link or upload of the combined log.
2024-04-05T03:33:25.221000Z TestFramework (ERROR): https://github.com/ElementsProject/elements/issues
2024-04-05T03:33:25.221000Z TestFramework (ERROR): 

[james@elements-workspace elements]$ git log
commit 1ed7f84f05b54234439fc85436bd2b8854353812 (HEAD, origin/issues-1297)

@delta1
Copy link
Member Author

delta1 commented Apr 5, 2024

strange, worked for me again. i'm going to bump up the iteration and run it again.

2024-04-04T15:10:11.209000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_1c04byhm
2024-04-04T15:53:58.589000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-04T15:54:27.777000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [0, 0, 0]
2024-04-04T17:00:24.418000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [0, 0, 0]
2024-04-04T18:06:32.628000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [0, 0, 0]
2024-04-04T19:12:57.430000Z TestFramework (INFO): Iteration 3, generating 2500 transactions [1, 1, 0]
2024-04-04T20:19:31.617000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-04T20:19:32.528000Z TestFramework (INFO): Restarted nodes: [1, 1, 0]; crashes on restart: 2
2024-04-04T20:19:32.528000Z TestFramework (WARNING): Node 2 never crashed during utxo flush!
2024-04-04T20:19:32.585000Z TestFramework (INFO): Stopping nodes
2024-04-04T20:19:32.889000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_1c04byhm on exit
2024-04-04T20:19:32.889000Z TestFramework (INFO): Tests successful

@delta1
Copy link
Member Author

delta1 commented Apr 5, 2024

5 iterations

> time test/functional/feature_dbcrash.py
2024-04-05T07:05:16.203000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_8_zlhs9q
2024-04-05T07:48:56.506000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-05T07:49:24.107000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [0, 0, 0]
2024-04-05T08:55:19.380000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [0, 0, 0]
2024-04-05T10:01:41.702000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [1, 0, 0]
2024-04-05T11:08:10.640000Z TestFramework (INFO): Iteration 3, generating 2500 transactions [1, 0, 0]
2024-04-05T12:15:03.874000Z TestFramework (INFO): Iteration 4, generating 2500 transactions [2, 0, 0]
2024-04-05T13:21:56.336000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-05T13:21:58.731000Z TestFramework (INFO): Restarted nodes: [3, 1, 0]; crashes on restart: 2
2024-04-05T13:21:58.731000Z TestFramework (WARNING): Node 2 never crashed during utxo flush!
2024-04-05T13:21:58.791000Z TestFramework (INFO): Stopping nodes
2024-04-05T13:21:59.045000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_8_zlhs9q on exit
2024-04-05T13:21:59.045000Z TestFramework (INFO): Tests successful
test/functional/feature_dbcrash.py  22436.98s user 118.79s system 99% cpu 6:16:42.95 total

and

> time test/functional/feature_dbcrash.py
2024-04-05T14:43:53.223000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_y3m7o8tu
2024-04-05T15:27:36.512000Z TestFramework (INFO): Prepped 5001 utxo entries
2024-04-05T15:28:06.834000Z TestFramework (INFO): Iteration 0, generating 2500 transactions [1, 0, 0]
2024-04-05T16:33:52.073000Z TestFramework (INFO): Iteration 1, generating 2500 transactions [1, 0, 0]
2024-04-05T17:40:09.012000Z TestFramework (INFO): Iteration 2, generating 2500 transactions [2, 0, 0]
2024-04-05T18:46:39.010000Z TestFramework (INFO): Iteration 3, generating 2500 transactions [3, 1, 0]
2024-04-05T19:53:15.120000Z TestFramework (INFO): Iteration 4, generating 2500 transactions [4, 1, 0]
2024-04-05T21:00:19.098000Z TestFramework (INFO): Verifying utxo hash matches for all nodes
2024-04-05T21:00:19.222000Z TestFramework (INFO): Restarted nodes: [4, 2, 0]; crashes on restart: 1
2024-04-05T21:00:19.222000Z TestFramework (WARNING): Node 2 never crashed during utxo flush!
2024-04-05T21:00:19.280000Z TestFramework (INFO): Stopping nodes
2024-04-05T21:00:19.734000Z TestFramework (INFO): Cleaning up /tmp/bitcoin_func_test_y3m7o8tu on exit
2024-04-05T21:00:19.734000Z TestFramework (INFO): Tests successful
test/functional/feature_dbcrash.py  22428.39s user 119.89s system 99% cpu 6:16:26.63 total

@delta1
Copy link
Member Author

delta1 commented Apr 7, 2024

@jamesdorfman had a similar failure. bumped the iterations to 6 and will run a number of tests

@whitslack
Copy link

@delta1: It looks as though you're trying to tune a probabilistic test. You're never going to find a number that gives the desired result on every machine from the slowest single-core SoC to the fastest NUMA supercomputer. Instead, the test case should be fixed so that it is deterministic. If that is not possible, then the test case should be discarded.

@delta1
Copy link
Member Author

delta1 commented Apr 8, 2024

I suspect there’s a threshold that works reliably like the Bitcoin test.

JBetz added a commit to SequentiaSEQ/SEQ-Core-Elements that referenced this pull request May 2, 2024
JBetz added a commit to SequentiaSEQ/SEQ-Core-Elements that referenced this pull request May 3, 2024
* Introduce CValue for distinguishing amounts from values

* Remove random character

* Define insertion operator for CValue

* Move CValue to separate module

* Remove consensus changes

* Remove CValue usage for fee rates, restrict to mempool

* More cleanup

* Use CValue for descendant and ancestor fee updates

* More mempool fixes

* Remove unused include

* Fix variable shadowing

* More conversions

* Factor in fee asset when validating package fees

* More conversions

* Convert dust relay fee from RFU

* More cleanup

* Cleanup

* Don't factor in asset in when calculating dust threshold

* Convert package fees

* Return CValue from CalculateExchangeValue

* Use CValue for reverse exchange rate conversion

* Update method documentation

* Rename currency conversion methods

* Fix typo

* Add justification for RFU to comments

* Use accessor instead of public instance variable unpacking CValue

* Use GetValue() in fee rate conversion

* Only apply dust check to outputs denominated in same asset as fee

* Interpret coin selection effective feerate in RFU

* Change GetModifiedFee() method to use RFU for summing with ancestor and descendant transactions

* Add functional test for "fee_rate" parameter in fundrawtransaction RPC and -mintxfee node configuration parameter

* Normalize fees to RFU during fee estimation

* Lint fixes

* More linting fixes

* More linting

* Apply method renams to fee amount conversions

* Add any asset fee rates test to test runner

* More amount conversion fixes

* Add compiler switch for currency constants

* Add new configuration flag to documentation

* Apply fixes to fee estimation functional test from ElementsProject/elements#1298

* Add tests for paytxfee parameter

* Cleanup

* Fix compiler flag

* Add tests for blockmintxfee

* Remove whitespace

* Add comments clarifying what's being demonstrated

* Remove nFeeValue, recompute as needed

* Revert "Remove nFeeValue, recompute as needed"

This reverts commit ab3a67b.

* Move No Coin configuration documentation to separate line

* Add constant for full name of currency atom

* Fix constant reference

* Fix typo
JBetz added a commit to SequentiaSEQ/SEQ-Core-Elements that referenced this pull request May 3, 2024
* Introduce CValue for distinguishing amounts from values

* Remove random character

* Define insertion operator for CValue

* Move CValue to separate module

* Remove consensus changes

* Remove CValue usage for fee rates, restrict to mempool

* More cleanup

* Use CValue for descendant and ancestor fee updates

* More mempool fixes

* Remove unused include

* Fix variable shadowing

* More conversions

* Factor in fee asset when validating package fees

* More conversions

* Convert dust relay fee from RFU

* More cleanup

* Cleanup

* Don't factor in asset in when calculating dust threshold

* Convert package fees

* Return CValue from CalculateExchangeValue

* Use CValue for reverse exchange rate conversion

* Update method documentation

* Rename currency conversion methods

* Fix typo

* Add justification for RFU to comments

* Use accessor instead of public instance variable unpacking CValue

* Use GetValue() in fee rate conversion

* Only apply dust check to outputs denominated in same asset as fee

* Interpret coin selection effective feerate in RFU

* Change GetModifiedFee() method to use RFU for summing with ancestor and descendant transactions

* Add functional test for "fee_rate" parameter in fundrawtransaction RPC and -mintxfee node configuration parameter

* Normalize fees to RFU during fee estimation

* Lint fixes

* More linting fixes

* More linting

* Apply method renams to fee amount conversions

* Add any asset fee rates test to test runner

* More amount conversion fixes

* Add compiler switch for currency constants

* Add new configuration flag to documentation

* Apply fixes to fee estimation functional test from ElementsProject/elements#1298

* Add tests for paytxfee parameter

* Cleanup

* Fix compiler flag

* Add tests for blockmintxfee

* Remove whitespace

* Add comments clarifying what's being demonstrated

* Remove nFeeValue, recompute as needed

* Revert "Remove nFeeValue, recompute as needed"

This reverts commit ab3a67b.

* Move No Coin configuration documentation to separate line

* Add constant for full name of currency atom

* Fix constant reference

* Fix typo
JBetz added a commit to SequentiaSEQ/SEQ-Core-Elements that referenced this pull request May 3, 2024
* Introduce CValue for distinguishing amounts from values

* Remove random character

* Define insertion operator for CValue

* Move CValue to separate module

* Remove consensus changes

* Remove CValue usage for fee rates, restrict to mempool

* More cleanup

* Use CValue for descendant and ancestor fee updates

* More mempool fixes

* Remove unused include

* Fix variable shadowing

* More conversions

* Factor in fee asset when validating package fees

* More conversions

* Convert dust relay fee from RFU

* More cleanup

* Cleanup

* Don't factor in asset in when calculating dust threshold

* Convert package fees

* Return CValue from CalculateExchangeValue

* Use CValue for reverse exchange rate conversion

* Update method documentation

* Rename currency conversion methods

* Fix typo

* Add justification for RFU to comments

* Use accessor instead of public instance variable unpacking CValue

* Use GetValue() in fee rate conversion

* Only apply dust check to outputs denominated in same asset as fee

* Interpret coin selection effective feerate in RFU

* Change GetModifiedFee() method to use RFU for summing with ancestor and descendant transactions

* Add functional test for "fee_rate" parameter in fundrawtransaction RPC and -mintxfee node configuration parameter

* Normalize fees to RFU during fee estimation

* Lint fixes

* More linting fixes

* More linting

* Apply method renams to fee amount conversions

* Add any asset fee rates test to test runner

* More amount conversion fixes

* Add compiler switch for currency constants

* Add new configuration flag to documentation

* Apply fixes to fee estimation functional test from ElementsProject/elements#1298

* Add tests for paytxfee parameter

* Cleanup

* Fix compiler flag

* Add tests for blockmintxfee

* Remove whitespace

* Add comments clarifying what's being demonstrated

* Remove nFeeValue, recompute as needed

* Revert "Remove nFeeValue, recompute as needed"

This reverts commit ab3a67b.

* Move No Coin configuration documentation to separate line

* Add constant for full name of currency atom

* Fix constant reference

* Fix typo
Mixa84 pushed a commit to SequentiaSEQ/SEQ-Core-Elements that referenced this pull request May 6, 2024
* Introduce CValue for distinguishing amounts from values

* Remove random character

* Define insertion operator for CValue

* Move CValue to separate module

* Remove consensus changes

* Remove CValue usage for fee rates, restrict to mempool

* More cleanup

* Use CValue for descendant and ancestor fee updates

* More mempool fixes

* Remove unused include

* Fix variable shadowing

* More conversions

* Factor in fee asset when validating package fees

* More conversions

* Convert dust relay fee from RFU

* More cleanup

* Cleanup

* Don't factor in asset in when calculating dust threshold

* Convert package fees

* Return CValue from CalculateExchangeValue

* Use CValue for reverse exchange rate conversion

* Update method documentation

* Rename currency conversion methods

* Fix typo

* Add justification for RFU to comments

* Use accessor instead of public instance variable unpacking CValue

* Use GetValue() in fee rate conversion

* Only apply dust check to outputs denominated in same asset as fee

* Interpret coin selection effective feerate in RFU

* Change GetModifiedFee() method to use RFU for summing with ancestor and descendant transactions

* Add functional test for "fee_rate" parameter in fundrawtransaction RPC and -mintxfee node configuration parameter

* Normalize fees to RFU during fee estimation

* Lint fixes

* More linting fixes

* More linting

* Apply method renams to fee amount conversions

* Add any asset fee rates test to test runner

* More amount conversion fixes

* Add compiler switch for currency constants

* Add new configuration flag to documentation

* Apply fixes to fee estimation functional test from ElementsProject/elements#1298

* Add tests for paytxfee parameter

* Cleanup

* Fix compiler flag

* Add tests for blockmintxfee

* Remove whitespace

* Add comments clarifying what's being demonstrated

* Remove nFeeValue, recompute as needed

* Revert "Remove nFeeValue, recompute as needed"

This reverts commit ab3a67b.

* Move No Coin configuration documentation to separate line

* Add constant for full name of currency atom

* Fix constant reference

* Fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants