diff --git a/USER-INTERFACE-INTERFACE.md b/USER-INTERFACE-INTERFACE.md index 81f743023..c53e420cc 100644 --- a/USER-INTERFACE-INTERFACE.md +++ b/USER-INTERFACE-INTERFACE.md @@ -343,7 +343,7 @@ Another reason the secrets might be missing is that there are not yet any secret , , ... ], - "PaymentThresholds": { + "paymentThresholds": { "debtThresholdGwei": , "maturityThresholdSec": , "paymentGracePeriodSec": , @@ -392,7 +392,7 @@ database password. If you want to know whether the password you have is the corr * `earningWalletAddressOpt`: The wallet address for the earning wallet. This is not secret, so if you don't get this field, it's because it hasn't been set yet. -* `gasPrice`: The Node will not pay more than this number of Gwei for gas to complete a transaction. +* `gasPrice`: The Node will not pay more than this number of gwei for gas to complete a transaction. * `neighborhoodMode`: The neighborhood mode being currently used, this parameter has nothing to do with descriptors which may have been used in order to set the Node's nearest neighborhood. It is only informative, to know what mode is running @@ -417,7 +417,7 @@ database password. If you want to know whether the password you have is the corr allowed to remain unpaid, or a pending receivable that won’t cause a ban, decreases linearly from the debtThresholdGwei to permanentDebtAllowedGwei or unbanBelowGwei. -* `debtThresholdGwei`: Payables higher than this -- in Gwei of MASQ -- will be suggested for payment immediately upon +* `debtThresholdGwei`: Payables higher than this -- in gwei of MASQ -- will be suggested for payment immediately upon passing the maturityThresholdSec age. Payables less than this can stay unpaid longer. Receivables higher than this will be expected to be settled by other Nodes, but will never cause bans until they pass the maturityThresholdSec + paymentGracePeriodSec age. Receivables less than this will survive longer without banning. @@ -428,15 +428,15 @@ database password. If you want to know whether the password you have is the corr * `paymentGracePeriodSec`: A large receivable can get as old as maturityThresholdSec + paymentGracePeriodSec -- in seconds -- before the Node that owes it will be banned. -* `permanentDebtAllowedGwei`: Receivables this small and smaller -- in Gwei of MASQ -- will not cause bans no matter +* `permanentDebtAllowedGwei`: Receivables this small and smaller -- in gwei of MASQ -- will not cause bans no matter how old they get. * `unbanBelowGwei`: When a delinquent Node has been banned due to non-payment, the receivables balance must be paid - below this level -- in Gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the + below this level -- in gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the same as permanentDebtAllowedGwei. * `ratePack`: These four parameters specify your rates that your Node will use for charging other Nodes for your provided - services. They are currently denominated in Gwei of MASQ, but will be improved to allow denomination in Wei units. + services. They are currently denominated in gwei of MASQ, but will be improved to allow denomination in wei units. These are ever present values, no matter if the user's set any value, they have defaults. * `exitByteRate`: This parameter indicates an amount of MASQ demanded to process 1 byte of routed payload while the Node @@ -595,14 +595,91 @@ field will be null or absent. ##### Correspondent: Node ##### Layout: ``` -"payload": {} +"payload": { + "statsRequired": , + "topRecordsOpt": , + "orderedBy": + }>, + "customQueriesOpt": , + "maxAgeS": , + "minBalanceGwei": , + "maxBalanceGwei": + }>, + "receivableOpt": , + "maxAgeS": , + "minBalanceGwei": , + "maxBalanceGwei": + }> + }> +} ``` ##### Description: -Requests financial statistics from the Node. +This command requests financial statistics from the Node. Without any parameters specified, it produces a brief summary +of financial information; but the output can be customized to show more details. + +The details can bear on two different account types stored in two tables of the persistent database. Those types are +payables and receivables. + +One option to get more information about them is a query of the top N records from these financial tables, ordered by +one's preferences, either by balance or age. This command setup always returns two sets of accounts, one set for each +type. + +In a different query mode, the user provides a customized query for either payables or receivables or both. The user +will be requested to specify ranges of age and balance to constrain the result set. + +The limits for the age range can vary from 0 (as recent as possible) to 9,223,372,036,854,775,807 seconds ago (well +beyond any reasonable scientific estimate of the age of the universe). The limits for the balance range are similarly +generous, except that receivable balances can be negative as well as positive. + +It needs to be stated that each mode excludes the other one. + +While statistics has been considered the foundation of this command, it reports back structured information about +the Node's historical financial operations. This will include services ordered from other Nodes as well as the opposite, +services provided for other Nodes, represented by monetary values owed and paid to and from this Node's wallets. + +`statsRequired` should be true if historical total balances (see below) are required, false if they are not necessary. +If topRecordsOpt and customQueriesOpt are both null, statsRequired must not be false, since it would produce an empty +response otherwise. + +`topRecordsOpt` initiates the request for a given number of top accounts from the tables. An optional feature. +It is important to note that only accounts with positive balances can be displayed by this mode. Some values might stay +out of scope since there might be cases where receivable balances are negative. + +`count` is the maximum number of records to be returned. It should be a number between 1 and 65535, inclusive. + +`orderedBy` allows you to choose whether the results are sorted in descending order either by `Balance` or `Age`. +It is compulsory to use at least one parameter. + +`customQueriesOpt` provides another way to get a subset of accounts, this time by more specific metrics. It works for +both account types, payables and receivables. Possibly only one of them is queried. This query responds well also for +balances with negative values, and therefore this is a good fit for a complete check of receivable accounts going +possibly negative. + +`payableOpt` is an optional field with values that configure the customized search for payables. It holds four numbers +that define two ranges of balance and age used as the scope of the search. + +`receivableOpt` is an optional field with values that configure the customized search for receivables. It holds four +numbers that define two ranges of balance and age used as the scope of the search. + +Both `payableOpt` and `receivableOpt` are complex structures containing four subfields. There is a number in each of +these subfields that represents the end point of the range of either balance or age. Regarding the names of the +subfields, both structures are identical. The only difference lies in the values that the balance parameters can take +on. While receivables are valid between -9223372036854775808 and 9223372036854775807, payables only between 0 and +9223372036854775807. + +`minAgeS` is measured in seconds before the present time and sets a time constraint for the accounts we will be +searching over; this is the lower limit for the debt's age, or how long it has been since the last payment. + +`maxAgeS` is measured in seconds before the present time and sets a constraint for the accounts we will be +searching over; this is the upper limit for the debt's age, or how long it has been since the last payment. -This request will report back information about a Node's historical financial operations. This will include both -services ordered from other Nodes and services provided for other Nodes, represented as monetary values owed and -paid to and from this Node's wallets. +`minBalanceGwei` is represented as an amount of gwei. Any records with balance below this value will not be returned. + +`maxBalanceGwei` is represented as an amount of gwei. Any records with balance above this value will not be returned. #### `financials` ##### Direction: Response @@ -610,30 +687,88 @@ paid to and from this Node's wallets. ##### Layout: ``` "payload": { - "totalUnpaidAndPendingPayable": - "totalPaidPayable": - "totalUnpaidReceivable": - "totalPaidReceivable": + "statsOpt": + "totalPaidPayableGwei": + "totalUnpaidReceivableGwei": + "totalPaidReceivableGwei": + }>, + "queryResultsOpt":, + "ageS": , + "balanceGwei": , + "pendingPayableHashOpt": + }, + [...] + ], + "receivableOpt": [ + { + "wallet": , + "ageS": , + "balanceGwei": + }, + [...] + ] + }> } ``` ##### Description: -Contains the requested financial statistics. +Contains the requested financial statistics or subsets (views) of the database tables and their records. + +`statsOpt` provides a collection of metrics on services consumed and provided. The current span of the tracked data +is since the start of the still running Node. Later on, we'd like to change it to all-time values. -`totalUnpaidAndPendingPayable` is the number of Gwei we believe we owe to other Nodes and that those other Nodes have -not yet received, as far as we know. This includes both bills we haven't yet paid and bills we have paid, but whose +`totalUnpaidAndPendingPayableGwei` is the number of gwei we believe we owe to other Nodes and that those other Nodes +have not yet received, as far as we know. This includes both bills we haven't yet paid and bills we have paid, but whose transactions we have not yet seen confirmed on the blockchain. -`totalPaidPayable` is the number of Gwei we have successfully paid to our creditors and seen confirmed during the time -the current instance of the Node has been running. In the future, this number may become cumulative over more time than -just the current Node run. +`totalPaidPayableGwei` is the number of gwei we have successfully paid to our creditors and seen confirmed. -`totalUnpaidReceivable` is the number of Gwei we believe other Nodes owe to us, but have not yet been included in +`totalUnpaidReceivableGwei` is the number of gwei we believe other Nodes owe to us, but have not yet been included in payments we have seen confirmed on the blockchain. This includes both payments that have never been made and also payments that have been made but not yet confirmed. -`totalPaidReceivable` is the number of Gwei we have successfully received in confirmed payments from our debtors during -the time the current instance of the Node has been running. In the future, this number may become cumulative over more -time than just the current Node run. +`totalPaidReceivableGwei` is the number of gwei we have successfully received in confirmed payments from our debtors. + +`queryResultsOpt` with no respect to which mode of record retrieval was requested, this is always the field that will +hold the records found. If there are no records matching the query, the response will bring an empty array. + +If the `topRecords` parameter is used, the results will be sorted in descending order by either balance or age, +depending on the value of the orderedBy parameter. The number of results returned will be no greater than the value +of the `topRecords` parameter. + +With `customQueryOpt`, the limiting age and balance ranges depend on the user's choices and constrain the subset of +records being returned. The results are always ordered by balance in this mode, and it is the case here too, that +an empty array is handed back if no records can be retrieved. + +This query mode is just optional, and it is also a valid option to request just a single table view. Therefore, null +should be anticipated at various places, either at the position of individual tables (`payableOpt`, `receivableOpt`) +or in place of the whole thing (`customQueryOpt`), which implies that a command with these arguments was not used. + +`payableOpt` is the part referring to payable records if any exist, null or an empty array are also possible. + +`wallet` is the wallet of the Node that we owe to. + +`ageS` is a number of seconds that elapsed since our payment to this particular Node was sent out last time, and the +payment was later also confirmed on the blockchain. + +`balanceGwei` is a number of gwei we owe to this particular Node. + +`pendingPayableHashOpt` is present only sporadically. When it is, it denotes that we've recently sent a payment to the +blockchain, but our confirmation detector has not yet determined that the payment has been confirmed. The value is +either null or stores a transaction hash of the pending transaction. + +`receivable` is the field devoted to receivable records if any exist. + +`wallet` is the wallet of the Node that owes money to us for the services we provided to this Node in the past. + +`age` is a number of seconds that elapsed since this particular debtor made the last payment to our account in order to +redeem his liabilities. + +`balanceGwei` is a number of gwei that this debtor owes to us. + #### `generateWallets` ##### Direction: Request diff --git a/automap/Cargo.lock b/automap/Cargo.lock index 977fc189e..97ee98688 100644 --- a/automap/Cargo.lock +++ b/automap/Cargo.lock @@ -125,7 +125,7 @@ checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" [[package]] name = "automap" -version = "0.6.2" +version = "0.7.0" dependencies = [ "crossbeam-channel 0.5.1", "flexi_logger", @@ -908,7 +908,7 @@ dependencies = [ [[package]] name = "masq_lib" -version = "0.6.3" +version = "0.7.0" dependencies = [ "actix", "clap", diff --git a/automap/Cargo.toml b/automap/Cargo.toml index 2cbc0fb0c..e3c98ef7f 100644 --- a/automap/Cargo.toml +++ b/automap/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "automap" -version = "0.6.2" +version = "0.7.0" authors = ["Dan Wiebe ", "MASQ"] license = "GPL-3.0-only" copyright = "Copyright (c) 2019-2021, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." diff --git a/dns_utility/Cargo.lock b/dns_utility/Cargo.lock index 2a9e74790..fbe59b66d 100644 --- a/dns_utility/Cargo.lock +++ b/dns_utility/Cargo.lock @@ -407,7 +407,7 @@ dependencies = [ [[package]] name = "dns_utility" -version = "0.6.3" +version = "0.7.0" dependencies = [ "core-foundation", "ipconfig 0.2.2", @@ -811,7 +811,7 @@ dependencies = [ [[package]] name = "masq_lib" -version = "0.6.3" +version = "0.7.0" dependencies = [ "actix", "clap", diff --git a/dns_utility/Cargo.toml b/dns_utility/Cargo.toml index 5235768d2..2df8c6820 100644 --- a/dns_utility/Cargo.toml +++ b/dns_utility/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "dns_utility" -version = "0.6.3" +version = "0.7.0" license = "GPL-3.0-only" authors = ["Dan Wiebe ", "MASQ"] copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." diff --git a/masq/Cargo.toml b/masq/Cargo.toml index 9a8ea9e9c..9da81d27d 100644 --- a/masq/Cargo.toml +++ b/masq/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "masq" -version = "0.6.3" +version = "0.7.0" authors = ["Dan Wiebe ", "MASQ"] license = "GPL-3.0-only" copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." @@ -18,13 +18,18 @@ itertools = "0.8.0" lazy_static = "1.4.0" linefeed = "0.6.0" masq_lib = { path = "../masq_lib" } +num = "0.4.0" regex = "1.5.4" +thousands = "0.2.0" websocket = {version = "0.26.2", default-features = false, features = ["sync"]} ctrlc = "3.2.1" [target.'cfg(not(target_os = "windows"))'.dependencies] nix = "0.23.0" +[dev-dependencies] +atty = "0.2.14" + [lib] name = "masq_cli_lib" path = "src/lib.rs" diff --git a/masq/src/command_factory.rs b/masq/src/command_factory.rs index 309d33d95..21222c454 100644 --- a/masq/src/command_factory.rs +++ b/masq/src/command_factory.rs @@ -52,7 +52,10 @@ impl CommandFactory for CommandFactoryReal { Err(msg) => return Err(CommandSyntax(msg)), }, "descriptor" => Box::new(DescriptorCommand::new()), - "financials" => Box::new(FinancialsCommand::new()), + "financials" => match FinancialsCommand::new(pieces) { + Ok(command) => Box::new(command), + Err(msg) => return Err(CommandSyntax(msg)), + }, "generate-wallets" => match GenerateWalletsCommand::new(pieces) { Ok(command) => Box::new(command), Err(msg) => return Err(CommandSyntax(msg)), @@ -367,6 +370,33 @@ mod tests { ); } + #[test] + fn complains_about_financials_command_with_bad_syntax() { + let subject = CommandFactoryReal::new(); + + let result = subject + .make(&[ + "financials".to_string(), + "--make-me-rich".to_string(), + "slowly".to_string(), + ]) + .err() + .unwrap(); + + let msg = match result { + CommandSyntax(msg) => msg, + x => panic!("Expected syntax error, got {:?}", x), + }; + assert_eq!(msg.contains("Found argument"), true, "{}", msg); + assert_eq!(msg.contains("--make-me-rich"), true, "{}", msg); + assert_eq!( + msg.contains("which wasn't expected, or isn't valid in this context"), + true, + "{}", + msg + ); + } + #[test] fn testing_command_factory_with_good_command() { let subject = CommandFactoryReal::new(); diff --git a/masq/src/commands/change_password_command.rs b/masq/src/commands/change_password_command.rs index 6f9e79a82..bbc753d8b 100644 --- a/masq/src/commands/change_password_command.rs +++ b/masq/src/commands/change_password_command.rs @@ -20,11 +20,11 @@ pub struct ChangePasswordCommand { pub new_password: String, } -const CHANGE_PASSWORD_ABOUT: &str = "Changes the existing password on the Node database"; -const OLD_DB_PASSWORD_HELP: &str = "The existing password"; -const NEW_DB_PASSWORD_HELP: &str = "The new password to set"; -const SET_PASSWORD_ABOUT: &str = "Sets an initial password on the Node database"; -const SET_PASSWORD_HELP: &str = "Password to be set; must not already exist"; +const CHANGE_PASSWORD_ABOUT: &str = "Changes the existing password on the Node database."; +const OLD_DB_PASSWORD_HELP: &str = "The existing password."; +const NEW_DB_PASSWORD_HELP: &str = "The new password to set."; +const SET_PASSWORD_ABOUT: &str = "Sets an initial password on the Node database."; +const SET_PASSWORD_HELP: &str = "Password to be set; must not already exist."; impl ChangePasswordCommand { pub fn new_set(pieces: &[String]) -> Result { @@ -129,17 +129,17 @@ mod tests { fn constants_have_correct_values() { assert_eq!( CHANGE_PASSWORD_ABOUT, - "Changes the existing password on the Node database" + "Changes the existing password on the Node database." ); - assert_eq!(OLD_DB_PASSWORD_HELP, "The existing password"); - assert_eq!(NEW_DB_PASSWORD_HELP, "The new password to set"); + assert_eq!(OLD_DB_PASSWORD_HELP, "The existing password."); + assert_eq!(NEW_DB_PASSWORD_HELP, "The new password to set."); assert_eq!( SET_PASSWORD_ABOUT, - "Sets an initial password on the Node database" + "Sets an initial password on the Node database." ); assert_eq!( SET_PASSWORD_HELP, - "Password to be set; must not already exist" + "Password to be set; must not already exist." ); } diff --git a/masq/src/commands/check_password_command.rs b/masq/src/commands/check_password_command.rs index 42d6fb4aa..cec3ac421 100644 --- a/masq/src/commands/check_password_command.rs +++ b/masq/src/commands/check_password_command.rs @@ -17,9 +17,9 @@ pub struct CheckPasswordCommand { } const CHECK_PASSWORD_ABOUT: &str = - "Checks whether the supplied db-password (if any) is the correct password for the Node's database"; + "Checks whether the supplied db-password (if any) is the correct password for the Node's database."; const DB_PASSWORD_ARG_HELP: &str = - "Password to check--leave it out if you think the database doesn't have a password yet"; + "Password to check--leave it out if you think the database doesn't have a password yet."; pub fn check_password_subcommand() -> App<'static, 'static> { SubCommand::with_name("check-password") @@ -81,11 +81,11 @@ mod tests { fn constants_have_correct_values() { assert_eq!( CHECK_PASSWORD_ABOUT, - "Checks whether the supplied db-password (if any) is the correct password for the Node's database" + "Checks whether the supplied db-password (if any) is the correct password for the Node's database." ); assert_eq!( DB_PASSWORD_ARG_HELP, - "Password to check--leave it out if you think the database doesn't have a password yet" + "Password to check--leave it out if you think the database doesn't have a password yet." ); } diff --git a/masq/src/commands/configuration_command.rs b/masq/src/commands/configuration_command.rs index c1c29b1fc..78d3d6ee2 100644 --- a/masq/src/commands/configuration_command.rs +++ b/masq/src/commands/configuration_command.rs @@ -15,6 +15,7 @@ use std::any::Any; use std::fmt::{Debug, Display}; use std::io::Write; use std::iter::once; +use thousands::Separable; const COLUMN_WIDTH: usize = 33; @@ -25,7 +26,7 @@ pub struct ConfigurationCommand { const CONFIGURATION_ABOUT: &str = "Displays a running Node's current configuration."; const CONFIGURATION_ARG_HELP: &str = - "Password of the database from which the configuration will be read"; + "Password of the database from which the configuration will be read."; pub fn configuration_subcommand() -> App<'static, 'static> { SubCommand::with_name("configuration") @@ -129,26 +130,26 @@ impl ConfigurationCommand { let payment_thresholds = Self::preprocess_combined_parameters({ let p_c = &configuration.payment_thresholds; &[ - ("Debt threshold:", &p_c.debt_threshold_gwei, "Gwei"), + ("Debt threshold:", &p_c.debt_threshold_gwei, "gwei"), ("Maturity threshold:", &p_c.maturity_threshold_sec, "s"), ("Payment grace period:", &p_c.payment_grace_period_sec, "s"), ( "Permanent debt allowed:", &p_c.permanent_debt_allowed_gwei, - "Gwei", + "gwei", ), ("Threshold interval:", &p_c.threshold_interval_sec, "s"), - ("Unban below:", &p_c.unban_below_gwei, "Gwei"), + ("Unban below:", &p_c.unban_below_gwei, "gwei"), ] }); Self::dump_value_list(stream, "Payment thresholds:", &payment_thresholds); let rate_pack = Self::preprocess_combined_parameters({ let r_p = &configuration.rate_pack; &[ - ("Routing byte rate:", &r_p.routing_byte_rate, "Gwei"), - ("Routing service rate:", &r_p.routing_service_rate, "Gwei"), - ("Exit byte rate:", &r_p.exit_byte_rate, "Gwei"), - ("Exit service rate:", &r_p.exit_service_rate, "Gwei"), + ("Routing byte rate:", &r_p.routing_byte_rate, "wei"), + ("Routing service rate:", &r_p.routing_service_rate, "wei"), + ("Exit byte rate:", &r_p.exit_byte_rate, "wei"), + ("Exit service rate:", &r_p.exit_service_rate, "wei"), ] }); Self::dump_value_list(stream, "Rate pack:", &rate_pack); @@ -186,12 +187,14 @@ impl ConfigurationCommand { } } - fn preprocess_combined_parameters(parameters: &[(&str, &dyn Display, &str)]) -> Vec { + fn preprocess_combined_parameters( + parameters: &[(&str, &dyn DisplaySeparable, &str)], + ) -> Vec { let iter_of_strings = parameters.iter().map(|(description, value, unit)| { format!( "{:width$} {} {}", description, - value, + value.separate_with_commas(), unit, width = COLUMN_WIDTH ) @@ -200,6 +203,10 @@ impl ConfigurationCommand { } } +trait DisplaySeparable: Display + Separable {} +impl DisplaySeparable for u64 {} +impl DisplaySeparable for String {} + #[cfg(test)] mod tests { use super::*; @@ -223,7 +230,7 @@ mod tests { ); assert_eq!( CONFIGURATION_ARG_HELP, - "Password of the database from which the configuration will be read" + "Password of the database from which the configuration will be read." ); } @@ -306,23 +313,23 @@ mod tests { past_neighbors: vec!["neighbor 1".to_string(), "neighbor 2".to_string()], payment_thresholds: UiPaymentThresholds { threshold_interval_sec: 11111, - debt_threshold_gwei: 1212, + debt_threshold_gwei: 1201412000, payment_grace_period_sec: 4578, - permanent_debt_allowed_gwei: 11222, + permanent_debt_allowed_gwei: 112000, maturity_threshold_sec: 3333, - unban_below_gwei: 12000, + unban_below_gwei: 120000, }, rate_pack: UiRatePack { - routing_byte_rate: 8, - routing_service_rate: 9, - exit_byte_rate: 12, - exit_service_rate: 14, + routing_byte_rate: 99025000, + routing_service_rate: 138000000, + exit_byte_rate: 129000000, + exit_service_rate: 160000000, }, start_block: 3456, scan_intervals: UiScanIntervals { - pending_payable_sec: 150, - payable_sec: 155, - receivable_sec: 250, + pending_payable_sec: 150500, + payable_sec: 155000, + receivable_sec: 250666, }, }; let mut context = CommandContextMock::new() @@ -366,21 +373,21 @@ mod tests { |Past neighbors: neighbor 1\n\ | neighbor 2\n\ |Payment thresholds: \n\ -| Debt threshold: 1212 Gwei\n\ -| Maturity threshold: 3333 s\n\ -| Payment grace period: 4578 s\n\ -| Permanent debt allowed: 11222 Gwei\n\ -| Threshold interval: 11111 s\n\ -| Unban below: 12000 Gwei\n\ +| Debt threshold: 1,201,412,000 gwei\n\ +| Maturity threshold: 3,333 s\n\ +| Payment grace period: 4,578 s\n\ +| Permanent debt allowed: 112,000 gwei\n\ +| Threshold interval: 11,111 s\n\ +| Unban below: 120,000 gwei\n\ |Rate pack: \n\ -| Routing byte rate: 8 Gwei\n\ -| Routing service rate: 9 Gwei\n\ -| Exit byte rate: 12 Gwei\n\ -| Exit service rate: 14 Gwei\n\ +| Routing byte rate: 99,025,000 wei\n\ +| Routing service rate: 138,000,000 wei\n\ +| Exit byte rate: 129,000,000 wei\n\ +| Exit service rate: 160,000,000 wei\n\ |Scan intervals: \n\ -| Pending payable: 150 s\n\ -| Payable: 155 s\n\ -| Receivable: 250 s\n" +| Pending payable: 150,500 s\n\ +| Payable: 155,000 s\n\ +| Receivable: 250,666 s\n" ) .replace('|', "") ); @@ -461,21 +468,21 @@ mod tests { |Start block: 3456\n\ |Past neighbors: [?]\n\ |Payment thresholds: \n\ -| Debt threshold: 2500 Gwei\n\ +| Debt threshold: 2,500 gwei\n\ | Maturity threshold: 500 s\n\ | Payment grace period: 666 s\n\ -| Permanent debt allowed: 1200 Gwei\n\ -| Threshold interval: 1000 s\n\ -| Unban below: 1400 Gwei\n\ +| Permanent debt allowed: 1,200 gwei\n\ +| Threshold interval: 1,000 s\n\ +| Unban below: 1,400 gwei\n\ |Rate pack: \n\ -| Routing byte rate: 15 Gwei\n\ -| Routing service rate: 17 Gwei\n\ -| Exit byte rate: 20 Gwei\n\ -| Exit service rate: 30 Gwei\n\ +| Routing byte rate: 15 wei\n\ +| Routing service rate: 17 wei\n\ +| Exit byte rate: 20 wei\n\ +| Exit service rate: 30 wei\n\ |Scan intervals: \n\ -| Pending payable: 1000 s\n\ -| Payable: 1000 s\n\ -| Receivable: 1000 s\n", +| Pending payable: 1,000 s\n\ +| Payable: 1,000 s\n\ +| Receivable: 1,000 s\n", ) .replace('|', "") ); diff --git a/masq/src/commands/connection_status_command.rs b/masq/src/commands/connection_status_command.rs index 419e84e5b..f4b88a696 100644 --- a/masq/src/commands/connection_status_command.rs +++ b/masq/src/commands/connection_status_command.rs @@ -21,7 +21,7 @@ pub struct ConnectionStatusCommand {} const CONNECTION_STATUS_ABOUT: &str = "Returns the current stage of the connection status. (NotConnected, ConnectedToNeighbor \ - or ThreeHopsRouteFound)"; + or ThreeHopsRouteFound)."; const NOT_CONNECTED_MSG: &str = "NotConnected: No external neighbor is connected to us."; const CONNECTED_TO_NEIGHBOR_MSG: &str = "ConnectedToNeighbor: External neighbor(s) are connected to us."; @@ -98,7 +98,7 @@ mod tests { assert_eq!( CONNECTION_STATUS_ABOUT, "Returns the current stage of the connection status. (NotConnected, ConnectedToNeighbor \ - or ThreeHopsRouteFound)" + or ThreeHopsRouteFound)." ); assert_eq!( NOT_CONNECTED_MSG, diff --git a/masq/src/commands/crash_command.rs b/masq/src/commands/crash_command.rs index 224960db3..4235b3181 100644 --- a/masq/src/commands/crash_command.rs +++ b/masq/src/commands/crash_command.rs @@ -14,9 +14,9 @@ pub struct CrashCommand { const CRASH_COMMAND_ABOUT: &str = "Causes an element of the Node to crash with a specified message. \ - Only valid if the Node has been started with '--crash-point message'"; -const ACTOR_ARG_HELP: &str = "Name of actor inside the Node that should be made to crash"; -const MESSAGE_ARG_HELP: &str = "Panic message that should be produced by the crash"; + Only valid if the Node has been started with '--crash-point message'."; +const ACTOR_ARG_HELP: &str = "Name of actor inside the Node that should be made to crash."; +const MESSAGE_ARG_HELP: &str = "Panic message that should be produced by the crash."; const ACTOR_ARG_POSSIBLE_VALUES: [&str; 5] = [ "BlockchainBridge", "Dispatcher", @@ -98,15 +98,15 @@ mod tests { assert_eq!( CRASH_COMMAND_ABOUT, "Causes an element of the Node to crash with a specified message. \ - Only valid if the Node has been started with '--crash-point message'" + Only valid if the Node has been started with '--crash-point message'." ); assert_eq!( ACTOR_ARG_HELP, - "Name of actor inside the Node that should be made to crash" + "Name of actor inside the Node that should be made to crash." ); assert_eq!( MESSAGE_ARG_HELP, - "Panic message that should be produced by the crash" + "Panic message that should be produced by the crash." ); assert_eq!( ACTOR_ARG_POSSIBLE_VALUES, diff --git a/masq/src/commands/financials_command.rs b/masq/src/commands/financials_command.rs deleted file mode 100644 index 10c6a1221..000000000 --- a/masq/src/commands/financials_command.rs +++ /dev/null @@ -1,175 +0,0 @@ -// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. - -use crate::command_context::CommandContext; -use crate::commands::commands_common::{ - dump_parameter_line, transaction, Command, CommandError, STANDARD_COMMAND_TIMEOUT_MILLIS, -}; -use clap::{App, SubCommand}; -use masq_lib::messages::{UiFinancialsRequest, UiFinancialsResponse}; -use masq_lib::short_writeln; -use std::fmt::Debug; - -const FINANCIALS_SUBCOMMAND_ABOUT: &str = - "Displays financial statistics of this Node. Only valid if Node is already running."; - -#[derive(Debug)] -pub struct FinancialsCommand {} - -pub fn financials_subcommand() -> App<'static, 'static> { - SubCommand::with_name("financials").about(FINANCIALS_SUBCOMMAND_ABOUT) -} - -impl Command for FinancialsCommand { - fn execute(&self, context: &mut dyn CommandContext) -> Result<(), CommandError> { - let input = UiFinancialsRequest {}; - let output: Result = - transaction(input, context, STANDARD_COMMAND_TIMEOUT_MILLIS); - match output { - Ok(response) => { - let stdout = context.stdout(); - short_writeln!(stdout, "Financial status totals in Gwei\n"); - dump_parameter_line( - stdout, - "Unpaid and pending payable:", - &response.total_unpaid_and_pending_payable.to_string(), - ); - dump_parameter_line( - stdout, - "Paid payable:", - &response.total_paid_payable.to_string(), - ); - dump_parameter_line( - stdout, - "Unpaid receivable:", - &response.total_unpaid_receivable.to_string(), - ); - dump_parameter_line( - stdout, - "Paid receivable:", - &response.total_paid_receivable.to_string(), - ); - Ok(()) - } - Err(e) => { - short_writeln!(context.stderr(), "Financials retrieval failed: {:?}", e); - Err(e) - } - } - } -} - -impl Default for FinancialsCommand { - fn default() -> Self { - Self::new() - } -} - -impl FinancialsCommand { - pub fn new() -> Self { - Self {} - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::command_context::ContextError::ConnectionDropped; - use crate::command_factory::{CommandFactory, CommandFactoryReal}; - use crate::commands::commands_common::CommandError::ConnectionProblem; - use crate::test_utils::mocks::CommandContextMock; - use masq_lib::messages::{ToMessageBody, UiFinancialsResponse}; - use std::sync::{Arc, Mutex}; - - #[test] - fn constants_have_correct_values() { - assert_eq!( - FINANCIALS_SUBCOMMAND_ABOUT, - "Displays financial statistics of this Node. Only valid if Node is already running." - ); - } - - #[test] - fn testing_command_factory_here() { - let factory = CommandFactoryReal::new(); - let mut context = CommandContextMock::new().transact_result(Ok(UiFinancialsResponse { - total_unpaid_and_pending_payable: 0, - total_paid_payable: 1111, - total_unpaid_receivable: 2222, - total_paid_receivable: 3333, - } - .tmb(0))); - let subject = factory.make(&["financials".to_string()]).unwrap(); - - let result = subject.execute(&mut context); - - assert_eq!(result, Ok(())); - } - - #[test] - fn financials_command_happy_path() { - let transact_params_arc = Arc::new(Mutex::new(vec![])); - let expected_response = UiFinancialsResponse { - total_unpaid_and_pending_payable: 116688, - total_paid_payable: 55555, - total_unpaid_receivable: 221144, - total_paid_receivable: 66555, - }; - let mut context = CommandContextMock::new() - .transact_params(&transact_params_arc) - .transact_result(Ok(expected_response.tmb(31))); - let stdout_arc = context.stdout_arc(); - let stderr_arc = context.stderr_arc(); - let subject = FinancialsCommand::new(); - - let result = subject.execute(&mut context); - - assert_eq!(result, Ok(())); - let transact_params = transact_params_arc.lock().unwrap(); - assert_eq!( - *transact_params, - vec![( - UiFinancialsRequest {}.tmb(0), - STANDARD_COMMAND_TIMEOUT_MILLIS - )] - ); - assert_eq!( - stdout_arc.lock().unwrap().get_string(), - "\ - Financial status totals in Gwei\n\ - \n\ - Unpaid and pending payable: 116688\n\ - Paid payable: 55555\n\ - Unpaid receivable: 221144\n\ - Paid receivable: 66555\n" - ); - assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); - } - - #[test] - fn financials_command_sad_path() { - let transact_params_arc = Arc::new(Mutex::new(vec![])); - let mut context = CommandContextMock::new() - .transact_params(&transact_params_arc) - .transact_result(Err(ConnectionDropped("Booga".to_string()))); - let stdout_arc = context.stdout_arc(); - let stderr_arc = context.stderr_arc(); - let subject = FinancialsCommand::new(); - - let result = subject.execute(&mut context); - - assert_eq!(result, Err(ConnectionProblem("Booga".to_string()))); - let transact_params = transact_params_arc.lock().unwrap(); - assert_eq!( - *transact_params, - vec![( - UiFinancialsRequest {}.tmb(0), - STANDARD_COMMAND_TIMEOUT_MILLIS - )] - ); - assert_eq!(stdout_arc.lock().unwrap().get_string(), String::new()); - assert_eq!( - stderr_arc.lock().unwrap().get_string(), - "Financials retrieval failed: ConnectionProblem(\"Booga\")\n" - ); - } -} diff --git a/masq/src/commands/financials_command/args_validation.rs b/masq/src/commands/financials_command/args_validation.rs new file mode 100644 index 000000000..31d485bbe --- /dev/null +++ b/masq/src/commands/financials_command/args_validation.rs @@ -0,0 +1,313 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use crate::commands::financials_command::parsing_and_value_dressing::restricted::{ + parse_masq_range_to_gwei, parse_time_params, +}; +use clap::{App, Arg, ArgGroup, SubCommand}; +use masq_lib::shared_schema::common_validators::validate_non_zero_u16; +use num::CheckedMul; +use std::fmt::{Debug, Display}; +use std::num::ParseIntError; +use std::str::FromStr; + +const FINANCIALS_SUBCOMMAND_ABOUT: &str = + "Displays financial statistics of this Node. Only valid if Node is already running."; +const TOP_ARG_HELP: &str = "Fetches the top N records (or fewer) from both payable and receivable. The default order is decreasing by balance, but can be changed with the additional '--ordered' argument."; +const PAYABLE_ARG_HELP: &str = "Enables querying payable records by two specified ranges, one for the age in seconds and another for the balance in MASQs (use the decimal notation to achieve the desired gwei precision). \ + The correct format consists of two ranges separated by | as in the example -|-. Leaving out , including the preceding hyphen, will default to maximum (2^64 - 1). If this \ + parameter is being set in the non-interactive mode, the value needs to be enclosed in quotes (single or double)."; +const RECEIVABLE_ARG_HELP: &str = "Enables querying receivable records by two specified ranges, one for the age in seconds and another for the balance in MASQs (use the decimal notation to achieve the desired gwei precision). \ + The correct format consists of two ranges separated by | as in the example -|-. Leaving out , including the preceding hyphen, will default to maximum (2^64 - 1). If this \ + parameter is being set in the non-interactive mode, the value needs to be enclosed in quotes (single or double)."; +const NO_STATS_ARG_HELP: &str = "Disables statistics that display by default, containing totals of paid and unpaid money from the perspective of debtors and creditors. This argument is not accepted alone and must be placed \ + before other arguments."; +const GWEI_HELP: &str = + "Orders money values rendering in gwei of MASQ instead of whole MASQs as the default."; +const ORDERED_HELP: &str = "Determines in what ordering the top records will be returned. This option works only with the '--top' argument."; + +pub fn financials_subcommand() -> App<'static, 'static> { + SubCommand::with_name("financials") + .about(FINANCIALS_SUBCOMMAND_ABOUT) + .arg( + Arg::with_name("top") + .help(TOP_ARG_HELP) + .value_name("TOP") + .long("top") + .short("t") + .required(false) + .case_insensitive(false) + .takes_value(true) + .validator(validate_non_zero_u16), + ) + .arg( + Arg::with_name("payable") + .help(PAYABLE_ARG_HELP) + .value_name("PAYABLE") + .long("payable") + .short("p") + .required(false) + .case_insensitive(false) + .takes_value(true) + .validator(validate_two_ranges::), + ) + .arg( + Arg::with_name("receivable") + .help(RECEIVABLE_ARG_HELP) + .value_name("RECEIVABLE") + .long("receivable") + .short("r") + .required(false) + .case_insensitive(false) + .takes_value(true) + .validator(validate_two_ranges::), + ) + .arg( + Arg::with_name("no-stats") + .help(NO_STATS_ARG_HELP) + .value_name("NO-STATS") + .long("no-stats") + .short("n") + .case_insensitive(false) + .takes_value(false) + .required(false), + ) + .arg( + Arg::with_name("gwei") + .help(GWEI_HELP) + .value_name("GWEI") + .long("gwei") + .short("g") + .case_insensitive(false) + .takes_value(false) + .required(false), + ) + .arg( + Arg::with_name("ordered") + .help(ORDERED_HELP) + .value_name("ORDERED") + .long("ordered") + .short("o") + .case_insensitive(false) + .default_value_if("top", None, "balance") + .possible_values(&["balance", "age"]) + .required(false), + ) + .groups(&[ + ArgGroup::with_name("at_least_one_query") + .args(&["receivable", "payable", "top"]) + .multiple(true), + ArgGroup::with_name("no-stats-requirement-group") + .arg("no-stats") + .requires("at_least_one_query"), + ArgGroup::with_name("custom-queries") + .args(&["payable", "receivable"]) + .multiple(true), + ArgGroup::with_name("top-records-conflicts") + .args(&["top"]) + .conflicts_with("custom-queries") + .requires("ordered"), + ArgGroup::with_name("ordered-conflicts") + .arg("ordered") + .conflicts_with("custom-queries"), + ]) +} + +fn validate_two_ranges(two_ranges: String) -> Result<(), String> +where + N: FromStr + + TryFrom + + TryFrom + + CheckedMul + + Display + + Copy + + PartialOrd, + i64: TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, +{ + fn checked_split<'a>( + str: &'a str, + delim: char, + err_msg_formatter: fn(&'a str) -> String, + ) -> Result<(&'a str, &'a str), String> { + let split_elems = str.split(delim).collect::>(); + if split_elems.len() != 2 { + return Err(err_msg_formatter(str)); + } + Ok((split_elems[0], split_elems[1])) + } + let (aga_range, balance_range) = checked_split(&two_ranges, '|', |wrong_input| { + format!("Vertical delimiter | should be used between age and balance ranges and only there. Example: '1234-2345|3456-4567', not '{}'", wrong_input) + })?; + let (min_age_str, max_age_str) = checked_split(aga_range, '-', |wrong_input| { + format!("Age range '{}' is formatted wrong", wrong_input) + })?; + let (min_age, max_age) = parse_time_params(min_age_str, max_age_str)?; + let (min_amount, max_amount, _, _): (N, N, _, _) = parse_masq_range_to_gwei(balance_range)?; + //Reasons why only a range input is allowed: + //There is no use trying to check an exact age because of its all time moving nature. + //The backend engine does the search always with a wei precision while at this end you cannot + //pick values more precisely than as 1 gwei, so it's quite impossible to guess an exact value anyway. + if min_age >= max_age || min_amount >= max_amount { + Err(format!("Both ranges '{}' must be low to high", two_ranges)) + } else { + Ok(()) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn constants_have_correct_values() { + assert_eq!( + FINANCIALS_SUBCOMMAND_ABOUT, + "Displays financial statistics of this Node. Only valid if Node is already running." + ); + assert_eq!( + TOP_ARG_HELP, + "Fetches the top N records (or fewer) from both payable and receivable. The default order is decreasing by balance, but can be changed with the additional '--ordered' argument." + ); + assert_eq!(PAYABLE_ARG_HELP, "Enables querying payable records by two specified ranges, one for the age in seconds and another for the balance in MASQs (use the decimal notation to achieve the desired gwei precision). \ + The correct format consists of two ranges separated by | as in the example -|-. Leaving out , including the preceding hyphen, will default to maximum (2^64 - 1). \ + If this parameter is being set in the non-interactive mode, the value needs to be enclosed in quotes (single or double)."); + assert_eq!(RECEIVABLE_ARG_HELP, "Enables querying receivable records by two specified ranges, one for the age in seconds and another for the balance in MASQs (use the decimal notation to achieve the desired gwei precision). \ + The correct format consists of two ranges separated by | as in the example -|-. Leaving out , including the preceding hyphen, will default to maximum (2^64 - 1). \ + If this parameter is being set in the non-interactive mode, the value needs to be enclosed in quotes (single or double)."); + assert_eq!(NO_STATS_ARG_HELP, "Disables statistics that display by default, containing totals of paid and unpaid money from the perspective of debtors and creditors. This argument is not accepted alone and must be placed \ + before other arguments."); + assert_eq!( + GWEI_HELP, + "Orders money values rendering in gwei of MASQ instead of whole MASQs as the default." + ); + assert_eq!(ORDERED_HELP, "Determines in what ordering the top records will be returned. This option works only with the '--top' argument."); + } + + #[test] + fn validate_two_ranges_also_integers_are_acceptable_for_masqs_range() { + let result = validate_two_ranges::("454-2000|2000-30000".to_string()); + + assert_eq!(result, Ok(())) + } + + #[test] + fn validate_two_ranges_one_side_negative_range_is_acceptable_for_masqs_range() { + let result = validate_two_ranges::("454-2000|-2000-30000".to_string()); + + assert_eq!(result, Ok(())) + } + + #[test] + fn validate_two_ranges_both_side_negative_range_is_acceptable_for_masqs_range() { + let result = validate_two_ranges::("454-2000|-2000--1000".to_string()); + + assert_eq!(result, Ok(())) + } + + #[test] + fn validate_two_ranges_with_decimal_part_longer_than_the_whole_gwei_range() { + let result = validate_two_ranges::("454-2000|100-1000.000111222333".to_string()); + + assert_eq!(result, Err("Value '1000.000111222333' exceeds the limit of maximally nine decimal digits (only gwei supported)".to_string())) + } + + #[test] + fn validate_two_ranges_with_decimal_part_fully_used_up() { + let result = validate_two_ranges::("454-2000|100-1000.000111222".to_string()); + + assert_eq!(result, Ok(())) + } + + #[test] + fn validate_two_ranges_with_misused_central_delimiter() { + let result = validate_two_ranges::("45-500545-006".to_string()); + + assert_eq!( + result, + Err("Vertical delimiter | should be used between age and balance ranges and only there. \ + Example: '1234-2345|3456-4567', not '45-500545-006'".to_string()) + ) + } + + #[test] + fn validate_two_ranges_with_misused_range_delimiter() { + let result = validate_two_ranges::("45+500|545+006".to_string()); + + assert_eq!( + result, + Err("Age range '45+500' is formatted wrong".to_string()) + ) + } + + #[test] + fn validate_two_ranges_second_value_smaller_than_the_first_for_time() { + let result = validate_two_ranges::("4545-2000|20000.0-30000.0".to_string()); + + assert_eq!( + result, + Err("Both ranges '4545-2000|20000.0-30000.0' must be low to high".to_string()) + ) + } + + #[test] + fn validate_two_ranges_both_values_the_same_for_time() { + let result = validate_two_ranges::("2000-2000|20000.0-30000.0".to_string()); + + assert_eq!( + result, + Err("Both ranges '2000-2000|20000.0-30000.0' must be low to high".to_string()) + ) + } + + #[test] + fn validate_two_ranges_both_values_the_same_for_masqs() { + let result = validate_two_ranges::("1000-2000|20000.0-20000.0".to_string()); + + assert_eq!( + result, + Err("Both ranges '1000-2000|20000.0-20000.0' must be low to high".to_string()) + ) + } + + #[test] + fn validate_two_ranges_second_value_smaller_than_the_first_for_masqs_but_not_in_decimals() { + let result = validate_two_ranges::("2000-4545|30.0-27.0".to_string()); + + assert_eq!( + result, + Err("Both ranges '2000-4545|30.0-27.0' must be low to high".to_string()) + ) + } + + #[test] + fn validate_two_ranges_second_value_smaller_than_the_first_for_masqs_in_decimals() { + let result = validate_two_ranges::("2000-4545|20.13-20.11".to_string()); + + assert_eq!( + result, + Err("Both ranges '2000-4545|20.13-20.11' must be low to high".to_string()) + ) + } + + #[test] + fn validate_two_ranges_non_numeric_value_for_first_range() { + let result = validate_two_ranges::("blah-1234|899-999".to_string()); + + assert_eq!( + result, + Err("Non numeric value 'blah', it must be a valid integer".to_string()) + ) + } + + #[test] + fn validate_two_ranges_non_numeric_value_for_second_range() { + let result = validate_two_ranges::("1000-1234|7878.0-a lot".to_string()); + + assert_eq!( + result, + Err("Balance range '7878.0-a lot' in improper format".to_string()) + ) + } +} diff --git a/masq/src/commands/financials_command/data_structures.rs b/masq/src/commands/financials_command/data_structures.rs new file mode 100644 index 000000000..4e1c44730 --- /dev/null +++ b/masq/src/commands/financials_command/data_structures.rs @@ -0,0 +1,29 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub(in crate::commands::financials_command) mod restricted { + use masq_lib::messages::{CustomQueries, RangeQuery}; + + #[derive(Debug, PartialEq, Eq)] + pub struct CustomQueryInput { + pub query: CustomQueries, + pub users_payable_format_opt: Option, + pub users_receivable_format_opt: Option, + } + + pub type UserOriginalTypingOfRanges = ((String, String), (String, String)); + + pub struct RangeQueryInput { + pub num_values: RangeQuery, + pub captured_literal_input: UserOriginalTypingOfRanges, + } + + pub struct ProcessAccountsMetadata { + pub table_type: &'static str, + pub headings: HeadingsHolder, + } + + pub struct HeadingsHolder { + pub words: Vec, + pub is_gwei: bool, + } +} diff --git a/masq/src/commands/financials_command/mod.rs b/masq/src/commands/financials_command/mod.rs new file mode 100644 index 000000000..d4d8806ed --- /dev/null +++ b/masq/src/commands/financials_command/mod.rs @@ -0,0 +1,1842 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub mod args_validation; +pub mod data_structures; +pub mod parsing_and_value_dressing; +pub mod pretty_print_utils; +#[cfg(test)] +pub mod test_utils; + +use crate::command_context::CommandContext; +use crate::commands::commands_common::{ + dump_parameter_line, transaction, Command, CommandError, STANDARD_COMMAND_TIMEOUT_MILLIS, +}; +use crate::commands::financials_command::args_validation::financials_subcommand; +use crate::commands::financials_command::data_structures::restricted::{ + CustomQueryInput, ProcessAccountsMetadata, RangeQueryInput, UserOriginalTypingOfRanges, +}; +use crate::commands::financials_command::parsing_and_value_dressing::restricted::{ + parse_masq_range_to_gwei, parse_time_params, split_time_range, +}; +use crate::commands::financials_command::pretty_print_utils::restricted::process_gwei_into_requested_format; +use crate::commands::financials_command::pretty_print_utils::restricted::{ + financial_status_totals_title, main_title_for_tops_opt, no_records_found, prepare_metadata, + render_accounts_generic, subtitle_for_tops, title_for_custom_query, + triple_or_single_blank_line, StringValuesFormattableAccount, +}; +use clap::ArgMatches; +use masq_lib::messages::{ + CustomQueries, QueryResults, RangeQuery, TopRecordsConfig, UiFinancialStatistics, + UiFinancialsRequest, UiFinancialsResponse, +}; +use masq_lib::short_writeln; +use masq_lib::utils::ExpectValue; +use num::CheckedMul; +use std::fmt::{Debug, Display}; +use std::io::Write; +use std::num::ParseIntError; +use std::str::FromStr; + +#[derive(Debug, PartialEq, Eq)] +pub struct FinancialsCommand { + stats_required: bool, + gwei_precision: bool, + top_records_opt: Option, + custom_queries_opt: Option, +} + +impl Command for FinancialsCommand { + fn execute(&self, context: &mut dyn CommandContext) -> Result<(), CommandError> { + let input = UiFinancialsRequest { + stats_required: self.stats_required, + top_records_opt: self.top_records_opt, + custom_queries_opt: self.custom_queries_opt.as_ref().map(|cq| cq.query.clone()), + }; + let output: Result = + transaction(input, context, STANDARD_COMMAND_TIMEOUT_MILLIS); + match output { + Ok(response) => self.process_command_response(response, context), + Err(e) => { + short_writeln!(context.stderr(), "Financials retrieval failed: {:?}", e); + Err(e) + } + } + } +} + +//$(:),+ means that the parameter can be supplied multiple times and is delimited by commas +//here there is use of the advantage twice right away, assuming that both sets have the same number of entries +macro_rules! dump_statistics_lines { + ($stats: expr, $($parameter_name: literal),+, $($gwei: ident),+; $gwei_flag: expr, $stdout: ident) => { + $(dump_parameter_line( + $stdout, + $parameter_name, + &process_gwei_into_requested_format($stats.$gwei, $gwei_flag), + ) + );+ + } +} + +impl FinancialsCommand { + pub fn new(pieces: &[String]) -> Result { + let matches = match financials_subcommand().get_matches_from_safe(pieces) { + Ok(matches) => matches, + Err(e) => return Err(e.to_string()), + }; + let stats_required = !matches.is_present("no-stats"); + let top_records_opt = Self::parse_top_records_args(&matches); + let gwei_precision = matches.is_present("gwei"); + let custom_queries_opt = Self::parse_custom_query_args(&matches); + Ok(Self { + stats_required, + top_records_opt, + custom_queries_opt, + gwei_precision, + }) + } + + fn process_command_response( + &self, + response: UiFinancialsResponse, + context: &mut dyn CommandContext, + ) -> Result<(), CommandError> { + let stdout = context.stdout(); + if let Some(ref stats) = response.stats_opt { + self.process_financial_statistics(stdout, stats, self.gwei_precision) + }; + if let Some(results) = response.query_results_opt { + self.process_queried_records( + stdout, + results, + response.stats_opt.is_none(), + self.gwei_precision, + ) + } + Ok(()) + } + + fn process_financial_statistics( + &self, + stdout: &mut dyn Write, + stats: &UiFinancialStatistics, + gwei_flag: bool, + ) { + financial_status_totals_title(stdout, gwei_flag); + dump_statistics_lines!( + stats, + "Unpaid and pending payable:", + "Paid payable:", + "Unpaid receivable:", + "Paid receivable:", + total_unpaid_and_pending_payable_gwei, + total_paid_payable_gwei, + total_unpaid_receivable_gwei, + total_paid_receivable_gwei; + gwei_flag, + stdout + ); + } + + fn process_queried_records( + &self, + stdout: &mut dyn Write, + returned_records: QueryResults, + is_first_printed_thing: bool, + gwei_flag: bool, + ) { + let is_both_sets = self.are_both_sets_to_be_displayed(); + let (payable_metadata, receivable_metadata) = prepare_metadata(gwei_flag); + + triple_or_single_blank_line(stdout, is_first_printed_thing); + main_title_for_tops_opt(self, stdout); + self.process_returned_records_in_requested_mode( + returned_records.payable_opt, + stdout, + payable_metadata, + |custom_query_input| &custom_query_input.users_payable_format_opt, + ); + if is_both_sets { + triple_or_single_blank_line(stdout, false) + } + self.process_returned_records_in_requested_mode( + returned_records.receivable_opt, + stdout, + receivable_metadata, + |custom_query_input| &custom_query_input.users_receivable_format_opt, + ); + } + + fn are_both_sets_to_be_displayed(&self) -> bool { + self.top_records_opt.is_some() + || (if let Some(custom_queries) = self.custom_queries_opt.as_ref() { + custom_queries.users_payable_format_opt.is_some() + && custom_queries.users_receivable_format_opt.is_some() + } else { + false + }) + } + + fn process_returned_records_in_requested_mode( + &self, + returned_records_opt: Option>, + stdout: &mut dyn Write, + metadata: ProcessAccountsMetadata, + user_range_format_fetcher: fn(&CustomQueryInput) -> &Option, + ) where + A: StringValuesFormattableAccount, + { + if self.top_records_opt.is_some() { + subtitle_for_tops(stdout, metadata.table_type); + let accounts = returned_records_opt.expectv(metadata.table_type); + if !accounts.is_empty() { + render_accounts_generic(stdout, accounts, &metadata.headings); + } else { + no_records_found(stdout, metadata.headings.words.as_slice()) + } + } else if let Some(custom_queries) = self.custom_queries_opt.as_ref() { + if let Some(user_range_format) = user_range_format_fetcher(custom_queries) { + title_for_custom_query(stdout, metadata.table_type, user_range_format); + if let Some(accounts) = returned_records_opt { + render_accounts_generic(stdout, accounts, &metadata.headings) + } else { + no_records_found(stdout, metadata.headings.words.as_slice()) + } + } + } + } + + fn parse_top_records_args(matches: &ArgMatches) -> Option { + matches.value_of("top").map(|str| TopRecordsConfig { + count: str + .parse::() + .expect("top records count not properly validated"), + ordered_by: matches + .value_of("ordered") + .expect("should be required and defaulted") + .try_into() + .expect("Clap did not catch invalid value"), + }) + } + + fn parse_custom_query_args(matches: &ArgMatches) -> Option { + fn decompose_optional_inputs( + composed_parameters_opt: Option>, + ) -> (Option>, Option) { + composed_parameters_opt + .map(|inputs| (Some(inputs.num_values), Some(inputs.captured_literal_input))) + .unwrap_or_default() + } + fn handle_inputs_for_custom_query( + payable_range_inputs_opt: Option>, + receivable_range_inputs_opt: Option>, + ) -> Option { + { + let (payable_opt, users_payable_format_opt) = + decompose_optional_inputs(payable_range_inputs_opt); + + let (receivable_opt, users_receivable_format_opt) = + decompose_optional_inputs(receivable_range_inputs_opt); + + Some(CustomQueryInput { + query: CustomQueries { + payable_opt, + receivable_opt, + }, + users_payable_format_opt, + users_receivable_format_opt, + }) + } + } + + match ( + Self::parse_range_for_query::(matches, "payable"), + Self::parse_range_for_query::(matches, "receivable"), + ) { + (None, None) => None, + (payable_range_inputs_opt, receivable_range_inputs_opt) => { + handle_inputs_for_custom_query( + payable_range_inputs_opt, + receivable_range_inputs_opt, + ) + } + } + } + + fn parse_range_for_query<'a, N>( + matches: &'a ArgMatches, + parameter_name: &'a str, + ) -> Option> + where + N: FromStr + TryFrom + TryFrom + CheckedMul + Display + Copy, + i64: TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, + { + let parse_params = |age_min: &str, + age_max: &str, + money_range: &str| + -> ((u64, u64), (N, N, String, String)) { + ( + parse_time_params(age_min, age_max).expect("blew up after validation"), + parse_masq_range_to_gwei(money_range).expect("blew up after validation"), + ) + }; + + matches.value_of(parameter_name).map(|pair_of_ranges| { + //already after thoroughfare Clap validation + let mut separated_ranges = pair_of_ranges.split('|'); + let (min_age_str, max_age_str) = + split_time_range(separated_ranges.next().expectv("first range")); + let ( + (min_age, max_age), + (min_balance_num, max_balance_num, min_balance_str, max_balance_str), + ) = parse_params( + min_age_str, + max_age_str, + separated_ranges.next().expectv("second range"), + ); + + RangeQueryInput { + num_values: RangeQuery { + min_age_s: min_age, + max_age_s: max_age, + min_amount_gwei: min_balance_num, + max_amount_gwei: max_balance_num, + }, + captured_literal_input: ( + (min_age_str.to_string(), max_age_str.to_string()), + (min_balance_str, max_balance_str), + ), + } + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::command_context::ContextError::ConnectionDropped; + use crate::command_factory::{CommandFactory, CommandFactoryError, CommandFactoryReal}; + use crate::commands::commands_common::CommandError::ConnectionProblem; + use crate::commands::financials_command::args_validation::financials_subcommand; + use crate::commands::financials_command::test_utils::transpose_inputs_to_nested_tuples; + use crate::test_utils::mocks::CommandContextMock; + use atty::Stream; + use masq_lib::messages::{ + ToMessageBody, TopRecordsOrdering, UiFinancialStatistics, UiFinancialsResponse, + UiPayableAccount, UiReceivableAccount, + }; + use masq_lib::ui_gateway::MessageBody; + use masq_lib::utils::array_of_borrows_to_vec; + use regex::Regex; + use std::sync::{Arc, Mutex}; + + fn meaningless_financials_response() -> MessageBody { + UiFinancialsResponse { + stats_opt: None, + query_results_opt: None, + } + .tmb(0) + } + + #[test] + fn command_factory_default_command() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let factory = CommandFactoryReal::new(); + let mut context = CommandContextMock::new() + .transact_result(Ok(meaningless_financials_response())) + .transact_params(&transact_params_arc); + let subject = factory.make(&["financials".to_string()]).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn command_factory_top_records_without_stats() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let factory = CommandFactoryReal::new(); + let mut context = CommandContextMock::new() + .transact_result(Ok(meaningless_financials_response())) + .transact_params(&transact_params_arc); + let subject = factory + .make(&array_of_borrows_to_vec(&[ + "financials", + "--top", + "20", + "--no-stats", + ])) + .unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 20, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn command_factory_everything_demanded_with_top_records() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let factory = CommandFactoryReal::new(); + let mut context = CommandContextMock::new() + .transact_result(Ok(meaningless_financials_response())) + .transact_params(&transact_params_arc); + let subject = factory + .make(&array_of_borrows_to_vec(&[ + "financials", + "--top", + "10", + "--gwei", + ])) + .unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: Some(TopRecordsConfig { + count: 10, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn command_factory_everything_demanded_with_custom_queries() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let factory = CommandFactoryReal::new(); + let mut context = CommandContextMock::new() + .transact_result(Ok(meaningless_financials_response())) + .transact_params(&transact_params_arc); + let subject = factory + .make(&array_of_borrows_to_vec(&[ + "financials", + "--payable", + "200-450|480000-158000008", + "--receivable", + "5000-10000|0.003000000-5.600070000", + "--gwei", + ])) + .unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 200, + max_age_s: 450, + min_amount_gwei: 480000000000000, + max_amount_gwei: 158000008000000000 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 3000000, + max_amount_gwei: 5600070000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn supplied_big_masq_values_are_not_fatal_for_non_decimal_values() { + let factory = CommandFactoryReal::new(); + let result = factory + .make(&array_of_borrows_to_vec(&[ + "financials", + "--payable", + "200-450|480000-15800000800045", + ])) + .unwrap_err(); + let err = match result { + CommandFactoryError::CommandSyntax(msg) => msg, + x => panic!("we expected CommandSyntax error but got: {:?}", x), + }; + + assert!(err.contains("Amount bigger than the MASQ total supply: 15800000800045")) + } + + #[test] + fn supplied_big_masq_values_are_not_fatal_for_decimal_values() { + let factory = CommandFactoryReal::new(); + let result = factory + .make(&array_of_borrows_to_vec(&[ + "financials", + "--payable", + "200-450|480045454455.00-158000008000455", + ])) + .unwrap_err(); + let err = match result { + CommandFactoryError::CommandSyntax(msg) => msg, + x => panic!("we expected CommandSyntax error but got: {:?}", x), + }; + + assert!( + err.contains("Amount bigger than the MASQ total supply: 480045454455.00"), + "{}", + err + ) + } + + #[test] + fn command_factory_no_stats_arg_is_forbidden_if_no_other_arg_present() { + let factory = CommandFactoryReal::new(); + + let result = factory.make(&array_of_borrows_to_vec(&["financials", "--no-stats"])); + + let err = match result { + Err(CommandFactoryError::CommandSyntax(err_msg)) => err_msg, + x => panic!("we expected CommandSyntax error but got: {:?}", x), + }; + assert!( + err.contains("The following required arguments were not provided:"), + "{}", + err + ); + assert!(err.contains( + "financials <--receivable |--payable |--top > <--no-stats>" + ),"{}",err); + } + + fn top_records_mutual_exclusivity_assertion( + args: &[&str], + affected_parameters: &[(&str, bool)], + ) { + let factory = CommandFactoryReal::new(); + + let result = factory.make(&array_of_borrows_to_vec(args)); + + let err = match result { + Ok(_) => panic!("we expected error but got ok"), + Err(CommandFactoryError::CommandSyntax(err_msg)) => err_msg, + Err(e) => panic!("we expected CommandSyntax error but got: {:?}", e), + }; + assert_on_text_simply_in_ide_and_otherwise_in_terminal(&err, affected_parameters); + assert!( + err.contains("cannot be used with one or more of the other specified arguments"), + "{}", + err + ); + assert!(err.contains("financials <--receivable |--payable |--top > <--payable |--receivable > <--ordered > <--top >"),"{}",err) + } + + fn assert_on_text_simply_in_ide_and_otherwise_in_terminal( + err: &str, + searched_words: &[(&str, bool)], + ) { + fn with_quotes(quotes: bool) -> &'static str { + if quotes { + "'" + } else { + "" + } + } + searched_words.iter().for_each(|(string, quotes)| { + if atty::is(Stream::Stderr) { + let regex = Regex::new(&format!("\x1B\\[.*m{}\x1B\\[0m", string)).unwrap(); + assert!( + regex.is_match(&err), + "the regex didn't chase {} down here: {}", + string, + err + ) + } else { + assert!( + err.contains(&format!( + "{}{}{}", + with_quotes(*quotes), + string, + with_quotes(*quotes) + )), + "{} is not in {}", + string, + err + ) + } + }) + } + + #[test] + fn command_factory_top_records_and_payable_custom_query_are_mutually_exclusive() { + top_records_mutual_exclusivity_assertion( + &["financials", "--top", "15", "--payable", "5-100|600-7000"], + &[("--payable ", true)], + ) + } + + #[test] + fn command_factory_top_records_and_receivable_custom_query_are_mutually_exclusive() { + top_records_mutual_exclusivity_assertion( + &[ + "financials", + "--top", + "15", + "--receivable", + "5-100|600-7000", + ], + &[("--receivable ", true)], + ) + } + + #[test] + fn ordered_can_be_combined_with_top_records_only() { + let factory = CommandFactoryReal::new(); + + let result = factory.make(&array_of_borrows_to_vec(&[ + "financials", + "--receivable", + "5-100|600-7000", + "--ordered", + "age", + ])); + + let err = match result { + Ok(_) => panic!("we expected error but got ok"), + Err(CommandFactoryError::CommandSyntax(err_msg)) => err_msg, + Err(e) => panic!("we expected CommandSyntax error but got: {:?}", e), + }; + assert_on_text_simply_in_ide_and_otherwise_in_terminal( + &err, + &[("--receivable ", true)], + ); + assert!( + err.contains("cannot be used with one or more of the other specified arguments"), + "{}", + err + ); + assert!(err.contains("financials <--receivable |--payable |--top > <--payable |--receivable > <--ordered >"),"{}",err) + } + + #[test] + fn ordered_have_just_two_possible_values() { + let args = + array_of_borrows_to_vec(&["financials", "--top", "11", "--ordered", "upside-down"]); + + let result = financials_subcommand() + .get_matches_from_safe(args) + .unwrap_err(); + + assert_on_text_simply_in_ide_and_otherwise_in_terminal( + &result.message, + &[ + ("upside-down", true), + ("--ordered ", true), + ("age", false), + ("balance", false), + ], + ); + assert!( + result.message.contains("isn't a valid value for"), + "{}", + result + ); + assert!(result.message.contains("[possible values: "), "{}", result) + } + + #[test] + fn financials_command_allows_shorthands_including_top_records() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let args = + array_of_borrows_to_vec(&["financials", "-g", "-t", "123", "-o", "balance", "-n"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(meaningless_financials_response())); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 123, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn financials_command_allows_shorthands_including_custom_query() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let args = array_of_borrows_to_vec(&[ + "financials", + "-g", + "-p", + "0-350000|0.005-9.000000000", + "-r", + "5000-10000|0.000004000-50.003000000", + "-n", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(meaningless_financials_response())); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 350000, + min_amount_gwei: 5000000, + max_amount_gwei: 9000000000 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 4000, + max_amount_gwei: 50003000000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn financials_command_top_records_ordered_by_age_instead_of_balance() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let args = + array_of_borrows_to_vec(&["financials", "--no-stats", "--top", "7", "-o", "age"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(meaningless_financials_response())); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 7, + ordered_by: TopRecordsOrdering::Age + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + } + + #[test] + fn parse_top_records_arg_with_ordered_defaulted_to_balance() { + let args = array_of_borrows_to_vec(&["financials", "--top", "11"]); + let matches = financials_subcommand().get_matches_from_safe(args).unwrap(); + + let result = FinancialsCommand::parse_top_records_args(&matches); + + assert_eq!( + result, + Some(TopRecordsConfig { + count: 11, + ordered_by: TopRecordsOrdering::Balance + }) + ) + } + + #[test] + fn financials_command_allows_obscure_leading_zeros_in_positive_numbers() { + let args = + array_of_borrows_to_vec(&["financials", "--receivable", "05000-0010000|040-050"]); + + let result = FinancialsCommand::new(&args).unwrap(); + + assert_eq!( + result, + FinancialsCommand { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueryInput { + query: CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 40000000000, + max_amount_gwei: 50000000000 + }) + }, + users_payable_format_opt: None, + users_receivable_format_opt: Some(transpose_inputs_to_nested_tuples([ + "05000", "0010000", "040", "050" + ])) + }), + gwei_precision: false + } + ); + } + + #[test] + fn financials_command_allows_obscure_leading_zeros_in_negative_numbers() { + let args = array_of_borrows_to_vec(&["financials", "--receivable", "5000-10000|-050--040"]); + + let result = FinancialsCommand::new(&args).unwrap(); + + assert_eq!( + result, + FinancialsCommand { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueryInput { + query: CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: -50000000000, + max_amount_gwei: -40000000000 + }) + }, + users_payable_format_opt: None, + users_receivable_format_opt: Some(transpose_inputs_to_nested_tuples([ + "5000", "10000", "-050", "-040" + ])) + }), + gwei_precision: false + } + ); + } + + #[test] + fn default_financials_command_happy_path() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 1_166_880_215, + total_paid_payable_gwei: 78_455_555, + total_unpaid_receivable_gwei: -55_000_400, + total_paid_receivable_gwei: 1_278_766_555_456, + }), + query_results_opt: None, + }; + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let args = &["financials".to_string()]; + let subject = FinancialsCommand::new(args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\ + \n\ + Financial status totals in MASQ\n\ + \n\ + Unpaid and pending payable: 1.16\n\ + Paid payable: 0.07\n\ + Unpaid receivable: -0.05\n\ + Paid receivable: 1,278.76\n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn are_both_sets_to_be_displayed_works_for_top_records() { + //top records always print as a pair so it always consists of both sets + let subject = + FinancialsCommand::new(&array_of_borrows_to_vec(&["financials", "--top", "20"])) + .unwrap(); + + let result = subject.are_both_sets_to_be_displayed(); + + assert_eq!(result, true) + } + + #[test] + fn are_both_sets_to_be_displayed_works_for_custom_query_with_only_payable() { + let subject = FinancialsCommand::new(&array_of_borrows_to_vec(&[ + "financials", + "--payable", + "20-40|60-120", + ])) + .unwrap(); + + let result = subject.are_both_sets_to_be_displayed(); + + assert_eq!(result, false) + } + + #[test] + fn are_both_sets_to_be_displayed_works_for_custom_query_with_only_receivable() { + let subject = FinancialsCommand::new(&array_of_borrows_to_vec(&[ + "financials", + "--receivable", + "20-40|-50-120", + ])) + .unwrap(); + + let result = subject.are_both_sets_to_be_displayed(); + + assert_eq!(result, false) + } + + #[test] + fn are_both_sets_to_be_displayed_works_for_custom_query_with_both_parts() { + let subject = FinancialsCommand::new(&array_of_borrows_to_vec(&[ + "financials", + "--receivable", + "20-40|-50-120", + "--payable", + "15-55|667-800", + ])) + .unwrap(); + + let result = subject.are_both_sets_to_be_displayed(); + + assert_eq!(result, true) + } + + fn response_with_stats_and_either_top_records_or_top_queries( + for_top_records: bool, + ) -> UiFinancialsResponse { + UiFinancialsResponse { + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 116688555, + total_paid_payable_gwei: 235555554578, + total_unpaid_receivable_gwei: 0, + total_paid_receivable_gwei: 665557, + }), + query_results_opt: Some(if for_top_records { + QueryResults { + payable_opt: Some(vec![ + UiPayableAccount { + wallet: "0xA884A2F1A5Ec6C2e499644666a5E6af97B966888".to_string(), + age_s: 5645405400, + balance_gwei: 68843325667, + pending_payable_hash_opt: None, + }, + UiPayableAccount { + wallet: "0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440".to_string(), + age_s: 150000, + balance_gwei: 8, + pending_payable_hash_opt: Some( + "0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e" + .to_string(), + ), + }, + ]), + receivable_opt: Some(vec![ + UiReceivableAccount { + wallet: "0x6e250504DdfFDb986C4F0bb8Df162503B4118b05".to_string(), + age_s: 22000, + balance_gwei: 2444533124512, + }, + UiReceivableAccount { + wallet: "0x8bA50675e590b545D2128905b89039256Eaa24F6".to_string(), + age_s: 19000, + balance_gwei: -328123256546, + }, + ]), + } + } else { + QueryResults { + payable_opt: Some(vec![UiPayableAccount { + wallet: "0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440".to_string(), + age_s: 150000, + balance_gwei: 8, + pending_payable_hash_opt: Some( + "0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e" + .to_string(), + ), + }]), + receivable_opt: None, + } + }), + } + } + + #[test] + fn financials_command_stats_and_top_records_default_units_as_masq() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = response_with_stats_and_either_top_records_or_top_queries(true); + let args = array_of_borrows_to_vec(&["financials", "--top", "123"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: Some(TopRecordsConfig { + count: 123, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), + "\ + \n\ + Financial status totals in MASQ\n\ + \n\ + Unpaid and pending payable: 0.11\n\ + Paid payable: 235.55\n\ + Unpaid receivable: < 0.01\n\ + Paid receivable: < 0.01\n\ + \n\ + \n\ + \n\ + Up to 123 top accounts\n\ + \n\ + Payable\n\ + \n\ + # Wallet Age [s] Balance [MASQ] Pending tx \n\ + 1 0xA884A2F1A5Ec6C2e499644666a5E6af97B966888 5,645,405,400 68.84 None \n\ + 2 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 150,000 < 0.01 0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e\n\ + \n\ + \n\ + \n\ + Receivable\n\ + \n\ + # Wallet Age [s] Balance [MASQ]\n\ + 1 0x6e250504DdfFDb986C4F0bb8Df162503B4118b05 22,000 2,444.53 \n\ + 2 0x8bA50675e590b545D2128905b89039256Eaa24F6 19,000 -328.12 \n"); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_stats_and_custom_query_demanded_default_units_as_masq() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = response_with_stats_and_either_top_records_or_top_queries(false); + let args = array_of_borrows_to_vec(&[ + "financials", + "--payable", + "0-350000|0.005-9", + "--receivable", + "5000-10000|0.003000000-5.600070000", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 350000, + min_amount_gwei: 5000000, + max_amount_gwei: 9000000000 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 3000000, + max_amount_gwei: 5600070000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), + "\ + \n\ + Financial status totals in MASQ\n\ + \n\ + Unpaid and pending payable: 0.11\n\ + Paid payable: 235.55\n\ + Unpaid receivable: < 0.01\n\ + Paid receivable: < 0.01\n\ + \n\ + \n\ + \n\ + Specific payable query: 0-350000 sec 0.005-9 MASQ\n\ + \n\ + # Wallet Age [s] Balance [MASQ] Pending tx \n\ + 1 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 150,000 < 0.01 0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e\n\ + \n\ + \n\ + \n\ + Specific receivable query: 5000-10000 sec 0.003-5.60007 MASQ\n\ + \n\ + # Wallet Age [s] Balance [MASQ]\n\ + \n\ + No records found\n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_statistics_and_top_records_with_gwei_precision() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = response_with_stats_and_either_top_records_or_top_queries(true); + let args = array_of_borrows_to_vec(&["financials", "--top", "123", "--gwei"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: Some(TopRecordsConfig { + count: 123, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), + "\ + \n\ + Financial status totals in gwei\n\ + \n\ + Unpaid and pending payable: 116,688,555\n\ + Paid payable: 235,555,554,578\n\ + Unpaid receivable: 0\n\ + Paid receivable: 665,557\n\ + \n\ + \n\ + \n\ + Up to 123 top accounts\n\ + \n\ + Payable\n\ + \n\ + # Wallet Age [s] Balance [gwei] Pending tx \n\ + 1 0xA884A2F1A5Ec6C2e499644666a5E6af97B966888 5,645,405,400 68,843,325,667 None \n\ + 2 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 150,000 8 0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e\n\ + \n\ + \n\ + \n\ + Receivable\n\ + \n\ + # Wallet Age [s] Balance [gwei] \n\ + 1 0x6e250504DdfFDb986C4F0bb8Df162503B4118b05 22,000 2,444,533,124,512\n\ + 2 0x8bA50675e590b545D2128905b89039256Eaa24F6 19,000 -328,123,256,546 \n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_statistics_and_custom_query_with_gwei_precision() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = response_with_stats_and_either_top_records_or_top_queries(false); + let args = array_of_borrows_to_vec(&[ + "financials", + "--payable", + "0-350000|0.005-9", + "--receivable", + "5000-10000|0.000004-0.4550", + "--gwei", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 350000, + min_amount_gwei: 5000000, + max_amount_gwei: 9000000000 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 4000, + max_amount_gwei: 455000000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), "\ + \n\ + Financial status totals in gwei\n\ + \n\ + Unpaid and pending payable: 116,688,555\n\ + Paid payable: 235,555,554,578\n\ + Unpaid receivable: 0\n\ + Paid receivable: 665,557\n\ + \n\ + \n\ + \n\ + Specific payable query: 0-350000 sec 0.005-9 MASQ\n\ + \n\ + # Wallet Age [s] Balance [gwei] Pending tx \n\ + 1 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 150,000 8 0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e\n\ + \n\ + \n\ + \n\ + Specific receivable query: 5000-10000 sec 0.000004-0.455 MASQ\n\ + \n\ + # Wallet Age [s] Balance [gwei]\n\ + \n\ + No records found\n"); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn custom_query_balance_range_can_be_shorthanded() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![UiPayableAccount { + wallet: "0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440".to_string(), + age_s: 150000, + balance_gwei: 1200000000000, + pending_payable_hash_opt: Some( + "0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e" + .to_string(), + ), + }]), + receivable_opt: Some(vec![UiReceivableAccount { + wallet: "0x8bA50675e590b545D2128905b89039256Eaa24F6".to_string(), + age_s: 45700, + balance_gwei: 5050330000, + }]), + }), + }; + let args = array_of_borrows_to_vec(&[ + "financials", + "--payable", + "0-350000|5", + "--receivable", + "5000-10000|0.8", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 350000, + min_amount_gwei: 5000000000, + max_amount_gwei: i64::MAX as u64 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 10000, + min_amount_gwei: 800000000, + max_amount_gwei: i64::MAX + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), + "\n\ + Specific payable query: 0-350000 sec 5-UNLIMITED MASQ\n\ + \n\ + # Wallet Age [s] Balance [MASQ] Pending tx \n\ + 1 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 150,000 1,200.00 0x0290db1d56121112f4d45c1c3f36348644f6afd20b759b762f1dba9c4949066e\n\ + \n\ + \n\ + \n\ + Specific receivable query: 5000-10000 sec 0.8-UNLIMITED MASQ\n\ + \n\ + # Wallet Age [s] Balance [MASQ]\n\ + 1 0x8bA50675e590b545D2128905b89039256Eaa24F6 45,700 5.05 \n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_no_records_found_with_stats_and_top_records() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 116688, + total_paid_payable_gwei: 55555, + total_unpaid_receivable_gwei: 221144, + total_paid_receivable_gwei: 66555, + }), + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![]), + receivable_opt: Some(vec![]), + }), + }; + let args = array_of_borrows_to_vec(&["financials", "--top", "10"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: Some(TopRecordsConfig { + count: 10, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\ +| +|Financial status totals in MASQ +| +|Unpaid and pending payable: < 0.01 +|Paid payable: < 0.01 +|Unpaid receivable: < 0.01 +|Paid receivable: < 0.01 +| +| +| +|Up to 10 top accounts +| +|Payable +| +|# Wallet Age [s] Balance [MASQ] Pending tx +| +|No records found +| +| +| +|Receivable +| +|# Wallet Age [s] Balance [MASQ] +| +|No records found\n" + .lines() + .map(|line| format!("{}\n", line.strip_prefix("|").unwrap())) + .collect::() + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_no_records_found_with_stats_and_custom_query() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 116688, + total_paid_payable_gwei: 55555, + total_unpaid_receivable_gwei: 221144, + total_paid_receivable_gwei: 66555, + }), + query_results_opt: Some(QueryResults { + payable_opt: None, + receivable_opt: None, + }), + }; + let args = array_of_borrows_to_vec(&[ + "financials", + "--payable", + "0-400000|355-6000", + "--receivable", + "40000-80000|111-10000", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 400000, + min_amount_gwei: 355000000000, + max_amount_gwei: 6000000000000 + }), + receivable_opt: Some(RangeQuery { + min_age_s: 40000, + max_age_s: 80000, + min_amount_gwei: 111000000000, + max_amount_gwei: 10000000000000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\ +| +|Financial status totals in MASQ +| +|Unpaid and pending payable: < 0.01 +|Paid payable: < 0.01 +|Unpaid receivable: < 0.01 +|Paid receivable: < 0.01 +| +| +| +|Specific payable query: 0-400000 sec 355-6000 MASQ +| +|# Wallet Age [s] Balance [MASQ] Pending tx +| +|No records found +| +| +| +|Specific receivable query: 40000-80000 sec 111-10000 MASQ +| +|# Wallet Age [s] Balance [MASQ] +| +|No records found" + .lines() + .map(|line| format!("{}\n", line.strip_prefix("|").unwrap())) + .collect::() + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_only_top_records_demanded() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![ + UiPayableAccount { + wallet: "0xA884A2F1A5Ec6C2e499644666a5E6af97B966888".to_string(), + age_s: 5405400, + balance_gwei: 644000000, + pending_payable_hash_opt: Some( + "0x3648c8b8c7e067ac30b80b6936159326d564dd13b7ae465b26647154ada2c638" + .to_string(), + ), + }, + UiPayableAccount { + wallet: "0xEA674fdac714fd979de3EdF0F56AA9716B198ec8".to_string(), + age_s: 28120444, + balance_gwei: 97524120, + pending_payable_hash_opt: None, + }, + ]), + receivable_opt: Some(vec![ + UiReceivableAccount { + wallet: "0xaa22968a5263f165F014d3F21A443f10a116EDe0".to_string(), + age_s: 566668, + balance_gwei: 550, + }, + UiReceivableAccount { + wallet: "0x6e250504DdfFDb986C4F0bb8Df162503B4118b05".to_string(), + age_s: 11111111, + balance_gwei: -4551012, + }, + ]), + }), + }; + let args = array_of_borrows_to_vec(&["financials", "--no-stats", "--top", "7"]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 7, + ordered_by: TopRecordsOrdering::Balance + }), + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\n\ + Up to 7 top accounts\n\ + \n\ + Payable\n\ + \n\ + # Wallet Age [s] Balance [MASQ] Pending tx \n\ + 1 0xA884A2F1A5Ec6C2e499644666a5E6af97B966888 5,405,400 0.64 0x3648c8b8c7e067ac30b80b6936159326d564dd13b7ae465b26647154ada2c638\n\ + 2 0xEA674fdac714fd979de3EdF0F56AA9716B198ec8 28,120,444 0.09 None \n\ + \n\ + \n\ + \n\ + Receivable\n\ + \n\ + # Wallet Age [s] Balance [MASQ]\n\ + 1 0xaa22968a5263f165F014d3F21A443f10a116EDe0 566,668 < 0.01 \n\ + 2 0x6e250504DdfFDb986C4F0bb8Df162503B4118b05 11,111,111 -0.01 < x < 0 \n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_only_payable_demanded() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![ + UiPayableAccount { + wallet: "0x6e250504DdfFDb986C4F0bb8Df162503B4118b05".to_string(), + age_s: 4445, + balance_gwei: 3862654858938090, + pending_payable_hash_opt: Some( + "0x5fe272ed1e941cc05fbd624ec4b1546cd03c25d53e24ba2c18b11feb83cd4581" + .to_string(), + ), + }, + UiPayableAccount { + wallet: "0xA884A2F1A5Ec6C2e499644666a5E6af97B966888".to_string(), + age_s: 70000, + balance_gwei: 708090, + pending_payable_hash_opt: None, + }, + UiPayableAccount { + wallet: "0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440".to_string(), + age_s: 6089909, + balance_gwei: 66658, + pending_payable_hash_opt: None, + }, + ]), + receivable_opt: None, + }), + }; + let args = array_of_borrows_to_vec(&[ + "financials", + "--payable", + "3000-40000|88-1000", + "--no-stats", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 3000, + max_age_s: 40000, + min_amount_gwei: 88000000000, + max_amount_gwei: 1000000000000 + }), + receivable_opt: None + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\n\ + Specific payable query: 3000-40000 sec 88-1000 MASQ\n\ + \n\ + # Wallet Age [s] Balance [MASQ] Pending tx \n\ + 1 0x6e250504DdfFDb986C4F0bb8Df162503B4118b05 4,445 3,862,654.85 0x5fe272ed1e941cc05fbd624ec4b1546cd03c25d53e24ba2c18b11feb83cd4581\n\ + 2 0xA884A2F1A5Ec6C2e499644666a5E6af97B966888 70,000 < 0.01 None \n\ + 3 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 6,089,909 < 0.01 None \n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_only_receivable_demanded() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let expected_response = UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: None, + receivable_opt: Some(vec![ + UiReceivableAccount { + wallet: "0x6e250504DdfFDb986C4F0bb8Df162503B4118b05".to_string(), + age_s: 4445, + balance_gwei: 9898999888, + }, + UiReceivableAccount { + wallet: "0xA884A2F1A5Ec6C2e499644666a5E6af97B966888".to_string(), + age_s: 70000, + balance_gwei: 708090, + }, + UiReceivableAccount { + wallet: "0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440".to_string(), + age_s: 6089909, + balance_gwei: 66658, + }, + ]), + }), + }; + let args = array_of_borrows_to_vec(&[ + "financials", + "--no-stats", + "--receivable", + "3000-40000|66-980", + "--gwei", + ]); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Ok(expected_response.tmb(31))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let subject = FinancialsCommand::new(&args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Ok(())); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 3000, + max_age_s: 40000, + min_amount_gwei: 66000000000, + max_amount_gwei: 980000000000 + }) + }) + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!( + stdout_arc.lock().unwrap().get_string(), + "\n\ + Specific receivable query: 3000-40000 sec 66-980 MASQ\n\ + \n\ + # Wallet Age [s] Balance [gwei]\n\ + 1 0x6e250504DdfFDb986C4F0bb8Df162503B4118b05 4,445 9,898,999,888 \n\ + 2 0xA884A2F1A5Ec6C2e499644666a5E6af97B966888 70,000 708,090 \n\ + 3 0x6DbcCaC5596b7ac986ff8F7ca06F212aEB444440 6,089,909 66,658 \n" + ); + assert_eq!(stderr_arc.lock().unwrap().get_string(), String::new()); + } + + #[test] + fn financials_command_sad_path() { + let transact_params_arc = Arc::new(Mutex::new(vec![])); + let mut context = CommandContextMock::new() + .transact_params(&transact_params_arc) + .transact_result(Err(ConnectionDropped("Booga".to_string()))); + let stdout_arc = context.stdout_arc(); + let stderr_arc = context.stderr_arc(); + let args = &["financials".to_string()]; + let subject = FinancialsCommand::new(args).unwrap(); + + let result = subject.execute(&mut context); + + assert_eq!(result, Err(ConnectionProblem("Booga".to_string()))); + let transact_params = transact_params_arc.lock().unwrap(); + assert_eq!( + *transact_params, + vec![( + UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None + } + .tmb(0), + STANDARD_COMMAND_TIMEOUT_MILLIS + )] + ); + assert_eq!(stdout_arc.lock().unwrap().get_string(), String::new()); + assert_eq!( + stderr_arc.lock().unwrap().get_string(), + "Financials retrieval failed: ConnectionProblem(\"Booga\")\n" + ); + } +} diff --git a/masq/src/commands/financials_command/parsing_and_value_dressing.rs b/masq/src/commands/financials_command/parsing_and_value_dressing.rs new file mode 100644 index 000000000..30c9b9a10 --- /dev/null +++ b/masq/src/commands/financials_command/parsing_and_value_dressing.rs @@ -0,0 +1,519 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub(in crate::commands::financials_command) mod restricted { + use crate::commands::financials_command::data_structures::restricted::UserOriginalTypingOfRanges; + use masq_lib::constants::{MASQ_TOTAL_SUPPLY, WEIS_OF_GWEI}; + use masq_lib::utils::ExpectValue; + use num::CheckedMul; + use regex::{Captures, Regex}; + use std::any::type_name; + use std::collections::VecDeque; + use std::fmt::{Debug, Display}; + use std::num::{IntErrorKind, ParseIntError}; + use std::str::FromStr; + use thousands::Separable; + + pub fn convert_masq_from_gwei_and_dress_well(balance_gwei: i64) -> String { + const MASK_FOR_NON_SIGNIFICANT_DIGITS: i64 = 10_000_000; + let balance_masq_int = (balance_gwei / WEIS_OF_GWEI as i64).abs(); + let balance_masq_frac = (balance_gwei % WEIS_OF_GWEI as i64).abs(); + let balance_masq_frac_trunc = balance_masq_frac / MASK_FOR_NON_SIGNIFICANT_DIGITS; + match ( + (balance_masq_int == 0) && (balance_masq_frac_trunc == 0), + balance_gwei >= 0, + ) { + (true, true) => "< 0.01".to_string(), + (true, false) => "-0.01 < x < 0".to_string(), + _ => { + format!( + "{}{}.{:0>2}", + if balance_gwei < 0 { "-" } else { "" }, + balance_masq_int.separate_with_commas(), + balance_masq_frac_trunc + ) + } + } + } + + pub fn neaten_users_writing_if_possible( + user_ranges: &UserOriginalTypingOfRanges, + ) -> (String, String) { + fn collect_captured_parts_of_a_number(captures: Captures) -> [Option; 3] { + let fetch_group = |idx: usize| -> Option { single_capture(&captures, idx) }; + [fetch_group(1), fetch_group(2), fetch_group(3)] + } + fn assemble_string_segments_into_single_number(strings: [Option; 3]) -> String { + strings.into_iter().flatten().collect() + } + fn assemble_age_and_balance_string_ranges( + mut numbers: VecDeque, + ) -> (String, String) { + if numbers.get(3).expectv("fourth element").is_empty() { + //meaning the 4th limit deliberately omitted by the user + numbers[3] = "UNLIMITED".to_string() + }; + let mut pop = || numbers.pop_front(); + ( + format!("{}-{}", pop().expectv("age min"), pop().expectv("age max")), + format!( + "{}-{}", + pop().expectv("balance min"), + pop().expectv("balance max") + ), + ) + } + + //the 4th capture group is inactivated by ?: + let simplifying_extractor = + Regex::new(r#"^(-?)0*(\d+)(?:(\.\d*[1-9])0*$|\.0|$)"#).expect("wrong regex"); + let apply_care = |cached_user_input: &str| -> [Option; 3] { + simplifying_extractor + .captures(cached_user_input) + .map(collect_captured_parts_of_a_number) + .unwrap_or_else( + || panic!("Broken code: value must have been present during a check but yet wrong: {}", cached_user_input) + ) + }; + let ((time_min, time_max), (amount_min, amount_max)) = user_ranges; + let vec_of_possibly_corrected_values = [ + apply_care(time_min), + apply_care(time_max), + apply_care(amount_min), + if amount_max.is_empty() { + Default::default() + } else { + apply_care(amount_max) + }, + ] + .into_iter() + .map(assemble_string_segments_into_single_number) + .collect::>(); + assemble_age_and_balance_string_ranges(vec_of_possibly_corrected_values) + } + + pub fn parse_masq_range_to_gwei(range: &str) -> Result<(N, N, String, String), String> + where + N: FromStr + TryFrom + TryFrom + CheckedMul + Display + Copy, + i64: TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, + { + let regex = Regex::new(r#"^((-?\d+\.?\d*)\s*-\s*(-?\d+\.?\d*))|(-?\d+\.?\d*)$"#) + .expect("wrong regex"); + let (first, second_opt) = extract_individual_masq_values(range, regex)?; + let first_numeral = process_optionally_fractional_number(&first)?; + let second_numeral = if let Some(second) = second_opt.as_ref() { + process_optionally_fractional_number(second)? + } else { + N::try_from(i64::MAX).expect("must fit in between limits") + }; + Ok(( + first_numeral, + second_numeral, + first, + second_opt.unwrap_or_default(), //None signifies unlimited bounds + )) + } + + pub fn parse_time_params(min_age: &str, max_age: &str) -> Result<(u64, u64), String> { + Ok(( + parse_integer_within_limits(min_age)?, + parse_integer_within_limits(max_age)?, + )) + } + + pub fn split_time_range(range: &str) -> (&str, &str) { + let age_args: Vec<&str> = range.split('-').collect(); + ( + age_args.first().expectv("age min"), + age_args.get(1).expectv("age max"), + ) + } + + pub(super) fn parse_integer_within_limits(str_gwei: &str) -> Result + where + N: FromStr + Copy + TryFrom, + u64: TryFrom, + { + fn error_msg( + gwei: &str, + lower_expected_limit: N, + higher_expected_limit: N, + ) -> String { + let numbers = [ + &gwei as &dyn Separable, + &lower_expected_limit, + &higher_expected_limit, + ] + .into_iter() + .map(|value| value.separate_with_commas()) + .collect::>(); + format!( + "Supplied value of {} gwei overflows the tech limits. You probably want one between {} and {} MASQ", numbers[0], numbers[1], numbers[2] + ) + } + let handle_parsing_error = |str_gwei: &str, e: ParseIntError| -> String { + let minus_sign_regex = Regex::new(r#"\s*-\s*\d+"#).expect("bad regex"); + match (e.kind(), minus_sign_regex.is_match(str_gwei)) { + (IntErrorKind::NegOverflow | IntErrorKind::PosOverflow, _) => { + if type_name::() == type_name::() { + error_msg(str_gwei, 0, MASQ_TOTAL_SUPPLY) + } else { + error_msg( + str_gwei, + -(MASQ_TOTAL_SUPPLY as i64), + MASQ_TOTAL_SUPPLY as i64, + ) + } + } + (IntErrorKind::InvalidDigit, true) if type_name::() == type_name::() => { + error_msg(str_gwei, 0, MASQ_TOTAL_SUPPLY) + } + _ => format!( + "Non numeric value '{}', it must be a valid integer", + str_gwei + ), + } + }; + + match str::parse::(str_gwei) { + Ok(int) => match u64::try_from(int) { + Ok(int_as_u64) => { + if int_as_u64 <= i64::MAX as u64 { + Ok(int) + } else { + Err(error_msg(str_gwei, 0, MASQ_TOTAL_SUPPLY)) + } + } + Err(_) => { + //This error can only signalize a negative number + //because we always expect N to be u64 or i64 + Ok(int) + } + }, + Err(e) => Err(handle_parsing_error(str_gwei, e)), + } + } + + fn single_capture(captures: &Captures, idx: usize) -> Option { + captures.get(idx).map(|catch| catch.as_str().to_owned()) + } + + pub(super) fn extract_individual_masq_values( + masq_in_range_str: &str, + masq_values_in_range_regex: Regex, + ) -> Result<(String, Option), String> { + fn handle_captures(captures: Captures) -> (Option, Option) { + let fetch_group = |idx: usize| single_capture(&captures, idx); + match (fetch_group(2), fetch_group(3)) { + (Some(second), Some(third)) => (Some(second), Some(third)), + (None, None) => { + let four = fetch_group(4).expect("the regex is wrong if it allows this panic"); + (Some(four), None) + } + (x, y) => { + unreachable!( + "the regex was designed not to allow '{:?}' for the second and \ + '{:?}' for the third capture group", + x, y + ) + } + } + } + + match masq_values_in_range_regex + .captures(masq_in_range_str) + .map(handle_captures) + { + Some((Some(first), Some(second))) => Ok((first, Some(second))), + Some((Some(first), None)) => Ok((first, None)), + _ => Err(format!( + "Balance range '{}' in improper format", + masq_in_range_str + )), + } + } + + pub(super) fn process_optionally_fractional_number(num: &str) -> Result + where + N: FromStr + TryFrom + TryFrom + CheckedMul + Display + Copy, + i64: TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, + { + const DIGITS_IN_BILLION: u32 = 9; + fn all_digits_with_dot_removed(num: &str) -> String { + num.chars().filter(|char| *char != '.').collect() + } + fn number_of_decimal_digits_unchecked(num: &str, dot_idx: usize) -> u32 { + let int_part_plus_dot_length = dot_idx + 1; + u32::try_from(num.chars().count() - int_part_plus_dot_length) + .expect("previous check of maximally 9 decimal digits failed") + } + fn decimal_digits_count(num: &str, dot_idx: usize) -> Result { + let decimal_digits_count = number_of_decimal_digits_unchecked(num, dot_idx); + if decimal_digits_count <= DIGITS_IN_BILLION { + Ok(decimal_digits_count) + } else { + Err(format!("Value '{}' exceeds the limit of maximally nine decimal digits (only gwei supported)", num)) + } + } + let decimal_shift_to_wei = |parse_result: N, exponent: u32| { + parse_result + .checked_mul(&N::try_from(10_i64.pow(exponent)).expect("no fear")) + .ok_or_else(|| { + format!( + "Amount bigger than the MASQ total supply: {}, total supply: {}", + num, MASQ_TOTAL_SUPPLY + ) + }) + }; + + let dot_opt = num.chars().position(|char| char == '.'); + if let Some(dot_idx) = dot_opt { + check_right_dot_usage(num, dot_idx)?; + let full_range_of_digits = all_digits_with_dot_removed(num); + let all_digits_with_dot_removed_parsed: N = + parse_integer_within_limits(&full_range_of_digits)?; + let decimal_digits_count = decimal_digits_count(num, dot_idx)?; + decimal_shift_to_wei( + all_digits_with_dot_removed_parsed, + DIGITS_IN_BILLION - decimal_digits_count, + ) + } else { + let integer_parsed = parse_integer_within_limits::(num)?; + decimal_shift_to_wei(integer_parsed, DIGITS_IN_BILLION) + } + } + + fn check_right_dot_usage(num: &str, dot_idx: usize) -> Result<(), String> { + if dot_idx == (num.len() - 1) { + Err(format!( + "Ending dot at decimal number, like here '{}', is unsupported", + num + )) + } else if num.chars().filter(|char| *char == '.').count() != 1 { + Err(format!("Misused decimal number dot delimiter at '{}'", num)) + } else { + Ok(()) + } + } +} + +#[cfg(test)] +mod tests { + use crate::commands::financials_command::parsing_and_value_dressing::restricted::{ + convert_masq_from_gwei_and_dress_well, extract_individual_masq_values, + neaten_users_writing_if_possible, parse_integer_within_limits, + process_optionally_fractional_number, + }; + use crate::commands::financials_command::test_utils::transpose_inputs_to_nested_tuples; + use masq_lib::constants::{MASQ_TOTAL_SUPPLY, WEIS_OF_GWEI}; + use regex::Regex; + + #[test] + fn convert_masq_from_gwei_and_dress_well_handles_values_smaller_than_one_hundredth_of_masq_and_bigger_than_zero( + ) { + let gwei: i64 = 9999999; + + let result = convert_masq_from_gwei_and_dress_well(gwei); + + assert_eq!(result, "< 0.01") + } + + #[test] + fn convert_masq_from_gwei_and_dress_well_handles_values_bigger_than_minus_one_hundredth_of_masq_and_smaller_than_zero( + ) { + let gwei: i64 = -9999999; + + let result = convert_masq_from_gwei_and_dress_well(gwei); + + assert_eq!(result, "-0.01 < x < 0") + } + + #[test] + fn convert_masq_from_gwei_and_dress_well_handles_positive_number() { + let gwei: i64 = 987654321987654; + + let result = convert_masq_from_gwei_and_dress_well(gwei); + + assert_eq!(result, "987,654.32") + } + + #[test] + fn convert_masq_from_gwei_and_dress_well_handles_negative_number() { + let gwei: i64 = -1234567891234; + + let result = convert_masq_from_gwei_and_dress_well(gwei); + + assert_eq!(result, "-1,234.56") + } + + #[test] + fn neaten_users_writing_handles_leading_and_tailing_zeros() { + let result = neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "00045656", + "0354865.1500000", + "000124856", + "01561785.3300", + ])); + + assert_eq!( + result, + ( + "45656-354865.15".to_string(), + "124856-1561785.33".to_string() + ) + ) + } + + #[test] + fn neaten_users_writing_handles_plain_zero_after_the_dot() { + let result = neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "45656.0", + "354865.0", + "124856.0", + "1561785.0", + ])); + + assert_eq!( + result, + ("45656-354865".to_string(), "124856-1561785".to_string()) + ) + } + + #[test] + fn neaten_users_writing_returns_same_thing_if_no_change_needed() { + let result = neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "456500", "35481533", "-500", "0.4545", + ])); + + assert_eq!( + result, + ("456500-35481533".to_string(), "-500-0.4545".to_string()) + ) + } + + #[test] + fn neaten_users_writing_returns_well_formatted_range_for_negative_values() { + let result = neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "456500", "35481533", "-500", "-45", + ])); + + assert_eq!( + result, + ("456500-35481533".to_string(), "-500--45".to_string()) + ) + } + + #[test] + fn neaten_users_writing_treats_zero_followed_decimal_numbers_gently() { + let result = neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "0.45545000", + "000.333300", + "000.00010000", + "565.454500", + ])); + + assert_eq!( + result, + ("0.45545-0.3333".to_string(), "0.0001-565.4545".to_string()) + ) + } + + #[test] + #[should_panic( + expected = "Broken code: value must have been present during a check but yet wrong: 0.4554booooga45" + )] + fn neaten_users_writing_complains_about_leaked_string_with_bad_syntax() { + neaten_users_writing_if_possible(&transpose_inputs_to_nested_tuples([ + "0.4554booooga45", + "333300", + "0.0001", + "565", + ])); + } + + #[test] + fn parse_integer_overflow_indicates_too_big_number_supplied_for_i64() { + let err_msg_i64: Result = + parse_integer_within_limits(&(i64::MAX as u64 + 1).to_string()); + + assert_eq!(err_msg_i64.unwrap_err(), "Supplied value of 9,223,372,036,854,775,808 gwei overflows the tech limits. You probably want one between -37,500,000 and 37,500,000 MASQ"); + + let err_msg_i64: Result = + parse_integer_within_limits(&(i64::MIN as i128 - 1).to_string()); + + assert_eq!(err_msg_i64.unwrap_err(), "Supplied value of -9,223,372,036,854,775,809 gwei overflows the tech limits. You probably want one between -37,500,000 and 37,500,000 MASQ") + } + + #[test] + fn parse_integer_overflow_indicates_too_big_number_supplied_for_u64() { + let err_msg_u64: Result = + parse_integer_within_limits(&(i64::MAX as u64 + 1).to_string()); + + assert_eq!(err_msg_u64.unwrap_err(), "Supplied value of 9,223,372,036,854,775,808 gwei overflows the tech limits. You probably want one between 0 and 37,500,000 MASQ"); + + let err_msg_u64: Result = parse_integer_within_limits("-1"); + + assert_eq!(err_msg_u64.unwrap_err(), "Supplied value of -1 gwei overflows the tech limits. You probably want one between 0 and 37,500,000 MASQ") + } + + #[test] + fn unparsable_u64_but_not_because_of_minus_sign() { + let err_msg: Result = parse_integer_within_limits(".1"); + + assert_eq!( + err_msg.unwrap_err(), + "Non numeric value '.1', it must be a valid integer" + ) + } + + #[test] + fn u64_detect_minus_sign_error_with_different_white_spaces_around() { + ["- 5", " -8", " - 1"].into_iter().for_each(|example|{ + let err_msg: Result = + parse_integer_within_limits(example); + assert_eq!(err_msg.unwrap_err(), format!("Supplied value of {} gwei overflows the tech limits. You probably want one between 0 and 37,500,000 MASQ", example)) + }) + } + + #[test] + fn i64_interpretation_capabilities_are_good_enough_for_masq_total_supply_in_gwei() { + let _: i64 = (MASQ_TOTAL_SUPPLY * WEIS_OF_GWEI as u64) + .try_into() + .unwrap(); + } + + #[test] + #[should_panic( + expected = "entered unreachable code: the regex was designed not to allow 'None' for the second and 'Some(\"ghi\")' for the third capture group" + )] + fn extract_individual_masq_values_regex_is_wrong() { + let regex = Regex::new("(abc)?(def)?(ghi)").unwrap(); + + let _ = extract_individual_masq_values("ghi", regex); + } + + #[test] + fn process_optionally_fractional_number_dislikes_dot_as_the_last_char() { + let result = process_optionally_fractional_number::("4556."); + + assert_eq!( + result, + Err("Ending dot at decimal number, like here '4556.', is unsupported".to_string()) + ) + } + + #[test] + fn process_optionally_fractional_number_dislikes_more_than_one_dot() { + let result = process_optionally_fractional_number::("45.056.000"); + + assert_eq!( + result, + Err("Misused decimal number dot delimiter at '45.056.000'".to_string()) + ) + } +} diff --git a/masq/src/commands/financials_command/pretty_print_utils.rs b/masq/src/commands/financials_command/pretty_print_utils.rs new file mode 100644 index 000000000..16088b7d5 --- /dev/null +++ b/masq/src/commands/financials_command/pretty_print_utils.rs @@ -0,0 +1,313 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub(in crate::commands::financials_command) mod restricted { + use crate::commands::financials_command::data_structures::restricted::{ + HeadingsHolder, ProcessAccountsMetadata, UserOriginalTypingOfRanges, + }; + use crate::commands::financials_command::parsing_and_value_dressing::restricted::{ + convert_masq_from_gwei_and_dress_well, neaten_users_writing_if_possible, + }; + use crate::commands::financials_command::FinancialsCommand; + use masq_lib::constants::WALLET_ADDRESS_LENGTH; + use masq_lib::messages::{UiPayableAccount, UiReceivableAccount}; + use masq_lib::short_writeln; + use std::fmt::{Debug, Display}; + use std::io::Write; + use thousands::Separable; + + pub trait StringValuesFormattableAccount { + fn convert_to_strings(&self, ordinal_num: usize, is_gwei: bool) -> Vec; + } + + impl StringValuesFormattableAccount for UiPayableAccount { + fn convert_to_strings(&self, ordinal_num: usize, is_gwei: bool) -> Vec { + vec![ + ordinal_num.to_string(), + self.wallet.to_string(), + self.age_s.separate_with_commas(), + process_gwei_into_requested_format(self.balance_gwei, is_gwei), + if let Some(hash) = &self.pending_payable_hash_opt { + hash.to_string() + } else { + "None".to_string() + }, + ] + } + } + + impl StringValuesFormattableAccount for UiReceivableAccount { + fn convert_to_strings(&self, ordinal_num: usize, is_gwei: bool) -> Vec { + vec![ + ordinal_num.to_string(), + self.wallet.to_string(), + self.age_s.separate_with_commas(), + process_gwei_into_requested_format(self.balance_gwei, is_gwei), + ] + } + } + + pub fn financial_status_totals_title(stdout: &mut dyn Write, is_gwei: bool) { + short_writeln!( + stdout, + "\nFinancial status totals in {}\n", + &gwei_or_masq_balance(is_gwei)[9..13] + ); + } + + pub fn main_title_for_tops_opt(fin_com: &FinancialsCommand, stdout: &mut dyn Write) { + if let Some(tr_config) = fin_com.top_records_opt.as_ref() { + short_writeln!(stdout, "Up to {} top accounts\n", tr_config.count) + } + } + + pub fn subtitle_for_tops(stdout: &mut dyn Write, account_type: &str) { + fn capitalize(name: &str) -> String { + let mut letter_iterator = name.chars(); + let first = letter_iterator + .next() + .expect("empty string instead of name"); + first.to_uppercase().chain(letter_iterator).collect() + } + short_writeln!(stdout, "{}\n", capitalize(account_type)) + } + + pub fn title_for_custom_query( + stdout: &mut dyn Write, + table_type: &str, + user_written_ranges: &UserOriginalTypingOfRanges, + ) { + let (age_range, balance_range) = neaten_users_writing_if_possible(user_written_ranges); + short_writeln!( + stdout, + "Specific {} query: {} sec {} MASQ\n", + table_type, + age_range, + balance_range + ) + } + + pub fn render_accounts_generic( + stdout: &mut dyn Write, + accounts: Vec, + headings: &HeadingsHolder, + ) { + let preformatted_subset = &accounts + .iter() + .enumerate() + .map(|(idx, account)| account.convert_to_strings(idx + 1, headings.is_gwei)) + .collect::>(); + let optimal_widths = width_precise_calculation(headings, preformatted_subset); + let headings_and_widths = &zip_them(headings.words.as_slice(), &optimal_widths); + write_column_formatted(stdout, headings_and_widths); + preformatted_subset.iter().for_each(|account| { + let zipped_inputs = zip_them(account, &optimal_widths); + write_column_formatted(stdout, &zipped_inputs); + }); + } + + pub fn process_gwei_into_requested_format(gwei: N, should_stay_gwei: bool) -> String + where + N: From + Separable + Display, + i64: TryFrom, + >::Error: Debug, + { + if should_stay_gwei { + gwei.separate_with_commas() + } else { + let gwei_as_i64 = i64::try_from(gwei) + .expect("Clap validation failed: value bigger than i64::MAX is forbidden"); + convert_masq_from_gwei_and_dress_well(gwei_as_i64) + } + } + + pub fn triple_or_single_blank_line(stdout: &mut dyn Write, leading_dump: bool) { + if leading_dump { + short_writeln!(stdout) + } else { + short_writeln!(stdout, "\n\n") + } + } + + pub fn no_records_found(stdout: &mut dyn Write, headings: &[String]) { + let mut headings_widths = widths_of_str_values(headings); + headings_widths[1] = WALLET_ADDRESS_LENGTH; + write_column_formatted(stdout, &zip_them(headings, &headings_widths)); + short_writeln!(stdout, "\nNo records found",) + } + + pub fn prepare_metadata(is_gwei: bool) -> (ProcessAccountsMetadata, ProcessAccountsMetadata) { + let (payable_headings, receivable_headings) = prepare_headings_of_records(is_gwei); + ( + ProcessAccountsMetadata { + table_type: "payable", + headings: payable_headings, + }, + ProcessAccountsMetadata { + table_type: "receivable", + headings: receivable_headings, + }, + ) + } + + fn gwei_or_masq_balance(is_gwei: bool) -> String { + format!("Balance {}", gwei_or_masq_units(is_gwei)) + } + + fn gwei_or_masq_units(is_gwei: bool) -> &'static str { + if is_gwei { + "[gwei]" + } else { + "[MASQ]" + } + } + + fn prepare_headings_of_records(is_gwei: bool) -> (HeadingsHolder, HeadingsHolder) { + fn to_owned_strings(words: Vec<&str>) -> Vec { + words.iter().map(|str| str.to_string()).collect() + } + let balance = gwei_or_masq_balance(is_gwei); + ( + HeadingsHolder { + words: to_owned_strings(vec!["#", "Wallet", "Age [s]", &balance, "Pending tx"]), + is_gwei, + }, + HeadingsHolder { + words: to_owned_strings(vec!["#", "Wallet", "Age [s]", &balance]), + is_gwei, + }, + ) + } + + fn width_precise_calculation( + headings: &HeadingsHolder, + values_of_accounts: &[Vec], + ) -> Vec { + let headings_widths = widths_of_str_values(headings.words.as_slice()); + let values_widths = figure_out_max_widths(values_of_accounts); + yield_bigger_values_from_vecs(headings_widths, &values_widths) + } + + fn widths_of_str_values>(headings: &[T]) -> Vec { + headings + .iter() + .map(|phrase| phrase.as_ref().len()) + .collect() + } + + fn zip_them<'a>( + words: &'a [String], + optimal_widths: &'a [usize], + ) -> Vec<(&'a String, &'a usize)> { + words.iter().zip(optimal_widths.iter()).collect() + } + + fn write_column_formatted( + stdout: &mut dyn Write, + account_segments_values_as_strings_and_widths: &[(&String, &usize)], + ) { + let column_count = account_segments_values_as_strings_and_widths.len(); + account_segments_values_as_strings_and_widths + .iter() + .enumerate() + .for_each(|(idx, (value, optimal_width))| { + write!( + stdout, + "{:]) -> Vec { + //two-dimensional set of strings; measuring their lengths and saving the largest value for each column + //the first value (ordinal number) and the second (wallet) are processed specifically, in shortcut + let init = vec![0_usize; values_of_accounts[0].len() - 2]; + let widths_except_ordinal_num = values_of_accounts.iter().fold(init, |acc, record| { + yield_bigger_values_from_vecs(acc, &widths_of_str_values(record)[2..]) + }); + let mut result = vec![ + (values_of_accounts.len() as f64).log10() as usize + 1, + WALLET_ADDRESS_LENGTH, + ]; + result.extend(widths_except_ordinal_num); + result + } + + fn yield_bigger_values_from_vecs(first: Vec, second: &[usize]) -> Vec { + (0..first.len()).fold(vec![], |mut acc, idx| { + acc.push(first[idx].max(second[idx])); + acc + }) + } +} + +#[cfg(test)] +mod tests { + use crate::commands::financials_command::pretty_print_utils::restricted::{ + figure_out_max_widths, StringValuesFormattableAccount, + }; + + #[derive(Clone)] + struct TestAccount { + a: &'static str, + b: &'static str, + c: &'static str, + } + + impl StringValuesFormattableAccount for TestAccount { + fn convert_to_strings(&self, ordinal_num: usize, _gwei: bool) -> Vec { + vec![ + ordinal_num.to_string(), + self.a.to_string(), + self.b.to_string(), + self.c.to_string(), + ] + } + } + + #[test] + fn figure_out_max_widths_works() { + let mut vec_of_accounts = vec![ + TestAccount { + a: "all", + b: "howdy", + c: "15489", + }, + TestAccount { + a: "whoooooo", + b: "the", + c: "meow", + }, + TestAccount { + a: "ki", + b: "", + c: "baabaalooo", + }, + ]; + //filling used to reach an ordinal number with more than just one digit, here three digits + vec_of_accounts.append(&mut vec![ + TestAccount { + a: "", + b: "", + c: "" + }; + 100 + ]); + let preformatted_subset = &vec_of_accounts + .iter() + .enumerate() + .map(|(idx, account)| account.convert_to_strings(idx, false)) + .collect::>(); + + let result = figure_out_max_widths(&preformatted_subset); + + //the first number means number of digits within the biggest ordinal number + //the second number is always 42 as the length of wallet address + assert_eq!(result, vec![3, 42, 5, 10]) + } +} diff --git a/masq/src/commands/financials_command/test_utils.rs b/masq/src/commands/financials_command/test_utils.rs new file mode 100644 index 000000000..fa7762e07 --- /dev/null +++ b/masq/src/commands/financials_command/test_utils.rs @@ -0,0 +1,10 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub fn transpose_inputs_to_nested_tuples( + inputs: [&str; 4], +) -> ((String, String), (String, String)) { + ( + (inputs[0].to_string(), inputs[1].to_string()), + (inputs[2].to_string(), inputs[3].to_string()), + ) +} diff --git a/masq/src/commands/recover_wallets_command.rs b/masq/src/commands/recover_wallets_command.rs index 15ff61728..c1feae2a6 100644 --- a/masq/src/commands/recover_wallets_command.rs +++ b/masq/src/commands/recover_wallets_command.rs @@ -125,24 +125,24 @@ impl Command for RecoverWalletsCommand { } const RECOVER_WALLETS_ABOUT: &str = - "Recovers a pair of wallets (consuming and earning) for the Node if they haven't been recovered already"; + "Recovers a pair of wallets (consuming and earning) for the Node if they haven't been recovered already."; const DB_PASSWORD_ARG_HELP: &str = - "The current database password (a password must be set to use this command)"; + "The current database password (a password must be set to use this command)."; const MNEMONIC_PHRASE_ARG_HELP: &str = "The mnemonic phrase upon which the consuming wallet (and possibly the earning wallet) is based. \ Surround with double quotes."; const PASSPHRASE_ARG_HELP: &str = - "An additional word--any word--to place at the end of the mnemonic phrase to recover the wallet pair"; -const LANGUAGE_ARG_HELP: &str = "The language in which the wallets' mnemonic phrase is written"; + "An additional word--any word--to place at the end of the mnemonic phrase to recover the wallet pair."; +const LANGUAGE_ARG_HELP: &str = "The language in which the wallets' mnemonic phrase is written."; const CONSUMING_PATH_ARG_HELP: &str = "Derivation that was used to generate the consuming wallet from which your bills will be paid. \ - Remember to put it in double quotes; otherwise the single quotes will cause problems"; + Remember to put it in double quotes; otherwise the single quotes will cause problems."; const CONSUMING_KEY_ARG_HELP: &str = "The private key of the consuming wallet. Represent it as a 64-character string of hexadecimal digits."; const EARNING_PATH_ARG_HELP: &str = "Derivation path that was used to generate the earning wallet from which your bills will be paid. \ Can be the same as consuming-path. Remember to put it in double quotes; otherwise the single \ - quotes will cause problems"; + quotes will cause problems."; const EARNING_ADDRESS_ARG_HELP: &str = "The address of the earning wallet. Represent it as '0x' followed by 40 hexadecimal digits."; const LANGUAGE_ARG_POSSIBLE_VALUES: [&str; 8] = [ @@ -254,11 +254,11 @@ mod tests { assert_eq!( RECOVER_WALLETS_ABOUT, "Recovers a pair of wallets (consuming and earning) for the Node if they haven't been \ - recovered already" + recovered already." ); assert_eq!( DB_PASSWORD_ARG_HELP, - "The current database password (a password must be set to use this command)" + "The current database password (a password must be set to use this command)." ); assert_eq!( MNEMONIC_PHRASE_ARG_HELP, @@ -267,16 +267,16 @@ mod tests { assert_eq!( PASSPHRASE_ARG_HELP, "An additional word--any word--to place at the end of the mnemonic phrase to recover \ - the wallet pair" + the wallet pair." ); assert_eq!( LANGUAGE_ARG_HELP, - "The language in which the wallets' mnemonic phrase is written" + "The language in which the wallets' mnemonic phrase is written." ); assert_eq!( CONSUMING_PATH_ARG_HELP, "Derivation that was used to generate the consuming wallet from which your bills will \ - be paid. Remember to put it in double quotes; otherwise the single quotes will cause problems" + be paid. Remember to put it in double quotes; otherwise the single quotes will cause problems." ); assert_eq!( CONSUMING_KEY_ARG_HELP, @@ -287,7 +287,7 @@ mod tests { EARNING_PATH_ARG_HELP, "Derivation path that was used to generate the earning wallet from which your bills \ will be paid. Can be the same as consuming-path. Remember to put it in double quotes; \ - otherwise the single quotes will cause problems" + otherwise the single quotes will cause problems." ); assert_eq!( EARNING_ADDRESS_ARG_HELP, diff --git a/masq/src/commands/scan_command.rs b/masq/src/commands/scan_command.rs index 0034cf187..82359d2a5 100644 --- a/masq/src/commands/scan_command.rs +++ b/masq/src/commands/scan_command.rs @@ -15,8 +15,8 @@ pub struct ScanCommand { } const SCAN_SUBCOMMAND_ABOUT: &str = - "Orders the Node to perform an immediate scan of the indicated type"; -const SCAN_SUBCOMMAND_HELP: &str = "Type of the scan that should be triggered"; + "Orders the Node to perform an immediate scan of the indicated type."; +const SCAN_SUBCOMMAND_HELP: &str = "Type of the scan that should be triggered."; pub fn scan_subcommand() -> App<'static, 'static> { SubCommand::with_name("scan") @@ -79,11 +79,11 @@ mod tests { fn constants_have_correct_values() { assert_eq!( SCAN_SUBCOMMAND_ABOUT, - "Orders the Node to perform an immediate scan of the indicated type" + "Orders the Node to perform an immediate scan of the indicated type." ); assert_eq!( SCAN_SUBCOMMAND_HELP, - "Type of the scan that should be triggered" + "Type of the scan that should be triggered." ); } diff --git a/masq/src/commands/set_configuration_command.rs b/masq/src/commands/set_configuration_command.rs index 758f22e90..fc6fc948f 100644 --- a/masq/src/commands/set_configuration_command.rs +++ b/masq/src/commands/set_configuration_command.rs @@ -6,7 +6,7 @@ use masq_lib::messages::{UiSetConfigurationRequest, UiSetConfigurationResponse}; use masq_lib::shared_schema::common_validators; use masq_lib::shared_schema::GAS_PRICE_HELP; use masq_lib::short_writeln; -use masq_lib::utils::{ExpectValue, WrapResult}; +use masq_lib::utils::ExpectValue; #[cfg(test)] use std::any::Any; @@ -18,18 +18,17 @@ pub struct SetConfigurationCommand { impl SetConfigurationCommand { pub fn new(pieces: &[String]) -> Result { - let parameter_opt = pieces.get(1).map(|s| s.replace("--", "")); + let parameter_opt = pieces.get(1).map(|s| &s[2..]); match set_configuration_subcommand().get_matches_from_safe(pieces) { Ok(matches) => { let parameter = parameter_opt.expectv("required param"); - SetConfigurationCommand { - name: parameter.clone(), + Ok(SetConfigurationCommand { + name: parameter.to_string(), value: matches .value_of(parameter) .expectv("required param") .to_string(), - } - .wrap_to_ok() + }) } Err(e) => Err(format!("{}", e)), @@ -60,7 +59,7 @@ impl Command for SetConfigurationCommand { } const SET_CONFIGURATION_ABOUT: &str = - "Sets Node configuration parameters being enabled for this operation when the Node is running"; + "Sets Node configuration parameters being enabled for this operation when the Node is running."; const START_BLOCK_HELP: &str = "Ordinal number of the Ethereum block where scanning for transactions will start."; @@ -105,7 +104,7 @@ mod tests { fn constants_have_correct_values() { assert_eq!( SET_CONFIGURATION_ABOUT, - "Sets Node configuration parameters being enabled for this operation when the Node is running" + "Sets Node configuration parameters being enabled for this operation when the Node is running." ); assert_eq!( START_BLOCK_HELP, diff --git a/masq/src/commands/wallet_addresses_command.rs b/masq/src/commands/wallet_addresses_command.rs index c834b0d37..60d99a976 100644 --- a/masq/src/commands/wallet_addresses_command.rs +++ b/masq/src/commands/wallet_addresses_command.rs @@ -33,9 +33,9 @@ impl WalletAddressesCommand { const WALLET_ADDRESS_SUBCOMMAND_ABOUT: &str = "Provides addresses of consuming and earning wallets.\ Only valid if the wallets were successfully generated (generate-wallets) or \ - recovered (recover-wallets)"; + recovered (recover-wallets)."; const DB_PASSWORD_ARG_HELP: &str = - "The current database password (a password must be set to use this command)"; + "The current database password (a password must be set to use this command)."; pub fn wallet_addresses_subcommand() -> App<'static, 'static> { SubCommand::with_name("wallet-addresses") @@ -87,11 +87,11 @@ mod tests { WALLET_ADDRESS_SUBCOMMAND_ABOUT, "Provides addresses of consuming and earning wallets.\ Only valid if the wallets were successfully generated \ - (generate-wallets) or recovered (recover-wallets)" + (generate-wallets) or recovered (recover-wallets)." ); assert_eq!( DB_PASSWORD_ARG_HELP, - "The current database password (a password must be set to use this command)" + "The current database password (a password must be set to use this command)." ); } diff --git a/masq/src/communications/connection_manager.rs b/masq/src/communications/connection_manager.rs index 3cceb20eb..956e1e8f6 100644 --- a/masq/src/communications/connection_manager.rs +++ b/masq/src/communications/connection_manager.rs @@ -555,7 +555,8 @@ mod tests { use crate::test_utils::client_utils::make_client; use crossbeam_channel::TryRecvError; use masq_lib::messages::{ - CrashReason, FromMessageBody, ToMessageBody, UiNodeCrashedBroadcast, UiSetupBroadcast, + CrashReason, FromMessageBody, ToMessageBody, UiFinancialStatistics, UiNodeCrashedBroadcast, + UiSetupBroadcast, }; use masq_lib::messages::{ UiFinancialsRequest, UiFinancialsResponse, UiRedirect, UiSetupRequest, UiSetupResponse, @@ -1170,10 +1171,13 @@ mod tests { let node_port = find_free_port(); let node_server = MockWebSocketsServer::new(node_port).queue_response( UiFinancialsResponse { - total_unpaid_and_pending_payable: 10, - total_paid_payable: 22, - total_unpaid_receivable: 29, - total_paid_receivable: 32, + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 10, + total_paid_payable_gwei: 22, + total_unpaid_receivable_gwei: 29, + total_paid_receivable_gwei: 32, + }), + query_results_opt: None, } .tmb(1), ); @@ -1187,7 +1191,12 @@ mod tests { payload: r#"{"payableMinimumAmount":12,"payableMaximumAge":23,"receivableMinimumAmount":34,"receivableMaximumAge":45}"#.to_string() }.tmb(0)); let daemon_stop_handle = daemon_server.start(); - let request = UiFinancialsRequest {}.tmb(1); + let request = UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None, + } + .tmb(1); let send_params_arc = Arc::new(Mutex::new(vec![])); let broadcast_handler = BroadcastHandleMock::new().send_params(&send_params_arc); let mut subject = ConnectionManager::new(); @@ -1204,10 +1213,13 @@ mod tests { assert_eq!( response, UiFinancialsResponse { - total_unpaid_and_pending_payable: 10, - total_paid_payable: 22, - total_unpaid_receivable: 29, - total_paid_receivable: 32 + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 10, + total_paid_payable_gwei: 22, + total_unpaid_receivable_gwei: 29, + total_paid_receivable_gwei: 32, + }), + query_results_opt: None } ); assert_eq!(context_id, 1); diff --git a/masq/src/schema.rs b/masq/src/schema.rs index d5a5d2d43..c03a3ea39 100644 --- a/masq/src/schema.rs +++ b/masq/src/schema.rs @@ -8,7 +8,7 @@ use crate::commands::configuration_command::configuration_subcommand; use crate::commands::connection_status_command::connection_status_subcommand; use crate::commands::crash_command::crash_subcommand; use crate::commands::descriptor_command::descriptor_subcommand; -use crate::commands::financials_command::financials_subcommand; +use crate::commands::financials_command::args_validation::financials_subcommand; use crate::commands::generate_wallets_command::generate_wallets_subcommand; use crate::commands::recover_wallets_command::recover_wallets_subcommand; use crate::commands::scan_command::scan_subcommand; diff --git a/masq/tests/subcommand_help_test_integration.rs b/masq/tests/subcommand_help_test_integration.rs index c25d0c478..8df209ccb 100644 --- a/masq/tests/subcommand_help_test_integration.rs +++ b/masq/tests/subcommand_help_test_integration.rs @@ -1,6 +1,9 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use crate::utils::MasqProcess; +use masq_lib::test_utils::utils::check_if_source_code_is_attached; +use masq_lib::test_utils::utils::ShouldWeRunTheTest::Skip; +use regex::Regex; use std::env::current_dir; use std::fs; @@ -8,23 +11,28 @@ mod utils; #[test] fn assure_that_all_subcommands_hang_on_the_help_root_integration() { + let current_dir = current_dir().unwrap(); + if Skip == check_if_source_code_is_attached(¤t_dir) { + return; + } let help_handle = MasqProcess::new().start_noninteractive(vec!["--help"]); let (stdout, _, _) = help_handle.stop(); let index = stdout.find("SUBCOMMANDS:").unwrap(); let trimmed_help = (&stdout[index..]).to_owned(); - let commands_files = current_dir().unwrap().join("src").join("commands"); + let commands_files = current_dir.join("src").join("commands"); let list: Vec<_> = fs::read_dir(&commands_files) .unwrap() .flat_map(|file| { - let file_name_long = file.unwrap().path(); - let file_name_long = file_name_long.file_name().unwrap().to_str().unwrap(); - if !file_name_long.starts_with("mod") && !file_name_long.starts_with("commands_common") + let entry_name_path = file.unwrap().path(); + let entry_name_long = entry_name_path.file_name().unwrap().to_str().unwrap(); + if !entry_name_long.starts_with("mod") + && !entry_name_long.starts_with("commands_common") { - let mut short_name = String::from(" "); - let suffix = file_name_long - .trim_end_matches("_command.rs") - .replace("_", "-"); - short_name.push_str(&suffix); + let regex = Regex::new(r#"(.+)_command(\.rs|$)"#) + .unwrap_or_else(|_| panic!("didn't find a match for {}", entry_name_long)); + let entity_key_word_captures = regex.captures(entry_name_long).unwrap(); + let entity_key_word = entity_key_word_captures.get(1).unwrap().as_str(); + let short_name = format!(" {}", entity_key_word.replace("_", "-")); if trimmed_help.contains(&short_name) { None } else { diff --git a/masq_lib/Cargo.toml b/masq_lib/Cargo.toml index 0ac0b19fa..b871d2334 100644 --- a/masq_lib/Cargo.toml +++ b/masq_lib/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "masq_lib" -version = "0.6.3" +version = "0.7.0" authors = ["Dan Wiebe ", "MASQ"] license = "GPL-3.0-only" copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." diff --git a/masq_lib/src/constants.rs b/masq_lib/src/constants.rs index 9ccd10166..2c63818c4 100644 --- a/masq_lib/src/constants.rs +++ b/masq_lib/src/constants.rs @@ -8,14 +8,20 @@ pub const DEFAULT_CHAIN: Chain = Chain::PolyMainnet; pub const HIGHEST_RANDOM_CLANDESTINE_PORT: u16 = 9999; pub const HTTP_PORT: u16 = 80; pub const TLS_PORT: u16 = 443; -pub const MASQ_URL_PREFIX: &str = "masq://"; -pub const DEFAULT_GAS_PRICE: u64 = 1; pub const LOWEST_USABLE_INSECURE_PORT: u16 = 1025; pub const HIGHEST_USABLE_PORT: u16 = 65535; pub const DEFAULT_UI_PORT: u16 = 5333; + +pub const MASQ_URL_PREFIX: &str = "masq://"; pub const CURRENT_LOGFILE_NAME: &str = "MASQNode_rCURRENT.log"; pub const MASQ_PROMPT: &str = "masq> "; +pub const DEFAULT_GAS_PRICE: u64 = 1; //TODO ?? Really + +pub const WALLET_ADDRESS_LENGTH: usize = 42; +pub const MASQ_TOTAL_SUPPLY: u64 = 37_500_000; +pub const WEIS_OF_GWEI: i128 = 1_000_000_000; + pub const ETH_MAINNET_CONTRACT_CREATION_BLOCK: u64 = 11_170_708; pub const ROPSTEN_TESTNET_CONTRACT_CREATION_BLOCK: u64 = 8_688_171; pub const MULTINODE_TESTNET_CONTRACT_CREATION_BLOCK: u64 = 0; @@ -52,6 +58,12 @@ pub const SETUP_ERROR: u64 = UI_NODE_COMMUNICATION_PREFIX | 5; pub const TIMEOUT_ERROR: u64 = UI_NODE_COMMUNICATION_PREFIX | 6; pub const SCAN_ERROR: u64 = UI_NODE_COMMUNICATION_PREFIX | 7; +//accountant +pub const ACCOUNTANT_PREFIX: u64 = 0x0040_0000_0000_0000; +pub const REQUEST_WITH_NO_VALUES: u64 = ACCOUNTANT_PREFIX | 1; +pub const REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS: u64 = ACCOUNTANT_PREFIX | 2; +pub const VALUE_EXCEEDS_ALLOWED_LIMIT: u64 = ACCOUNTANT_PREFIX | 3; + //////////////////////////////////////////////////////////////////////////////////////////////////// pub const COMBINED_PARAMETERS_DELIMITER: char = '|'; @@ -81,13 +93,16 @@ mod tests { assert_eq!(HIGHEST_RANDOM_CLANDESTINE_PORT, 9999); assert_eq!(HTTP_PORT, 80); assert_eq!(TLS_PORT, 443); - assert_eq!(MASQ_URL_PREFIX, "masq://"); - assert_eq!(DEFAULT_GAS_PRICE, 1); assert_eq!(LOWEST_USABLE_INSECURE_PORT, 1025); assert_eq!(HIGHEST_USABLE_PORT, 65535); assert_eq!(DEFAULT_UI_PORT, 5333); + assert_eq!(MASQ_URL_PREFIX, "masq://"); assert_eq!(CURRENT_LOGFILE_NAME, "MASQNode_rCURRENT.log"); assert_eq!(MASQ_PROMPT, "masq> "); + assert_eq!(DEFAULT_GAS_PRICE, 1); + assert_eq!(WALLET_ADDRESS_LENGTH, 42); + assert_eq!(MASQ_TOTAL_SUPPLY, 37_500_000); + assert_eq!(WEIS_OF_GWEI, 1_000_000_000); assert_eq!(ETH_MAINNET_CONTRACT_CREATION_BLOCK, 11_170_708); assert_eq!(ROPSTEN_TESTNET_CONTRACT_CREATION_BLOCK, 8_688_171); assert_eq!(MULTINODE_TESTNET_CONTRACT_CREATION_BLOCK, 0); @@ -118,6 +133,14 @@ mod tests { assert_eq!(UNMARSHAL_ERROR, UI_NODE_COMMUNICATION_PREFIX | 4); assert_eq!(SETUP_ERROR, UI_NODE_COMMUNICATION_PREFIX | 5); assert_eq!(TIMEOUT_ERROR, UI_NODE_COMMUNICATION_PREFIX | 6); + assert_eq!(SCAN_ERROR, UI_NODE_COMMUNICATION_PREFIX | 7); + assert_eq!(ACCOUNTANT_PREFIX, 0x0040_0000_0000_0000); + assert_eq!(REQUEST_WITH_NO_VALUES, ACCOUNTANT_PREFIX | 1); + assert_eq!( + REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS, + ACCOUNTANT_PREFIX | 2 + ); + assert_eq!(VALUE_EXCEEDS_ALLOWED_LIMIT, ACCOUNTANT_PREFIX | 3); assert_eq!(CENTRAL_DELIMITER, '@'); assert_eq!(CHAIN_IDENTIFIER_DELIMITER, ':'); assert_eq!(MAINNET, "mainnet"); diff --git a/masq_lib/src/logger.rs b/masq_lib/src/logger.rs index c2cde53b6..a0eef154a 100644 --- a/masq_lib/src/logger.rs +++ b/masq_lib/src/logger.rs @@ -1,3 +1,4 @@ +use std::fmt::{Debug, Formatter}; // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use crate::messages::SerializableLogLevel; #[cfg(not(feature = "log_recipient_test"))] @@ -47,6 +48,12 @@ pub struct Logger { level_limit: Level, } +impl Debug for Logger { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!(f, "Logger{{ name: \"{}\" }}", self.name) + } +} + #[macro_export] macro_rules! trace { ($logger: expr, $($arg:tt)*) => { @@ -872,6 +879,13 @@ mod tests { tlh.exists_log_containing("error! 42"); } + #[test] + fn debug_for_logger() { + let logger = Logger::new("my new logger"); + + assert_eq!(format!("{:?}", logger), "Logger{ name: \"my new logger\" }") + } + fn timestamp_as_string(timestamp: OffsetDateTime) -> String { timestamp .format(&parse(TIME_FORMATTING_STRING).unwrap()) diff --git a/masq_lib/src/messages.rs b/masq_lib/src/messages.rs index 2f6e0119c..412076e0d 100644 --- a/masq_lib/src/messages.rs +++ b/masq_lib/src/messages.rs @@ -184,7 +184,6 @@ macro_rules! conversation_message { /////////////////////////////////////////////////////////////////////// // These messages are sent only to and/or by the Daemon, not the Node /////////////////////////////////////////////////////////////////////// - // if a fire-and-forget message for the Node was detected but the Node is down #[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] pub struct UiUndeliveredFireAndForget { @@ -534,17 +533,17 @@ pub struct UiScanIntervals { #[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] pub struct UiPaymentThresholds { #[serde(rename = "thresholdIntervalSec")] - pub threshold_interval_sec: i64, + pub threshold_interval_sec: u64, #[serde(rename = "debtThresholdGwei")] - pub debt_threshold_gwei: i64, + pub debt_threshold_gwei: u64, #[serde(rename = "paymentGracePeriodSec")] - pub payment_grace_period_sec: i64, + pub payment_grace_period_sec: u64, #[serde(rename = "maturityThresholdSec")] - pub maturity_threshold_sec: i64, + pub maturity_threshold_sec: u64, #[serde(rename = "permanentDebtAllowedGwei")] - pub permanent_debt_allowed_gwei: i64, + pub permanent_debt_allowed_gwei: u64, #[serde(rename = "unbanBelowGwei")] - pub unban_below_gwei: i64, + pub unban_below_gwei: u64, } #[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq)] @@ -583,11 +582,98 @@ pub struct UiDescriptorResponse { } conversation_message!(UiDescriptorResponse, "descriptor"); +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] +pub struct UiFinancialsRequest { + #[serde(rename = "statsRequired")] + pub stats_required: bool, + #[serde(rename = "topRecordsOpt")] + pub top_records_opt: Option, + #[serde(rename = "customQueriesOpt")] + pub custom_queries_opt: Option, +} +conversation_message!(UiFinancialsRequest, "financials"); + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone, Copy)] +pub struct TopRecordsConfig { + pub count: u16, + #[serde(rename = "orderedBy")] + pub ordered_by: TopRecordsOrdering, +} + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone, Copy)] +pub enum TopRecordsOrdering { + Age, + Balance, +} + +impl TryFrom<&str> for TopRecordsOrdering { + type Error = String; + + fn try_from(value: &str) -> Result { + Ok(match value { + "balance" => Self::Balance, + "age" => Self::Age, + x => return Err(format!("Unrecognized ordering: '{}'", x)), + }) + } +} + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] +pub struct CustomQueries { + #[serde(rename = "payableOpt")] + pub payable_opt: Option>, + #[serde(rename = "receivableOpt")] + pub receivable_opt: Option>, +} + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] +pub struct RangeQuery { + #[serde(rename = "minAgeS")] + pub min_age_s: u64, + #[serde(rename = "maxAgeS")] + pub max_age_s: u64, + #[serde(rename = "minAmountGwei")] + pub min_amount_gwei: T, + #[serde(rename = "maxAmountGwei")] + pub max_amount_gwei: T, +} + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] +pub struct UiFinancialsResponse { + #[serde(rename = "statsOpt")] + pub stats_opt: Option, + #[serde(rename = "queryResultsOpt")] + pub query_results_opt: Option, +} +conversation_message!(UiFinancialsResponse, "financials"); + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] +pub struct UiFinancialStatistics { + #[serde(rename = "totalUnpaidAndPendingPayableGwei")] + pub total_unpaid_and_pending_payable_gwei: u64, + #[serde(rename = "totalPaidPayableGwei")] + pub total_paid_payable_gwei: u64, + #[serde(rename = "totalUnpaidReceivableGwei")] + pub total_unpaid_receivable_gwei: i64, + #[serde(rename = "totalPaidReceivableGwei")] + pub total_paid_receivable_gwei: u64, +} + +#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] +pub struct QueryResults { + #[serde(rename = "payableOpt")] + pub payable_opt: Option>, + #[serde(rename = "receivableOpt")] + pub receivable_opt: Option>, +} + #[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] pub struct UiPayableAccount { pub wallet: String, - pub age: u64, - pub amount: u64, + #[serde(rename = "ageS")] + pub age_s: u64, + #[serde(rename = "balanceGwei")] + pub balance_gwei: u64, #[serde(rename = "pendingPayableHashOpt")] pub pending_payable_hash_opt: Option, } @@ -595,26 +681,11 @@ pub struct UiPayableAccount { #[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] pub struct UiReceivableAccount { pub wallet: String, - pub age: u64, - pub amount: u64, -} - -#[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] -pub struct UiFinancialsRequest {} -conversation_message!(UiFinancialsRequest, "financials"); - -#[derive(Serialize, Deserialize, Debug, PartialEq, Eq)] -pub struct UiFinancialsResponse { - #[serde(rename = "totalUnpaidAndPendingPayable")] - pub total_unpaid_and_pending_payable: u64, - #[serde(rename = "totalPaidPayable")] - pub total_paid_payable: u64, - #[serde(rename = "totalUnpaidReceivable")] - pub total_unpaid_receivable: i64, - #[serde(rename = "totalPaidReceivable")] - pub total_paid_receivable: u64, + #[serde(rename = "ageS")] + pub age_s: u64, + #[serde(rename = "balanceGwei")] + pub balance_gwei: i64, } -conversation_message!(UiFinancialsResponse, "financials"); #[derive(Serialize, Deserialize, Debug, PartialEq, Eq, Clone)] pub struct UiGenerateSeedSpec { @@ -1117,4 +1188,24 @@ mod tests { Err("Unrecognized ScanType: 'unrecognized'".to_string()) ); } + + #[test] + fn top_records_ordering_from_str() { + assert_eq!( + TopRecordsOrdering::try_from("balance").unwrap(), + TopRecordsOrdering::Balance + ); + assert_eq!( + TopRecordsOrdering::try_from("age").unwrap(), + TopRecordsOrdering::Age + ) + } + + #[test] + fn top_records_ordering_from_str_error() { + assert_eq!( + TopRecordsOrdering::try_from("upside-down"), + Err("Unrecognized ordering: 'upside-down'".to_string()) + ); + } } diff --git a/masq_lib/src/shared_schema.rs b/masq_lib/src/shared_schema.rs index d7eafac99..2d77ea687 100644 --- a/masq_lib/src/shared_schema.rs +++ b/masq_lib/src/shared_schema.rs @@ -122,13 +122,13 @@ pub const RATE_PACK_HELP: &str = "\ These four parameters specify your rates that your Node will use for charging other Nodes for your provided \ services. These are ever present values, defaulted if left unspecified. The parameters must be always supplied \ all together, delimited by vertical bars and in the right order.\n\n\ - 1. Routing Byte Rate: This parameter indicates an amount of MASQ demanded to process 1 byte of routed payload \ + 1. Routing Byte Rate: This parameter indicates an amount of MASQ in wei demanded to process 1 byte of routed payload \ while the Node is a common relay Node.\n\n\ - 2. Routing Service Rate: This parameter indicates an amount of MASQ demanded to provide services, unpacking \ + 2. Routing Service Rate: This parameter indicates an amount of MASQ in wei demanded to provide services, unpacking \ and repacking 1 CORES package, while the Node is a common relay Node.\n\n\ - 3. Exit Byte Rate: This parameter indicates an amount of MASQ demanded to process 1 byte of routed payload \ + 3. Exit Byte Rate: This parameter indicates an amount of MASQ in wei demanded to process 1 byte of routed payload \ while the Node acts as the exit Node.\n\n\ - 4. Exit Service Rate: This parameter indicates an amount of MASQ demanded to provide services, unpacking and \ + 4. Exit Service Rate: This parameter indicates an amount of MASQ in wei demanded to provide services, unpacking and \ repacking 1 CORES package, while the Node acts as the exit Node."; pub const PAYMENT_THRESHOLDS_HELP: &str = "\ These are parameters that define thresholds to determine when and how much to pay other Nodes for routing and \ @@ -137,7 +137,7 @@ pub const PAYMENT_THRESHOLDS_HELP: &str = "\ since they have not paid mature debts. These are ever present values, no matter if the user's set any value, as \ they have defaults. The parameters must be always supplied all together, delimited by vertical bars and in the right \ order.\n\n\ - 1. Debt Threshold Gwei: Payables higher than this -- in Gwei of MASQ -- will be suggested for payment immediately \ + 1. Debt Threshold gwei: Payables higher than this -- in gwei of MASQ -- will be suggested for payment immediately \ upon passing the Maturity Threshold Sec age. Payables less than this can stay unpaid longer. Receivables higher than \ this will be expected to be settled by other Nodes, but will never cause bans until they pass the Maturity Threshold Sec \ + Payment Grace Period Sec age. Receivables less than this will survive longer without banning.\n\n\ @@ -145,15 +145,15 @@ pub const PAYMENT_THRESHOLDS_HELP: &str = "\ that it be paid.\n\n\ 3. Payment Grace Period Sec: A large receivable can get as old as Maturity Threshold Sec + Payment Grace Period Sec \ -- in seconds -- before the Node that owes it will be banned.\n\n\ - 4. Permanent Debt Allowed Gwei: Receivables this small and smaller -- in Gwei of MASQ -- will not cause bans no \ + 4. Permanent Debt Allowed gwei: Receivables this small and smaller -- in gwei of MASQ -- will not cause bans no \ matter how old they get.\n\n\ 5. Threshold Interval Sec: This interval -- in seconds -- begins after Maturity Threshold Sec for payables and after \ Maturity Threshold Sec + Payment Grace Period Sec for receivables. During the interval, the amount of a payable that is \ - allowed to remain unpaid, or a pending receivable that won’t cause a ban, decreases linearly from the Debt Threshold Gwei \ - to Permanent Debt Allowed Gwei or Unban Below Gwei.\n\n\ - 6. Unban Below Gwei: When a delinquent Node has been banned due to non-payment, the receivables balance must be paid \ - below this level -- in Gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the same \ - as Permanent Debt Allowed Gwei."; + allowed to remain unpaid, or a pending receivable that won’t cause a ban, decreases linearly from the Debt Threshold gwei \ + to Permanent Debt Allowed gwei or Unban Below gwei.\n\n\ + 6. Unban Below gwei: When a delinquent Node has been banned due to non-payment, the receivables balance must be paid \ + below this level -- in gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the same \ + as Permanent Debt Allowed gwei."; pub const SCAN_INTERVALS_HELP:&str = "\ These three intervals describe the length of three different scan cycles running automatically in the background \ since the Node has connected to a qualified neighborhood that consists of neighbors enabling a complete 3-hop \ @@ -191,7 +191,7 @@ lazy_static! { LOWEST_USABLE_INSECURE_PORT, HIGHEST_USABLE_PORT ); pub static ref GAS_PRICE_HELP: String = format!( - "The Gas Price is the amount of Gwei you will pay per unit of gas used in a transaction. \ + "The Gas Price is the amount of gwei you will pay per unit of gas used in a transaction. \ If left unspecified, MASQ Node will use the previously stored value (Default {}).", DEFAULT_GAS_PRICE); } @@ -564,12 +564,19 @@ pub mod common_validators { } } + pub fn validate_non_zero_u16(str: String) -> Result<(), String> { + match str::parse::(&str) { + Ok(num) if num > 0 => Ok(()), + _ => Err(str), + } + } + pub fn validate_separate_u64_values(values_with_delimiters: String) -> Result<(), String> { values_with_delimiters.split('|').try_for_each(|segment| { segment .parse::() .map_err(|_| { - "Wrong format, supply positive numeric values separated by vertical bars like 111|222|333|..." + "Supply positive numeric values separated by vertical bars like 111|222|333|..." .to_string() }) .map(|_| ()) @@ -631,6 +638,7 @@ mod tests { use super::*; use crate::blockchains::chains::Chain; + use crate::shared_schema::common_validators::validate_non_zero_u16; use crate::shared_schema::{common_validators, official_chain_names}; #[test] @@ -791,7 +799,7 @@ mod tests { assert_eq!( GAS_PRICE_HELP.to_string(), format!( - "The Gas Price is the amount of Gwei you will pay per unit of gas used in a transaction. \ + "The Gas Price is the amount of gwei you will pay per unit of gas used in a transaction. \ If left unspecified, MASQ Node will use the previously stored value (Default {}).", DEFAULT_GAS_PRICE ) @@ -801,13 +809,13 @@ mod tests { "These four parameters specify your rates that your Node will use for charging other Nodes for your provided \ services. These are ever present values, defaulted if left unspecified. The parameters must be always supplied \ all together, delimited by vertical bars and in the right order.\n\n\ - 1. Routing Byte Rate: This parameter indicates an amount of MASQ demanded to process 1 byte of routed payload \ + 1. Routing Byte Rate: This parameter indicates an amount of MASQ in wei demanded to process 1 byte of routed payload \ while the Node is a common relay Node.\n\n\ - 2. Routing Service Rate: This parameter indicates an amount of MASQ demanded to provide services, unpacking \ + 2. Routing Service Rate: This parameter indicates an amount of MASQ in wei demanded to provide services, unpacking \ and repacking 1 CORES package, while the Node is a common relay Node.\n\n\ - 3. Exit Byte Rate: This parameter indicates an amount of MASQ demanded to process 1 byte of routed payload \ + 3. Exit Byte Rate: This parameter indicates an amount of MASQ in wei demanded to process 1 byte of routed payload \ while the Node acts as the exit Node.\n\n\ - 4. Exit Service Rate: This parameter indicates an amount of MASQ demanded to provide services, unpacking and \ + 4. Exit Service Rate: This parameter indicates an amount of MASQ in wei demanded to provide services, unpacking and \ repacking 1 CORES package, while the Node acts as the exit Node." ); assert_eq!( @@ -817,7 +825,7 @@ mod tests { exit services. The thresholds are also used to determine whether to offer services to other Nodes or enact a ban \ since they have not paid mature debts. These are ever present values, no matter if the user's set any value, as \ they have defaults. The parameters must be always supplied all together, delimited by vertical bars and in the right order.\n\n\ - 1. Debt Threshold Gwei: Payables higher than this -- in Gwei of MASQ -- will be suggested for payment immediately \ + 1. Debt Threshold gwei: Payables higher than this -- in gwei of MASQ -- will be suggested for payment immediately \ upon passing the Maturity Threshold Sec age. Payables less than this can stay unpaid longer. Receivables higher than \ this will be expected to be settled by other Nodes, but will never cause bans until they pass the Maturity Threshold Sec \ + Payment Grace Period Sec age. Receivables less than this will survive longer without banning.\n\n\ @@ -825,15 +833,15 @@ mod tests { that it be paid.\n\n\ 3. Payment Grace Period Sec: A large receivable can get as old as Maturity Threshold Sec + Payment Grace Period Sec \ -- in seconds -- before the Node that owes it will be banned.\n\n\ - 4. Permanent Debt Allowed Gwei: Receivables this small and smaller -- in Gwei of MASQ -- will not cause bans no \ + 4. Permanent Debt Allowed gwei: Receivables this small and smaller -- in gwei of MASQ -- will not cause bans no \ matter how old they get.\n\n\ 5. Threshold Interval Sec: This interval -- in seconds -- begins after Maturity Threshold Sec for payables and after \ Maturity Threshold Sec + Payment Grace Period Sec for receivables. During the interval, the amount of a payable that is \ - allowed to remain unpaid, or a pending receivable that won’t cause a ban, decreases linearly from the Debt Threshold Gwei \ - to Permanent Debt Allowed Gwei or Unban Below Gwei.\n\n\ - 6. Unban Below Gwei: When a delinquent Node has been banned due to non-payment, the receivables balance must be paid \ - below this level -- in Gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the same \ - as Permanent Debt Allowed Gwei." + allowed to remain unpaid, or a pending receivable that won’t cause a ban, decreases linearly from the Debt Threshold gwei \ + to Permanent Debt Allowed gwei or Unban Below gwei.\n\n\ + 6. Unban Below gwei: When a delinquent Node has been banned due to non-payment, the receivables balance must be paid \ + below this level -- in gwei of MASQ -- to cause them to be unbanned. In most cases, you'll want this to be set the same \ + as Permanent Debt Allowed gwei." ); assert_eq!( SCAN_INTERVALS_HELP, @@ -966,19 +974,12 @@ mod tests { } #[test] - fn validate_gas_price_normal_ropsten() { + fn validate_gas_price_normal() { let result = common_validators::validate_gas_price("2".to_string()); assert_eq!(result, Ok(())); } - #[test] - fn validate_gas_price_normal_mainnet() { - let result = common_validators::validate_gas_price("20".to_string()); - - assert_eq!(result, Ok(())); - } - #[test] fn validate_gas_price_max() { let max = 0xFFFFFFFFFFFFFFFFu64; @@ -1015,7 +1016,7 @@ mod tests { assert_eq!( result, Err(String::from( - "Wrong format, supply positive numeric values separated by vertical bars like 111|222|333|..." + "Supply positive numeric values separated by vertical bars like 111|222|333|..." )) ) } @@ -1027,7 +1028,7 @@ mod tests { assert_eq!( result, Err(String::from( - "Wrong format, supply positive numeric values separated by vertical bars like 111|222|333|..." + "Supply positive numeric values separated by vertical bars like 111|222|333|..." )) ) } @@ -1039,11 +1040,46 @@ mod tests { assert_eq!( result, Err(String::from( - "Wrong format, supply positive numeric values separated by vertical bars like 111|222|333|..." + "Supply positive numeric values separated by vertical bars like 111|222|333|..." )) ) } + #[test] + fn validate_non_zero_u16_happy_path() { + let result = validate_non_zero_u16("456".to_string()); + + assert_eq!(result, Ok(())) + } + + #[test] + fn validate_non_zero_u16_sad_path_with_zero() { + let result = validate_non_zero_u16("0".to_string()); + + assert_eq!(result, Err("0".to_string())) + } + + #[test] + fn validate_non_zero_u16_sad_path_with_negative() { + let result = validate_non_zero_u16("-123".to_string()); + + assert_eq!(result, Err("-123".to_string())) + } + + #[test] + fn validate_non_zero_u16_too_big() { + let result = validate_non_zero_u16("65536".to_string()); + + assert_eq!(result, Err("65536".to_string())) + } + + #[test] + fn validate_non_zero_u16_sad_path_just_junk() { + let result = validate_non_zero_u16("garbage".to_string()); + + assert_eq!(result, Err("garbage".to_string())) + } + #[test] fn official_chain_names_are_reliable() { let mut iterator = official_chain_names().iter(); diff --git a/masq_lib/src/test_utils/utils.rs b/masq_lib/src/test_utils/utils.rs index baa9b6f83..ddc32e7bc 100644 --- a/masq_lib/src/test_utils/utils.rs +++ b/masq_lib/src/test_utils/utils.rs @@ -1,13 +1,15 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use crate::blockchains::chains::Chain; +use crate::test_utils::environment_guard::EnvironmentGuard; use std::fs; -use std::path::PathBuf; +use std::path::{Path, PathBuf}; use std::time::Duration; pub const TEST_DEFAULT_CHAIN: Chain = Chain::EthRopsten; pub const TEST_DEFAULT_MULTINODE_CHAIN: Chain = Chain::Dev; pub const BASE_TEST_DIR: &str = "generated/test"; +const MASQ_SOURCE_CODE_UNAVAILABLE: &str = "MASQ_SOURCE_CODE_UNAVAILABLE"; pub fn node_home_directory(module: &str, name: &str) -> PathBuf { let home_dir_string = format!("{}/{}/{}/home", BASE_TEST_DIR, module, name); @@ -35,6 +37,33 @@ pub fn is_running_under_github_actions() -> bool { } } +#[derive(PartialEq, Eq)] +pub enum ShouldWeRunTheTest { + GoAhead, + Skip, +} + +pub fn check_if_source_code_is_attached(current_dir: &Path) -> ShouldWeRunTheTest { + let _guard = EnvironmentGuard::new(); + if current_dir.join("src").exists() && current_dir.join("Cargo.toml").exists() { + ShouldWeRunTheTest::GoAhead + } else if std::env::var(MASQ_SOURCE_CODE_UNAVAILABLE).is_ok() { + eprintln!( + "Trying to run a test dependent on reading the source code which wasn't \ + found. Nevertheless, MASQ_SOURCE_CODE_UNAVAILABLE environment variable has been \ + supplied; skipping this test." + ); + ShouldWeRunTheTest::Skip + } else { + panic!( + "Test depending on interaction with the source code, but it was not found at \ + {:?}. If that does not surprise you, set the environment variable \ + MASQ_SOURCE_CODE_UNAVAILABLE to some non-blank value and run the tests again.", + current_dir + ) + } +} + pub fn to_millis(dur: &Duration) -> u64 { (dur.as_secs() * 1000) + (u64::from(dur.subsec_nanos()) / 1_000_000) } diff --git a/masq_lib/src/utils.rs b/masq_lib/src/utils.rs index 35dbc6ce1..ba1df4ad4 100644 --- a/masq_lib/src/utils.rs +++ b/masq_lib/src/utils.rs @@ -348,6 +348,8 @@ where where F: FnOnce(&T, &mut Self) -> Self::Result, { + //TODO we should seriously think about rewriting this in well tested unsafe code, + // Rust is unnecessarily strict as for this conflicting situation let helper = self.helper_access().take().expectv("helper"); let result = closure(&helper, self); self.helper_access().replace(helper); diff --git a/multinode_integration_tests/Cargo.toml b/multinode_integration_tests/Cargo.toml index 89943ee8c..674220dad 100644 --- a/multinode_integration_tests/Cargo.toml +++ b/multinode_integration_tests/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "multinode_integration_tests" -version = "0.6.3" +version = "0.7.0" authors = ["Dan Wiebe ", "MASQ"] license = "GPL-3.0-only" copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." @@ -23,7 +23,7 @@ node = { path = "../node", features = [ "expose_test_privates" ] } pretty-hex = "0.2.1" primitive-types = {version = "0.5.0", default-features = false, features = ["default", "rlp", "serde"] } regex = "1.5.4" -rusqlite = {version = "0.26.1", features = ["bundled"]} +rusqlite = {version = "0.28.0", features = ["bundled"]} rustc-hex = "2.1.0" serde = "1.0.130" serde_cbor = "0.11.2" diff --git a/multinode_integration_tests/src/utils.rs b/multinode_integration_tests/src/utils.rs index 960f12244..7916243a0 100644 --- a/multinode_integration_tests/src/utils.rs +++ b/multinode_integration_tests/src/utils.rs @@ -8,8 +8,9 @@ use masq_lib::utils::NeighborhoodModeLight; use node_lib::accountant::payable_dao::{PayableDao, PayableDaoReal}; use node_lib::accountant::receivable_dao::{ReceivableDao, ReceivableDaoReal}; use node_lib::database::connection_wrapper::ConnectionWrapper; -use node_lib::database::db_initializer::{DbInitializer, DbInitializerReal}; -use node_lib::database::db_migrations::{ExternalData, MigratorConfig}; +use node_lib::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, ExternalData, +}; use node_lib::db_config::config_dao::{ConfigDao, ConfigDaoReal}; use node_lib::neighborhood::node_record::NodeRecordInner_0v1; use node_lib::neighborhood::AccessibleGossipRecord; @@ -76,8 +77,7 @@ pub fn database_conn(node_name: &str) -> Box { db_initializer .initialize( &path, - true, - MigratorConfig::create_or_migrate(ExternalData { + DbInitializationConfig::create_or_migrate(ExternalData { chain: TEST_DEFAULT_MULTINODE_CHAIN, neighborhood_mode: NeighborhoodModeLight::Standard, db_password_opt: None, diff --git a/multinode_integration_tests/tests/blockchain_interaction_test.rs b/multinode_integration_tests/tests/blockchain_interaction_test.rs index c1cf32d4f..79dbc637c 100644 --- a/multinode_integration_tests/tests/blockchain_interaction_test.rs +++ b/multinode_integration_tests/tests/blockchain_interaction_test.rs @@ -21,6 +21,7 @@ use multinode_integration_tests_lib::mock_blockchain_client_server::MBCSBuilder; use multinode_integration_tests_lib::utils::{ config_dao, open_all_file_permissions, receivable_dao, UrlHolder, }; +use node_lib::accountant::dao_utils::CustomQuery; use node_lib::sub_lib::wallet::Wallet; #[test] @@ -85,16 +86,24 @@ fn debtors_are_credited_once_but_not_twice() { .more_money_receivable( SystemTime::UNIX_EPOCH.add(Duration::from_secs(15_000_000)), &Wallet::new("0x3333333333333333333333333333333333333333"), - 1_000_000, + 9_000_000_000, ) .unwrap(); } // Use the receivable DAO to verify that the receivable's balance has been initialized { let receivable_dao = receivable_dao(&node_name); - let receivable_accounts = receivable_dao.receivables(); + let receivable_accounts = receivable_dao + .custom_query(CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: i64::MAX as u64, + min_amount_gwei: 0, + max_amount_gwei: i64::MAX, + timestamp: SystemTime::now(), + }) + .unwrap_or_default(); assert_eq!(receivable_accounts.len(), 1); - assert_eq!(receivable_accounts[0].balance, 1_000_000); + assert_eq!(receivable_accounts[0].balance_wei, 9_000_000_000); } // Use the config DAO to verify that the start block has been set to 1000 { @@ -121,9 +130,17 @@ fn debtors_are_credited_once_but_not_twice() { // Use the receivable DAO to verify that the receivable's balance has been adjusted { let receivable_dao = receivable_dao(&node_name); - let receivable_accounts = receivable_dao.receivables(); + let receivable_accounts = receivable_dao + .custom_query(CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: i64::MAX as u64, + min_amount_gwei: i64::MIN, + max_amount_gwei: i64::MAX, + timestamp: SystemTime::now(), + }) + .unwrap_or_default(); assert_eq!(receivable_accounts.len(), 1); - assert_eq!(receivable_accounts[0].balance, 1_000_000); + assert_eq!(receivable_accounts[0].balance_wei, 9_000_000_000); } // Use the config DAO to verify that the start block has been advanced to 2001 { diff --git a/multinode_integration_tests/tests/bookkeeping_test.rs b/multinode_integration_tests/tests/bookkeeping_test.rs index 51780bd43..98b4d3630 100644 --- a/multinode_integration_tests/tests/bookkeeping_test.rs +++ b/multinode_integration_tests/tests/bookkeeping_test.rs @@ -1,7 +1,3 @@ -use std::collections::HashMap; -use std::thread; -use std::time::Duration; - // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use multinode_integration_tests_lib::masq_node::{MASQNode, NodeReference}; use multinode_integration_tests_lib::masq_node_cluster::MASQNodeCluster; @@ -9,9 +5,13 @@ use multinode_integration_tests_lib::masq_real_node::{ make_consuming_wallet_info, make_earning_wallet_info, MASQRealNode, NodeStartupConfigBuilder, }; use multinode_integration_tests_lib::utils::{payable_dao, receivable_dao}; +use node_lib::accountant::dao_utils::CustomQuery; use node_lib::accountant::payable_dao::PayableAccount; use node_lib::accountant::receivable_dao::ReceivableAccount; use node_lib::sub_lib::wallet::Wallet; +use std::collections::HashMap; +use std::thread; +use std::time::{Duration, SystemTime}; #[test] fn provided_and_consumed_services_are_recorded_in_databases() { @@ -48,9 +48,11 @@ fn provided_and_consumed_services_are_recorded_in_databases() { .flat_map(|node| { receivables(node) .into_iter() - .map(move |receivable_account| (node.earning_wallet(), receivable_account.balance)) + .map(move |receivable_account| { + (node.earning_wallet(), receivable_account.balance_wei) + }) }) - .collect::>(); + .collect::>(); // check that each payable has a receivable assert_eq!( @@ -68,8 +70,8 @@ fn provided_and_consumed_services_are_recorded_in_databases() { payables.iter().for_each(|payable| { assert_eq!( - &payable.balance, - receivable_balances.get(&payable.wallet).unwrap(), + payable.balance_wei, + *receivable_balances.get(&payable.wallet).unwrap() as u128, ); }); } @@ -81,7 +83,15 @@ fn non_pending_payables(node: &MASQRealNode) -> Vec { fn receivables(node: &MASQRealNode) -> Vec { let receivable_dao = receivable_dao(node.name()); - receivable_dao.receivables() + receivable_dao + .custom_query(CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: i64::MAX as u64, + min_amount_gwei: i64::MIN, + max_amount_gwei: i64::MAX, + timestamp: SystemTime::now(), + }) + .unwrap_or_default() } pub fn start_lonely_real_node(cluster: &mut MASQNodeCluster) -> MASQRealNode { diff --git a/multinode_integration_tests/tests/verify_bill_payment.rs b/multinode_integration_tests/tests/verify_bill_payment.rs index 287240ae9..b56ca1cfc 100644 --- a/multinode_integration_tests/tests/verify_bill_payment.rs +++ b/multinode_integration_tests/tests/verify_bill_payment.rs @@ -2,6 +2,7 @@ use bip39::{Language, Mnemonic, Seed}; use futures::Future; use masq_lib::blockchains::chains::Chain; +use masq_lib::constants::WEIS_OF_GWEI; use masq_lib::utils::{derivation_path, NeighborhoodModeLight}; use multinode_integration_tests_lib::blockchain::BlockchainServer; use multinode_integration_tests_lib::masq_node::{MASQNode, MASQNodeUtils}; @@ -17,16 +18,17 @@ use node_lib::blockchain::bip32::Bip32ECKeyProvider; use node_lib::blockchain::blockchain_interface::{ BlockchainInterface, BlockchainInterfaceNonClandestine, REQUESTS_IN_PARALLEL, }; -use node_lib::database::db_initializer::{DbInitializer, DbInitializerReal}; -use node_lib::database::db_migrations::{ExternalData, MigratorConfig}; +use node_lib::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, ExternalData, +}; use node_lib::sub_lib::accountant::PaymentThresholds; use node_lib::sub_lib::wallet::Wallet; use node_lib::test_utils; use rustc_hex::{FromHex, ToHex}; use std::convert::TryFrom; use std::path::{Path, PathBuf}; -use std::thread; use std::time::{Duration, Instant, SystemTime}; +use std::{thread, u128}; use tiny_hderive::bip32::ExtendedPrivKey; use web3::transports::Http; use web3::types::{Address, Bytes, TransactionParameters}; @@ -94,7 +96,7 @@ fn verify_bill_payment() { derivation_path(0, 3), ); - let amount = 10u64 * u64::try_from(payment_thresholds.permanent_debt_allowed_gwei).unwrap(); + let amount = 10 * payment_thresholds.permanent_debt_allowed_gwei as u128 * WEIS_OF_GWEI as u128; let project_root = MASQNodeUtils::find_project_root(); let (consuming_node_name, consuming_node_index) = cluster.prepare_real_node(&consuming_config); @@ -102,8 +104,7 @@ fn verify_bill_payment() { let consuming_node_connection = DbInitializerReal::default() .initialize( Path::new(&consuming_node_path), - true, - make_migrator_config(cluster.chain), + make_init_config(cluster.chain), ) .unwrap(); let consuming_payable_dao = PayableDaoReal::new(consuming_node_connection); @@ -141,8 +142,7 @@ fn verify_bill_payment() { let serving_node_1_connection = DbInitializerReal::default() .initialize( Path::new(&serving_node_1_path), - true, - make_migrator_config(cluster.chain), + make_init_config(cluster.chain), ) .unwrap(); let serving_node_1_receivable_dao = ReceivableDaoReal::new(serving_node_1_connection); @@ -157,8 +157,7 @@ fn verify_bill_payment() { let serving_node_2_connection = DbInitializerReal::default() .initialize( Path::new(&serving_node_2_path), - true, - make_migrator_config(cluster.chain), + make_init_config(cluster.chain), ) .unwrap(); let serving_node_2_receivable_dao = ReceivableDaoReal::new(serving_node_2_connection); @@ -173,8 +172,7 @@ fn verify_bill_payment() { let serving_node_3_connection = DbInitializerReal::default() .initialize( Path::new(&serving_node_3_path), - true, - make_migrator_config(cluster.chain), + make_init_config(cluster.chain), ) .unwrap(); let serving_node_3_receivable_dao = ReceivableDaoReal::new(serving_node_3_connection); @@ -245,21 +243,21 @@ fn verify_bill_payment() { &serving_node_1_wallet, &blockchain_interface, "100000000000000000000", - (1_000_000_000 * amount).to_string().as_str(), + amount.to_string().as_str(), ); assert_balances( &serving_node_2_wallet, &blockchain_interface, "100000000000000000000", - (1_000_000_000 * amount).to_string().as_str(), + amount.to_string().as_str(), ); assert_balances( &serving_node_3_wallet, &blockchain_interface, "100000000000000000000", - (1_000_000_000 * amount).to_string().as_str(), + amount.to_string().as_str(), ); let serving_node_1 = cluster.start_named_real_node( @@ -290,29 +288,29 @@ fn verify_bill_payment() { test_utils::wait_for(Some(1000), Some(15000), || { if let Some(status) = serving_node_1_receivable_dao.account_status(&contract_owner_wallet) { - status.balance == 0 + status.balance_wei == 0 } else { false } }); test_utils::wait_for(Some(1000), Some(15000), || { if let Some(status) = serving_node_2_receivable_dao.account_status(&contract_owner_wallet) { - status.balance == 0 + status.balance_wei == 0 } else { false } }); test_utils::wait_for(Some(1000), Some(15000), || { if let Some(status) = serving_node_3_receivable_dao.account_status(&contract_owner_wallet) { - status.balance == 0 + status.balance_wei == 0 } else { false } }); } -fn make_migrator_config(chain: Chain) -> MigratorConfig { - MigratorConfig::create_or_migrate(ExternalData::new( +fn make_init_config(chain: Chain) -> DbInitializationConfig { + DbInitializationConfig::create_or_migrate(ExternalData::new( chain, NeighborhoodModeLight::Standard, None, @@ -422,7 +420,7 @@ fn build_config( fn expire_payables(path: PathBuf) { let conn = DbInitializerReal::default() - .initialize(&path, true, MigratorConfig::panic_on_migration()) + .initialize(&path, DbInitializationConfig::panic_on_migration()) .unwrap(); let mut statement = conn .prepare("update payable set last_paid_timestamp = 0 where pending_payable_rowid is null") @@ -437,7 +435,7 @@ fn expire_payables(path: PathBuf) { fn expire_receivables(path: PathBuf) { let conn = DbInitializerReal::default() - .initialize(&path, true, MigratorConfig::panic_on_migration()) + .initialize(&path, DbInitializationConfig::panic_on_migration()) .unwrap(); let mut statement = conn .prepare("update receivable set last_received_timestamp = 0") diff --git a/node/Cargo.lock b/node/Cargo.lock index 8423c67d0..86a081080 100644 --- a/node/Cargo.lock +++ b/node/Cargo.lock @@ -182,7 +182,7 @@ checksum = "cdb031dd78e28731d87d56cc8ffef4a8f36ca26c38fe2de700543e627f8a464a" [[package]] name = "automap" -version = "0.6.2" +version = "0.7.0" dependencies = [ "crossbeam-channel 0.5.1", "flexi_logger 0.17.1", @@ -271,9 +271,9 @@ dependencies = [ [[package]] name = "blake2b_simd" -version = "0.5.10" +version = "0.5.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8fb2d74254a3a0b5cac33ac9f8ed0e44aa50378d9dbb2e5d83bd21ed1dc2c8a" +checksum = "afa748e348ad3be8263be728124b24a24f268266f6f5d58af9d75f6a40b5c587" dependencies = [ "arrayref", "arrayvec", @@ -652,12 +652,12 @@ dependencies = [ [[package]] name = "ctrlc" -version = "3.2.1" +version = "3.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a19c6cedffdc8c03a3346d723eb20bd85a13362bb96dc2ac000842c6381ec7bf" +checksum = "1631ca6e3c59112501a9d87fd86f21591ff77acd31331e8a73f8d80a65bbdd71" dependencies = [ - "nix 0.23.1", - "winapi 0.3.9", + "nix 0.26.1", + "windows-sys", ] [[package]] @@ -1211,20 +1211,20 @@ checksum = "d7afe4a420e3fe79967a00898cc1f4db7c8a49a9333a29f8a4bd76a253d5cd04" [[package]] name = "hashbrown" -version = "0.11.2" +version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab5ef0d4909ef3724cc8cce6ccc8572c5c817592e9285f5464f8e86f8bd3726e" +checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" dependencies = [ "ahash", ] [[package]] name = "hashlink" -version = "0.7.0" +version = "0.8.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7249a3129cbc1ffccd74857f81464a323a152173cdb134e0fd81bc803b29facf" +checksum = "69fe1fcf8b4278d860ad0548329f892a3631fb63f82574df68275f34cdbe0ffa" dependencies = [ - "hashbrown 0.11.2", + "hashbrown 0.12.3", ] [[package]] @@ -1616,9 +1616,9 @@ checksum = "dc6f3ad7b9d11a0c00842ff8de1b60ee58661048eb8049ed33c73594f359d7e6" [[package]] name = "itoa" -version = "1.0.1" +version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1aab8fc367588b89dcee83ab0fd66b72b50b72fa1904d7095045ace2b0c81c35" +checksum = "4217ad341ebadf8d8e724e264f13e593e0648f5b3e94b3896a5df283be015ecc" [[package]] name = "js-sys" @@ -1666,9 +1666,9 @@ checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "libc" -version = "0.2.126" +version = "0.2.138" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "349d5a591cd28b49e1d1037471617a32ddcda5731b99419008085f72d5a53836" +checksum = "db6d7e329c562c5dfab7a46a2afabc8b987ab9a4834c9d1ca04dc54c1546cef8" [[package]] name = "libsecp256k1" @@ -1747,9 +1747,9 @@ dependencies = [ [[package]] name = "libsqlite3-sys" -version = "0.23.2" +version = "0.25.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2cafc7c74096c336d9d27145f7ebd4f4b6f95ba16aa5a282387267e6925cb58" +checksum = "9f0455f2c1bc9a7caa792907026e469c1d91761fb0ea37cbb16427c77280cf35" dependencies = [ "cc", "pkg-config", @@ -1827,8 +1827,9 @@ dependencies = [ [[package]] name = "masq" -version = "0.6.3" +version = "0.7.0" dependencies = [ + "atty", "clap", "crossbeam-channel 0.5.1", "ctrlc", @@ -1837,14 +1838,16 @@ dependencies = [ "linefeed", "masq_lib", "nix 0.23.1", + "num", "regex", + "thousands", "time 0.3.11", "websocket", ] [[package]] name = "masq_lib" -version = "0.6.3" +version = "0.7.0" dependencies = [ "actix", "clap", @@ -2004,13 +2007,13 @@ dependencies = [ [[package]] name = "mortal" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "998fd6a991497275567703b6f435e27958b633878ec991f5734b96dd46675e9f" +checksum = "8d3b281c45a2dbb0609b854de9df94694fb77eab2fa2933c07d07001dcb29377" dependencies = [ "bitflags", "libc", - "nix 0.17.0", + "nix 0.23.1", "smallstr", "terminfo", "unicode-normalization", @@ -2020,7 +2023,7 @@ dependencies = [ [[package]] name = "multinode_integration_tests" -version = "0.6.3" +version = "0.7.0" dependencies = [ "base64 0.13.0", "crossbeam-channel 0.5.1", @@ -2088,33 +2091,32 @@ checksum = "c8d77f3db4bce033f4d04db08079b2ef1c3d02b44e86f25d08886fafa7756ffa" [[package]] name = "nix" -version = "0.17.0" +version = "0.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50e4785f2c3b7589a0d0c1dd60285e1188adac4006e8abd6dd578e1567027363" +checksum = "9f866317acbd3a240710c63f065ffb1e4fd466259045ccb504130b7f668f35c6" dependencies = [ "bitflags", "cc", - "cfg-if 0.1.10", + "cfg-if 1.0.0", "libc", - "void", + "memoffset 0.6.5", ] [[package]] name = "nix" -version = "0.23.1" +version = "0.26.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f866317acbd3a240710c63f065ffb1e4fd466259045ccb504130b7f668f35c6" +checksum = "46a58d1d356c6597d08cde02c2f09d785b09e28711837b1ed667dc652c08a694" dependencies = [ "bitflags", - "cc", "cfg-if 1.0.0", "libc", - "memoffset 0.6.5", + "static_assertions 1.1.0", ] [[package]] name = "node" -version = "0.6.3" +version = "0.7.0" dependencies = [ "actix", "automap", @@ -2169,6 +2171,7 @@ dependencies = [ "sodiumoxide", "sysinfo", "system-configuration", + "thousands", "time 0.3.11", "tiny-bip39", "tiny-hderive", @@ -2203,6 +2206,40 @@ dependencies = [ "winapi 0.3.9", ] +[[package]] +name = "num" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43db66d1170d347f9a065114077f7dccb00c1b9478c89384490a3425279a4606" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f93ab6289c7b344a8a9f60f88d80aa20032336fe78da341afc91c8a2341fc75f" +dependencies = [ + "autocfg 1.0.1", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ae39348c8bc5fbd7f40c727a9925f03517afd2ab27d46702108b6a7e5414c19" +dependencies = [ + "num-traits", +] + [[package]] name = "num-integer" version = "0.1.45" @@ -2213,11 +2250,34 @@ dependencies = [ "num-traits", ] +[[package]] +name = "num-iter" +version = "0.1.43" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d03e6c028c5dc5cac6e2dec0efda81fc887605bb3d884578bb6d6bf7514e252" +dependencies = [ + "autocfg 1.0.1", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0638a1c9d0a3c0914158145bc76cff373a75a627e6ecbfb71cbe6f453a5a19b0" +dependencies = [ + "autocfg 1.0.1", + "num-bigint", + "num-integer", + "num-traits", +] + [[package]] name = "num-traits" -version = "0.2.12" +version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac267bcc07f48ee5f8935ab0d24f316fb722d7a1292e2913f0cc196b29ffd611" +checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd" dependencies = [ "autocfg 1.0.1", ] @@ -3011,29 +3071,28 @@ dependencies = [ [[package]] name = "rusqlite" -version = "0.26.3" +version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ba4d3462c8b2e4d7f4fcfcf2b296dc6b65404fbbc7b63daa37fd485c149daf7" +checksum = "01e213bc3ecb39ac32e81e51ebe31fd888a940515173e3a18a35f8c6e896422a" dependencies = [ "bitflags", "fallible-iterator", "fallible-streaming-iterator", "hashlink", "libsqlite3-sys", - "memchr", "smallvec 1.6.1", ] [[package]] name = "rust-argon2" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9dab61250775933275e84053ac235621dfb739556d5c54a2f2e9313b7cf43a19" +checksum = "4b18820d944b33caa75a71378964ac46f58517c92b6ae5f762636247c09e78fb" dependencies = [ - "base64 0.12.3", + "base64 0.13.0", "blake2b_simd", "constant_time_eq", - "crossbeam-utils 0.7.2", + "crossbeam-utils 0.8.5", ] [[package]] @@ -3244,7 +3303,7 @@ version = "1.0.79" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e8d9fa5c3b304765ce1fd9c4c8a3de2c8db365a5b91be52f186efc675681d95" dependencies = [ - "itoa 1.0.1", + "itoa 1.0.4", "ryu", "serde", ] @@ -3352,9 +3411,9 @@ dependencies = [ [[package]] name = "siphasher" -version = "0.3.5" +version = "0.3.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cbce6d4507c7e4a3962091436e56e95290cb71fa302d0d270e32130b75fbff27" +checksum = "7bd3e3206899af3f8b12af284fafc038cc1dc2b41d1b89dd17297221c5d225de" [[package]] name = "slab" @@ -3605,6 +3664,12 @@ dependencies = [ "syn 1.0.85", ] +[[package]] +name = "thousands" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3bf63baf9f5039dadc247375c29eb13706706cfde997d0330d05aa63a77d8820" + [[package]] name = "time" version = "0.1.44" @@ -3622,7 +3687,7 @@ version = "0.3.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72c91f41dcb2f096c05f0873d667dceec1087ce5bcf984ec8ffb19acddbb3217" dependencies = [ - "itoa 1.0.1", + "itoa 1.0.4", "libc", "num_threads", "time-macros", @@ -4304,12 +4369,6 @@ version = "0.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5a972e5669d67ba988ce3dc826706fb0a8b01471c088cb0b6110b805cc36aed" -[[package]] -name = "void" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a02e4885ed3bc0f2de90ea6dd45ebcbb66dacffe03547fadbb0eeae2770887d" - [[package]] name = "want" version = "0.2.0" @@ -4538,6 +4597,63 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" +[[package]] +name = "windows-sys" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a3e1820f08b8513f676f7ab6c1f99ff312fb97b553d30ff4dd86f9f15728aa7" +dependencies = [ + "windows_aarch64_gnullvm", + "windows_aarch64_msvc", + "windows_i686_gnu", + "windows_i686_msvc", + "windows_x86_64_gnu", + "windows_x86_64_gnullvm", + "windows_x86_64_msvc", +] + +[[package]] +name = "windows_aarch64_gnullvm" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d2aa71f6f0cbe00ae5167d90ef3cfe66527d6f613ca78ac8024c3ccab9a19e" + +[[package]] +name = "windows_aarch64_msvc" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dd0f252f5a35cac83d6311b2e795981f5ee6e67eb1f9a7f64eb4500fbc4dcdb4" + +[[package]] +name = "windows_i686_gnu" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fbeae19f6716841636c28d695375df17562ca208b2b7d0dc47635a50ae6c5de7" + +[[package]] +name = "windows_i686_msvc" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "84c12f65daa39dd2babe6e442988fc329d6243fdce47d7d2d155b8d874862246" + +[[package]] +name = "windows_x86_64_gnu" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf7b1b21b5362cbc318f686150e5bcea75ecedc74dd157d874d754a2ca44b0ed" + +[[package]] +name = "windows_x86_64_gnullvm" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09d525d2ba30eeb3297665bd434a54297e4170c7f1a44cad4ef58095b4cd2028" + +[[package]] +name = "windows_x86_64_msvc" +version = "0.42.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f40009d85759725a34da6d89a94e63d7bdc50a862acf0dbc7c8e488f1edcb6f5" + [[package]] name = "winreg" version = "0.5.1" diff --git a/node/Cargo.toml b/node/Cargo.toml index 541af7cd9..44e752515 100644 --- a/node/Cargo.toml +++ b/node/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "node" -version = "0.6.3" +version = "0.7.0" license = "GPL-3.0-only" authors = ["Dan Wiebe ", "MASQ"] copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved." @@ -37,13 +37,14 @@ libc = "0.2.107" libsecp256k1 = "0.7.0" log = "0.4.14" masq_lib = { path = "../masq_lib"} +paste = "1.0.6" pretty-hex = "0.2.1" primitive-types = {version = "0.5.0", default-features = false, features = ["default", "rlp", "serde"]} rand = {version = "0.8.4", features = ["getrandom", "small_rng"]} regex = "1.5.4" rlp = "0.4.6" rpassword = "5.0.1" -rusqlite = {version = "0.26.1", features = ["bundled"]} +rusqlite = {version = "0.28.0", features = ["bundled","functions"]} rustc-hex = "2.1.0" serde = "1.0.136" serde_derive = "1.0.136" @@ -54,6 +55,7 @@ sodiumoxide = "0.2.2" sysinfo = "0.21.1" tiny-bip39 = "0.8.2" tiny-hderive = "0.3.0" +thousands = "0.2.0" tokio = "0.1.22" tokio-core = "0.1.18" toml = "0.5.8" @@ -63,7 +65,6 @@ unindent = "0.1.7" web3 = {version = "0.11.0", default-features = false, features = ["http", "tls"]} websocket = {version = "0.26.2", default-features = false, features = ["async", "sync"]} secp256k1secrets = {package = "secp256k1", version = "0.17.2"} -paste = "1.0.6" [target.'cfg(target_os = "macos")'.dependencies] system-configuration = "0.4.0" diff --git a/node/src/accountant/big_int_processing/big_int_db_processor.rs b/node/src/accountant/big_int_processing/big_int_db_processor.rs new file mode 100644 index 000000000..2be4dbf19 --- /dev/null +++ b/node/src/accountant/big_int_processing/big_int_db_processor.rs @@ -0,0 +1,1480 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::checked_conversion; +use crate::accountant::payable_dao::PayableDaoError; +use crate::accountant::receivable_dao::ReceivableDaoError; +use crate::database::connection_wrapper::ConnectionWrapper; +use crate::sub_lib::wallet::Wallet; +use itertools::Either; +use rusqlite::{Error, Row, Statement, ToSql, Transaction}; +use std::fmt::{Debug, Display, Formatter}; +use std::iter::once; +use std::marker::PhantomData; +use std::ops::Neg; + +#[derive(Debug)] +pub struct BigIntDbProcessor { + overflow_handler: Box>, +} + +impl<'a, T: TableNameDAO> BigIntDbProcessor { + pub fn execute( + &self, + conn: Either<&dyn ConnectionWrapper, &Transaction>, + config: BigIntSqlConfig<'a, T>, + ) -> Result<(), BigIntDbError> { + let main_sql = config.main_sql; + let mut stm = Self::prepare_statement(conn, main_sql); + let params = config + .params + .merge_pure_rusqlite_and_wei_params((&config.params.wei_change_params).into()); + match stm.execute(params.as_slice()) { + Ok(_) => Ok(()), + //SQLITE_CONSTRAINT_DATATYPE (3091), + //the moment of Sqlite trying to store the number as REAL in a strict INT column + Err(Error::SqliteFailure(e, _)) if e.extended_code == 3091 => { + self.overflow_handler.update_with_overflow(conn, config) + } + Err(e) => Err(BigIntDbError(format!( + "Error from invalid {} command for {} table and change of {} wei to '{} = {}' with error '{}'", + config.determine_command(), + T::table_name(), + config.balance_change(), + config.params.table_unique_key_name, + config.key_param_value(), + e + ))), + } + } +} + +impl Default for BigIntDbProcessor { + fn default() -> BigIntDbProcessor { + Self { + overflow_handler: Box::new(UpdateOverflowHandlerReal::default()), + } + } +} + +impl BigIntDbProcessor { + fn prepare_statement<'a>( + form_of_conn: Either<&'a dyn ConnectionWrapper, &'a Transaction>, + sql: &'a str, + ) -> Statement<'a> { + match form_of_conn { + Either::Left(conn) => conn.prepare(sql), + Either::Right(tx) => tx.prepare(sql), + } + .expect("internal rusqlite error") + } +} + +pub trait UpdateOverflowHandler: Debug + Send { + fn update_with_overflow<'a>( + &self, + conn: Either<&dyn ConnectionWrapper, &Transaction>, + config: BigIntSqlConfig<'a, T>, + ) -> Result<(), BigIntDbError>; +} + +#[derive(Debug)] +struct UpdateOverflowHandlerReal { + phantom: PhantomData, +} + +impl Default for UpdateOverflowHandlerReal { + fn default() -> Self { + Self { + phantom: Default::default(), + } + } +} + +impl UpdateOverflowHandler for UpdateOverflowHandlerReal { + fn update_with_overflow<'a>( + &self, + conn: Either<&dyn ConnectionWrapper, &Transaction>, + config: BigIntSqlConfig<'a, T>, + ) -> Result<(), BigIntDbError> { + let update_divided_integer = |row: &Row| -> Result<(), rusqlite::Error> { + let high_bytes_result = row.get::(0); + let low_bytes_result = row.get::(1); + + match [high_bytes_result, low_bytes_result] { + [Ok(former_high_bytes), Ok(former_low_bytes)] => { + let requested_wei_change = &config.params.wei_change_params; + let (high_bytes_corrected, low_bytes_corrected) = Self::correct_bytes( + former_high_bytes, + former_low_bytes, + requested_wei_change, + ); + let wei_update_array: [(&str, &dyn ToSql); 2] = [ + ( + requested_wei_change.high.name.as_str(), + &high_bytes_corrected, + ), + (requested_wei_change.low.name.as_str(), &low_bytes_corrected), + ]; + + let execute_params = config + .params + .merge_pure_rusqlite_and_wei_params(wei_update_array); + + Self::execute_update(conn, &config, &execute_params); + Ok(()) + } + array_of_results => Self::return_first_error(array_of_results), + } + }; + + let select_sql = config.select_sql(); + let mut select_stm = BigIntDbProcessor::::prepare_statement(conn, &select_sql); + match select_stm.query_row([], update_divided_integer) { + Ok(()) => Ok(()), + Err(e) => Err(BigIntDbError(format!( + "Updating balance for {} table and change of {} wei to '{} = {}' with error '{}'", + T::table_name(), + config.balance_change(), + config.params.table_unique_key_name, + config.key_param_value(), + e + ))), + } + } +} + +impl UpdateOverflowHandlerReal { + fn execute_update<'a>( + conn: Either<&dyn ConnectionWrapper, &Transaction>, + config: &BigIntSqlConfig<'a, T>, + execute_params: &[(&str, &dyn ToSql)], + ) { + match BigIntDbProcessor::::prepare_statement(conn, config.overflow_update_clause) + .execute(execute_params) + .expect("logic broken given the previous non-overflow call accepted right") + { + 1 => (), + x => panic!( + "Broken code: this code was written to handle one changed row a time, not {}", + x + ), + } + } + + fn correct_bytes( + former_high_bytes: i64, + former_low_bytes: i64, + requested_wei_change: &WeiChangeAsHighAndLowBytes, + ) -> (i64, i64) { + let high_bytes_correction = former_high_bytes + requested_wei_change.high.value + 1; + let low_bytes_correction = ((former_low_bytes as i128 + + requested_wei_change.low.value as i128) + & 0x7FFFFFFFFFFFFFFF) as i64; + (high_bytes_correction, low_bytes_correction) + } + + fn return_first_error(two_results: [rusqlite::Result; 2]) -> rusqlite::Result<()> { + let cached = format!("{:?}", two_results); + match two_results.into_iter().find(|result| result.is_err()) { + Some(err) => Err(err.expect_err("we just said it is an error")), + None => panic!( + "Broken code: being called to process an error but none was found in {}", + cached + ), + } + } +} + +pub struct BigIntSqlConfig<'a, T> { + main_sql: &'a str, + overflow_update_clause: &'a str, + pub params: SQLParams<'a>, + phantom: PhantomData, +} + +impl<'a, T: TableNameDAO> BigIntSqlConfig<'a, T> { + pub fn new( + main_sql: &'a str, + overflow_update_clause: &'a str, + params: SQLParams<'a>, + ) -> BigIntSqlConfig<'a, T> { + Self { + main_sql, + overflow_update_clause, + params, + phantom: Default::default(), + } + } + + fn select_sql(&self) -> String { + format!( + "select {}, {} from {} where {} = '{}'", + &self.params.wei_change_params.high.name[1..], + &self.params.wei_change_params.low.name[1..], + T::table_name(), + self.params.table_unique_key_name, + self.key_param_value() + ) + } + + fn key_param_value(&self) -> &'a dyn ExtendedParamsMarker { + self.params.params_except_wei_change[0].1 + } + + fn balance_change(&self) -> i128 { + let wei_params = &self.params.wei_change_params; + BigIntDivider::reconstitute(wei_params.high.value, wei_params.low.value) + } + + fn determine_command(&self) -> String { + let keyword = self + .main_sql + .chars() + .skip_while(|char| char.is_whitespace()) + .take_while(|char| !char.is_whitespace()) + .collect::(); + match keyword.as_str() { + "insert" => { + if self.main_sql.contains("update") { + "upsert".to_string() + } else { + panic!("Sql with simple insert. The processor of big integers is correctly used only if combined with update") + } + } + "update" => keyword, + _ => panic!( + "broken code: unexpected or misplaced command \"{}\" \ + in upsert or update, respectively", + keyword + ), + } + } +} + +//to be able to display things that implement ToSql +pub trait ExtendedParamsMarker: ToSql + Display {} + +macro_rules! impl_of_extended_params_marker{ + ($($implementer: ty),+) => { + $(impl ExtendedParamsMarker for $implementer {})+ + } +} + +impl_of_extended_params_marker!(i64, &str, Wallet); + +#[derive(Default)] +pub struct SQLParamsBuilder<'a> { + key_spec_opt: Option>, + wei_change_spec_opt: Option, + other_params: Vec<(&'a str, &'a dyn ExtendedParamsMarker)>, +} + +impl<'a> SQLParamsBuilder<'a> { + pub fn key(mut self, key_variant: KnownKeyVariants<'a>) -> Self { + let (definition_name, substitution_name_in_sql, value_itself) = key_variant.into(); + self.key_spec_opt = Some(UniqueKeySpec { + definition_name, + substitution_name_in_sql, + value_itself, + }); + self + } + + pub fn wei_change(mut self, wei_change: WeiChange) -> Self { + self.wei_change_spec_opt = Some(wei_change); + self + } + + pub fn other(mut self, params: Vec<(&'a str, &'a (dyn ExtendedParamsMarker + 'a))>) -> Self { + self.other_params = params; + self + } + + pub fn build(self) -> SQLParams<'a> { + let key_spec = self + .key_spec_opt + .unwrap_or_else(|| panic!("SQLparams cannot miss the component of a key")); + let wei_change_spec = self + .wei_change_spec_opt + .unwrap_or_else(|| panic!("SQLparams cannot miss the component of wei change")); + let ((high_bytes_param_name, low_bytes_param_name), (high_bytes_value, low_bytes_value)) = + Self::expand_wei_params(wei_change_spec); + let key_as_the_first_param = (key_spec.substitution_name_in_sql, key_spec.value_itself); + let params = once(key_as_the_first_param) + .chain(self.other_params.into_iter()) + .collect(); + SQLParams { + table_unique_key_name: key_spec.definition_name, + wei_change_params: WeiChangeAsHighAndLowBytes { + high: StdNumParamFormNamed::new(high_bytes_param_name, high_bytes_value), + low: StdNumParamFormNamed::new(low_bytes_param_name, low_bytes_value), + }, + params_except_wei_change: params, + } + } + + fn expand_wei_params(wei_change_spec: WeiChange) -> ((String, String), (i64, i64)) { + let (name, num): (&'static str, i128) = match wei_change_spec { + WeiChange::Addition(name, num) => (name, checked_conversion::(num)), + WeiChange::Subtraction(name, num) => { + (name, checked_conversion::(num).neg()) + } + }; + let (high_bytes, low_bytes) = BigIntDivider::deconstruct(num); + let param_sub_name_for_high_bytes = + Self::proper_wei_change_param_name(name, ByteMagnitude::High); + let param_sub_name_for_low_bytes = + Self::proper_wei_change_param_name(name, ByteMagnitude::Low); + ( + (param_sub_name_for_high_bytes, param_sub_name_for_low_bytes), + (high_bytes, low_bytes), + ) + } + + fn proper_wei_change_param_name(base_word: &str, byte_magnitude: ByteMagnitude) -> String { + format!(":{}_{}_b", base_word, byte_magnitude) + } +} + +struct UniqueKeySpec<'a> { + definition_name: &'a str, + substitution_name_in_sql: &'a str, + value_itself: &'a dyn ExtendedParamsMarker, +} + +pub enum KnownKeyVariants<'a> { + WalletAddress(&'a dyn ExtendedParamsMarker), + PendingPayableRowid(&'a dyn ExtendedParamsMarker), + #[cfg(test)] + TestKey { + var_name: &'static str, + sub_name: &'static str, + val: &'a dyn ExtendedParamsMarker, + }, +} + +impl<'a> From> for (&'static str, &'static str, &'a dyn ExtendedParamsMarker) { + fn from(variant: KnownKeyVariants<'a>) -> Self { + match variant { + KnownKeyVariants::WalletAddress(val) => ("wallet_address", ":wallet", val), + KnownKeyVariants::PendingPayableRowid(val) => ("pending_payable_rowid", ":rowid", val), + #[cfg(test)] + KnownKeyVariants::TestKey { + var_name, + sub_name, + val, + } => (var_name, sub_name, val), + } + } +} + +enum ByteMagnitude { + High, + Low, +} + +impl Display for ByteMagnitude { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + match self { + Self::High => write!(f, "high"), + Self::Low => write!(f, "low"), + } + } +} + +pub struct SQLParams<'a> { + table_unique_key_name: &'a str, + wei_change_params: WeiChangeAsHighAndLowBytes, + params_except_wei_change: Vec<(&'a str, &'a dyn ExtendedParamsMarker)>, +} + +#[derive(Debug, PartialEq)] +struct WeiChangeAsHighAndLowBytes { + high: StdNumParamFormNamed, + low: StdNumParamFormNamed, +} + +#[derive(Debug, PartialEq)] +struct StdNumParamFormNamed { + name: String, + value: i64, +} + +impl StdNumParamFormNamed { + fn new(name: String, value: i64) -> Self { + Self { name, value } + } +} + +impl<'a> From<&'a WeiChangeAsHighAndLowBytes> for [(&'a str, &'a dyn ToSql); 2] { + fn from(wei_change: &'a WeiChangeAsHighAndLowBytes) -> Self { + [ + (wei_change.high.name.as_str(), &wei_change.high.value), + (wei_change.low.name.as_str(), &wei_change.low.value), + ] + } +} + +impl<'a> SQLParams<'a> { + fn merge_pure_rusqlite_and_wei_params( + &'a self, + wei_change_params: [(&'a str, &'a dyn ToSql); 2], + ) -> Vec<(&'a str, &'a dyn ToSql)> { + self.pure_rusqlite_params(wei_change_params.into_iter()) + .collect() + } + + fn pure_rusqlite_params( + &'a self, + wei_change_params: impl Iterator, + ) -> impl Iterator { + self.params_except_wei_change + .iter() + .map(|(name, value)| (*name, value as &dyn ToSql)) + .chain(wei_change_params) + } +} + +pub trait TableNameDAO: Debug + Send { + fn table_name() -> String; +} + +#[derive(Debug, PartialEq, Eq, Clone)] +pub enum WeiChange { + Addition(&'static str, u128), + Subtraction(&'static str, u128), +} + +#[derive(Debug, PartialEq, Eq)] +pub struct BigIntDbError(pub String); + +macro_rules! big_int_db_error_from { + ($($implementer: ident),+) => { + $(impl From for $implementer { + fn from(iu_err: BigIntDbError) -> Self { + $implementer::RusqliteError(iu_err.0) + } + })+ + } +} + +big_int_db_error_from!(PayableDaoError, ReceivableDaoError); + +#[cfg(test)] +mod tests { + use super::*; + use crate::accountant::big_int_processing::big_int_db_processor::ByteMagnitude::{High, Low}; + use crate::accountant::big_int_processing::big_int_db_processor::KnownKeyVariants::TestKey; + use crate::accountant::big_int_processing::big_int_db_processor::WeiChange::{ + Addition, Subtraction, + }; + use crate::accountant::big_int_processing::test_utils::restricted::{ + create_new_empty_db, test_database_key, + }; + use crate::database::connection_wrapper::{ConnectionWrapper, ConnectionWrapperReal}; + use crate::test_utils::make_wallet; + use itertools::Either; + use itertools::Either::Left; + use rusqlite::{Connection, ToSql}; + use std::cell::RefCell; + use std::sync::{Arc, Mutex}; + + #[derive(Debug)] + struct DummyDao {} + + impl TableNameDAO for DummyDao { + fn table_name() -> String { + String::from("test_table") + } + } + + #[test] + fn conversion_from_local_error_to_particular_payable_dao_error_works() { + let subject = BigIntDbError(String::from("whatever")); + + let result: PayableDaoError = subject.into(); + + assert_eq!( + result, + PayableDaoError::RusqliteError("whatever".to_string()) + ) + } + + #[test] + fn conversion_from_local_error_to_particular_receivable_dao_error_works() { + let subject = BigIntDbError(String::from("whatever")); + + let result: ReceivableDaoError = subject.into(); + + assert_eq!( + result, + ReceivableDaoError::RusqliteError("whatever".to_string()) + ) + } + + #[test] + fn display_for_byte_magnitude_works() { + assert_eq!(High.to_string(), "high".to_string()); + assert_eq!(Low.to_string(), "low".to_string()) + } + + #[test] + fn known_key_variants_to_tuple_of_names_and_val_works() { + let (wallet_name, wallet_sub_name, wallet_val): (&str, &str, &dyn ExtendedParamsMarker) = + KnownKeyVariants::WalletAddress(&"blah").into(); + let (rowid_name, rowid_sub_name, rowid_val): (&str, &str, &dyn ExtendedParamsMarker) = + KnownKeyVariants::PendingPayableRowid(&123).into(); + + assert_eq!(wallet_name, "wallet_address"); + assert_eq!(wallet_sub_name, ":wallet"); + assert_eq!(wallet_val.to_string(), "blah".to_string()); + assert_eq!(rowid_name, "pending_payable_rowid"); + assert_eq!(rowid_sub_name, ":rowid"); + assert_eq!(rowid_val.to_string(), 123.to_string()) + //cannot be compared by values directly + } + + #[test] + fn sql_params_builder_is_nicely_populated_inside_before_calling_build() { + let subject = SQLParamsBuilder::default(); + + let result = subject + .wei_change(Addition("balance", 4546)) + .key(TestKey { + var_name: "some_key", + sub_name: ":some_key", + val: &"blah", + }) + .other(vec![("other_thing", &46565)]); + + assert_eq!(result.wei_change_spec_opt, Some(Addition("balance", 4546))); + let key_spec = result.key_spec_opt.unwrap(); + assert_eq!(key_spec.definition_name, "some_key"); + assert_eq!(key_spec.substitution_name_in_sql, ":some_key"); + assert_eq!(key_spec.value_itself.to_string(), "blah".to_string()); + assert!(matches!(result.other_params[0], ("other_thing", _))); + assert_eq!(result.other_params.len(), 1) + } + + #[test] + fn sql_params_builder_builds_correct_params() { + let subject = SQLParamsBuilder::default(); + + let result = subject + .wei_change(Addition("balance", 115898)) + .key(TestKey { + var_name: "some_key", + sub_name: ":some_key", + val: &"blah", + }) + .other(vec![(":other_thing", &11111)]) + .build(); + + assert_eq!(result.table_unique_key_name, "some_key"); + assert_eq!( + result.wei_change_params, + WeiChangeAsHighAndLowBytes { + high: StdNumParamFormNamed::new(":balance_high_b".to_string(), 0), + low: StdNumParamFormNamed::new(":balance_low_b".to_string(), 115898) + } + ); + assert_eq!(result.params_except_wei_change[0].0, ":some_key"); + assert_eq!( + result.params_except_wei_change[0].1.to_string(), + "blah".to_string() + ); + assert_eq!(result.params_except_wei_change[1].0, ":other_thing"); + assert_eq!( + result.params_except_wei_change[1].1.to_string(), + "11111".to_string() + ); + assert_eq!(result.params_except_wei_change.len(), 2) + } + + #[test] + fn sql_params_builder_builds_correct_params_with_negative_wei_change() { + let subject = SQLParamsBuilder::default(); + + let result = subject + .wei_change(Subtraction("balance", 454684)) + .key(TestKey { + var_name: "some_key", + sub_name: ":some_key", + val: &"wooow", + }) + .other(vec![(":other_thing", &46565)]) + .build(); + + assert_eq!(result.table_unique_key_name, "some_key"); + assert_eq!( + result.wei_change_params, + WeiChangeAsHighAndLowBytes { + high: StdNumParamFormNamed::new(":balance_high_b".to_string(), -1), + low: StdNumParamFormNamed::new(":balance_low_b".to_string(), 9223372036854321124) + } + ); + assert_eq!(result.params_except_wei_change[0].0, ":some_key"); + assert_eq!( + result.params_except_wei_change[0].1.to_string(), + "wooow".to_string() + ); + assert_eq!(result.params_except_wei_change[1].0, ":other_thing"); + assert_eq!( + result.params_except_wei_change[1].1.to_string(), + "46565".to_string() + ); + assert_eq!(result.params_except_wei_change.len(), 2) + } + + #[test] + #[should_panic(expected = "SQLparams cannot miss the component of a key")] + fn sql_params_builder_cannot_be_built_without_key_spec() { + let subject = SQLParamsBuilder::default(); + + let _ = subject + .wei_change(Addition("balance", 4546)) + .other(vec![("laughter", &"hahaha")]) + .build(); + } + + #[test] + #[should_panic(expected = "SQLparams cannot miss the component of wei change")] + fn sql_params_builder_cannot_be_built_without_wei_change_spec() { + let subject = SQLParamsBuilder::default(); + + let _ = subject + .key(TestKey { + var_name: "wallet", + sub_name: ":wallet", + val: &make_wallet("wallet"), + }) + .other(vec![("other_thing", &46565)]) + .build(); + } + + #[test] + fn sql_params_builder_can_be_built_without_other_params_present() { + let subject = SQLParamsBuilder::default(); + + let _ = subject + .wei_change(Addition("balance", 4546)) + .key(TestKey { + var_name: "id", + sub_name: ":id", + val: &45, + }) + .build(); + } + + #[test] + fn return_first_error_works_for_first_error() { + let results = [Err(Error::GetAuxWrongType), Ok(45465)]; + + let err = UpdateOverflowHandlerReal::::return_first_error(results); + + assert_eq!(err, Err(Error::GetAuxWrongType)) + } + + #[test] + fn return_first_error_works_for_second_error() { + let results = [Ok(45465), Err(Error::QueryReturnedNoRows)]; + + let err = UpdateOverflowHandlerReal::::return_first_error(results); + + assert_eq!(err, Err(Error::QueryReturnedNoRows)) + } + + #[test] + #[should_panic( + expected = "Broken code: being called to process an error but none was found in [Ok(-45465), Ok(898)]" + )] + fn return_first_error_needs_some_error() { + let results = [Ok(-45465), Ok(898)]; + + let err = UpdateOverflowHandlerReal::::return_first_error(results); + + assert_eq!(err, Err(Error::QueryReturnedNoRows)) + } + + fn make_empty_sql_params<'a>() -> SQLParams<'a> { + SQLParams { + table_unique_key_name: "", + wei_change_params: WeiChangeAsHighAndLowBytes { + high: StdNumParamFormNamed::new("".to_string(), 0), + low: StdNumParamFormNamed::new("".to_string(), 0), + }, + params_except_wei_change: vec![], + } + } + + #[test] + fn determine_command_works_for_upsert() { + let subject: BigIntSqlConfig<'_, DummyDao> = BigIntSqlConfig { + main_sql: + "insert into table (a,b) values ('a','b') on conflict (rowid) do update set etc.", + overflow_update_clause: "side clause", + params: make_empty_sql_params(), + phantom: Default::default(), + }; + + let result = subject.determine_command(); + + assert_eq!(result, "upsert".to_string()) + } + + #[test] + fn determine_command_works_for_update() { + let subject: BigIntSqlConfig<'_, DummyDao> = BigIntSqlConfig { + main_sql: "update table set a='a',b='b' where a = 'e'", + overflow_update_clause: "update with overflow sql", + params: make_empty_sql_params(), + phantom: Default::default(), + }; + + let result = subject.determine_command(); + + assert_eq!(result, "update".to_string()) + } + + #[test] + #[should_panic( + expected = "Sql with simple insert. The processor of big integers is correctly used only if combined with update" + )] + fn determine_command_does_not_now_simple_insert() { + let subject: BigIntSqlConfig<'_, DummyDao> = BigIntSqlConfig { + main_sql: "insert into table (blah) values ('double blah')", + overflow_update_clause: "update with overflow sql", + params: make_empty_sql_params(), + phantom: Default::default(), + }; + + let _ = subject.determine_command(); + } + + #[test] + #[should_panic( + expected = "broken code: unexpected or misplaced command \"some\" in upsert or update, respectively" + )] + fn determine_command_panics_if_unknown_command() { + let subject: BigIntSqlConfig<'_, DummyDao> = BigIntSqlConfig { + main_sql: "some other sql command", + overflow_update_clause: "", + params: make_empty_sql_params(), + phantom: Default::default(), + }; + + let _ = subject.determine_command(); + } + + #[test] + fn determine_command_allows_preceding_spaces() { + let subject: BigIntSqlConfig<'_, DummyDao> = BigIntSqlConfig { + main_sql: " update into table (a,b) values ('a','b')", + overflow_update_clause: "", + params: make_empty_sql_params(), + phantom: Default::default(), + }; + + let result = subject.determine_command(); + + assert_eq!(result, "update".to_string()) + } + + fn insert_single_record(conn: &dyn ConnectionWrapper, params: [&dyn ToSql; 3]) { + conn.prepare( + "insert into test_table (name,balance_high_b, balance_low_b) values (?, ?, ?)", + ) + .unwrap() + .execute(params.as_slice()) + .unwrap(); + } + + #[derive(Debug, Default)] + struct UpdateOverflowHandlerMock { + update_with_overflow_params: Arc>>, + update_with_overflow_results: RefCell>>, + } + + impl UpdateOverflowHandler for UpdateOverflowHandlerMock { + fn update_with_overflow<'a>( + &self, + _conn: Either<&dyn ConnectionWrapper, &Transaction>, + _config: BigIntSqlConfig<'a, T>, + ) -> Result<(), BigIntDbError> { + self.update_with_overflow_params.lock().unwrap().push(()); + self.update_with_overflow_results.borrow_mut().remove(0) + } + } + + impl UpdateOverflowHandlerMock { + fn update_with_overflow_params(mut self, params: &Arc>>) -> Self { + self.update_with_overflow_params = params.clone(); + self + } + + fn update_with_overflow_result(self, result: Result<(), BigIntDbError>) -> Self { + self.update_with_overflow_results.borrow_mut().push(result); + self + } + } + + #[derive(Debug, PartialEq)] + struct ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: bool, + final_database_values: ReadFinalRow, + } + + #[derive(Debug, PartialEq)] + struct ReadFinalRow { + high_bytes: i64, + low_bytes: i64, + as_i128: i128, + } + + fn analyse_sql_commands_execution_without_details_of_overflow( + test_name: &str, + main_sql: &str, + requested_wei_change: WeiChange, + init_record: i128, + ) -> ConventionalUpsertUpdateAnalysisData { + let update_with_overflow_params_arc = Arc::new(Mutex::new(vec![])); + let overflow_handler = UpdateOverflowHandlerMock::default() + .update_with_overflow_params(&update_with_overflow_params_arc) + .update_with_overflow_result(Ok(())); + let mut subject = BigIntDbProcessor::::default(); + subject.overflow_handler = Box::new(overflow_handler); + + let act = |conn: &mut dyn ConnectionWrapper| { + subject.execute( + Left(conn), + BigIntSqlConfig::new( + main_sql, + "", + SQLParamsBuilder::default() + .key(test_database_key(&"Joe")) + .wei_change(requested_wei_change.clone()) + .build(), + ), + ) + }; + + precise_upsert_or_update_assertion_test_environment( + test_name, + init_record, + act, + update_with_overflow_params_arc, + ) + } + + fn precise_upsert_or_update_assertion_test_environment( + test_name: &str, + init_record: i128, + act: F, + update_with_overflow_params_arc: Arc>>, + ) -> ConventionalUpsertUpdateAnalysisData + where + F: Fn(&mut dyn ConnectionWrapper) -> Result<(), BigIntDbError>, + { + let mut conn = initiate_simple_connection_and_test_table("big_int_db_processor", test_name); + if init_record != 0 { + let (init_high, init_low) = BigIntDivider::deconstruct(init_record); + insert_single_record(conn.as_ref(), [&"Joe", &init_high, &init_low]) + }; + + let result = act(conn.as_mut()); + + assert_eq!(result, Ok(())); + let update_with_overflow_params = update_with_overflow_params_arc.lock().unwrap(); + let was_update_with_overflow = !update_with_overflow_params.is_empty(); + assert_on_whole_row(was_update_with_overflow, &*conn, "Joe") + } + + fn assert_on_whole_row( + was_update_with_overflow: bool, + conn: &dyn ConnectionWrapper, + expected_name: &str, + ) -> ConventionalUpsertUpdateAnalysisData { + let final_database_values = conn + .prepare("select name, balance_high_b, balance_low_b from test_table") + .unwrap() + .query_row([], |row| { + let name = row.get::(0).unwrap(); + assert_eq!(name, expected_name.to_string()); + let high_bytes = row.get::(1).unwrap(); + let low_bytes = row.get::(2).unwrap(); + let single_numbered_balance = BigIntDivider::reconstitute(high_bytes, low_bytes); + Ok(ReadFinalRow { + high_bytes, + low_bytes, + as_i128: single_numbered_balance, + }) + }) + .unwrap(); + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow, + final_database_values, + } + } + + fn create_test_table(conn: &Connection) { + conn.execute( + "create table test_table (name text primary key, balance_high_b integer not null, balance_low_b integer not null) strict", + [], + ) + .unwrap(); + } + + fn initiate_simple_connection_and_test_table( + module: &str, + test_name: &str, + ) -> Box { + let conn = create_new_empty_db(module, test_name); + create_test_table(&conn); + Box::new(ConnectionWrapperReal::new(conn)) + } + + const STANDARD_EXAMPLE_OF_UPDATE_CLAUSE: &str = "update test_table set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b where name = :name"; + const STANDARD_EXAMPLE_OF_INSERT_CLAUSE: &str = "insert into test_table (name, balance_high_b, balance_low_b) values (:name, :balance_high_b, :balance_low_b)"; + const STANDARD_EXAMPLE_OF_INSERT_WITH_CONFLICT_CLAUSE: &str = "insert into test_table (name, balance_high_b, balance_low_b) values (:name, :balance_high_b, :balance_low_b) on conflict (name) do update set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b where name = :name"; + const STANDARD_EXAMPLE_OF_OVERFLOW_UPDATE_CLAUSE: &str = "update test_table set balance_high_b = :balance_high_b, balance_low_b = :balance_low_b where name = :name"; + + #[test] + fn update_alone_works_for_addition() { + let initial = BigIntDivider::reconstitute(55, 1234567); + let wei_change = BigIntDivider::reconstitute(1, 22222); + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "update_alone_works_for_addition", + STANDARD_EXAMPLE_OF_UPDATE_CLAUSE, + Addition("balance", wei_change as u128), + initial, + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: 56, + low_bytes: 1256789, + as_i128: initial + wei_change + } + } + ) + } + + #[test] + fn update_alone_works_for_addition_with_overflow() { + let initial = BigIntDivider::reconstitute(55, i64::MAX - 5); + let wei_change = BigIntDivider::reconstitute(1, 6); + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "update_alone_works_for_addition_with_overflow", + STANDARD_EXAMPLE_OF_UPDATE_CLAUSE, + Addition("balance", wei_change as u128), + initial, + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: true, + //overflow halts the update machinery within this specific test, no numeric change + final_database_values: ReadFinalRow { + high_bytes: 55, + low_bytes: i64::MAX - 5, + as_i128: initial + } + } + ) + } + + #[test] + fn update_alone_works_for_subtraction() { + let initial = BigIntDivider::reconstitute(55, i64::MAX - 5); + let wei_change = -(i64::MAX - 3) as i128; + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "update_alone_works_for_subtraction", + STANDARD_EXAMPLE_OF_UPDATE_CLAUSE, + Subtraction("balance", wei_change.abs() as u128), + initial, + ); + + assert_eq!(BigIntDivider::deconstruct(wei_change), (-1, 4)); + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: 54, + low_bytes: i64::MAX - 1, + as_i128: initial - (-wei_change) + } + } + ) + } + + #[test] + fn update_alone_works_for_subtraction_with_overflow() { + let initial = BigIntDivider::reconstitute(55, 4588288282); + let wei_change: i128 = -12; + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "update_alone_works_for_subtraction_with_overflow", + STANDARD_EXAMPLE_OF_UPDATE_CLAUSE, + Subtraction("balance", wei_change.abs() as u128), + initial, + ); + + assert_eq!( + BigIntDivider::deconstruct(wei_change), + (-1, 9223372036854775796) + ); + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: true, + //overflow halts the update machinery within this specific test, no numeric change + final_database_values: ReadFinalRow { + high_bytes: 55, + low_bytes: 4588288282, + as_i128: initial + } + } + ) + } + + #[test] + fn early_return_for_successful_insert_works_for_addition() { + let initial = BigIntDivider::reconstitute(0, 0); + let wei_change = BigIntDivider::reconstitute(845, 7788); + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "early_return_for_successful_insert_works", + STANDARD_EXAMPLE_OF_INSERT_CLAUSE, + Addition("balance", wei_change as u128), + initial, + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: 845, + low_bytes: 7788, + as_i128: wei_change + } + } + ) + } + + #[test] + fn early_return_for_successful_insert_works_for_subtraction() { + let initial = BigIntDivider::reconstitute(0, 0); + let wei_change: i128 = -987654; + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "early_return_for_successful_insert_works_for_subtraction", + STANDARD_EXAMPLE_OF_INSERT_CLAUSE, + Subtraction("balance", wei_change.abs() as u128), + initial, + ); + + assert_eq!( + BigIntDivider::deconstruct(wei_change), + (-1, 9223372036853788154) + ); + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: -1, + low_bytes: 9223372036853788154, + as_i128: wei_change + } + } + ) + } + + #[test] + fn insert_blocked_simple_update_succeeds_for_addition() { + let initial = BigIntDivider::reconstitute(-50, 20); + let wei_change = BigIntDivider::reconstitute(3, 4); + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "insert_blocked_simple_update_succeeds_for_addition", + STANDARD_EXAMPLE_OF_INSERT_WITH_CONFLICT_CLAUSE, + Addition("balance", wei_change as u128), + initial, + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: -47, + low_bytes: 24, + as_i128: initial + wei_change + } + } + ) + } + + #[test] + fn insert_blocked_simple_update_succeeds_for_subtraction() { + let initial = BigIntDivider::reconstitute(-50, 20); + let wei_change: i128 = -27670116110564327418; + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "insert_blocked_simple_update_succeeds_for_subtraction", + STANDARD_EXAMPLE_OF_INSERT_WITH_CONFLICT_CLAUSE, + Subtraction("balance", wei_change.abs() as u128), + initial, + ); + + assert_eq!(BigIntDivider::deconstruct(wei_change), (-3, 6)); + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: -53, + low_bytes: 26, + as_i128: initial - (-wei_change) + } + } + ) + } + + #[test] + fn insert_blocked_update_with_overflow_for_addition() { + let initial = BigIntDivider::reconstitute(-50, 20); + let wei_change = BigIntDivider::reconstitute(8, i64::MAX - 19); + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "insert_blocked_update_with_overflow_for_addition", + STANDARD_EXAMPLE_OF_INSERT_WITH_CONFLICT_CLAUSE, + Addition("balance", wei_change as u128), + initial, + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: true, + //overflow halts the update machinery within this specific test, no numeric change + final_database_values: ReadFinalRow { + high_bytes: -50, + low_bytes: 20, + as_i128: initial + } + } + ) + } + + #[test] + fn insert_blocked_update_with_overflow_for_subtraction() { + let initial = BigIntDivider::reconstitute(-44, 11); + let wei_change: i128 = -7; + + let result = analyse_sql_commands_execution_without_details_of_overflow( + "insert_blocked_update_with_overflow_for_subtraction", + STANDARD_EXAMPLE_OF_INSERT_WITH_CONFLICT_CLAUSE, + Subtraction("balance", wei_change.abs() as u128), + initial, + ); + + assert_eq!( + BigIntDivider::deconstruct(wei_change), + (-1, 9223372036854775801) + ); + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: true, + //overflow halts the update machinery within this specific test, no numeric change + final_database_values: ReadFinalRow { + high_bytes: -44, + low_bytes: 11, + as_i128: initial + } + } + ); + } + + #[test] + fn update_alone_works_also_for_transaction_instead_of_connection() { + let initial = BigIntDivider::reconstitute(10, 20); + let wei_change = BigIntDivider::reconstitute(0, 30); + let subject = BigIntDbProcessor::::default(); + let act = |conn: &mut dyn ConnectionWrapper| { + let tx = conn.transaction().unwrap(); + let result = subject.execute( + Either::Right(&tx), + BigIntSqlConfig::new( + STANDARD_EXAMPLE_OF_UPDATE_CLAUSE, + "", + SQLParamsBuilder::default() + .key(test_database_key(&"Joe")) + .wei_change(Addition("balance", wei_change as u128)) + .build(), + ), + ); + tx.commit().unwrap(); + result + }; + let result = precise_upsert_or_update_assertion_test_environment( + "update_alone_works_also_for_transaction_instead_of_connection", + initial, + act, + Arc::new(Mutex::new(vec![])), + ); + + assert_eq!( + result, + ConventionalUpsertUpdateAnalysisData { + was_update_with_overflow: false, + final_database_values: ReadFinalRow { + high_bytes: 10, + low_bytes: 50, + as_i128: initial + wei_change + } + } + ) + } + + #[test] + fn main_sql_clause_error_handled() { + let conn = initiate_simple_connection_and_test_table( + "big_int_db_processor", + "main_sql_clause_error_handled", + ); + let subject = BigIntDbProcessor::::default(); + let balance_change = Addition("balance", 4879898145125); + let config = BigIntSqlConfig::new( + "insert into test_table (name, balance_high_b, balance_low_b) values (:name, :balance_wrong_a, :balance_wrong_b) on conflict (name) do \ + update set balance_high_b = balance_high_b + 5, balance_low_b = balance_low_b + 10 where name = :name", + "", + SQLParamsBuilder::default() + .key(test_database_key(&"Joe")) + .wei_change(balance_change) + .build(), + ); + + let result = subject.execute(Left(conn.as_ref()), config); + + assert_eq!( + result, + Err(BigIntDbError( + "Error from invalid upsert command for test_table table and change of 4879898145125 \ + wei to 'name = Joe' with error 'Invalid parameter name: :balance_high_b'" + .to_string() + )) + ); + } + + fn update_with_overflow_shared_test_body( + test_name: &str, + init_big_initial: i128, + balance_change: WeiChange, + ) -> (i64, i64) { + let conn = initiate_simple_connection_and_test_table("big_int_db_processor", test_name); + let (init_high_bytes, init_low_bytes) = BigIntDivider::deconstruct(init_big_initial); + insert_single_record(&*conn, [&"Joe", &init_high_bytes, &init_low_bytes]); + let update_config = BigIntSqlConfig::new( + "", + STANDARD_EXAMPLE_OF_OVERFLOW_UPDATE_CLAUSE, + SQLParamsBuilder::default() + .wei_change(balance_change) + .key(test_database_key(&"Joe")) + .build(), + ); + + let result = BigIntDbProcessor::::default() + .overflow_handler + .update_with_overflow(Left(&*conn), update_config); + + assert_eq!(result, Ok(())); + let (final_high_bytes, final_low_bytes) = conn + .prepare("select balance_high_b, balance_low_b from test_table where name = 'Joe'") + .unwrap() + .query_row([], |row| { + let high_bytes = row.get::(0).unwrap(); + let low_bytes = row.get::(1).unwrap(); + Ok((high_bytes, low_bytes)) + }) + .unwrap(); + (final_high_bytes, final_low_bytes) + } + + #[test] + fn update_with_overflow_for_addition() { + let big_initial = i64::MAX as i128 * 3; + let big_addend = i64::MAX as i128 + 454; + let big_sum = big_initial + big_addend; + + let (final_high_bytes, final_low_bytes) = update_with_overflow_shared_test_body( + "update_with_overflow_for_addition", + big_initial, + Addition("balance", big_addend as u128), + ); + + assert_eq!( + BigIntDivider::deconstruct(big_initial), + (2, 9223372036854775805) + ); + assert_eq!(BigIntDivider::deconstruct(big_addend), (1, 453)); + let result = BigIntDivider::reconstitute(final_high_bytes, final_low_bytes); + assert_eq!(result, big_sum) + } + + #[test] + fn update_with_overflow_for_subtraction_from_positive_num() { + let big_initial = i64::MAX as i128 * 2; + let big_subtrahend = i64::MAX as i128 + 120; + let big_sum = big_initial - big_subtrahend; + + let (final_high_bytes, final_low_bytes) = update_with_overflow_shared_test_body( + "update_with_overflow_for_subtraction_from_positive_num", + big_initial, + Subtraction("balance", big_subtrahend as u128), + ); + + assert_eq!( + BigIntDivider::deconstruct(big_initial), + (1, 9223372036854775806) + ); + assert_eq!( + BigIntDivider::deconstruct(-big_subtrahend), + (-2, 9223372036854775689) + ); + let result = BigIntDivider::reconstitute(final_high_bytes, final_low_bytes); + assert_eq!(result, big_sum) + } + + #[test] + fn update_with_overflow_for_subtraction_from_negative_num() { + let big_initial = i64::MAX as i128 * 3 + 200; + let big_subtrahend = i64::MAX as i128 + 120; + let big_sum = -big_initial - big_subtrahend; + + let (final_high_bytes, final_low_bytes) = update_with_overflow_shared_test_body( + "update_with_overflow_for_subtraction_from_negative_num", + -big_initial, + Subtraction("balance", big_subtrahend as u128), + ); + + assert_eq!( + BigIntDivider::deconstruct(-big_initial), + (-4, 9223372036854775611) + ); + assert_eq!( + BigIntDivider::deconstruct(-big_subtrahend), + (-2, 9223372036854775689) + ); + let result = BigIntDivider::reconstitute(final_high_bytes, final_low_bytes); + assert_eq!(result, big_sum) + } + + #[test] + fn update_with_overflow_handles_unspecific_error() { + let conn = initiate_simple_connection_and_test_table( + "big_int_db_processor", + "update_with_overflow_handles_unspecific_error", + ); + let balance_change = Addition("balance", 100); + let update_config = BigIntSqlConfig::new( + "this can be whatever because the test fails earlier on the select stm", + STANDARD_EXAMPLE_OF_OVERFLOW_UPDATE_CLAUSE, + SQLParamsBuilder::default() + .wei_change(balance_change) + .key(test_database_key(&"Joe")) + .build(), + ); + + let result = BigIntDbProcessor::::default() + .overflow_handler + .update_with_overflow(Left(conn.as_ref()), update_config); + + //this kind of error is impossible in the real use case but is easiest regarding an arrangement of the test + assert_eq!( + result, + Err(BigIntDbError( + "Updating balance for test_table table and change of 100 wei to 'name = Joe' with \ + error 'Query returned no rows'" + .to_string() + )) + ); + } + + #[test] + #[should_panic( + expected = "Broken code: this code was written to handle one changed row a time, not 2" + )] + fn update_with_overflow_is_designed_to_handle_one_record_a_time() { + let conn = initiate_simple_connection_and_test_table( + "big_int_db_processor", + "update_with_overflow_is_designed_to_handle_one_record_a_time", + ); + insert_single_record(&*conn, [&"Joe", &60, &5555]); + insert_single_record(&*conn, [&"Jodie", &77, &0]); + let balance_change = Addition("balance", 100); + let update_config = BigIntSqlConfig::new( + "", + "update test_table set balance_high_b = balance_high_b + :balance_high_b, \ + balance_low_b = balance_low_b + :balance_low_b where name in (:name, 'Jodie')", + SQLParamsBuilder::default() + .wei_change(balance_change) + .key(test_database_key(&"Joe")) + .build(), + ); + + let _ = BigIntDbProcessor::::default() + .overflow_handler + .update_with_overflow(Left(conn.as_ref()), update_config); + } + + #[test] + fn update_with_overflow_handles_error_from_executing_the_initial_select_stm() { + let conn = initiate_simple_connection_and_test_table( + "big_int_db_processor", + "update_with_overflow_handles_error_from_executing_the_initial_select_stm", + ); + conn.prepare("alter table test_table drop column balance_low_b") + .unwrap() + .execute([]) + .unwrap(); + conn.prepare("alter table test_table add column balance_low_b text") + .unwrap() + .execute([]) + .unwrap(); + insert_single_record(&*conn, [&"Joe", &60, &"bad type"]); + let balance_change = Addition("balance", 100); + let update_config = BigIntSqlConfig::new( + "this can be whatever because the test fails earlier on the select stm", + "", + SQLParamsBuilder::default() + .wei_change(balance_change) + .key(test_database_key(&"Joe")) + .build(), + ); + + let result = BigIntDbProcessor::::default() + .overflow_handler + .update_with_overflow(Left(conn.as_ref()), update_config); + + assert_eq!( + result, + Err(BigIntDbError( + "Updating balance for test_table table and change of 100 wei to 'name = Joe' with error \ + 'Invalid column type Text at index: 1, name: balance_low_b'" + .to_string() + )) + ); + } +} diff --git a/node/src/accountant/big_int_processing/big_int_divider.rs b/node/src/accountant/big_int_processing/big_int_divider.rs new file mode 100644 index 000000000..40d5a15f1 --- /dev/null +++ b/node/src/accountant/big_int_processing/big_int_divider.rs @@ -0,0 +1,580 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use crate::accountant::big_int_processing::big_int_divider::UserDefinedFunctionError::InvalidInputValue; +use crate::accountant::gwei_to_wei; +use rusqlite::functions::{Context, FunctionFlags}; +use rusqlite::Connection; +use rusqlite::Error::UserFunctionError; +use std::fmt::{Display, Formatter}; + +macro_rules! create_big_int_sqlite_fns { + ($conn: expr, $($sqlite_fn_name: expr),+; $($intern_fn_name: ident),+) => { + $( + $conn.create_scalar_function::<_, i64>($sqlite_fn_name, 3, FunctionFlags::SQLITE_UTF8 | FunctionFlags::SQLITE_DETERMINISTIC, + move |ctx| { + Ok(BigIntDivider::$intern_fn_name(common_arg_distillation( + ctx, + $sqlite_fn_name, + )?)) + } + )?; + )+ + } +} + +pub struct BigIntDivider {} + +impl BigIntDivider { + pub fn deconstruct(num: i128) -> (i64, i64) { + ( + Self::deconstruct_high_bytes(num), + Self::deconstruct_low_bytes(num), + ) + } + + fn deconstruct_high_bytes(num: i128) -> i64 { + Self::deconstruct_range_check(num); + (num >> 63) as i64 + } + + fn deconstruct_low_bytes(num: i128) -> i64 { + (num & 0x7FFFFFFFFFFFFFFFi128) as i64 + } + + pub fn reconstitute(high_bytes: i64, low_bytes: i64) -> i128 { + Self::forbidden_low_bytes_negativity_check(low_bytes); + let low_bytes = low_bytes as i128; + let high_bytes = high_bytes as i128; + (high_bytes << 63) | low_bytes + } + + fn deconstruct_range_check(num: i128) { + let top_two_bits = num >> 126 & 0b11; + if top_two_bits == 0b01 { + panic!("Dividing big integer for special database storage: {:#X} is too big, maximally 0x3FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF allowed",num) + } else if top_two_bits == 0b10 { + panic!("Dividing big integer for special database storage: {:#X} is too small, minimally 0xC0000000000000000000000000000000 allowed",num) + } + } + + fn forbidden_low_bytes_negativity_check(low_bytes: i64) { + if low_bytes < 0 { + panic!("Reconstituting big integer from special database storage: the second, lower integer {:#X} is signed despite the requirement to be all-time positive",low_bytes) + } + } + + pub fn register_big_int_deconstruction_for_sqlite_connection( + conn: &Connection, + ) -> rusqlite::Result<()> { + Self::register_deconstruct_guts(conn, "slope_drop_high_bytes", "slope_drop_low_bytes") + } + + fn register_deconstruct_guts( + conn: &Connection, + fn_name_1: &'static str, + fn_name_2: &'static str, + ) -> rusqlite::Result<()> { + fn common_arg_distillation( + rusqlite_fn_ctx: &Context, + fn_name: &str, + ) -> rusqlite::Result { + const ERR_MSG_BEGINNINGS: [&str; 3] = ["First", "Second", "Third"]; + let error_msg = |msg: String| -> rusqlite::Error { + UserFunctionError(Box::new(InvalidInputValue(fn_name.to_string(), msg))) + }; + let get_i64_from_args = |arg_idx: usize| -> rusqlite::Result { + let raw_value = rusqlite_fn_ctx.get_raw(arg_idx); + raw_value.as_i64().map_err(|_| { + error_msg(format!( + "{} argument takes only i64, not: {:?}", + ERR_MSG_BEGINNINGS[arg_idx], raw_value + )) + }) + }; + let start_point_to_decrease_from_gwei = get_i64_from_args(0)?; + let slope = get_i64_from_args(1)?; + let time_parameter = get_i64_from_args(2)?; + match (slope.is_negative(), time_parameter.is_positive()) { + (true, true) => Ok(gwei_to_wei::(start_point_to_decrease_from_gwei) + slope as i128 * time_parameter as i128), + (false, _) => Err(error_msg(format!( + "Nonnegative slope {}; delinquency slope must be negative, since debts must become more delinquent over time.", + slope + ))), + _ => Err(error_msg(format!( + "Negative time parameter {}; debt age cannot go negative.", + time_parameter + ))), + } + } + + create_big_int_sqlite_fns!( + conn, fn_name_1, fn_name_2; + deconstruct_high_bytes, deconstruct_low_bytes + ); + Ok(()) + } +} + +#[derive(Debug, PartialEq)] +enum UserDefinedFunctionError { + InvalidInputValue(String, String), +} + +impl std::error::Error for UserDefinedFunctionError {} + +impl Display for UserDefinedFunctionError { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + match self { + InvalidInputValue(fn_name, err_msg) => { + write!(f, "Error from {}: {}", fn_name, err_msg) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::accountant::big_int_processing::test_utils::restricted::create_new_empty_db; + use rusqlite::Error::SqliteFailure; + use rusqlite::ErrorCode; + + fn assert_reconstitution(as_two_integers: (i64, i64), expected_number: i128) { + let result = BigIntDivider::reconstitute(as_two_integers.0, as_two_integers.1); + + assert_eq!(result, expected_number) + } + + #[test] + fn deconstruct_and_reconstitute_works_for_huge_number() { + let tested_number = (0x3FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFu128) as i128; + + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (i64::MAX, i64::MAX)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_and_reconstitute_works_for_number_just_slightly_bigger_than_the_low_b_type_size() + { + let tested_number = i64::MAX as i128 + 1; + + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (1, 0)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_big_number() { + let tested_number = i64::MAX as i128; + let result = BigIntDivider::deconstruct(i64::MAX as i128); + + assert_eq!(result, (0, 9223372036854775807)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_small_positive_number() { + let tested_number = 1; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (0, 1)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_zero() { + let tested_number = 0; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (0, 0)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_small_negative_number() { + let tested_number = -1; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (-1, i64::MAX)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_big_negative_number() { + let tested_number = i64::MIN as i128; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (-1, 0)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_and_reconstitute_works_for_number_just_slightly_smaller_than_the_low_b_type_size( + ) { + let tested_number = i64::MIN as i128 - 1; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (-2, 9223372036854775807)); + + assert_reconstitution(result, tested_number) + } + + #[test] + fn deconstruct_works_for_huge_negative_number() { + let tested_number = 0xC0000000000000000000000000000000u128 as i128; + let result = BigIntDivider::deconstruct(tested_number); + + assert_eq!(result, (-9223372036854775808, 0)); + + assert_reconstitution(result, tested_number) + } + + #[test] + #[should_panic( + expected = "Dividing big integer for special database storage: 0x40000000000000000000000000000000 is too big, maximally 0x3FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF allowed" + )] + fn deconstruct_has_its_limits_up() { + let _ = BigIntDivider::deconstruct(0x3FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF + 1); + } + + #[test] + #[should_panic( + expected = "Dividing big integer for special database storage: 0xBFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF is too small, minimally 0xC0000000000000000000000000000000 allowed" + )] + fn deconstruct_has_its_limits_down() { + let _ = BigIntDivider::deconstruct((0xC0000000000000000000000000000000u128 as i128) - 1); + } + + #[test] + #[should_panic( + expected = "Reconstituting big integer from special database storage: the second, lower integer 0xFFFFFFFFFFFFFFFF is signed despite the requirement to be all-time positive" + )] + fn reconstitute_should_reject_lower_half_with_high_bit_set() { + let _ = BigIntDivider::reconstitute(0, -1); + } + + #[test] + fn divided_integers_can_be_ordered() { + let init = i64::MAX as i128 * 23; + let numbers_ordered = vec![ + i64::MAX as i128 + 1, + i64::MAX as i128, + (i64::MAX - 1) as i128, + 7654, + 0, + -4567, + (i64::MIN + 1) as i128, + i64::MIN as i128, + i64::MIN as i128 - 1, + i64::MIN as i128 * 32, + ]; + + let _ = numbers_ordered.into_iter().enumerate().fold( + init, + |previous_big_int, (idx, current_big_int): (usize, i128)| { + let (previous_high_b, previous_low_b) = BigIntDivider::deconstruct(previous_big_int); + let (current_high_b, current_low_b) = BigIntDivider::deconstruct(current_big_int); + assert!( + (previous_high_b > current_high_b) || (previous_high_b == current_high_b && previous_low_b > current_low_b) , + "previous_high_b: {}, current_high_b: {} and previous_low_b: {}, current_low_b: {} for {} and {} which is idx {}", + previous_high_b, + current_high_b, + previous_low_b, + current_low_b, + BigIntDivider::reconstitute(previous_high_b, previous_low_b), + BigIntDivider::reconstitute(current_high_b, current_low_b), + idx + ); + current_big_int + }, + ); + } + + fn create_test_table_and_run_register_deconstruction_for_sqlite_connection( + test_name: &str, + ) -> Connection { + let conn = create_new_empty_db("big_int_db_processor", test_name); + BigIntDivider::register_big_int_deconstruction_for_sqlite_connection(&conn).unwrap(); + conn.execute("create table test_table (computed_high_bytes int, computed_low_bytes int, database_parameter int not null)",[]).unwrap(); + conn + } + + #[test] + fn register_deconstruct_for_sqlite_connection_works() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "register_deconstruct_for_sqlite_connection_works", + ); + + let database_value_1: i64 = 12222; + let database_value_2: i64 = 23333444; + let database_value_3: i64 = 5555; + let slope: i64 = -35_000_000; + conn.execute( + "insert into test_table (database_parameter) values (?),(?),(?)", + &[&database_value_1, &database_value_2, &database_value_3], + ) + .unwrap(); + let arbitrary_constant = 111222333444_i64; + conn.execute( + "update test_table set computed_high_bytes = slope_drop_high_bytes(:my_constant, :slope, database_parameter),\ + computed_low_bytes = slope_drop_low_bytes(:my_constant, :slope, database_parameter)", + &[(":my_constant", &arbitrary_constant), (":slope", &slope)], + ) + .unwrap(); + let mut stm = conn + .prepare("select computed_high_bytes, computed_low_bytes from test_table") + .unwrap(); + let computed_values = stm + .query_map([], |row| { + let high_bytes = row.get::(0).unwrap(); + let low_bytes = row.get::(1).unwrap(); + Ok((high_bytes, low_bytes)) + }) + .unwrap() + .flatten() + .collect::>(); + assert_eq!( + computed_values, + vec![ + BigIntDivider::deconstruct( + gwei_to_wei::(arbitrary_constant) + (slope * database_value_1) as i128 + ), + BigIntDivider::deconstruct( + gwei_to_wei::(arbitrary_constant) + (slope * database_value_2) as i128 + ), + BigIntDivider::deconstruct( + gwei_to_wei::(arbitrary_constant) + (slope * database_value_3) as i128 + ) + ] + ); + } + + #[test] + fn user_defined_functions_error_implements_display() { + assert_eq!( + InvalidInputValue("CoolFn".to_string(), "error message".to_string()).to_string(), + "Error from CoolFn: error message".to_string() + ) + } + + #[test] + fn register_deconstruct_for_sqlite_connection_returns_error_at_setting_the_first_function() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "register_deconstruct_for_sqlite_connection_returns_error_at_setting_the_first_function", + ); + + let result = conn + .execute( + "insert into test_table (computed_high_bytes) values (slope_drop_high_bytes('hello', -4005000000, 712))", + [], + ) + .unwrap_err(); + + assert_eq!( + result, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_high_bytes: First argument takes only i64, not: Text([104, 101, 108, 108, 111])" + .to_string() + ) + ) + ) + } + + #[test] + fn register_deconstruct_for_sqlite_connection_returns_error_at_setting_the_second_function() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "register_deconstruct_for_sqlite_connection_returns_error_at_setting_the_second_function", + ); + + let result = conn + .execute( + "insert into test_table (computed_high_bytes) values (slope_drop_low_bytes('bye', -10000000000, 44233))", + [], + ) + .unwrap_err(); + + assert_eq!( + result, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_low_bytes: First argument takes only i64, not: Text([98, 121, 101])".to_string() + ) + ) + ) + } + + #[test] + fn our_sqlite_functions_are_specialized_and_thus_should_not_take_positive_number_for_the_second_parameter( + ) { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "our_sqlite_functions_are_specialized_and_thus_should_not_take_positive_number_for_the_second_parameter" + ); + let error_invoker = |bytes_type: &str| { + let sql = format!( + "insert into test_table (computed_{0}_bytes) values (slope_drop_{0}_bytes(45656, 5656, 11111))", + bytes_type + ); + conn.execute(&sql, []).unwrap_err() + }; + + let high_bytes_error = error_invoker("high"); + let low_bytes_error = error_invoker("low"); + + assert_eq!( + high_bytes_error, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_high_bytes: Nonnegative slope 5656; delinquency \ + slope must be negative, since debts must become more delinquent over time." + .to_string() + ) + ) + ); + assert_eq!( + low_bytes_error, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_low_bytes: Nonnegative slope 5656; delinquency \ + slope must be negative, since debts must become more delinquent over time." + .to_string() + ) + ) + ); + } + + #[test] + fn our_sqlite_functions_are_specialized_thus_should_not_take_negative_number_for_the_third_parameter( + ) { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "our_sqlite_functions_are_specialized_thus_should_not_take_negative_number_for_the_third_parameter" + ); + let error_invoker = |bytes_type: &str| { + let sql = format!( + "insert into test_table (computed_{0}_bytes) values (slope_drop_{0}_bytes(45656, -500000, -11111))", + bytes_type + ); + conn.execute(&sql, []).unwrap_err() + }; + + let high_bytes_error = error_invoker("high"); + let low_bytes_error = error_invoker("low"); + + assert_eq!( + high_bytes_error, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_high_bytes: Negative time parameter -11111; debt age cannot go negative." + .to_string() + ) + ) + ); + assert_eq!( + low_bytes_error, + SqliteFailure( + rusqlite::ffi::Error { + code: ErrorCode::Unknown, + extended_code: 1 + }, + Some( + "Error from slope_drop_low_bytes: Negative time parameter -11111; debt age cannot go negative." + .to_string() + ) + ) + ); + } + + #[test] + fn third_argument_error() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "third_argument_error", + ); + + let result = conn + .execute( + "insert into test_table (computed_high_bytes) values (slope_drop_low_bytes(15464646, 7866, 'time'))", + [], + ) + .unwrap_err(); + + assert_eq!( + result, + SqliteFailure( + rusqlite::ffi::Error{ code: ErrorCode::Unknown, extended_code: 1 }, + Some("Error from slope_drop_low_bytes: Third argument takes only i64, not: Text([116, 105, 109, 101])".to_string() + )) + ) + } + + #[test] + fn first_fn_returns_internal_error_from_create_scalar_function() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "first_fn_returns_internal_error_from_create_scalar_function", + ); + + let result = BigIntDivider::register_deconstruct_guts( + &conn, + "badly\u{0000}named", + "slope_drop_low_bytes", + ) + .unwrap_err(); + + //not asserting on the exact fit because the error + //would involve some unstable code at reproducing it + assert_eq!( + result.to_string(), + "nul byte found in provided data at position: 5".to_string() + ) + } + + #[test] + fn second_fn_returns_internal_error_from_create_scalar_function() { + let conn = create_test_table_and_run_register_deconstruction_for_sqlite_connection( + "second_fn_returns_internal_error_from_create_scalar_function", + ); + + let result = BigIntDivider::register_deconstruct_guts( + &conn, + "slope_drop_high_bytes", + "also\u{0000}badlynamed", + ) + .unwrap_err(); + + //not asserting on the exact fit because the error + //would involve some unstable code at reproducing it + assert_eq!( + result.to_string(), + "nul byte found in provided data at position: 4".to_string() + ) + } +} diff --git a/node/src/accountant/big_int_processing/mod.rs b/node/src/accountant/big_int_processing/mod.rs new file mode 100644 index 000000000..c602cea42 --- /dev/null +++ b/node/src/accountant/big_int_processing/mod.rs @@ -0,0 +1,6 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub mod big_int_db_processor; +pub mod big_int_divider; +#[cfg(test)] +mod test_utils; diff --git a/node/src/accountant/big_int_processing/test_utils.rs b/node/src/accountant/big_int_processing/test_utils.rs new file mode 100644 index 000000000..b231db20c --- /dev/null +++ b/node/src/accountant/big_int_processing/test_utils.rs @@ -0,0 +1,24 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub(in crate::accountant::big_int_processing) mod restricted { + use crate::accountant::big_int_processing::big_int_db_processor::KnownKeyVariants::TestKey; + use crate::accountant::big_int_processing::big_int_db_processor::{ + ExtendedParamsMarker, KnownKeyVariants, + }; + use masq_lib::test_utils::utils::ensure_node_home_directory_exists; + use rusqlite::Connection; + + pub fn create_new_empty_db(module: &str, test_name: &str) -> Connection { + let home_dir = ensure_node_home_directory_exists(module, test_name); + let db_path = home_dir.join("test_table.db"); + Connection::open(db_path.as_path()).unwrap() + } + + pub fn test_database_key<'a>(val: &'a dyn ExtendedParamsMarker) -> KnownKeyVariants<'a> { + TestKey { + var_name: "name", + sub_name: ":name", + val, + } + } +} diff --git a/node/src/accountant/dao_utils.rs b/node/src/accountant/dao_utils.rs new file mode 100644 index 000000000..514a0306e --- /dev/null +++ b/node/src/accountant/dao_utils.rs @@ -0,0 +1,548 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::payable_dao::PayableAccount; +use crate::accountant::receivable_dao::ReceivableAccount; +use crate::accountant::{checked_conversion, sign_conversion}; +use crate::database::connection_wrapper::ConnectionWrapper; +use crate::database::db_initializer::{ + connection_or_panic, DbInitializationConfig, DbInitializerReal, +}; +use masq_lib::constants::WEIS_OF_GWEI; +use masq_lib::messages::{ + RangeQuery, TopRecordsConfig, TopRecordsOrdering, UiPayableAccount, UiReceivableAccount, +}; +use masq_lib::utils::ExpectValue; +use rusqlite::{Row, ToSql}; +use std::cell::RefCell; +use std::fmt::Display; +use std::iter::FlatMap; +use std::path::{Path, PathBuf}; +use std::time::Duration; +use std::time::SystemTime; + +pub fn to_time_t(system_time: SystemTime) -> i64 { + match system_time.duration_since(SystemTime::UNIX_EPOCH) { + Ok(d) => sign_conversion::(d.as_secs()).expect("MASQNode has expired"), + Err(e) => panic!( + "Must be wrong, moment way far in the past: {:?}, {}", + system_time, e + ), + } +} + +pub fn now_time_t() -> i64 { + to_time_t(SystemTime::now()) +} + +pub fn from_time_t(time_t: i64) -> SystemTime { + let interval = Duration::from_secs(time_t as u64); + SystemTime::UNIX_EPOCH + interval +} + +pub struct DaoFactoryReal { + pub data_directory: PathBuf, + pub init_config: RefCell>, +} + +impl DaoFactoryReal { + pub fn new(data_directory: &Path, init_config: DbInitializationConfig) -> Self { + Self { + data_directory: data_directory.to_path_buf(), + init_config: RefCell::new(Some(init_config)), + } + } + + pub fn make_connection(&self) -> Box { + connection_or_panic( + &DbInitializerReal::default(), + &self.data_directory, + self.init_config.take().expectv("Db init config"), + ) + } +} + +impl From for CustomQuery { + fn from(config: TopRecordsConfig) -> Self { + CustomQuery::TopRecords { + count: config.count, + ordered_by: config.ordered_by, + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CustomQuery { + TopRecords { + count: u16, + ordered_by: TopRecordsOrdering, + }, + RangeQuery { + min_age_s: u64, + max_age_s: u64, + min_amount_gwei: N, + max_amount_gwei: N, + timestamp: SystemTime, + }, +} + +type RusqliteParamsWithOwnedToSql = Vec<(&'static str, Box)>; + +pub struct TopStmConfig { + pub limit_clause: &'static str, + pub gwei_min_resolution_clause: &'static str, + pub age_ordering_clause: &'static str, +} + +pub struct RangeStmConfig { + pub where_clause: &'static str, + pub gwei_min_resolution_clause: &'static str, + pub secondary_order_param: &'static str, +} + +pub struct AssemblerFeeder { + pub main_where_clause: &'static str, + pub where_clause_extension: &'static str, + pub order_by_first_param: &'static str, + pub order_by_second_param: &'static str, + pub limit_clause: &'static str, +} + +//be aware that balances smaller than one gwei won't be shown, +//if there aren't any bigger ones the function returns None +impl CustomQuery { + pub fn query( + self, + conn: &dyn ConnectionWrapper, + stm_assembler: F1, + variant_top: TopStmConfig, + variant_range: RangeStmConfig, + value_fetcher: F2, + ) -> Option> + where + F1: Fn(AssemblerFeeder) -> String, + F2: Fn(&Row) -> rusqlite::Result, + S: TryFrom, + i128: From, + { + let (finalized_stm, params): (String, RusqliteParamsWithOwnedToSql) = match self { + Self::TopRecords { count, ordered_by } => { + let (order_by_first_param, order_by_second_param) = + Self::ordering(ordered_by, variant_top.age_ordering_clause); + ( + stm_assembler(AssemblerFeeder { + main_where_clause: variant_top.gwei_min_resolution_clause, + where_clause_extension: "", + order_by_first_param, + order_by_second_param, + limit_clause: variant_top.limit_clause, + }), + vec![(":limit_count", Box::new(count as i64))], + ) + } + Self::RangeQuery { + min_age_s: min_age, + max_age_s: max_age, + min_amount_gwei: min_amount, + max_amount_gwei: max_amount, + timestamp, + } => ( + stm_assembler(AssemblerFeeder { + main_where_clause: variant_range.where_clause, + where_clause_extension: variant_range.gwei_min_resolution_clause, + order_by_first_param: "balance_high_b desc, balance_low_b desc", + order_by_second_param: variant_range.secondary_order_param, + limit_clause: "", + }), + Self::set_age_constraints(min_age, max_age, timestamp) + .into_iter() + .chain(Self::set_wei_constraints(min_amount, max_amount)) + .collect::)>>(), + ), + }; + let accounts = Self::execute_query(conn, &finalized_stm, params, value_fetcher); + (!accounts.is_empty()).then_some(accounts) + } + + fn execute_query<'a, R, F1>( + conn: &'a dyn ConnectionWrapper, + stm: &'a str, + params: RusqliteParamsWithOwnedToSql, + value_fetcher: F1, + ) -> Vec + where + F1: Fn(&Row) -> rusqlite::Result, + { + conn.prepare(stm) + .expect("select statement is wrong") + .query_map( + params + .iter() + .map(|(param_name, value)| (*param_name, value.as_ref())) + .collect::>() + .as_slice(), + value_fetcher, + ) + .unwrap_or_else(|e| panic!("database corrupt: {}", e)) + .vigilant_flatten() + .collect::>() + } + + fn set_age_constraints( + min_age: u64, + max_age: u64, + timestamp: SystemTime, + ) -> RusqliteParamsWithOwnedToSql { + let now = to_time_t(timestamp); + let age_to_time_t = |age_limit| now - checked_conversion::(age_limit); + vec![ + (":min_timestamp", Box::new(age_to_time_t(max_age))), + (":max_timestamp", Box::new(age_to_time_t(min_age))), + ] + } + + fn set_wei_constraints(min_amount: N, max_amount: N) -> RusqliteParamsWithOwnedToSql + where + i128: From, + { + [ + (":min_balance_high_b", ":min_balance_low_b"), + (":max_balance_high_b", ":max_balance_low_b"), + ] + .into_iter() + .zip([min_amount, max_amount].into_iter()) + .flat_map(|(param_names, gwei_num)| { + let wei_num = i128::from(gwei_num) * WEIS_OF_GWEI; + let big_int_divided = BigIntDivider::deconstruct(wei_num); + Self::balance_constraint_as_integer_pair(param_names, big_int_divided) + }) + .collect() + } + + fn balance_constraint_as_integer_pair<'a>( + param_names: (&'a str, &'a str), + big_int_divided: (i64, i64), + ) -> Vec<(&'a str, Box)> { + let (high_bytes_param_name, low_bytes_param_name) = param_names; + let (high_bytes_value, low_bytes_value) = big_int_divided; + vec![ + (high_bytes_param_name, Box::new(high_bytes_value)), + (low_bytes_param_name, Box::new(low_bytes_value)), + ] + } + + fn ordering( + ordering: TopRecordsOrdering, + age_param: &'static str, + ) -> (&'static str, &'static str) { + match ordering { + TopRecordsOrdering::Age => (age_param, "balance_high_b desc, balance_low_b desc"), + TopRecordsOrdering::Balance => ("balance_high_b desc, balance_low_b desc", age_param), + } + } +} + +impl From<&RangeQuery> for CustomQuery { + fn from(user_input: &RangeQuery) -> Self { + Self::RangeQuery { + min_age_s: user_input.min_age_s, + max_age_s: user_input.max_age_s, + min_amount_gwei: user_input.min_amount_gwei, + max_amount_gwei: user_input.max_amount_gwei, + timestamp: SystemTime::now(), + } + } +} + +pub fn remap_payable_accounts(accounts: Vec) -> Vec { + accounts + .into_iter() + .map(|account| UiPayableAccount { + wallet: account.wallet.to_string(), + age_s: to_age(account.last_paid_timestamp), + balance_gwei: { + let gwei = (account.balance_wei / (WEIS_OF_GWEI as u128)) as u64; + if gwei > 0 { + gwei + } else { + panic!( + "Broken code: PayableAccount with less than 1 gwei passed through db query \ + constraints; wallet: {}, balance: {}", + account.wallet, account.balance_wei + ) + } + }, + pending_payable_hash_opt: account + .pending_payable_opt + .map(|full_id| full_id.hash.to_string()), + }) + .collect() +} + +pub fn remap_receivable_accounts(accounts: Vec) -> Vec { + accounts + .into_iter() + .map(|account| UiReceivableAccount { + wallet: account.wallet.to_string(), + age_s: to_age(account.last_received_timestamp), + balance_gwei:{ + let gwei = (account.balance_wei / (WEIS_OF_GWEI as i128)) as i64; + if gwei != 0 {gwei} else {panic!("Broken code: ReceivableAccount with balance \ + between {} and 0 gwei passed through db query constraints; wallet: {}, balance: {}", + if account.balance_wei.is_positive() {"1"}else{"-1"}, + account.wallet, + account.balance_wei + )} + }, + }) + .collect() +} + +fn to_age(timestamp: SystemTime) -> u64 { + (to_time_t(SystemTime::now()) - to_time_t(timestamp)) as u64 +} + +#[allow(clippy::type_complexity)] +pub trait VigilantRusqliteFlatten { + fn vigilant_flatten( + self, + ) -> FlatMap, fn(rusqlite::Result) -> rusqlite::Result> + where + Self: Iterator> + Sized, + { + self.flat_map(|item: rusqlite::Result| { + item.map_err(|err| { + panic!( + "discovered an error from a preceding operation when flattening produced \ + Result structures: {:?}", + err + ) + }) + }) + } +} + +impl>, R> VigilantRusqliteFlatten for T {} + +pub fn sum_i128_values_from_table( + conn: &dyn ConnectionWrapper, + table: &str, + param_name: &str, + value_completer: fn(usize, &Row) -> rusqlite::Result, +) -> i128 { + let mut row_number = 0; + let select_stm = format!("select {param_name}_high_b, {param_name}_low_b from {table}"); + conn.prepare(&select_stm) + .expect("select stm error") + .query_map([], |row| { + row_number += 1; + value_completer(row_number, row) + }) + .expect("select query failed") + .vigilant_flatten() + .sum() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::database::connection_wrapper::ConnectionWrapperReal; + use crate::test_utils::make_wallet; + use masq_lib::messages::TopRecordsOrdering::Balance; + use masq_lib::test_utils::utils::ensure_node_home_directory_exists; + use rusqlite::types::{ToSqlOutput, Value}; + use rusqlite::{Connection, OpenFlags}; + use std::time::UNIX_EPOCH; + + #[test] + fn set_age_constraints_works() { + let min_age = 5555; + let max_age = 10000; + let now = SystemTime::now(); + + let result = CustomQuery::::set_age_constraints(min_age, max_age, now); + + assert_eq!(result.len(), 2); + let param_pair_1 = &result[0]; + let param_pair_2 = &result[1]; + assert_eq!(param_pair_1.0, ":min_timestamp"); + assert_eq!(param_pair_2.0, ":max_timestamp"); + let get_assigned_value = |value| match value { + ToSqlOutput::Owned(Value::Integer(num)) => num, + x => panic!("we expected integer and got this: {:?}", x), + }; + let assigned_value_1 = get_assigned_value(param_pair_1.1.to_sql().unwrap()); + let assigned_value_2 = get_assigned_value(param_pair_2.1.to_sql().unwrap()); + assert_eq!(assigned_value_1, to_time_t(now) - 10000); + assert_eq!(assigned_value_2, to_time_t(now) - 5555) + } + + #[test] + #[should_panic(expected = "database corrupt: Invalid parameter name: :limit_count")] + fn erroneous_query_leads_to_panic() { + let home_dir = + ensure_node_home_directory_exists("dao_utils", "erroneous_query_leads_to_panic"); + let db_path = home_dir.join("test.db"); + let creation_conn = Connection::open(db_path.as_path()).unwrap(); + creation_conn + .execute( + "create table fruits (kind text primary key, price integer not null)", + [], + ) + .unwrap(); + let conn_read_only = + Connection::open_with_flags(db_path, OpenFlags::SQLITE_OPEN_READ_ONLY).unwrap(); + let conn_wrapped = ConnectionWrapperReal::new(conn_read_only); + let subject = CustomQuery::::TopRecords { + count: 12, + ordered_by: Balance, + }; + + let _ = subject.query::<_, i64, _, _>( + &conn_wrapped, + |_feeder: AssemblerFeeder| "select kind, price from fruits".to_string(), + TopStmConfig { + limit_clause: "", + gwei_min_resolution_clause: "", + age_ordering_clause: "", + }, + RangeStmConfig { + where_clause: "", + gwei_min_resolution_clause: "", + secondary_order_param: "", + }, + |_row| Ok(()), + ); + } + + #[test] + #[should_panic( + expected = "Broken code: PayableAccount with less than 1 gwei passed through db query constraints; \ + wallet: 0x0000000000000000000000000061633336363563, balance: 565122333" + )] + fn remap_payable_accounts_getting_record_below_one_gwei_means_broken_database_query() { + let accounts = vec![ + PayableAccount { + wallet: make_wallet("abc123"), + balance_wei: 4_888_123_457, + last_paid_timestamp: SystemTime::now(), + pending_payable_opt: None, + }, + PayableAccount { + wallet: make_wallet("ac3665c"), + balance_wei: 565_122_333, + last_paid_timestamp: SystemTime::now(), + pending_payable_opt: None, + }, + ]; + remap_payable_accounts(accounts); + } + + #[test] + #[should_panic( + expected = "Broken code: ReceivableAccount with balance between 1 and 0 gwei passed through db query \ + constraints; wallet: 0x0000000000000000000000000061633336363563, balance: 300122333" + )] + fn remap_receivable_accounts_getting_record_between_one_and_zero_gwei_means_broken_database_query( + ) { + let accounts = vec![ + ReceivableAccount { + wallet: make_wallet("ac45123"), + balance_wei: 4_888_123_457, + last_received_timestamp: SystemTime::now(), + }, + ReceivableAccount { + wallet: make_wallet("ac3665c"), + balance_wei: 300_122_333, + last_received_timestamp: SystemTime::now(), + }, + ]; + remap_receivable_accounts(accounts); + } + + #[test] + #[should_panic( + expected = "Broken code: ReceivableAccount with balance between -1 and 0 gwei passed through db query \ + constraints; wallet: 0x0000000000000000000000000061633336363563, balance: -290122333" + )] + fn remap_receivable_accounts_getting_record_between_minus_one_and_zero_gwei_means_broken_database_query( + ) { + let accounts = vec![ + ReceivableAccount { + wallet: make_wallet("ac45123"), + balance_wei: -4_000_123_457, + last_received_timestamp: SystemTime::now(), + }, + ReceivableAccount { + wallet: make_wallet("ac3665c"), + balance_wei: -290_122_333, + last_received_timestamp: SystemTime::now(), + }, + ]; + remap_receivable_accounts(accounts); + } + + #[test] + fn custom_query_from_range_query_works() { + let subject = RangeQuery { + min_age_s: 12, + max_age_s: 55, + min_amount_gwei: 89_i64, + max_amount_gwei: 12222, + }; + let before = SystemTime::now(); + + let result: CustomQuery = (&subject).into(); + + let after = SystemTime::now(); + if let CustomQuery::RangeQuery { + min_age_s, + max_age_s, + min_amount_gwei, + max_amount_gwei, + timestamp, + } = result + { + assert_eq!(min_age_s, 12); + assert_eq!(max_age_s, 55); + assert_eq!(min_amount_gwei, 89); + assert_eq!(max_amount_gwei, 12222); + assert!(before <= timestamp && timestamp <= after) + } else { + panic!("we expected range query but got something else") + } + } + + #[test] + #[should_panic(expected = "Must be wrong, moment way far in the past")] + fn to_time_t_does_not_like_time_traveling() { + let far_far_before = UNIX_EPOCH.checked_sub(Duration::from_secs(1)).unwrap(); + + let _ = to_time_t(far_far_before); + } + + #[test] + fn vigilant_flatten_can_flatten() { + let collection = vec![Ok(56_u16), Ok(0), Ok(6789)]; + let iterator = collection.into_iter(); + + let result = iterator.vigilant_flatten().collect::>(); + + assert_eq!(result, vec![56, 0, 6789]) + } + + #[test] + #[should_panic( + expected = "discovered an error from a preceding operation when flattening produced Result structures: QueryReturnedNoRows" + )] + fn vigilant_flatten_discovers_error() { + let collection = vec![ + Ok(56_u16), + Err(rusqlite::Error::QueryReturnedNoRows), + Err(rusqlite::Error::UnwindingPanic), + ]; + let iterator = collection.into_iter(); + + let _ = iterator.vigilant_flatten().collect::>(); + } +} diff --git a/node/src/accountant/financials.rs b/node/src/accountant/financials.rs new file mode 100644 index 000000000..88ae70855 --- /dev/null +++ b/node/src/accountant/financials.rs @@ -0,0 +1,259 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use std::fmt::Debug; + +const OPCODE_FINANCIALS: &str = "financials"; + +fn fits_in_0_to_i64max_for_u64(num: &N) -> bool +where + N: Ord + Copy + TryFrom, + u64: TryFrom, + >::Error: Debug, +{ + match u64::try_from(*num) { + Ok(u64_num) => u64_num <= i64::MAX as u64, + Err(_) => { + let zero_as_t: N = 0_u64.try_into().expect("should be fine"); + if num < &zero_as_t { + true + } else { + unreachable!("only u64 and i64 values are expected") + } + } + } +} + +pub(in crate::accountant) mod visibility_restricted_module { + use crate::accountant::dao_utils::CustomQuery; + use crate::accountant::financials::{fits_in_0_to_i64max_for_u64, OPCODE_FINANCIALS}; + use masq_lib::constants::{ + REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS, REQUEST_WITH_NO_VALUES, VALUE_EXCEEDS_ALLOWED_LIMIT, + }; + use masq_lib::messages::UiFinancialsRequest; + use masq_lib::ui_gateway::{MessageBody, MessagePath}; + use std::fmt::{Debug, Display}; + + pub fn check_query_is_within_tech_limits( + query: &CustomQuery, + table: &str, + context_id: u64, + ) -> Result<(), MessageBody> + where + N: Ord + Copy + Display + TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, + { + let err = |param_name, num: &dyn Display| { + Err(MessageBody { + opcode: OPCODE_FINANCIALS.to_string(), + path: MessagePath::Conversation(context_id), + payload: Err((VALUE_EXCEEDS_ALLOWED_LIMIT, format!( + "Range query for {}: {} requested too big. Should be less than or equal to {}, not: {}", + table, + param_name, + i64::MAX, + num + ))) + }) + }; + if let CustomQuery::RangeQuery { + min_age_s, + max_age_s, + min_amount_gwei, + max_amount_gwei, + .. + } = query + { + match ( + min_age_s <= &(i64::MAX as u64), + max_age_s <= &(i64::MAX as u64), + fits_in_0_to_i64max_for_u64(min_amount_gwei), + fits_in_0_to_i64max_for_u64(max_amount_gwei), + ) { + (false, ..) => err("Min age", min_age_s), + (_, false, ..) => err("Max age", max_age_s), + (_, _, false, _) => err("Min amount", min_amount_gwei), + (_, _, _, false) => err("Max amount", max_amount_gwei), + _ => Ok(()), + } + } else { + panic!("Broken code: only range query belongs in here") + } + } + + pub fn financials_entry_check( + msg: &UiFinancialsRequest, + context_id: u64, + ) -> Result<(), MessageBody> { + if !msg.stats_required && msg.top_records_opt.is_none() && msg.custom_queries_opt.is_none() + { + Err(MessageBody { + opcode: OPCODE_FINANCIALS.to_string(), + path: MessagePath::Conversation(context_id), + payload: Err(( + REQUEST_WITH_NO_VALUES, + "Empty requests with missing queries not to be processed".to_string(), + )), + }) + } else if msg.top_records_opt.is_some() && msg.custom_queries_opt.is_some() { + Err(MessageBody { + opcode: OPCODE_FINANCIALS.to_string(), + path: MessagePath::Conversation(context_id), + payload: Err((REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS, "Requesting top records and the more customized subset of records is not allowed both at the same time".to_string())), + }) + } else { + Ok(()) + } + } +} + +#[cfg(test)] +mod tests { + use super::visibility_restricted_module::check_query_is_within_tech_limits; + use crate::accountant::dao_utils::CustomQuery; + use crate::accountant::financials::fits_in_0_to_i64max_for_u64; + use masq_lib::constants::VALUE_EXCEEDS_ALLOWED_LIMIT; + use masq_lib::messages::TopRecordsOrdering::Age; + use masq_lib::ui_gateway::{MessageBody, MessagePath}; + use std::fmt::{Debug, Display}; + use std::time::SystemTime; + + fn assert_excessive_values_in_check_query_is_within_tech_limits( + query: CustomQuery, + err_msg: &str, + ) where + T: Ord + Copy + Display + TryFrom, + u64: TryFrom, + >::Error: Debug, + >::Error: Debug, + { + let result = check_query_is_within_tech_limits(&query, "payable", 1234); + + assert_eq!( + result, + Err(MessageBody { + opcode: "financials".to_string(), + path: MessagePath::Conversation(1234), + payload: Err((VALUE_EXCEEDS_ALLOWED_LIMIT, err_msg.to_string())), + }) + ) + } + + #[test] + fn check_query_is_within_tech_limits_catches_error_at_age_min() { + let query = CustomQuery::RangeQuery { + min_age_s: i64::MAX as u64 + 1, + max_age_s: 4000000, + min_amount_gwei: 55, + max_amount_gwei: 6666, + timestamp: SystemTime::now(), + }; + + assert_excessive_values_in_check_query_is_within_tech_limits( + query, + "Range query for payable: Min age requested \ + too big. Should be less than or equal to 9223372036854775807, not: 9223372036854775808", + ) + } + + #[test] + fn check_query_is_within_tech_limits_catches_error_at_age_max() { + let query = CustomQuery::RangeQuery { + min_age_s: 32656, + max_age_s: i64::MAX as u64 + 1, + min_amount_gwei: 55, + max_amount_gwei: 6666, + timestamp: SystemTime::now(), + }; + + assert_excessive_values_in_check_query_is_within_tech_limits( + query, + "Range query for payable: Max age requested \ + too big. Should be less than or equal to 9223372036854775807, not: 9223372036854775808", + ) + } + + #[test] + fn check_query_is_within_tech_limits_catches_error_at_amount_min() { + let query = CustomQuery::RangeQuery { + min_age_s: 32656, + max_age_s: 4545555, + min_amount_gwei: i64::MAX as u64 + 1, + max_amount_gwei: 6666, + timestamp: SystemTime::now(), + }; + + assert_excessive_values_in_check_query_is_within_tech_limits( + query, + "Range query for payable: Min amount requested \ + too big. Should be less than or equal to 9223372036854775807, not: 9223372036854775808", + ) + } + + #[test] + fn check_query_is_within_tech_limits_catches_error_at_amount_max() { + let query = CustomQuery::RangeQuery { + min_age_s: 32656, + max_age_s: 4545555, + min_amount_gwei: 144, + max_amount_gwei: i64::MAX as u64 + 1, + timestamp: SystemTime::now(), + }; + + assert_excessive_values_in_check_query_is_within_tech_limits( + query, + "Range query for payable: Max amount requested \ + too big. Should be less than or equal to 9223372036854775807, not: 9223372036854775808", + ) + } + + #[test] + fn check_query_is_within_tech_limits_works_for_smaller_or_equal_values_than_max_limit() { + [i64::MAX as u64, (i64::MAX - 1) as u64, 1] + .into_iter() + .for_each(|val| { + let query = CustomQuery::RangeQuery { + min_age_s: val, + max_age_s: val, + min_amount_gwei: val, + max_amount_gwei: val, + timestamp: SystemTime::now(), + }; + let result = check_query_is_within_tech_limits(&query, "payable", 1234); + assert_eq!(result, Ok(())) + }) + } + + #[test] + fn check_query_is_within_tech_limits_works_for_negative_values() { + let query = CustomQuery::RangeQuery { + min_age_s: 32656, + max_age_s: 4545555, + min_amount_gwei: -500000, + max_amount_gwei: -500, + timestamp: SystemTime::now(), + }; + + let result = check_query_is_within_tech_limits(&query, "receivable", 789); + + assert_eq!(result, Ok(())) + } + + #[test] + #[should_panic(expected = "entered unreachable code: only u64 and i64 values are expected")] + fn compare_amount_param_unreachable_condition() { + let _ = fits_in_0_to_i64max_for_u64(&u128::MAX); + } + + #[test] + #[should_panic(expected = "Broken code: only range query belongs in here")] + fn check_query_is_within_tech_limits_blows_up_on_unexpected_query_type() { + let query = CustomQuery::::TopRecords { + count: 123, + ordered_by: Age, + }; + + let _ = check_query_is_within_tech_limits(&query, "payable", 1234); + } +} diff --git a/node/src/accountant/mod.rs b/node/src/accountant/mod.rs index 297784c9c..ff1e284d8 100644 --- a/node/src/accountant/mod.rs +++ b/node/src/accountant/mod.rs @@ -1,4 +1,8 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub mod big_int_processing; +pub mod dao_utils; +pub mod financials; pub mod payable_dao; pub mod pending_payable_dao; pub mod receivable_dao; @@ -7,11 +11,21 @@ pub mod tools; #[cfg(test)] pub mod test_utils; -use masq_lib::constants::SCAN_ERROR; +use core::fmt::Debug; +use masq_lib::constants::{SCAN_ERROR, WEIS_OF_GWEI}; -use masq_lib::messages::{ScanType, UiScanRequest, UiScanResponse}; +use masq_lib::messages::{ + QueryResults, ScanType, UiFinancialStatistics, UiPayableAccount, UiReceivableAccount, + UiScanRequest, UiScanResponse, +}; use masq_lib::ui_gateway::{MessageBody, MessagePath}; +use crate::accountant::dao_utils::{ + remap_payable_accounts, remap_receivable_accounts, CustomQuery, DaoFactoryReal, +}; +use crate::accountant::financials::visibility_restricted_module::{ + check_query_is_within_tech_limits, financials_entry_check, +}; use crate::accountant::payable_dao::{Payable, PayableAccount, PayableDaoError, PayableDaoFactory}; use crate::accountant::pending_payable_dao::{PendingPayableDao, PendingPayableDaoFactory}; use crate::accountant::receivable_dao::{ @@ -22,13 +36,12 @@ use crate::banned_dao::{BannedDao, BannedDaoFactory}; use crate::blockchain::blockchain_bridge::{PendingPayableFingerprint, RetrieveTransactions}; use crate::blockchain::blockchain_interface::{BlockchainError, BlockchainTransaction}; use crate::bootstrapper::BootstrapperConfig; -use crate::database::dao_utils::DaoFactoryReal; -use crate::database::db_migrations::MigratorConfig; -use crate::sub_lib::accountant::ReportRoutingServiceProvidedMessage; +use crate::database::db_initializer::DbInitializationConfig; use crate::sub_lib::accountant::{AccountantConfig, FinancialStatistics, PaymentThresholds}; use crate::sub_lib::accountant::{AccountantSubs, ReportServicesConsumedMessage}; +use crate::sub_lib::accountant::{MessageIdGenerator, MessageIdGeneratorReal}; use crate::sub_lib::accountant::{ - MessageIdGenerator, MessageIdGeneratorReal, ReportExitServiceProvidedMessage, + ReportExitServiceProvidedMessage, ReportRoutingServiceProvidedMessage, }; use crate::sub_lib::blockchain_bridge::ReportAccountsPayable; use crate::sub_lib::peer_actors::{BindMessage, StartMessage}; @@ -51,12 +64,15 @@ use masq_lib::ui_gateway::{NodeFromUiMessage, NodeToUiMessage}; use masq_lib::utils::{plus, ExpectValue}; use payable_dao::PayableDao; use receivable_dao::ReceivableDao; +use std::any::type_name; #[cfg(test)] use std::any::Any; use std::default::Default; -use std::ops::Add; +use std::fmt::Display; +use std::ops::{Div, Mul}; use std::path::Path; use std::time::{Duration, SystemTime}; +use thousands::Separable; use web3::types::{TransactionReceipt, H256}; pub const CRASH_KEY: &str = "ACCOUNTANT"; @@ -78,9 +94,9 @@ pub struct Accountant { report_accounts_payable_sub: Option>, retrieve_transactions_sub: Option>, report_new_payments_sub: Option>, - report_sent_payments_sub: Option>, + report_sent_payments_sub: Option>, ui_message_sub: Option>, - payable_threshold_tools: Box, + payable_threshold_gauge: Box, message_id_generator: Box, logger: Logger, } @@ -103,7 +119,7 @@ pub struct ReceivedPayments { } #[derive(Debug, Message, PartialEq, Eq)] -pub struct SentPayable { +pub struct SentPayables { pub timestamp: SystemTime, pub payable: Vec>, pub response_skeleton_opt: Option, @@ -175,11 +191,11 @@ impl Handler for Accountant { } } -impl Handler for Accountant { +impl Handler for Accountant { type Result = (); - fn handle(&mut self, msg: SentPayable, _ctx: &mut Self::Context) -> Self::Result { - self.handle_sent_payable(msg); + fn handle(&mut self, msg: SentPayables, _ctx: &mut Self::Context) -> Self::Result { + self.handle_sent_payables(msg); } } @@ -315,7 +331,7 @@ impl Handler for Accountant { msg.fingerprints_with_receipts.len() ); let statuses = self.handle_pending_transaction_with_its_receipt(&msg); - self.process_transaction_by_status(statuses, ctx); + self.process_transactions_by_their_status(statuses, ctx); if let Some(response_skeleton) = &msg.response_skeleton_opt { self.ui_message_sub .as_ref() @@ -371,8 +387,8 @@ impl Handler for Accountant { fn handle(&mut self, msg: NodeFromUiMessage, ctx: &mut Self::Context) -> Self::Result { let client_id = msg.client_id; - if let Ok((_, context_id)) = UiFinancialsRequest::fmb(msg.body.clone()) { - self.handle_financials(client_id, context_id); + if let Ok((request, context_id)) = UiFinancialsRequest::fmb(msg.body.clone()) { + self.handle_financials(&request, client_id, context_id) } else if let Ok((body, context_id)) = UiScanRequest::fmb(msg.body.clone()) { self.handle_externally_triggered_scan( ctx, @@ -417,7 +433,7 @@ impl Accountant { ui_message_sub: None, confirmation_tools: TransactionConfirmationTools::default(), message_id_generator: Box::new(MessageIdGeneratorReal::default()), - payable_threshold_tools: Box::new(PayableExceedThresholdToolsReal::default()), + payable_threshold_gauge: Box::new(PayableThresholdsGaugeReal::default()), logger: Logger::new("Accountant"), } } @@ -432,14 +448,14 @@ impl Accountant { report_new_payments: recipient!(addr, ReceivedPayments), pending_payable_fingerprint: recipient!(addr, PendingPayableFingerprint), report_transaction_receipts: recipient!(addr, ReportTransactionReceipts), - report_sent_payments: recipient!(addr, SentPayable), + report_sent_payments: recipient!(addr, SentPayables), scan_errors: recipient!(addr, ScanError), ui_message_sub: recipient!(addr, NodeFromUiMessage), } } pub fn dao_factory(data_directory: &Path) -> DaoFactoryReal { - DaoFactoryReal::new(data_directory, false, MigratorConfig::panic_on_migration()) + DaoFactoryReal::new(data_directory, DbInitializationConfig::panic_on_migration()) } fn handle_scan_message( @@ -470,11 +486,7 @@ impl Accountant { "Chose {} qualified debts to pay", qualified_payables.len() ); - debug!( - self.logger, - "{}", - self.payables_debug_summary(&qualified_payables) - ); + self.payables_debug_summary(&qualified_payables); if !qualified_payables.is_empty() { self.report_accounts_payable_sub .as_ref() @@ -559,7 +571,7 @@ impl Accountant { } fn balance_and_age(account: &ReceivableAccount) -> (String, Duration) { - let balance = format!("{}", (account.balance as f64) / 1_000_000_000.0); + let balance = format!("{}", account.balance_wei / WEIS_OF_GWEI); let age = account .last_received_timestamp .elapsed() @@ -571,32 +583,31 @@ impl Accountant { self.payable_exceeded_threshold(payable).is_some() } - fn payable_exceeded_threshold(&self, payable: &PayableAccount) -> Option { - // TODO: This calculation should be done in the database, if possible - let time_since_last_paid = SystemTime::now() + fn payable_exceeded_threshold(&self, payable: &PayableAccount) -> Option { + let debt_age = SystemTime::now() .duration_since(payable.last_paid_timestamp) .expect("Internal error") .as_secs(); - if self.payable_threshold_tools.is_innocent_age( - time_since_last_paid, + if self.payable_threshold_gauge.is_innocent_age( + debt_age, self.config.payment_thresholds.maturity_threshold_sec as u64, ) { return None; } - if self.payable_threshold_tools.is_innocent_balance( - payable.balance, - self.config.payment_thresholds.permanent_debt_allowed_gwei, + if self.payable_threshold_gauge.is_innocent_balance( + payable.balance_wei, + gwei_to_wei(self.config.payment_thresholds.permanent_debt_allowed_gwei), ) { return None; } let threshold = self - .payable_threshold_tools - .calculate_payout_threshold(self.config.payment_thresholds, time_since_last_paid); - if payable.balance as f64 > threshold { - Some(threshold as u64) + .payable_threshold_gauge + .calculate_payout_threshold_in_gwei(&self.config.payment_thresholds, debt_age); + if payable.balance_wei > threshold { + Some(threshold) } else { None } @@ -610,8 +621,8 @@ impl Accountant { payload_size: usize, wallet: &Wallet, ) { - let byte_charge = byte_rate * (payload_size as u64); - let total_charge = service_rate + byte_charge; + let byte_charge = byte_rate as u128 * (payload_size as u128); + let total_charge = service_rate as u128 + byte_charge; if !self.our_wallet(wallet) { match self.receivable_dao .as_ref() @@ -644,12 +655,12 @@ impl Accountant { payload_size: usize, wallet: &Wallet, ) { - let byte_charge = byte_rate * (payload_size as u64); - let total_charge = service_rate + byte_charge; + let byte_charge = byte_rate as u128 * (payload_size as u128); + let total_charge = service_rate as u128 + byte_charge; if !self.our_wallet(wallet) { match self.payable_dao .as_ref() - .more_money_payable(timestamp, wallet, total_charge) { + .more_money_payable(timestamp, wallet,total_charge){ Ok(_) => (), Err(PayableDaoError::SignConversion(_)) => error! ( self.logger, @@ -685,7 +696,7 @@ impl Accountant { "Payable scan found no debts".to_string() } else { struct PayableInfo { - balance: i64, + balance: u128, age: Duration, } let init = ( @@ -706,22 +717,23 @@ impl Accountant { { //look at a test if not understandable let check_age_parameter_if_the_first_is_the_same = - || -> bool { p.balance == biggest.balance && p_age > biggest.age }; + || -> bool { p.balance_wei == biggest.balance && p_age > biggest.age }; - if p.balance > biggest.balance || check_age_parameter_if_the_first_is_the_same() + if p.balance_wei > biggest.balance + || check_age_parameter_if_the_first_is_the_same() { biggest = PayableInfo { - balance: p.balance, + balance: p.balance_wei, age: p_age, } } let check_balance_parameter_if_the_first_is_the_same = - || -> bool { p_age == oldest.age && p.balance > oldest.balance }; + || -> bool { p_age == oldest.age && p.balance_wei > oldest.balance }; if p_age > oldest.age || check_balance_parameter_if_the_first_is_the_same() { oldest = PayableInfo { - balance: p.balance, + balance: p.balance_wei, age: p_age, } } @@ -734,27 +746,31 @@ impl Accountant { } } - fn payables_debug_summary(&self, qualified_payables: &[PayableAccount]) -> String { - let now = SystemTime::now(); - let list = qualified_payables - .iter() - .map(|payable| { - let p_age = now - .duration_since(payable.last_paid_timestamp) - .expect("Payable time is corrupt"); - let threshold = self - .payable_exceeded_threshold(payable) - .expect("Threshold suddenly changed!"); - format!( - "{} owed for {}sec exceeds threshold: {}; creditor: {}", - payable.balance, - p_age.as_secs(), - threshold, - payable.wallet - ) - }) - .join("\n"); - String::from("Paying qualified debts:\n").add(&list) + fn payables_debug_summary(&self, qualified_payables: &[PayableAccount]) { + if qualified_payables.is_empty() { + return; + } + debug!(self.logger, "Paying qualified debts:\n{}", { + let now = SystemTime::now(); + qualified_payables + .iter() + .map(|payable| { + let p_age = now + .duration_since(payable.last_paid_timestamp) + .expect("Payable time is corrupt"); + let threshold = self + .payable_exceeded_threshold(payable) + .expect("Threshold suddenly changed!"); + format!( + "{} wei owed for {} sec exceeds threshold: {} wei; creditor: {}", + payable.balance_wei.separate_with_commas(), + p_age.as_secs(), + threshold.separate_with_commas(), + payable.wallet + ) + }) + .join("\n") + }) } fn handle_bind_message(&mut self, msg: BindMessage) { @@ -779,11 +795,11 @@ impl Accountant { let total_newly_paid_receivable = msg .payments .iter() - .fold(0, |so_far, now| so_far + now.gwei_amount); + .fold(0, |so_far, now| so_far + now.wei_amount); self.receivable_dao .as_mut() .more_money_received(msg.timestamp, msg.payments); - self.financial_statistics.total_paid_receivable += total_newly_paid_receivable; + self.financial_statistics.total_paid_receivable_wei += total_newly_paid_receivable; } if let Some(response_skeleton) = msg.response_skeleton_opt { self.ui_message_sub @@ -797,8 +813,8 @@ impl Accountant { } } - fn handle_sent_payable(&self, sent_payable: SentPayable) { - let (ok, err) = Self::separate_early_errors(&sent_payable, &self.logger); + fn handle_sent_payables(&self, sent_payables: SentPayables) { + let (ok, err) = Self::separate_early_errors(&sent_payables, &self.logger); debug!(self.logger, "We gathered these errors at sending transactions for payable: {:?}, out of the total of {} attempts", err, ok.len() + err.len()); self.mark_pending_payable(ok); if !err.is_empty() { @@ -807,7 +823,7 @@ impl Accountant { self.discard_incomplete_transaction_with_a_failure(hash) } else {debug!(self.logger,"Forgetting a transaction attempt that even did not reach the signing stage")}) } - if let Some(response_skeleton) = &sent_payable.response_skeleton_opt { + if let Some(response_skeleton) = &sent_payables.response_skeleton_opt { self.ui_message_sub .as_ref() .expect("UIGateway is not bound") @@ -917,18 +933,8 @@ impl Accountant { }) } - fn handle_financials(&mut self, client_id: u64, context_id: u64) { - let total_unpaid_and_pending_payable = self.payable_dao.total(); - let total_paid_payable = self.financial_statistics.total_paid_payable; - let total_unpaid_receivable = self.receivable_dao.total(); - let total_paid_receivable = self.financial_statistics.total_paid_receivable; - let body = UiFinancialsResponse { - total_unpaid_and_pending_payable, - total_paid_payable, - total_unpaid_receivable, - total_paid_receivable, - } - .tmb(context_id); + fn handle_financials(&self, msg: &UiFinancialsRequest, client_id: u64, context_id: u64) { + let body: MessageBody = self.compute_financials(msg, context_id); self.ui_message_sub .as_ref() .expect("UiGateway not bound") @@ -939,6 +945,123 @@ impl Accountant { .expect("UiGateway is dead"); } + fn compute_financials(&self, msg: &UiFinancialsRequest, context_id: u64) -> MessageBody { + if let Err(message_body) = financials_entry_check(msg, context_id) { + return message_body; + }; + let stats_opt = self.process_stats(msg); + let query_results_opt = match self.process_queries_of_records(msg, context_id) { + Ok(results_opt) => results_opt, + Err(message_body) => return message_body, + }; + UiFinancialsResponse { + stats_opt, + query_results_opt, + } + .tmb(context_id) + } + + fn request_payable_accounts_by_specific_mode( + &self, + mode: CustomQuery, + ) -> Option> { + self.payable_dao + .custom_query(mode) + .map(remap_payable_accounts) + } + + fn request_receivable_accounts_by_specific_mode( + &self, + mode: CustomQuery, + ) -> Option> { + self.receivable_dao + .custom_query(mode) + .map(remap_receivable_accounts) + } + + fn process_stats(&self, msg: &UiFinancialsRequest) -> Option { + if msg.stats_required { + Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: wei_to_gwei(self.payable_dao.total()), + total_paid_payable_gwei: wei_to_gwei( + self.financial_statistics.total_paid_payable_wei, + ), + total_unpaid_receivable_gwei: wei_to_gwei(self.receivable_dao.total()), + total_paid_receivable_gwei: wei_to_gwei( + self.financial_statistics.total_paid_receivable_wei, + ), + }) + } else { + None + } + } + + fn process_top_records_query(&self, msg: &UiFinancialsRequest) -> Option { + msg.top_records_opt.map(|config| { + let payable = self + .request_payable_accounts_by_specific_mode(config.into()) + .unwrap_or_default(); + let receivable = self + .request_receivable_accounts_by_specific_mode(config.into()) + .unwrap_or_default(); + + QueryResults { + payable_opt: Some(payable), + receivable_opt: Some(receivable), + } + }) + } + + fn process_custom_queries( + &self, + msg: &UiFinancialsRequest, + context_id: u64, + ) -> Result, MessageBody> { + Ok(match msg.custom_queries_opt.as_ref() { + Some(specs) => { + let payable_opt = if let Some(query_specs) = specs.payable_opt.as_ref() { + let query = CustomQuery::from(query_specs); + check_query_is_within_tech_limits(&query, "payable", context_id)?; + self.request_payable_accounts_by_specific_mode(query) + } else { + None + }; + let receivable_opt = if let Some(query_specs) = specs.receivable_opt.as_ref() { + let query = CustomQuery::from(query_specs); + check_query_is_within_tech_limits(&query, "receivable", context_id)?; + self.request_receivable_accounts_by_specific_mode(query) + } else { + None + }; + + Some(QueryResults { + payable_opt, + receivable_opt, + }) + } + None => None, + }) + } + + fn process_queries_of_records( + &self, + msg: &UiFinancialsRequest, + context_id: u64, + ) -> Result, MessageBody> { + let top_records_opt = self.process_top_records_query(msg); + let custom_query_records_opt = match self.process_custom_queries(msg, context_id) { + Ok(query_results) => query_results, + Err(message_body) => return Err(message_body), + }; + match vec![top_records_opt, custom_query_records_opt] + .into_iter() + .find(|results| results.is_some()) + { + Some(results) => Ok(results), + None => Ok(None), + } + } + fn handle_externally_triggered_scan( &self, _ctx: &mut Context, @@ -980,7 +1103,8 @@ impl Accountant { msg.pending_payable_fingerprint.hash, e ) } else { - self.financial_statistics.total_paid_payable += msg.pending_payable_fingerprint.amount; + self.financial_statistics.total_paid_payable_wei += + msg.pending_payable_fingerprint.amount; debug!( self.logger, "Confirmation of transaction {:?}; record for payable was modified", @@ -1003,7 +1127,7 @@ impl Accountant { } fn separate_early_errors( - sent_payments: &SentPayable, + sent_payments: &SentPayables, logger: &Logger, ) -> (Vec, Vec) { sent_payments @@ -1078,7 +1202,11 @@ impl Accountant { max_pending_interval: u64, logger: &Logger, ) -> PendingTransactionStatus { - info!(logger,"Pending transaction '{:?}' couldn't be confirmed at attempt {} at {}ms after its sending",fingerprint.hash, fingerprint.attempt_opt.expectv("initialized attempt"), elapsed_in_ms(fingerprint.timestamp)); + info!(logger,"Pending transaction '{:?}' couldn't be confirmed at attempt {} at {}ms after its sending", + fingerprint.hash, + fingerprint.attempt_opt.expectv("initialized attempt"), + elapsed_in_ms(fingerprint.timestamp) + ); let elapsed = fingerprint .timestamp .elapsed() @@ -1112,7 +1240,11 @@ impl Accountant { fingerprint: &PendingPayableFingerprint, logger: &Logger, ) -> PendingTransactionStatus { - error!(logger,"Pending transaction '{:?}' announced as a failure, interpreting attempt {} after {}ms from the sending",fingerprint.hash,fingerprint.attempt_opt.expectv("initialized attempt"),elapsed_in_ms(fingerprint.timestamp)); + error!(logger,"Pending transaction '{:?}' announced as a failure, interpreting attempt {} after {}ms from the sending", + fingerprint.hash, + fingerprint.attempt_opt.expectv("initialized attempt"), + elapsed_in_ms(fingerprint.timestamp) + ); PendingTransactionStatus::Failure(fingerprint.into()) } match receipt.status{ @@ -1143,7 +1275,7 @@ impl Accountant { } } - fn process_transaction_by_status( + fn process_transactions_by_their_status( &self, statuses: Vec, ctx: &mut Context, @@ -1201,21 +1333,10 @@ impl Accountant { } } -pub fn unsigned_to_signed(unsigned: u64) -> Result { - i64::try_from(unsigned).map_err(|_| unsigned) -} - -fn elapsed_in_ms(timestamp: SystemTime) -> u128 { - timestamp - .elapsed() - .expect("time calculation for elapsed failed") - .as_millis() -} - #[derive(Debug, PartialEq, Eq, Clone)] enum PendingTransactionStatus { StillPending(PendingPayableId), //updates slightly the record, waits an interval and starts a new round - Failure(PendingPayableId), //official tx failure + Failure(PendingPayableId), //standard tx failure Confirmed(PendingPayableFingerprint), //tx was fully processed and successful } @@ -1236,43 +1357,168 @@ impl From<&PendingPayableFingerprint> for PendingPayableId { } } -//TODO the data types should change with GH-497 (including signed => unsigned) -trait PayableExceedThresholdTools { +trait PayableThresholdsGauge { fn is_innocent_age(&self, age: u64, limit: u64) -> bool; - fn is_innocent_balance(&self, balance: i64, limit: i64) -> bool; - fn calculate_payout_threshold(&self, payment_thresholds: PaymentThresholds, x: u64) -> f64; + fn is_innocent_balance(&self, balance: u128, limit: u128) -> bool; + fn calculate_payout_threshold_in_gwei( + &self, + payment_thresholds: &PaymentThresholds, + x: u64, + ) -> u128; as_any_dcl!(); } #[derive(Default)] -struct PayableExceedThresholdToolsReal {} +struct PayableThresholdsGaugeReal {} -impl PayableExceedThresholdTools for PayableExceedThresholdToolsReal { +impl PayableThresholdsGauge for PayableThresholdsGaugeReal { fn is_innocent_age(&self, age: u64, limit: u64) -> bool { age <= limit } - fn is_innocent_balance(&self, balance: i64, limit: i64) -> bool { + fn is_innocent_balance(&self, balance: u128, limit: u128) -> bool { balance <= limit } - fn calculate_payout_threshold(&self, payment_thresholds: PaymentThresholds, x: u64) -> f64 { - let m = -((payment_thresholds.debt_threshold_gwei as f64 - - payment_thresholds.permanent_debt_allowed_gwei as f64) - / (payment_thresholds.threshold_interval_sec as f64 - - payment_thresholds.maturity_threshold_sec as f64)); - let b = payment_thresholds.debt_threshold_gwei as f64 - - m * payment_thresholds.maturity_threshold_sec as f64; - m * x as f64 + b + fn calculate_payout_threshold_in_gwei( + &self, + payment_thresholds: &PaymentThresholds, + debt_age: u64, + ) -> u128 { + ThresholdUtils::calculate_finite_debt_limit_by_age(payment_thresholds, debt_age) } as_any_impl!(); } +pub struct ThresholdUtils {} + +impl ThresholdUtils { + pub fn slope(payment_thresholds: &PaymentThresholds) -> i128 { + /* + Slope is an integer, rather than a float, to improve performance. Since there are + computations that divide by the slope, it cannot be allowed to be zero; but since it's + an integer, it can't get any closer to zero than -1. + + If the numerator of this computation is less than the denominator, the slope will be + calculated as 0; therefore, .permanent_debt_allowed_gwei must be less than + .debt_threshold_gwei, so that the numerator will be no greater than -10^9 (-gwei_to_wei(1)), + and the denominator must be less than or equal to 10^9. + + These restrictions do not seem over-strict, since having .permanent_debt_allowed greater + than or equal to .debt_threshold_gwei would result in chaos, and setting + .threshold_interval_sec over 10^9 would mean continuing to declare debts delinquent after + more than 31 years. + + If payment_thresholds are ever configurable by the user, these validations should be done + on the values before they are accepted. + */ + + (gwei_to_wei::(payment_thresholds.permanent_debt_allowed_gwei) + - gwei_to_wei::(payment_thresholds.debt_threshold_gwei)) + / payment_thresholds.threshold_interval_sec as i128 + } + + fn calculate_finite_debt_limit_by_age( + payment_thresholds: &PaymentThresholds, + debt_age_s: u64, + ) -> u128 { + if Self::qualifies_for_permanent_debt_limit(debt_age_s, payment_thresholds) { + return gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei); + }; + let m = ThresholdUtils::slope(payment_thresholds); + let b = ThresholdUtils::compute_theoretical_interception_with_y_axis( + m, + payment_thresholds.maturity_threshold_sec as i128, + gwei_to_wei(payment_thresholds.debt_threshold_gwei), + ); + let y = m * debt_age_s as i128 + b; + y as u128 + } + + fn compute_theoretical_interception_with_y_axis( + m: i128, //is negative + maturity_threshold_sec: i128, + debt_threshold_wei: i128, + ) -> i128 { + debt_threshold_wei - (maturity_threshold_sec * m) + } + + fn qualifies_for_permanent_debt_limit( + debt_age_s: u64, + payment_thresholds: &PaymentThresholds, + ) -> bool { + debt_age_s + > (payment_thresholds.maturity_threshold_sec + + payment_thresholds.threshold_interval_sec) + } +} + +pub fn sign_conversion>(num: T) -> Result { + S::try_from(num).map_err(|_| num) +} + +pub fn politely_checked_conversion>(num: T) -> Result { + sign_conversion(num).map_err(|num| { + format!( + "Overflow detected with {}: cannot be converted from {} to {}", + num, + type_name::(), + type_name::() + ) + }) +} + +#[track_caller] +pub fn checked_conversion>(num: T) -> S { + politely_checked_conversion(num).unwrap_or_else(|msg| panic!("{}", msg)) +} + +pub fn gwei_to_wei + From + From, S>(gwei: S) -> T { + (T::from(gwei)).mul(T::from(WEIS_OF_GWEI as u32)) +} + +pub fn wei_to_gwei, S: Display + Copy + Div + From>(wei: S) -> T { + checked_conversion::(wei.div(S::from(WEIS_OF_GWEI as u32))) +} + +fn elapsed_in_ms(timestamp: SystemTime) -> u128 { + timestamp + .elapsed() + .expect("time calculation for elapsed failed") + .as_millis() +} + +#[cfg(test)] +pub mod check_sqlite_fns { + use super::*; + use crate::sub_lib::accountant::DEFAULT_PAYMENT_THRESHOLDS; + use actix::System; + + #[derive(Message)] + pub struct TestUserDefinedSqliteFnsForNewDelinquencies {} + + impl Handler for Accountant { + type Result = (); + + fn handle( + &mut self, + _msg: TestUserDefinedSqliteFnsForNewDelinquencies, + _ctx: &mut Self::Context, + ) -> Self::Result { + //will crash a test if our user-defined SQLite fns have been unregistered + self.receivable_dao + .new_delinquencies(SystemTime::now(), &DEFAULT_PAYMENT_THRESHOLDS); + System::current().stop(); + } + } +} + #[cfg(test)] mod tests { use super::*; use std::cell::RefCell; - use std::ops::Sub; + use std::collections::HashMap; + use std::ops::{Add, Sub}; use std::rc::Rc; use std::sync::Mutex; use std::sync::{Arc, MutexGuard}; @@ -1283,22 +1529,31 @@ mod tests { use ethereum_types::{BigEndianHash, U64}; use ethsign_crypto::Keccak256; use log::Level; - use masq_lib::constants::SCAN_ERROR; + use masq_lib::constants::{ + MASQ_TOTAL_SUPPLY, REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS, REQUEST_WITH_NO_VALUES, + SCAN_ERROR, VALUE_EXCEEDS_ALLOWED_LIMIT, + }; use web3::types::U256; - use masq_lib::messages::{ScanType, UiScanRequest, UiScanResponse}; + use masq_lib::messages::{ + CustomQueries, RangeQuery, ScanType, TopRecordsConfig, UiFinancialStatistics, + UiMessageError, UiPayableAccount, UiReceivableAccount, UiScanRequest, UiScanResponse, + }; use masq_lib::test_utils::logging::init_test_logging; use masq_lib::test_utils::logging::TestLogHandler; use masq_lib::ui_gateway::{MessageBody, MessagePath, NodeFromUiMessage, NodeToUiMessage}; + use crate::accountant::dao_utils::from_time_t; + use crate::accountant::dao_utils::{to_time_t, CustomQuery}; use crate::accountant::payable_dao::PayableDaoError; use crate::accountant::pending_payable_dao::PendingPayableDaoError; use crate::accountant::receivable_dao::ReceivableAccount; use crate::accountant::test_utils::{ - bc_from_ac_plus_earning_wallet, bc_from_ac_plus_wallets, make_pending_payable_fingerprint, - make_receivable_account, BannedDaoFactoryMock, MessageIdGeneratorMock, - PayableDaoFactoryMock, PayableDaoMock, PendingPayableDaoFactoryMock, PendingPayableDaoMock, - ReceivableDaoFactoryMock, ReceivableDaoMock, + bc_from_ac_plus_earning_wallet, bc_from_ac_plus_wallets, make_payable_account, + make_pending_payable_fingerprint, make_receivable_account, BannedDaoFactoryMock, + MessageIdGeneratorMock, PayableDaoFactoryMock, PayableDaoMock, + PendingPayableDaoFactoryMock, PendingPayableDaoMock, ReceivableDaoFactoryMock, + ReceivableDaoMock, }; use crate::accountant::test_utils::{AccountantBuilder, BannedDaoMock}; use crate::accountant::tools::accountant_tools::{NullScanner, ReceivablesScanner}; @@ -1309,8 +1564,6 @@ mod tests { use crate::blockchain::test_utils::BlockchainInterfaceMock; use crate::blockchain::tool_wrappers::SendTransactionToolsWrapperNull; use crate::bootstrapper::BootstrapperConfig; - use crate::database::dao_utils::from_time_t; - use crate::database::dao_utils::to_time_t; use crate::sub_lib::accountant::{ ExitServiceConsumed, RoutingServiceConsumed, ScanIntervals, DEFAULT_PAYMENT_THRESHOLDS, }; @@ -1326,19 +1579,21 @@ mod tests { SystemKillerActor, }; use crate::test_utils::{make_paying_wallet, make_wallet}; + use masq_lib::messages::TopRecordsOrdering::{Age, Balance}; + use masq_lib::ui_gateway::MessagePath::Conversation; use web3::types::{TransactionReceipt, H256}; #[derive(Default)] - struct PayableThresholdToolsMock { + struct PayableThresholdsGaugeMock { is_innocent_age_params: Arc>>, is_innocent_age_results: RefCell>, - is_innocent_balance_params: Arc>>, + is_innocent_balance_params: Arc>>, is_innocent_balance_results: RefCell>, - calculate_payout_threshold_params: Arc>>, - calculate_payout_threshold_results: RefCell>, + calculate_payout_threshold_in_gwei_params: Arc>>, + calculate_payout_threshold_in_gwei_results: RefCell>, } - impl PayableExceedThresholdTools for PayableThresholdToolsMock { + impl PayableThresholdsGauge for PayableThresholdsGaugeMock { fn is_innocent_age(&self, age: u64, limit: u64) -> bool { self.is_innocent_age_params .lock() @@ -1347,7 +1602,7 @@ mod tests { self.is_innocent_age_results.borrow_mut().remove(0) } - fn is_innocent_balance(&self, balance: i64, limit: i64) -> bool { + fn is_innocent_balance(&self, balance: u128, limit: u128) -> bool { self.is_innocent_balance_params .lock() .unwrap() @@ -1355,18 +1610,22 @@ mod tests { self.is_innocent_balance_results.borrow_mut().remove(0) } - fn calculate_payout_threshold(&self, payment_thresholds: PaymentThresholds, x: u64) -> f64 { - self.calculate_payout_threshold_params + fn calculate_payout_threshold_in_gwei( + &self, + payment_thresholds: &PaymentThresholds, + x: u64, + ) -> u128 { + self.calculate_payout_threshold_in_gwei_params .lock() .unwrap() - .push((payment_thresholds, x)); - self.calculate_payout_threshold_results + .push((*payment_thresholds, x)); + self.calculate_payout_threshold_in_gwei_results .borrow_mut() .remove(0) } } - impl PayableThresholdToolsMock { + impl PayableThresholdsGaugeMock { fn is_innocent_age_params(mut self, params: &Arc>>) -> Self { self.is_innocent_age_params = params.clone(); self @@ -1377,7 +1636,7 @@ mod tests { self } - fn is_innocent_balance_params(mut self, params: &Arc>>) -> Self { + fn is_innocent_balance_params(mut self, params: &Arc>>) -> Self { self.is_innocent_balance_params = params.clone(); self } @@ -1387,16 +1646,16 @@ mod tests { self } - fn calculate_payout_threshold_params( + fn calculate_payout_threshold_in_gwei_params( mut self, params: &Arc>>, ) -> Self { - self.calculate_payout_threshold_params = params.clone(); + self.calculate_payout_threshold_in_gwei_params = params.clone(); self } - fn calculate_payout_threshold_result(self, result: f64) -> Self { - self.calculate_payout_threshold_results + fn calculate_payout_threshold_in_gwei_result(self, result: u128) -> Self { + self.calculate_payout_threshold_in_gwei_results .borrow_mut() .push(result); self @@ -1502,13 +1761,13 @@ mod tests { .downcast_ref::() .unwrap(); result - .payable_threshold_tools + .payable_threshold_gauge .as_any() - .downcast_ref::() + .downcast_ref::() .unwrap(); assert_eq!(result.crashable, false); - assert_eq!(result.financial_statistics.total_paid_receivable, 0); - assert_eq!(result.financial_statistics.total_paid_payable, 0); + assert_eq!(result.financial_statistics.total_paid_receivable_wei, 0); + assert_eq!(result.financial_statistics.total_paid_payable_wei, 0); result .message_id_generator .as_any() @@ -1633,7 +1892,7 @@ mod tests { ); let payable_account = PayableAccount { wallet: make_wallet("wallet"), - balance: DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 1, + balance_wei: gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 1), last_paid_timestamp: SystemTime::now().sub(Duration::from_secs( (DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + 1) as u64, )), @@ -1700,7 +1959,7 @@ mod tests { let system = System::new("test"); let peer_actors = peer_actors_builder().ui_gateway(ui_gateway).build(); subject_addr.try_send(BindMessage { peer_actors }).unwrap(); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![], response_skeleton_opt: Some(ResponseSkeleton { @@ -1859,7 +2118,7 @@ mod tests { expected_hash.clone(), expected_timestamp, ); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![Ok(expected_payable.clone())], response_skeleton_opt: None, @@ -1890,7 +2149,7 @@ mod tests { .pending_payable_dao(pending_payable_dao) .build(); let hash = H256::from_uint(&U256::from(12345)); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![Err(BlockchainError::TransactionFailed { msg: "SQLite migraine".to_string(), @@ -1947,7 +2206,7 @@ mod tests { let wallet = make_wallet("blah"); let hash_tx_1 = H256::from_uint(&U256::from(5555)); let hash_tx_2 = H256::from_uint(&U256::from(12345)); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![ Ok(Payable { @@ -2002,20 +2261,24 @@ mod tests { let accounts = vec![ PayableAccount { wallet: make_wallet("blah"), - balance: DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 55, + balance_wei: gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 55), last_paid_timestamp: from_time_t( to_time_t(SystemTime::now()) - - DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + - checked_conversion::( + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec, + ) - 5, ), pending_payable_opt: None, }, PayableAccount { wallet: make_wallet("foo"), - balance: DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 66, + balance_wei: gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 66), last_paid_timestamp: from_time_t( to_time_t(SystemTime::now()) - - DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + - checked_conversion::( + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec, + ) - 500, ), pending_payable_opt: None, @@ -2113,12 +2376,12 @@ mod tests { let expected_receivable_1 = BlockchainTransaction { block_number: 7, from: make_wallet("wallet0"), - gwei_amount: 456, + wei_amount: 456, }; let expected_receivable_2 = BlockchainTransaction { block_number: 13, from: make_wallet("wallet1"), - gwei_amount: 10000, + wei_amount: 10000, }; let more_money_received_params_arc = Arc::new(Mutex::new(vec![])); let receivable_dao = ReceivableDaoMock::new() @@ -2257,7 +2520,7 @@ mod tests { ); let new_delinquent_account = ReceivableAccount { wallet: wallet_to_be_banned.clone(), - balance: 4567, + balance_wei: 4567, last_received_timestamp: from_time_t(200_000_000), }; let system = System::new("periodical_scanning_for_receivables_and_delinquencies_works"); @@ -2478,9 +2741,11 @@ mod tests { // slightly above minimum balance, to the right of the curve (time intersection) let account = PayableAccount { wallet: make_wallet("wallet"), - balance: DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 5, + balance_wei: gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 5), last_paid_timestamp: from_time_t( - now - DEFAULT_PAYMENT_THRESHOLDS.threshold_interval_sec - 10, + now - checked_conversion::( + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + 10, + ), ), pending_payable_opt: None, }; @@ -2488,7 +2753,7 @@ mod tests { .non_pending_payables_params(&non_pending_payables_params_arc) .non_pending_payables_result(vec![]) .non_pending_payables_result(vec![account.clone()]); - payable_dao.have_non_pending_payables_shut_down_the_system = true; + payable_dao.have_non_pending_payable_shut_down_the_system = true; let peer_actors = peer_actors_builder() .blockchain_bridge(blockchain_bridge) .build(); @@ -2607,27 +2872,33 @@ mod tests { // below minimum balance, to the right of time intersection (inside buffer zone) PayableAccount { wallet: make_wallet("wallet0"), - balance: payment_thresholds.permanent_debt_allowed_gwei - 1, + balance_wei: gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei - 1), last_paid_timestamp: from_time_t( - now - payment_thresholds.threshold_interval_sec - 10, + now - checked_conversion::( + payment_thresholds.threshold_interval_sec + 10, + ), ), pending_payable_opt: None, }, - // above balance intersection, to the left of minimum time (inside buffer zone) + // above balance intersection, to the left of minimum time (outside buffer zone) PayableAccount { wallet: make_wallet("wallet1"), - balance: payment_thresholds.debt_threshold_gwei + 1, + balance_wei: gwei_to_wei(payment_thresholds.debt_threshold_gwei + 1), last_paid_timestamp: from_time_t( - now - payment_thresholds.maturity_threshold_sec + 10, + now - checked_conversion::( + payment_thresholds.maturity_threshold_sec - 10, + ), ), pending_payable_opt: None, }, // above minimum balance, to the right of minimum time (not in buffer zone, below the curve) PayableAccount { wallet: make_wallet("wallet2"), - balance: payment_thresholds.debt_threshold_gwei - 1000, + balance_wei: gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei + 55), last_paid_timestamp: from_time_t( - now - payment_thresholds.maturity_threshold_sec - 1, + now - checked_conversion::( + payment_thresholds.maturity_threshold_sec + 15, + ), ), pending_payable_opt: None, }, @@ -2678,18 +2949,26 @@ mod tests { // slightly above minimum balance, to the right of the curve (time intersection) PayableAccount { wallet: make_wallet("wallet0"), - balance: DEFAULT_PAYMENT_THRESHOLDS.permanent_debt_allowed_gwei + 1, + balance_wei: gwei_to_wei( + DEFAULT_PAYMENT_THRESHOLDS.permanent_debt_allowed_gwei + 1, + ), last_paid_timestamp: from_time_t( - now - DEFAULT_PAYMENT_THRESHOLDS.threshold_interval_sec - 10, + now - checked_conversion::( + DEFAULT_PAYMENT_THRESHOLDS.threshold_interval_sec + + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + + 10, + ), ), pending_payable_opt: None, }, // slightly above the curve (balance intersection), to the right of minimum time PayableAccount { wallet: make_wallet("wallet1"), - balance: DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 1, + balance_wei: gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 1), last_paid_timestamp: from_time_t( - now - DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec - 10, + now - checked_conversion::( + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + 10, + ), ), pending_payable_opt: None, }, @@ -2697,7 +2976,7 @@ mod tests { let mut payable_dao = PayableDaoMock::default() .non_pending_payables_result(accounts.clone()) .non_pending_payables_result(vec![]); - payable_dao.have_non_pending_payables_shut_down_the_system = true; + payable_dao.have_non_pending_payable_shut_down_the_system = true; let (blockchain_bridge, _, blockchain_bridge_recordings_arc) = make_recorder(); let system = System::new("scan_for_payable_message_triggers_payment_for_balances_over_the_curve"); @@ -3255,7 +3534,7 @@ mod tests { fn assert_that_we_do_not_charge_our_own_wallet_for_consumed_services( config: BootstrapperConfig, message: ReportServicesConsumedMessage, - ) -> Arc>> { + ) -> Arc>> { let more_money_payable_parameters_arc = Arc::new(Mutex::new(vec![])); let payable_dao_mock = PayableDaoMock::new() .non_pending_payables_result(vec![]) @@ -3546,7 +3825,7 @@ mod tests { ) { let rowid = 4; let hash = H256::from_uint(&U256::from(123)); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![Err(BlockchainError::TransactionFailed { msg: "blah".to_string(), @@ -3563,7 +3842,7 @@ mod tests { .pending_payable_dao(pending_payable_dao) .build(); - let _ = subject.handle_sent_payable(sent_payable); + let _ = subject.handle_sent_payables(sent_payable); } #[test] @@ -3580,7 +3859,7 @@ mod tests { msg: "closing hours, sorry".to_string(), hash_opt: None, }); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![payable_1, Ok(payable_2.clone()), payable_3], response_skeleton_opt: None, @@ -3593,7 +3872,7 @@ mod tests { .pending_payable_dao(pending_payable_dao) .build(); - subject.handle_sent_payable(sent_payable); + subject.handle_sent_payables(sent_payable); let fingerprint_rowid_params = fingerprint_rowid_params_arc.lock().unwrap(); assert_eq!(*fingerprint_rowid_params, vec![payable_hash_2]); //we know the other two errors are associated with an initiated transaction having a backup @@ -3794,27 +4073,27 @@ mod tests { let payables = &[ PayableAccount { wallet: make_wallet("wallet0"), - balance: same_amount_significance, + balance_wei: same_amount_significance, last_paid_timestamp: from_time_t(now - 5000), pending_payable_opt: None, }, //this debt is more significant because beside being high in amount it's also older, so should be prioritized and picked PayableAccount { wallet: make_wallet("wallet1"), - balance: same_amount_significance, + balance_wei: same_amount_significance, last_paid_timestamp: from_time_t(now - 10000), pending_payable_opt: None, }, //similarly these two wallets have debts equally old but the second has a bigger balance and should be chosen PayableAccount { wallet: make_wallet("wallet3"), - balance: 100, + balance_wei: 100, last_paid_timestamp: same_age_significance, pending_payable_opt: None, }, PayableAccount { wallet: make_wallet("wallet2"), - balance: 330, + balance_wei: 330, last_paid_timestamp: same_age_significance, pending_payable_opt: None, }, @@ -3827,6 +4106,7 @@ mod tests { #[test] fn payables_debug_summary_prints_pretty_summary() { + init_test_logging(); let now = to_time_t(SystemTime::now()); let payment_thresholds = PaymentThresholds { threshold_interval_sec: 2_592_000, @@ -3839,17 +4119,22 @@ mod tests { let qualified_payables = &[ PayableAccount { wallet: make_wallet("wallet0"), - balance: payment_thresholds.permanent_debt_allowed_gwei + 1000, + balance_wei: gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei + 2000), last_paid_timestamp: from_time_t( - now - payment_thresholds.threshold_interval_sec - 1234, + now - checked_conversion::( + payment_thresholds.maturity_threshold_sec + + payment_thresholds.threshold_interval_sec, + ), ), pending_payable_opt: None, }, PayableAccount { wallet: make_wallet("wallet1"), - balance: payment_thresholds.permanent_debt_allowed_gwei + 1, + balance_wei: gwei_to_wei(payment_thresholds.debt_threshold_gwei - 1), last_paid_timestamp: from_time_t( - now - payment_thresholds.threshold_interval_sec - 1, + now - checked_conversion::( + payment_thresholds.maturity_threshold_sec + 55, + ), ), pending_payable_opt: None, }, @@ -3861,105 +4146,492 @@ mod tests { .build(); subject.config.payment_thresholds = payment_thresholds; - let result = subject.payables_debug_summary(qualified_payables); + subject.payables_debug_summary(qualified_payables); + + TestLogHandler::new().exists_log_containing("Paying qualified debts:\n\ + 10,002,000,000,000,000 wei owed for 2678400 sec exceeds threshold: 10,000,000,001,152,000 wei; creditor: 0x0000000000000000000000000077616c6c657430\n\ + 999,999,999,000,000,000 wei owed for 86455 sec exceeds threshold: 999,978,993,055,555,580 wei; creditor: 0x0000000000000000000000000077616c6c657431"); + } + + #[test] + fn payables_debug_summary_stays_still_if_no_qualified_payments() { + init_test_logging(); + let mut subject = AccountantBuilder::default().build(); + subject.logger = Logger::new("payables_debug_summary_stays_still_if_no_qualified_payments"); + + subject.payables_debug_summary(&vec![]); + + TestLogHandler::new().exists_no_log_containing("DEBUG: payables_debug_summary_prints_nothing_if_no_qualified_payments: Paying qualified debts:"); + } + + #[test] + fn payout_sloped_segment_in_payment_thresholds_goes_along_proper_line() { + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 333, + payment_grace_period_sec: 444, + permanent_debt_allowed_gwei: 4444, + debt_threshold_gwei: 8888, + threshold_interval_sec: 1111111, + unban_below_gwei: 0, + }; + let higher_corner_timestamp = payment_thresholds.maturity_threshold_sec; + let middle_point_timestamp = payment_thresholds.maturity_threshold_sec + + payment_thresholds.threshold_interval_sec / 2; + let lower_corner_timestamp = + payment_thresholds.maturity_threshold_sec + payment_thresholds.threshold_interval_sec; + let tested_fn = |payment_thresholds: &PaymentThresholds, time| { + PayableThresholdsGaugeReal {} + .calculate_payout_threshold_in_gwei(payment_thresholds, time) as i128 + }; - assert_eq!(result, - "Paying qualified debts:\n\ - 10001000 owed for 2593234sec exceeds threshold: 9512428; creditor: 0x0000000000000000000000000077616c6c657430\n\ - 10000001 owed for 2592001sec exceeds threshold: 9999604; creditor: 0x0000000000000000000000000077616c6c657431" + let higher_corner_point = tested_fn(&payment_thresholds, higher_corner_timestamp); + let middle_point = tested_fn(&payment_thresholds, middle_point_timestamp); + let lower_corner_point = tested_fn(&payment_thresholds, lower_corner_timestamp); + + let allowed_imprecision = WEIS_OF_GWEI; + let ideal_template_higher: i128 = gwei_to_wei(payment_thresholds.debt_threshold_gwei); + let ideal_template_middle: i128 = gwei_to_wei( + (payment_thresholds.debt_threshold_gwei + - payment_thresholds.permanent_debt_allowed_gwei) + / 2 + + payment_thresholds.permanent_debt_allowed_gwei, + ); + let ideal_template_lower: i128 = + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei); + assert!( + higher_corner_point <= ideal_template_higher + allowed_imprecision + && ideal_template_higher - allowed_imprecision <= higher_corner_point, + "ideal: {}, real: {}", + ideal_template_higher, + higher_corner_point + ); + assert!( + middle_point <= ideal_template_middle + allowed_imprecision + && ideal_template_middle - allowed_imprecision <= middle_point, + "ideal: {}, real: {}", + ideal_template_middle, + middle_point + ); + assert!( + lower_corner_point <= ideal_template_lower + allowed_imprecision + && ideal_template_lower - allowed_imprecision <= lower_corner_point, + "ideal: {}, real: {}", + ideal_template_lower, + lower_corner_point ) } + fn gap_tester(payment_thresholds: &PaymentThresholds) -> (u64, u64) { + let mut counts_of_unique_elements: HashMap = HashMap::new(); + (1_u64..20) + .map(|to_add| { + ThresholdUtils::calculate_finite_debt_limit_by_age( + &payment_thresholds, + 1500 + to_add, + ) as u64 + }) + .for_each(|point_height| { + counts_of_unique_elements + .entry(point_height) + .and_modify(|q| *q += 1) + .or_insert(1); + }); + + let mut heights_and_counts = counts_of_unique_elements.drain().collect::>(); + heights_and_counts.sort_by_key(|(height, _)| (u64::MAX - height)); + let mut counts_of_groups_of_the_same_size: HashMap = HashMap::new(); + let mut previous_height = + ThresholdUtils::calculate_finite_debt_limit_by_age(&payment_thresholds, 1500) as u64; + heights_and_counts + .into_iter() + .for_each(|(point_height, unique_count)| { + let height_change = if point_height <= previous_height { + previous_height - point_height + } else { + panic!("unexpected trend; previously: {previous_height}, now: {point_height}") + }; + counts_of_groups_of_the_same_size + .entry(unique_count) + .and_modify(|(_height_change, occurrence_so_far)| *occurrence_so_far += 1) + .or_insert((height_change, 1)); + previous_height = point_height; + }); + + let mut sortable = counts_of_groups_of_the_same_size + .drain() + .collect::>(); + sortable.sort_by_key(|(_key, (_height_change, occurrence))| *occurrence); + + let (number_of_seconds_detected, (height_change, occurrence)) = + sortable.last().expect("no values to analyze"); + //checking if the sample of undistorted results (consist size groups) has enough weight compared to 20 tries from the beginning + if number_of_seconds_detected * occurrence >= 15 { + (*number_of_seconds_detected as u64, *height_change) + } else { + panic!("couldn't provide a relevant amount of data for the analysis") + } + } + + fn assert_on_height_granularity_with_advancing_time( + description_of_given_pt: &str, + payment_thresholds: &PaymentThresholds, + expected_height_change_wei: u64, + ) { + const WE_EXPECT_ALWAYS_JUST_ONE_SECOND_TO_CHANGE_THE_HEIGHT: u64 = 1; + let (seconds_needed_for_smallest_change_in_height, absolute_height_change_wei) = + gap_tester(&payment_thresholds); + assert_eq!(seconds_needed_for_smallest_change_in_height, WE_EXPECT_ALWAYS_JUST_ONE_SECOND_TO_CHANGE_THE_HEIGHT, + "while testing {} we expected that these thresholds: {:?} will require only {} s until we see the height change but computed {} s instead", + description_of_given_pt, payment_thresholds, WE_EXPECT_ALWAYS_JUST_ONE_SECOND_TO_CHANGE_THE_HEIGHT, seconds_needed_for_smallest_change_in_height); + assert_eq!(absolute_height_change_wei, expected_height_change_wei, + "while testing {} we expected that these thresholds: {:?} will cause a height change of {} wei as a result of advancement in time by {} s but the true result is {}", + description_of_given_pt, payment_thresholds, expected_height_change_wei, seconds_needed_for_smallest_change_in_height, absolute_height_change_wei) + } + #[test] - fn threshold_calculation_depends_on_user_defined_payment_thresholds() { - let safe_age_params_arc = Arc::new(Mutex::new(vec![])); - let safe_balance_params_arc = Arc::new(Mutex::new(vec![])); - let calculate_payable_threshold_params_arc = Arc::new(Mutex::new(vec![])); - let balance = 5555; - let how_far_in_past = Duration::from_secs(1111 + 1); - let last_paid_timestamp = SystemTime::now().sub(how_far_in_past); - let payable_account = PayableAccount { - wallet: make_wallet("hi"), - balance, - last_paid_timestamp, - pending_payable_opt: None, + fn testing_granularity_calculate_sloped_threshold_by_time() { + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 1000, + payment_grace_period_sec: 0, + permanent_debt_allowed_gwei: 100, + debt_threshold_gwei: 10_000, + threshold_interval_sec: 10_000, + unban_below_gwei: 100, }; - let custom_payment_thresholds = PaymentThresholds { - maturity_threshold_sec: 1111, - payment_grace_period_sec: 2222, - permanent_debt_allowed_gwei: 3333, - debt_threshold_gwei: 4444, - threshold_interval_sec: 5555, - unban_below_gwei: 3333, + + assert_on_height_granularity_with_advancing_time( + "135° slope", + &payment_thresholds, + 990_000_000, + ); + + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 1000, + payment_grace_period_sec: 0, + permanent_debt_allowed_gwei: 100, + debt_threshold_gwei: 3_420, + threshold_interval_sec: 10_000, + unban_below_gwei: 100, }; - let mut bootstrapper_config = BootstrapperConfig::default(); - bootstrapper_config.accountant_config_opt = Some(AccountantConfig { - scan_intervals: Default::default(), - payment_thresholds: custom_payment_thresholds, - suppress_initial_scans: false, - when_pending_too_long_sec: DEFAULT_PENDING_TOO_LONG_SEC, - }); - let payable_thresholds_tools = PayableThresholdToolsMock::default() - .is_innocent_age_params(&safe_age_params_arc) - .is_innocent_age_result( - how_far_in_past.as_secs() - <= custom_payment_thresholds.maturity_threshold_sec as u64, - ) - .is_innocent_balance_params(&safe_balance_params_arc) - .is_innocent_balance_result( - balance <= custom_payment_thresholds.permanent_debt_allowed_gwei, - ) - .calculate_payout_threshold_params(&calculate_payable_threshold_params_arc) - .calculate_payout_threshold_result(4567.0); //made up value - let mut subject = AccountantBuilder::default() - .bootstrapper_config(bootstrapper_config) - .build(); - subject.payable_threshold_tools = Box::new(payable_thresholds_tools); - let result = subject.payable_exceeded_threshold(&payable_account); + assert_on_height_granularity_with_advancing_time( + "160° slope", + &payment_thresholds, + 332_000_000, + ); - assert_eq!(result, Some(4567)); - let mut safe_age_params = safe_age_params_arc.lock().unwrap(); - let safe_age_single_params = safe_age_params.remove(0); - assert_eq!(*safe_age_params, vec![]); - let (time_elapsed, curve_derived_time) = safe_age_single_params; - assert!( - (how_far_in_past.as_secs() - 3) < time_elapsed - && time_elapsed < (how_far_in_past.as_secs() + 3) + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 1000, + payment_grace_period_sec: 0, + permanent_debt_allowed_gwei: 100, + debt_threshold_gwei: 875, + threshold_interval_sec: 10_000, + unban_below_gwei: 100, + }; + + assert_on_height_granularity_with_advancing_time( + "175° slope", + &payment_thresholds, + 77_500_000, ); - assert_eq!( - curve_derived_time, - custom_payment_thresholds.maturity_threshold_sec as u64 + } + + #[test] + fn checking_chosen_values_for_the_payment_thresholds_defaults_on_height_values_granularity() { + let payment_thresholds = *DEFAULT_PAYMENT_THRESHOLDS; + + assert_on_height_granularity_with_advancing_time( + "default thresholds", + &payment_thresholds, + 23_148_148_148_148, ); - let safe_balance_params = safe_balance_params_arc.lock().unwrap(); - assert_eq!( - *safe_balance_params, - vec![( - payable_account.balance, - custom_payment_thresholds.permanent_debt_allowed_gwei - )] + } + + #[test] + fn slope_has_loose_enough_limitations_to_allow_work_with_number_bigger_than_masq_token_max_supply( + ) { + //max masq token supply by August 2022: 37,500,000 + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 20, + payment_grace_period_sec: 33, + permanent_debt_allowed_gwei: 1, + debt_threshold_gwei: MASQ_TOTAL_SUPPLY * WEIS_OF_GWEI as u64, + threshold_interval_sec: 1, + unban_below_gwei: 0, + }; + + let slope = ThresholdUtils::slope(&payment_thresholds); + + assert_eq!(slope, -37499999999999999000000000); + let check = { + let y_interception = ThresholdUtils::compute_theoretical_interception_with_y_axis( + slope, + payment_thresholds.maturity_threshold_sec as i128, + gwei_to_wei(payment_thresholds.debt_threshold_gwei), + ); + slope * (payment_thresholds.maturity_threshold_sec + 1) as i128 + y_interception + }; + assert_eq!(check, WEIS_OF_GWEI) + } + + #[test] + fn slope_after_its_end_turns_into_permanent_debt_allowed() { + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 1000, + payment_grace_period_sec: 444, + permanent_debt_allowed_gwei: 44, + debt_threshold_gwei: 8888, + threshold_interval_sec: 11111, + unban_below_gwei: 0, + }; + + let right_at_the_end = ThresholdUtils::calculate_finite_debt_limit_by_age( + &payment_thresholds, + payment_thresholds.maturity_threshold_sec + + payment_thresholds.threshold_interval_sec + + 1, ); - let mut calculate_payable_curves_params = - calculate_payable_threshold_params_arc.lock().unwrap(); - let calculate_payable_curves_single_params = calculate_payable_curves_params.remove(0); - assert_eq!(*calculate_payable_curves_params, vec![]); - let (payment_thresholds, time_elapsed) = calculate_payable_curves_single_params; - assert!( - (how_far_in_past.as_secs() - 3) < time_elapsed - && time_elapsed < (how_far_in_past.as_secs() + 3) + let a_certain_distance_further = ThresholdUtils::calculate_finite_debt_limit_by_age( + &payment_thresholds, + payment_thresholds.maturity_threshold_sec + + payment_thresholds.threshold_interval_sec + + 1234, ); - assert_eq!(payment_thresholds, custom_payment_thresholds) + + assert_eq!( + right_at_the_end, + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei) + ); + assert_eq!( + a_certain_distance_further, + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei) + ) } #[test] - fn pending_transaction_is_registered_and_monitored_until_it_gets_confirmed_or_canceled() { - init_test_logging(); - let mark_pending_payable_params_arc = Arc::new(Mutex::new(vec![])); - let transaction_confirmed_params_arc = Arc::new(Mutex::new(vec![])); - let get_transaction_receipt_params_arc = Arc::new(Mutex::new(vec![])); - let return_all_fingerprints_params_arc = Arc::new(Mutex::new(vec![])); - let non_pending_payables_params_arc = Arc::new(Mutex::new(vec![])); - let insert_fingerprint_params_arc = Arc::new(Mutex::new(vec![])); + fn is_innocent_age_works_for_age_smaller_than_innocent_age() { + let payable_age = 999; + + let result = PayableThresholdsGaugeReal::default().is_innocent_age(payable_age, 1000); + + assert_eq!(result, true) + } + + #[test] + fn is_innocent_age_works_for_age_equal_to_innocent_age() { + let payable_age = 1000; + + let result = PayableThresholdsGaugeReal::default().is_innocent_age(payable_age, 1000); + + assert_eq!(result, true) + } + + #[test] + fn is_innocent_age_works_for_excessive_age() { + let payable_age = 1001; + + let result = PayableThresholdsGaugeReal::default().is_innocent_age(payable_age, 1000); + + assert_eq!(result, false) + } + + #[test] + fn is_innocent_balance_works_for_balance_smaller_than_innocent_balance() { + let payable_balance = 999; + + let result = + PayableThresholdsGaugeReal::default().is_innocent_balance(payable_balance, 1000); + + assert_eq!(result, true) + } + + #[test] + fn is_innocent_balance_works_for_balance_equal_to_innocent_balance() { + let payable_balance = 1000; + + let result = + PayableThresholdsGaugeReal::default().is_innocent_balance(payable_balance, 1000); + + assert_eq!(result, true) + } + + #[test] + fn is_innocent_balance_works_for_excessive_balance() { + let payable_balance = 1001; + + let result = + PayableThresholdsGaugeReal::default().is_innocent_balance(payable_balance, 1000); + + assert_eq!(result, false) + } + + #[test] + fn payable_is_found_innocent_by_age_and_returns() { + let is_innocent_age_params_arc = Arc::new(Mutex::new(vec![])); + let payable_thresholds_gauge = PayableThresholdsGaugeMock::default() + .is_innocent_age_params(&is_innocent_age_params_arc) + .is_innocent_age_result(true); + let mut subject = AccountantBuilder::default().build(); + subject.payable_threshold_gauge = Box::new(payable_thresholds_gauge); + let last_paid_timestamp = SystemTime::now() + .checked_sub(Duration::from_secs(123456)) + .unwrap(); + let mut payable = make_payable_account(111); + payable.last_paid_timestamp = last_paid_timestamp; + let before = SystemTime::now(); + + let result = subject.payable_exceeded_threshold(&payable); + + let after = SystemTime::now(); + assert_eq!(result, None); + let mut is_innocent_age_params = is_innocent_age_params_arc.lock().unwrap(); + let (debt_age, threshold_value) = is_innocent_age_params.remove(0); + assert!(is_innocent_age_params.is_empty()); + let time_elapsed_before = before + .duration_since(last_paid_timestamp) + .unwrap() + .as_secs(); + let time_elapsed_after = after.duration_since(last_paid_timestamp).unwrap().as_secs(); + assert!(time_elapsed_before <= debt_age && debt_age <= time_elapsed_after); + assert_eq!( + threshold_value, + DEFAULT_PAYMENT_THRESHOLDS.maturity_threshold_sec + ) + //no other method was called (absence of panic) and that means we returned early + } + + #[test] + fn payable_is_found_innocent_by_balance_and_returns() { + let is_innocent_age_params_arc = Arc::new(Mutex::new(vec![])); + let is_innocent_balance_params_arc = Arc::new(Mutex::new(vec![])); + let payable_thresholds_gauge = PayableThresholdsGaugeMock::default() + .is_innocent_age_params(&is_innocent_age_params_arc) + .is_innocent_age_result(false) + .is_innocent_balance_params(&is_innocent_balance_params_arc) + .is_innocent_balance_result(true); + let mut subject = AccountantBuilder::default().build(); + subject.payable_threshold_gauge = Box::new(payable_thresholds_gauge); + let last_paid_timestamp = SystemTime::now() + .checked_sub(Duration::from_secs(111111)) + .unwrap(); + let mut payable = make_payable_account(222); + payable.last_paid_timestamp = last_paid_timestamp; + payable.balance_wei = 123456; + let before = SystemTime::now(); + + let result = subject.payable_exceeded_threshold(&payable); + + let after = SystemTime::now(); + assert_eq!(result, None); + let mut is_innocent_age_params = is_innocent_age_params_arc.lock().unwrap(); + let (debt_age, _) = is_innocent_age_params.remove(0); + assert!(is_innocent_age_params.is_empty()); + let time_elapsed_before = before + .duration_since(last_paid_timestamp) + .unwrap() + .as_secs(); + let time_elapsed_after = after.duration_since(last_paid_timestamp).unwrap().as_secs(); + assert!(time_elapsed_before <= debt_age && debt_age <= time_elapsed_after); + let is_innocent_balance_params = is_innocent_balance_params_arc.lock().unwrap(); + assert_eq!( + *is_innocent_balance_params, + vec![( + 123456_u128, + gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.permanent_debt_allowed_gwei) + )] + ) + //no other method was called (absence of panic) and that means we returned early + } + + #[test] + fn threshold_calculation_depends_on_user_defined_payment_thresholds() { + let is_innocent_age_params_arc = Arc::new(Mutex::new(vec![])); + let is_innocent_balance_params_arc = Arc::new(Mutex::new(vec![])); + let calculate_payable_threshold_params_arc = Arc::new(Mutex::new(vec![])); + let balance = gwei_to_wei(5555_u64); + let how_far_in_past = Duration::from_secs(1111 + 1); + let last_paid_timestamp = SystemTime::now().sub(how_far_in_past); + let payable_account = PayableAccount { + wallet: make_wallet("hi"), + balance_wei: balance, + last_paid_timestamp, + pending_payable_opt: None, + }; + let custom_payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 1111, + payment_grace_period_sec: 2222, + permanent_debt_allowed_gwei: 3333, + debt_threshold_gwei: 4444, + threshold_interval_sec: 5555, + unban_below_gwei: 5555, + }; + let mut bootstrapper_config = BootstrapperConfig::default(); + bootstrapper_config.accountant_config_opt = Some(AccountantConfig { + scan_intervals: Default::default(), + payment_thresholds: custom_payment_thresholds, + suppress_initial_scans: false, + when_pending_too_long_sec: DEFAULT_PENDING_TOO_LONG_SEC, + }); + let payable_thresholds_gauge = PayableThresholdsGaugeMock::default() + .is_innocent_age_params(&is_innocent_age_params_arc) + .is_innocent_age_result( + how_far_in_past.as_secs() + <= custom_payment_thresholds.maturity_threshold_sec as u64, + ) + .is_innocent_balance_params(&is_innocent_balance_params_arc) + .is_innocent_balance_result( + balance <= gwei_to_wei(custom_payment_thresholds.permanent_debt_allowed_gwei), + ) + .calculate_payout_threshold_in_gwei_params(&calculate_payable_threshold_params_arc) + .calculate_payout_threshold_in_gwei_result(4567898); //made up value + let mut subject = AccountantBuilder::default() + .bootstrapper_config(bootstrapper_config) + .build(); + subject.payable_threshold_gauge = Box::new(payable_thresholds_gauge); + let before = SystemTime::now(); + + let result = subject.payable_exceeded_threshold(&payable_account); + + let after = SystemTime::now(); + assert_eq!(result, Some(4567898)); + let mut is_innocent_age_params = is_innocent_age_params_arc.lock().unwrap(); + let (time_elapsed, curve_derived_time) = is_innocent_age_params.remove(0); + assert_eq!(*is_innocent_age_params, vec![]); + let time_elapsed_before = before + .duration_since(last_paid_timestamp) + .unwrap() + .as_secs(); + let time_elapsed_after = after.duration_since(last_paid_timestamp).unwrap().as_secs(); + assert!(time_elapsed_before <= time_elapsed && time_elapsed <= time_elapsed_after); + assert_eq!( + curve_derived_time, + custom_payment_thresholds.maturity_threshold_sec as u64 + ); + let is_innocent_balance_params = is_innocent_balance_params_arc.lock().unwrap(); + assert_eq!( + *is_innocent_balance_params, + vec![( + payable_account.balance_wei, + gwei_to_wei(custom_payment_thresholds.permanent_debt_allowed_gwei) + )] + ); + let mut calculate_payable_curves_params = + calculate_payable_threshold_params_arc.lock().unwrap(); + let (payment_thresholds, time_elapsed) = calculate_payable_curves_params.remove(0); + assert_eq!(*calculate_payable_curves_params, vec![]); + assert!(time_elapsed_before <= time_elapsed && time_elapsed <= time_elapsed_after); + assert_eq!(payment_thresholds, custom_payment_thresholds) + } + + #[test] + fn pending_transaction_is_registered_and_monitored_until_it_gets_confirmed_or_canceled() { + init_test_logging(); + let mark_pending_payable_params_arc = Arc::new(Mutex::new(vec![])); + let transaction_confirmed_params_arc = Arc::new(Mutex::new(vec![])); + let get_transaction_receipt_params_arc = Arc::new(Mutex::new(vec![])); + let return_all_fingerprints_params_arc = Arc::new(Mutex::new(vec![])); + let non_pending_payables_params_arc = Arc::new(Mutex::new(vec![])); + let insert_fingerprint_params_arc = Arc::new(Mutex::new(vec![])); let update_fingerprint_params_arc = Arc::new(Mutex::new(vec![])); let mark_failure_params_arc = Arc::new(Mutex::new(vec![])); let delete_record_params_arc = Arc::new(Mutex::new(vec![])); @@ -3985,8 +4657,10 @@ mod tests { )); let this_payable_timestamp_1 = now; let this_payable_timestamp_2 = now.add(Duration::from_millis(50)); - let payable_account_balance_1 = DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 10; - let payable_account_balance_2 = DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 666; + let payable_account_balance_1 = + gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 10); + let payable_account_balance_2 = + gwei_to_wei(DEFAULT_PAYMENT_THRESHOLDS.debt_threshold_gwei + 666); let transaction_receipt_tx_2_first_round = TransactionReceipt::default(); let transaction_receipt_tx_1_second_round = TransactionReceipt::default(); let transaction_receipt_tx_2_second_round = TransactionReceipt::default(); @@ -4025,14 +4699,14 @@ mod tests { let wallet_account_1 = make_wallet("creditor1"); let account_1 = PayableAccount { wallet: wallet_account_1.clone(), - balance: payable_account_balance_1, + balance_wei: payable_account_balance_1, last_paid_timestamp: past_payable_timestamp_1, pending_payable_opt: None, }; let wallet_account_2 = make_wallet("creditor2"); let account_2 = PayableAccount { wallet: wallet_account_2.clone(), - balance: payable_account_balance_2, + balance_wei: payable_account_balance_2, last_paid_timestamp: past_payable_timestamp_2, pending_payable_opt: None, }; @@ -4065,7 +4739,7 @@ mod tests { timestamp: this_payable_timestamp_1, hash: pending_tx_hash_1, attempt_opt: Some(1), - amount: payable_account_balance_1 as u64, + amount: payable_account_balance_1, process_error: None, }; let fingerprint_2_first_round = PendingPayableFingerprint { @@ -4073,7 +4747,7 @@ mod tests { timestamp: this_payable_timestamp_2, hash: pending_tx_hash_2, attempt_opt: Some(1), - amount: payable_account_balance_2 as u64, + amount: payable_account_balance_2, process_error: None, }; let fingerprint_1_second_round = PendingPayableFingerprint { @@ -4556,7 +5230,7 @@ mod tests { tx_hash: Default::default(), }; let error = BlockchainError::SignedValueConversion(666); - let sent_payable = SentPayable { + let sent_payable = SentPayables { timestamp: SystemTime::now(), payable: vec![Ok(payable_ok.clone()), Err(error.clone())], response_skeleton_opt: None, @@ -4645,12 +5319,98 @@ mod tests { ); } + #[test] + fn financials_request_with_nothing_to_respond_to_is_refused() { + let system = System::new("test"); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .build(); + let (ui_gateway, _, ui_gateway_recording_arc) = make_recorder(); + let subject_addr = subject.start(); + let peer_actors = peer_actors_builder().ui_gateway(ui_gateway).build(); + subject_addr.try_send(BindMessage { peer_actors }).unwrap(); + let ui_message = NodeFromUiMessage { + client_id: 1234, + body: UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: None, + } + .tmb(2222), + }; + + subject_addr.try_send(ui_message).unwrap(); + + System::current().stop(); + system.run(); + let ui_gateway_recording = ui_gateway_recording_arc.lock().unwrap(); + let response = ui_gateway_recording.get_record::(0); + assert_eq!(response.target, ClientId(1234)); + let error = UiFinancialsResponse::fmb(response.body.clone()).unwrap_err(); + let err_message_body = match error { + UiMessageError::PayloadError(payload) => payload, + x => panic!("we expected error message in the payload but got: {:?}", x), + }; + let (err_code, err_message) = err_message_body.payload.unwrap_err(); + assert_eq!(err_code, REQUEST_WITH_NO_VALUES); + assert_eq!( + err_message, + "Empty requests with missing queries not to be processed" + ); + assert!(matches!(err_message_body.path, Conversation(2222))); + } + + #[test] + fn financials_request_allows_only_one_kind_of_view_into_books_at_a_time() { + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .build(); + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 13, + ordered_by: Age, + }), + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 5000, + max_age_s: 11000, + min_amount_gwei: 1_454_050_000, + max_amount_gwei: 555_000_000_000, + }), + receivable_opt: None, + }), + }; + + let result = subject.compute_financials(&request, 4567); + + assert_eq!( + result, + MessageBody { + opcode: "financials".to_string(), + path: Conversation(4567), + payload: Err(( + REQUEST_WITH_MUTUALLY_EXCLUSIVE_PARAMS, + "Requesting top records and the more customized subset of \ + records is not allowed both at the same time" + .to_string() + )) + } + ); + } + #[test] fn financials_request_produces_financials_response() { - let payable_dao = PayableDaoMock::new().total_result(23456789); - let receivable_dao = ReceivableDaoMock::new().total_result(98765432); + let payable_dao = PayableDaoMock::new().total_result(264_567_894_578); + let receivable_dao = ReceivableDaoMock::new().total_result(987_654_328_996); let system = System::new("test"); - let mut subject = AccountantBuilder::default() + let subject = AccountantBuilder::default() .bootstrapper_config(bc_from_ac_plus_earning_wallet( make_populated_accountant_config_with_defaults(), make_wallet("some_wallet_address"), @@ -4658,15 +5418,18 @@ mod tests { .receivable_dao(receivable_dao) .payable_dao(payable_dao) .build(); - subject.financial_statistics.total_paid_payable = 123456; - subject.financial_statistics.total_paid_receivable = 334455; let (ui_gateway, _, ui_gateway_recording_arc) = make_recorder(); let subject_addr = subject.start(); let peer_actors = peer_actors_builder().ui_gateway(ui_gateway).build(); subject_addr.try_send(BindMessage { peer_actors }).unwrap(); let ui_message = NodeFromUiMessage { client_id: 1234, - body: UiFinancialsRequest {}.tmb(2222), + body: UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None, + } + .tmb(2222), }; subject_addr.try_send(ui_message).unwrap(); @@ -4681,14 +5444,572 @@ mod tests { assert_eq!( body, UiFinancialsResponse { - total_unpaid_and_pending_payable: 23456789, - total_paid_payable: 123456, - total_unpaid_receivable: 98765432, - total_paid_receivable: 334455 + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 264, + total_paid_payable_gwei: 0, + total_unpaid_receivable_gwei: 987, + total_paid_receivable_gwei: 0, + }), + query_results_opt: None, } ); } + #[test] + fn compute_financials_processes_defaulted_request() { + let payable_dao = PayableDaoMock::new().total_result(u64::MAX as u128 + 123456); + let receivable_dao = ReceivableDaoMock::new().total_result((i64::MAX as i128) * 3); + let mut subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .payable_dao(payable_dao) + .build(); + subject.financial_statistics.total_paid_payable_wei = 172_345_602_235_454_454; + subject.financial_statistics.total_paid_receivable_wei = 4_455_656_989_415_777_555; + let context_id = 1234; + let request = UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None, + }; + + let result = subject.compute_financials(&request, context_id); + + assert_eq!( + result, + UiFinancialsResponse { + stats_opt: Some(UiFinancialStatistics { + total_unpaid_and_pending_payable_gwei: 18446744073, + total_paid_payable_gwei: 172345602, + total_unpaid_receivable_gwei: 27670116110, + total_paid_receivable_gwei: 4455656989 + }), + query_results_opt: None, + } + .tmb(context_id) + ) + } + + macro_rules! extract_ages_from_accounts { + ($main_structure: expr, $account_specific_field_opt: ident) => {{ + let accounts_collection = &$main_structure + .query_results_opt + .as_ref() + .unwrap() + .$account_specific_field_opt + .as_ref() + .unwrap(); + accounts_collection + .iter() + .map(|account| account.age_s) + .collect::>() + }}; + } + + #[test] + fn compute_financials_processes_request_with_top_records_only_and_balance_ordering() { + //take that the tested logic doesn't contain anything about an actual process of ordering, + //that part is in the responsibility of the database manager, answering the specific SQL query + let payable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let receivable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let payable_accounts_retrieved = vec![PayableAccount { + wallet: make_wallet("abcd123"), + balance_wei: 58_568_686_005, + last_paid_timestamp: SystemTime::now().sub(Duration::from_secs(5000)), + pending_payable_opt: None, + }]; + let payable_dao = PayableDaoMock::new() + .custom_query_params(&payable_custom_query_params_arc) + .custom_query_result(Some(payable_accounts_retrieved)); + let receivable_accounts_retrieved = vec![ReceivableAccount { + wallet: make_wallet("efe4848"), + balance_wei: 3_788_455_600_556_898, + last_received_timestamp: SystemTime::now().sub(Duration::from_secs(6500)), + }]; + let receivable_dao = ReceivableDaoMock::new() + .custom_query_params(&receivable_custom_query_params_arc) + .custom_query_result(Some(receivable_accounts_retrieved)); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .payable_dao(payable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 6, + ordered_by: Balance, + }), + custom_queries_opt: None, + }; + let before = SystemTime::now(); + + let result = subject.compute_financials(&request, context_id_expected); + + let after = SystemTime::now(); + let (computed_response, context_id) = UiFinancialsResponse::fmb(result).unwrap(); + let extracted_payable_ages = extract_ages_from_accounts!(computed_response, payable_opt); + let extracted_receivable_ages = + extract_ages_from_accounts!(computed_response, receivable_opt); + assert_eq!(context_id, context_id_expected); + assert_eq!( + computed_response, + UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![UiPayableAccount { + wallet: make_wallet("abcd123").to_string(), + age_s: extracted_payable_ages[0], + balance_gwei: 58, + pending_payable_hash_opt: None + },]), + receivable_opt: Some(vec![UiReceivableAccount { + wallet: make_wallet("efe4848").to_string(), + age_s: extracted_receivable_ages[0], + balance_gwei: 3_788_455 + },]) + }), + } + ); + let time_needed_for_the_act_in_full_sec = + (after.duration_since(before).unwrap().as_millis() / 1000 + 1) as u64; + assert!( + extracted_payable_ages[0] >= 5000 + && extracted_payable_ages[0] <= 5000 + time_needed_for_the_act_in_full_sec + ); + assert!( + extracted_receivable_ages[0] >= 6500 + && extracted_receivable_ages[0] <= 6500 + time_needed_for_the_act_in_full_sec + ); + let payable_custom_query_params = payable_custom_query_params_arc.lock().unwrap(); + assert_eq!( + *payable_custom_query_params, + vec![CustomQuery::TopRecords { + count: 6, + ordered_by: Balance + }] + ); + let receivable_custom_query_params = receivable_custom_query_params_arc.lock().unwrap(); + assert_eq!( + *receivable_custom_query_params, + vec![CustomQuery::TopRecords { + count: 6, + ordered_by: Balance + }] + ) + } + + #[test] + fn compute_financials_processes_request_with_top_records_only_and_age_ordering() { + let payable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let receivable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let payable_dao = PayableDaoMock::new() + .custom_query_params(&payable_custom_query_params_arc) + .custom_query_result(None); + let receivable_dao = ReceivableDaoMock::new() + .custom_query_params(&receivable_custom_query_params_arc) + .custom_query_result(None); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .payable_dao(payable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 80, + ordered_by: Age, + }), + custom_queries_opt: None, + }; + + let result = subject.compute_financials(&request, context_id_expected); + + let (response, context_id) = UiFinancialsResponse::fmb(result).unwrap(); + assert_eq!(context_id, context_id_expected); + assert_eq!( + response, + UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![]), + receivable_opt: Some(vec![]) + }) + } + ); + let payable_custom_query_params = payable_custom_query_params_arc.lock().unwrap(); + assert_eq!( + *payable_custom_query_params, + vec![CustomQuery::TopRecords { + count: 80, + ordered_by: Age + }] + ); + let receivable_custom_query_params = receivable_custom_query_params_arc.lock().unwrap(); + assert_eq!( + *receivable_custom_query_params, + vec![CustomQuery::TopRecords { + count: 80, + ordered_by: Age + }] + ) + } + + #[test] + fn compute_financials_processes_request_with_range_queries_only() { + let payable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let receivable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let payable_accounts_retrieved = vec![PayableAccount { + wallet: make_wallet("abcd123"), + balance_wei: 5_686_860_056, + last_paid_timestamp: SystemTime::now().sub(Duration::from_secs(7580)), + pending_payable_opt: None, + }]; + let payable_dao = PayableDaoMock::new() + .custom_query_params(&payable_custom_query_params_arc) + .custom_query_result(Some(payable_accounts_retrieved)); + let receivable_accounts_retrieved = vec![ + ReceivableAccount { + wallet: make_wallet("efe4848"), + balance_wei: 20_456_056_055_600_789, + last_received_timestamp: SystemTime::now().sub(Duration::from_secs(3333)), + }, + ReceivableAccount { + wallet: make_wallet("bb123aa"), + balance_wei: 550_555_565_233, + last_received_timestamp: SystemTime::now().sub(Duration::from_secs(87000)), + }, + ]; + let receivable_dao = ReceivableDaoMock::new() + .custom_query_params(&receivable_custom_query_params_arc) + .custom_query_result(Some(receivable_accounts_retrieved)); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .payable_dao(payable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 0, + max_age_s: 8000, + min_amount_gwei: 0, + max_amount_gwei: 50_000_000, + }), + receivable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: 200000, + min_amount_gwei: 0, + max_amount_gwei: 60_000_000, + }), + }), + }; + let before = SystemTime::now(); + + let result = subject.compute_financials(&request, context_id_expected); + + let after = SystemTime::now(); + let (computed_response, context_id) = UiFinancialsResponse::fmb(result).unwrap(); + let extracted_payable_ages = extract_ages_from_accounts!(computed_response, payable_opt); + let extracted_receivable_ages = + extract_ages_from_accounts!(computed_response, receivable_opt); + assert_eq!(context_id, context_id_expected); + assert_eq!( + computed_response, + UiFinancialsResponse { + stats_opt: None, + query_results_opt: Some(QueryResults { + payable_opt: Some(vec![UiPayableAccount { + wallet: make_wallet("abcd123").to_string(), + age_s: extracted_payable_ages[0], + balance_gwei: 5, + pending_payable_hash_opt: None + },]), + receivable_opt: Some(vec![ + UiReceivableAccount { + wallet: make_wallet("efe4848").to_string(), + age_s: extracted_receivable_ages[0], + balance_gwei: 20_456_056 + }, + UiReceivableAccount { + wallet: make_wallet("bb123aa").to_string(), + age_s: extracted_receivable_ages[1], + balance_gwei: 550, + } + ]) + }) + } + ); + let time_needed_for_the_act_in_full_sec = + (after.duration_since(before).unwrap().as_millis() / 1000 + 1) as u64; + assert!( + 7580 <= extracted_payable_ages[0] + && extracted_payable_ages[0] <= 7580 + time_needed_for_the_act_in_full_sec + ); + assert!( + 3333 <= extracted_receivable_ages[0] + && extracted_receivable_ages[0] <= 3333 + time_needed_for_the_act_in_full_sec + ); + assert!( + 87000 <= extracted_receivable_ages[1] + && extracted_receivable_ages[1] <= 87000 + time_needed_for_the_act_in_full_sec + ); + let payable_custom_query_params = payable_custom_query_params_arc.lock().unwrap(); + let actual_timestamp = extract_timestamp_from_custom_query(&payable_custom_query_params[0]); + assert_eq!( + *payable_custom_query_params, + vec![CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: 8000, + min_amount_gwei: 0, + max_amount_gwei: 50000000, + timestamp: actual_timestamp + }] + ); + assert!( + before <= actual_timestamp && actual_timestamp <= after, + "before: {:?}, actual: {:?}, after: {:?}", + before, + actual_timestamp, + after + ); + let receivable_custom_query_params = receivable_custom_query_params_arc.lock().unwrap(); + let actual_timestamp = + extract_timestamp_from_custom_query(&receivable_custom_query_params[0]); + assert_eq!( + *receivable_custom_query_params, + vec![CustomQuery::RangeQuery { + min_age_s: 2000, + max_age_s: 200000, + min_amount_gwei: 0, + max_amount_gwei: 60000000, + timestamp: actual_timestamp + }] + ); + assert!( + before <= actual_timestamp && actual_timestamp <= after, + "before: {:?}, actual: {:?}, after: {:?}", + before, + actual_timestamp, + after + ) + } + + fn extract_timestamp_from_custom_query(captured_input: &CustomQuery) -> SystemTime { + if let CustomQuery::RangeQuery { timestamp, .. } = captured_input { + *timestamp + } else { + panic!("we expected range query whose part is also a timestamp") + } + } + + #[test] + fn compute_financials_allows_range_query_to_be_aimed_only_at_one_table() { + let receivable_custom_query_params_arc = Arc::new(Mutex::new(vec![])); + let receivable_accounts_retrieved = vec![ReceivableAccount { + wallet: make_wallet("efe4848"), + balance_wei: 60055600789, + last_received_timestamp: SystemTime::now().sub(Duration::from_secs(3333)), + }]; + let receivable_dao = ReceivableDaoMock::new() + .custom_query_params(&receivable_custom_query_params_arc) + .custom_query_result(Some(receivable_accounts_retrieved)); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: 200000, + min_amount_gwei: 0, + max_amount_gwei: 150000000000, + }), + }), + }; + + let result = subject.compute_financials(&request, context_id_expected); + + let (response, _) = UiFinancialsResponse::fmb(result).unwrap(); + let response_guts = response.query_results_opt.unwrap(); + assert_eq!(response_guts.payable_opt.is_some(), false); + assert_eq!(response_guts.receivable_opt.is_some(), true); + } + + fn assert_compute_financials_tests_range_query_on_too_big_values_in_input( + request: UiFinancialsRequest, + err_msg: &str, + ) { + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .build(); + let context_id_expected = 1234; + + let result = subject.compute_financials(&request, context_id_expected); + + assert_eq!( + result, + MessageBody { + opcode: "financials".to_string(), + path: Conversation(context_id_expected), + payload: Err((VALUE_EXCEEDS_ALLOWED_LIMIT, err_msg.to_string())) + } + ); + } + + #[test] + fn compute_financials_tests_range_query_of_payables_on_too_big_values_in_input() { + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: 50000, + min_amount_gwei: 0, + max_amount_gwei: u64::MAX, + }), + receivable_opt: None, + }), + }; + + assert_compute_financials_tests_range_query_on_too_big_values_in_input( + request, + "Range query for payable: Max amount requested too big. \ + Should be less than or equal to 9223372036854775807, not: 18446744073709551615", + ) + } + + #[test] + fn compute_financials_tests_range_query_of_receivables_on_too_big_values_in_input() { + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: u64::MAX, + min_amount_gwei: -55, + max_amount_gwei: 6666, + }), + }), + }; + + assert_compute_financials_tests_range_query_on_too_big_values_in_input( + request, + "Range query for receivable: Max age requested too big. \ + Should be less than or equal to 9223372036854775807, not: 18446744073709551615", + ) + } + + #[test] + #[should_panic( + expected = "Broken code: PayableAccount with less than 1 gwei passed through db query \ + constraints; wallet: 0x0000000000000000000000000061626364313233, balance: 8686005" + )] + fn compute_financials_blows_up_on_screwed_sql_query_for_payables_returning_balance_smaller_than_one_gwei( + ) { + let payable_accounts_retrieved = vec![PayableAccount { + wallet: make_wallet("abcd123"), + balance_wei: 8_686_005, + last_paid_timestamp: SystemTime::now().sub(Duration::from_secs(5000)), + pending_payable_opt: None, + }]; + let payable_dao = + PayableDaoMock::new().custom_query_result(Some(payable_accounts_retrieved)); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .payable_dao(payable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: 200000, + min_amount_gwei: 0, + max_amount_gwei: 150000000000, + }), + receivable_opt: None, + }), + }; + + subject.compute_financials(&request, context_id_expected); + } + + #[test] + #[should_panic( + expected = "Broken code: ReceivableAccount with balance between 1 and 0 gwei passed through \ + db query constraints; wallet: 0x0000000000000000000000000061626364313233, balance: 7686005" + )] + fn compute_financials_blows_up_on_screwed_sql_query_for_receivables_returning_balance_smaller_than_one_gwei( + ) { + let receivable_accounts_retrieved = vec![ReceivableAccount { + wallet: make_wallet("abcd123"), + balance_wei: 7_686_005, + last_received_timestamp: SystemTime::now().sub(Duration::from_secs(5000)), + }]; + let receivable_dao = + ReceivableDaoMock::new().custom_query_result(Some(receivable_accounts_retrieved)); + let subject = AccountantBuilder::default() + .bootstrapper_config(bc_from_ac_plus_earning_wallet( + make_populated_accountant_config_with_defaults(), + make_wallet("some_wallet_address"), + )) + .receivable_dao(receivable_dao) + .build(); + let context_id_expected = 1234; + let request = UiFinancialsRequest { + stats_required: false, + top_records_opt: None, + custom_queries_opt: Some(CustomQueries { + payable_opt: None, + receivable_opt: Some(RangeQuery { + min_age_s: 2000, + max_age_s: 200000, + min_amount_gwei: 0, + max_amount_gwei: 150000000000, + }), + }), + }; + + subject.compute_financials(&request, context_id_expected); + } + #[test] fn total_paid_payable_rises_with_each_bill_paid() { let transaction_confirmed_params_arc = Arc::new(Mutex::new(vec![])); @@ -4711,14 +6032,17 @@ mod tests { .pending_payable_dao(pending_payable_dao) .payable_dao(payable_dao) .build(); - subject.financial_statistics.total_paid_payable += 1111; + subject.financial_statistics.total_paid_payable_wei += 1111; let msg = ConfirmPendingTransaction { pending_payable_fingerprint: fingerprint.clone(), }; subject.handle_confirm_pending_transaction(msg); - assert_eq!(subject.financial_statistics.total_paid_payable, 1111 + 5478); + assert_eq!( + subject.financial_statistics.total_paid_payable_wei, + 1111 + 5478 + ); let transaction_confirmed_params = transaction_confirmed_params_arc.lock().unwrap(); assert_eq!(*transaction_confirmed_params, vec![fingerprint]) } @@ -4732,17 +6056,17 @@ mod tests { let mut subject = AccountantBuilder::default() .receivable_dao(receivable_dao) .build(); - subject.financial_statistics.total_paid_receivable += 2222; + subject.financial_statistics.total_paid_receivable_wei += 2222; let receivables = vec![ BlockchainTransaction { block_number: 4578910, from: make_wallet("wallet_1"), - gwei_amount: 45780, + wei_amount: 45780, }, BlockchainTransaction { block_number: 4569898, from: make_wallet("wallet_2"), - gwei_amount: 33345, + wei_amount: 33345, }, ]; let now = SystemTime::now(); @@ -4754,7 +6078,7 @@ mod tests { }); assert_eq!( - subject.financial_statistics.total_paid_receivable, + subject.financial_statistics.total_paid_receivable_wei, 2222 + 45780 + 33345 ); let more_money_received_params = more_money_received_params_arc.lock().unwrap(); @@ -4785,14 +6109,14 @@ mod tests { #[test] fn unsigned_to_signed_handles_zero() { - let result = unsigned_to_signed(0); + let result = sign_conversion::(0); assert_eq!(result, Ok(0i64)); } #[test] fn unsigned_to_signed_handles_max_allowable() { - let result = unsigned_to_signed(i64::MAX as u64); + let result = sign_conversion::(i64::MAX as u64); assert_eq!(result, Ok(i64::MAX)); } @@ -4800,8 +6124,45 @@ mod tests { #[test] fn unsigned_to_signed_handles_max_plus_one() { let attempt = (i64::MAX as u64) + 1; - let result = unsigned_to_signed((i64::MAX as u64) + 1); + let result = sign_conversion::((i64::MAX as u64) + 1); assert_eq!(result, Err(attempt)); } + + #[test] + #[should_panic( + expected = "Overflow detected with 170141183460469231731687303715884105728: cannot be converted from u128 to i128" + )] + fn checked_conversion_works_for_overflow() { + checked_conversion::(i128::MAX as u128 + 1); + } + + #[test] + fn checked_conversion_without_panic() { + let result = politely_checked_conversion::(u128::MAX); + + assert_eq!(result,Err("Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128".to_string())) + } + + #[test] + fn gwei_to_wei_works() { + let result: u128 = gwei_to_wei(12_546_u64); + + assert_eq!(result, 12_546_000_000_000) + } + + #[test] + fn wei_to_gwei_works() { + let result: u64 = wei_to_gwei(127_800_050_500_u128); + + assert_eq!(result, 127) + } + + #[test] + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431: cannot be converted from u128 to u64" + )] + fn wei_to_gwei_blows_up_on_overflow() { + let _: u64 = wei_to_gwei(u128::MAX); + } } diff --git a/node/src/accountant/payable_dao.rs b/node/src/accountant/payable_dao.rs index e07f74c0e..b1f0a0d7b 100644 --- a/node/src/accountant/payable_dao.rs +++ b/node/src/accountant/payable_dao.rs @@ -1,14 +1,32 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. -use crate::accountant::{unsigned_to_signed, PendingPayableId}; +use crate::accountant::big_int_processing::big_int_db_processor::KnownKeyVariants::{ + PendingPayableRowid, WalletAddress, +}; +use crate::accountant::big_int_processing::big_int_db_processor::WeiChange::{ + Addition, Subtraction, +}; +use crate::accountant::big_int_processing::big_int_db_processor::{ + BigIntDbProcessor, BigIntSqlConfig, SQLParamsBuilder, TableNameDAO, +}; +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::dao_utils; +use crate::accountant::dao_utils::{ + sum_i128_values_from_table, to_time_t, AssemblerFeeder, CustomQuery, DaoFactoryReal, + RangeStmConfig, TopStmConfig, VigilantRusqliteFlatten, +}; +use crate::accountant::{checked_conversion, sign_conversion, PendingPayableId}; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::dao_utils; -use crate::database::dao_utils::{to_time_t, DaoFactoryReal}; use crate::sub_lib::wallet::Wallet; +#[cfg(test)] +use ethereum_types::{BigEndianHash, U256}; +use itertools::Either::Left; use masq_lib::utils::ExpectValue; -use rusqlite::types::{ToSql, Type}; -use rusqlite::Error; +use rusqlite::types::ToSql; +#[cfg(test)] +use rusqlite::OptionalExtension; +use rusqlite::{Error, Row}; use std::fmt::Debug; use std::str::FromStr; use std::time::SystemTime; @@ -23,21 +41,23 @@ pub enum PayableDaoError { #[derive(Clone, Debug, PartialEq, Eq)] pub struct PayableAccount { pub wallet: Wallet, - pub balance: i64, + pub balance_wei: u128, pub last_paid_timestamp: SystemTime, pub pending_payable_opt: Option, } +//TODO two to three of these fields can be technically eliminated now but I think my old plan was not to do that because it could be potentially a useful set of information, +// I somehow didn't trust unconditionally to the pending payable record to be always secure - and so I still think this might wait for GH-576 #[derive(Clone, Debug, PartialEq, Eq)] pub struct Payable { pub to: Wallet, - pub amount: u64, + pub amount: u128, pub timestamp: SystemTime, pub tx_hash: H256, } impl Payable { - pub fn new(to: Wallet, amount: u64, txn: H256, timestamp: SystemTime) -> Self { + pub fn new(to: Wallet, amount: u128, txn: H256, timestamp: SystemTime) -> Self { Self { to, amount, @@ -52,7 +72,7 @@ pub trait PayableDao: Debug + Send { &self, now: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), PayableDaoError>; fn mark_pending_payable_rowid( @@ -66,13 +86,14 @@ pub trait PayableDao: Debug + Send { payment: &PendingPayableFingerprint, ) -> Result<(), PayableDaoError>; - //there used to be method 'accountant_status' but was turned into test utility since never used in the production code - fn non_pending_payables(&self) -> Vec; - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec; + fn custom_query(&self, custom_query: CustomQuery) -> Option>; - fn total(&self) -> u64; + fn total(&self) -> u128; + + #[cfg(test)] + fn account_status(&self, wallet: &Wallet) -> Option; } pub trait PayableDaoFactory { @@ -88,6 +109,7 @@ impl PayableDaoFactory for DaoFactoryReal { #[derive(Debug)] pub struct PayableDaoReal { conn: Box, + big_int_db_processor: BigIntDbProcessor, } impl PayableDao for PayableDaoReal { @@ -95,16 +117,21 @@ impl PayableDao for PayableDaoReal { &self, timestamp: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), PayableDaoError> { - let signed_amount = unsigned_to_signed(amount).map_err(PayableDaoError::SignConversion)?; - match self.try_increase_balance(timestamp, wallet, signed_amount) { - Ok(_) => Ok(()), - Err(e) => panic!( - "Database is corrupt: {}; processing payable for {}", - e, wallet - ), - } + Ok(self.big_int_db_processor.execute( + Left(self.conn.as_ref()), + BigIntSqlConfig::new( + "insert into payable (wallet_address, balance_high_b, balance_low_b, last_paid_timestamp, pending_payable_rowid) values (:wallet, :balance_high_b, :balance_low_b, :last_paid_timestamp, null) on conflict (wallet_address) do \ + update set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b where wallet_address = :wallet", + "update {} set balance_high_b = :balance_high_b, balance_low_b = :balance_low_b where wallet_address = :wallet", + SQLParamsBuilder::default() + .key(WalletAddress(wallet)) + .wei_change( Addition("balance",amount)) + .other(vec![(":last_paid_timestamp",&to_time_t(timestamp))]) + .build() + ))? + ) } fn mark_pending_payable_rowid( @@ -135,224 +162,255 @@ impl PayableDao for PayableDaoReal { &self, fingerprint: &PendingPayableFingerprint, ) -> Result<(), PayableDaoError> { - let signed_amount = - unsigned_to_signed(fingerprint.amount).map_err(PayableDaoError::SignConversion)?; - self.try_decrease_balance( - fingerprint.rowid_opt.expectv("initialized rowid"), - signed_amount, - fingerprint.timestamp, - ) - .map_err(PayableDaoError::RusqliteError) + let key = + checked_conversion::(fingerprint.rowid_opt.expectv("initialized rowid")); + Ok(self + .big_int_db_processor + .execute(Left(self.conn.as_ref()), BigIntSqlConfig::new( + "update payable set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b, last_paid_timestamp = :last_paid, pending_payable_rowid = null where pending_payable_rowid = :rowid", + "update payable set balance_high_b = :balance_high_b, balance_low_b = :balance_low_b, last_paid_timestamp = :last_paid, pending_payable_rowid = null where pending_payable_rowid = :rowid", + SQLParamsBuilder::default() + .key( PendingPayableRowid(&key)) + .wei_change(Subtraction("balance",fingerprint.amount)) + .other(vec![(":last_paid", &to_time_t(fingerprint.timestamp))]) + .build()))?) } fn non_pending_payables(&self) -> Vec { let mut stmt = self.conn - .prepare("select wallet_address, balance, last_paid_timestamp from payable where pending_payable_rowid is null") + .prepare("select wallet_address, balance_high_b, balance_low_b, last_paid_timestamp from payable where pending_payable_rowid is null") .expect("Internal error"); - stmt.query_map([], |row| { let wallet_result: Result = row.get(0); - let balance_result = row.get(1); - let last_paid_timestamp_result = row.get(2); - match (wallet_result, balance_result, last_paid_timestamp_result) { - (Ok(wallet), Ok(balance), Ok(last_paid_timestamp)) => Ok(PayableAccount { - wallet, - balance, - last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), - pending_payable_opt: None, - }), + let high_b_result: Result = row.get(1); + let low_b_result: Result = row.get(2); + let last_paid_timestamp_result = row.get(3); + match ( + wallet_result, + high_b_result, + low_b_result, + last_paid_timestamp_result, + ) { + (Ok(wallet), Ok(high_b), Ok(low_b), Ok(last_paid_timestamp)) => { + Ok(PayableAccount { + wallet, + balance_wei: checked_conversion::(BigIntDivider::reconstitute( + high_b, low_b, + )), + last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), + pending_payable_opt: None, + }) + } _ => panic!("Database is corrupt: PAYABLE table columns and/or types"), } }) .expect("Database is corrupt") - .flatten() + .vigilant_flatten() .collect() } - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec { - let min_amt = unsigned_to_signed(minimum_amount).unwrap_or(0x7FFF_FFFF_FFFF_FFFF); - let max_age = unsigned_to_signed(maximum_age).unwrap_or(0x7FFF_FFFF_FFFF_FFFF); - let min_timestamp = dao_utils::now_time_t() - max_age; - let mut stmt = self - .conn - .prepare( - r#" - select - balance, - last_paid_timestamp, - wallet_address, - pending_payable_rowid, - pending_payable.transaction_hash - from - payable - left join pending_payable on - pending_payable.rowid = payable.pending_payable_rowid - where - balance >= ? and - last_paid_timestamp >= ? - order by - balance desc, - last_paid_timestamp desc - "#, + fn custom_query(&self, custom_query: CustomQuery) -> Option> { + let variant_top = TopStmConfig{ + limit_clause: "limit :limit_count", + gwei_min_resolution_clause: "where (balance_high_b > 0) or ((balance_high_b = 0) and (balance_low_b >= 1000000000))", + age_ordering_clause: "last_paid_timestamp asc", + }; + let variant_range = RangeStmConfig { + where_clause: "where ((last_paid_timestamp <= :max_timestamp) and (last_paid_timestamp >= :min_timestamp)) \ + and ((balance_high_b > :min_balance_high_b) or ((balance_high_b = :min_balance_high_b) and (balance_low_b >= :min_balance_low_b))) \ + and ((balance_high_b < :max_balance_high_b) or ((balance_high_b = :max_balance_high_b) and (balance_low_b <= :max_balance_low_b)))", + gwei_min_resolution_clause: "and ((balance_high_b > 0) or ((balance_high_b = 0) and (balance_low_b >= 1000000000)))", + secondary_order_param: "last_paid_timestamp asc" + }; + + custom_query.query::<_, i64, _, _>( + self.conn.as_ref(), + Self::stm_assembler_of_payable_cq, + variant_top, + variant_range, + Self::create_payable_account, + ) + } + + fn total(&self) -> u128 { + let value_completer = |row_number: usize, row: &Row| { + let high_bytes = row.get::(0).expectv("high bytes"); + let low_bytes = row.get::(1).expectv("low_bytes"); + let big_int = BigIntDivider::reconstitute(high_bytes, low_bytes); + if high_bytes < 0 { + panic!( + "database corrupted: found negative value {} in payable table for row id {}", + big_int, row_number + ) + }; + Ok(big_int) + }; + sign_conversion::(sum_i128_values_from_table( + self.conn.as_ref(), + &Self::table_name(), + "balance", + value_completer, + )) + .unwrap_or_else(|num| { + panic!( + "database corrupted: negative sum ({}) in payable table", + num ) - .expect("Internal error"); - let params: &[&dyn ToSql] = &[&min_amt, &min_timestamp]; - stmt.query_map(params, |row| { - let balance_result = row.get(0); - let last_paid_timestamp_result = row.get(1); - let wallet_result: Result = row.get(2); - let pending_payable_rowid_result_opt: Result, Error> = row.get(3); - let pending_payable_hash_result_opt: Result, Error> = row.get(4); - match ( - wallet_result, - balance_result, - last_paid_timestamp_result, - pending_payable_rowid_result_opt, - pending_payable_hash_result_opt, - ) { - ( - Ok(wallet), - Ok(balance), - Ok(last_paid_timestamp), - Ok(pending_payable_rowid_opt), - Ok(pending_payable_hash_opt), - ) => Ok(PayableAccount { - wallet, - balance, - last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), - pending_payable_opt: pending_payable_rowid_opt.map(|rowid| PendingPayableId { - rowid, - hash: pending_payable_hash_opt - .map(|s| H256::from_str(&s[2..]).expectv("string tx hash")) - .expectv("tx hash"), - }), - }), - x => panic!( - "Database is corrupt: PAYABLE table columns and/or types {:?}", - x - ), - } }) - .expect("Database is corrupt") - .flatten() - .collect() } - fn total(&self) -> u64 { - let mut stmt = self - .conn - .prepare("select sum(balance) from payable") - .expect("Internal error"); - match stmt.query_row([], |row| { - let total_balance_result: Result = row.get(0); - match total_balance_result { - Ok(total_balance) => Ok(total_balance), - Err(e) - if e == Error::InvalidColumnType(0, "sum(balance)".to_string(), Type::Null) => - { - Ok(0) + #[cfg(test)] + fn account_status(&self, wallet: &Wallet) -> Option { + let mut stmt = self.conn + .prepare("select balance_high_b, balance_low_b, last_paid_timestamp, pending_payable_rowid from payable where wallet_address = ?") + .unwrap(); + stmt.query_row(&[&wallet], |row| { + let high_bytes_result = row.get(0); + let low_bytes_result = row.get(1); + let last_paid_timestamp_result = row.get(2); + let pending_payable_rowid_result: Result, Error> = row.get(3); + match ( + high_bytes_result, + low_bytes_result, + last_paid_timestamp_result, + pending_payable_rowid_result, + ) { + (Ok(high_bytes), Ok(low_bytes), Ok(last_paid_timestamp), Ok(rowid)) => { + Ok(PayableAccount { + wallet: wallet.clone(), + balance_wei: checked_conversion::(BigIntDivider::reconstitute( + high_bytes, low_bytes, + )), + last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), + pending_payable_opt: match rowid { + Some(rowid) => Some(PendingPayableId { + rowid: u64::try_from(rowid).unwrap(), + hash: H256::from_uint(&U256::from(0)), //garbage + }), + None => None, + }, + }) } - Err(e) => panic!( + e => panic!( "Database is corrupt: PAYABLE table columns and/or types: {:?}", e ), } - }) { - Ok(value) => value, - Err(e) => panic!("Database is corrupt: {:?}", e), - } + }) + .optional() + .unwrap() } } impl PayableDaoReal { pub fn new(conn: Box) -> PayableDaoReal { - PayableDaoReal { conn } - } - - fn try_increase_balance( - &self, - timestamp: SystemTime, - wallet: &Wallet, - amount: i64, - ) -> Result { - let mut stmt = self - .conn - .prepare("insert into payable (wallet_address, balance, last_paid_timestamp, pending_payable_rowid) values (:address, :balance, :timestamp, null) on conflict (wallet_address) do update set balance = balance + :balance where wallet_address = :address") - .expect("Internal error"); - let params: &[(&str, &dyn ToSql)] = &[ - (":address", &wallet), - (":balance", &amount), - (":timestamp", &to_time_t(timestamp)), - ]; - match stmt.execute(params) { - Ok(0) => Ok(false), - Ok(_) => Ok(true), - Err(e) => Err(format!("{}", e)), + PayableDaoReal { + conn, + big_int_db_processor: BigIntDbProcessor::default(), } } - fn try_decrease_balance( - &self, - rowid: u64, - amount: i64, - last_paid_timestamp: SystemTime, - ) -> Result<(), String> { - let mut stmt = self - .conn - .prepare("update payable set balance = balance - :balance, last_paid_timestamp = :last_paid, pending_payable_rowid = null where pending_payable_rowid = :referential_rowid") - .expect("Internal error"); - let params: &[(&str, &dyn ToSql)] = &[ - (":balance", &amount), - (":last_paid", &dao_utils::to_time_t(last_paid_timestamp)), + fn create_payable_account(row: &Row) -> rusqlite::Result { + let wallet_result: Result = row.get(0); + let balance_high_bytes_result = row.get(1); + let balance_low_bytes_result = row.get(2); + let last_paid_timestamp_result = row.get(3); + let pending_payable_rowid_result: Result, Error> = row.get(4); + let pending_payable_hash_result: Result, Error> = row.get(5); + match ( + wallet_result, + balance_high_bytes_result, + balance_low_bytes_result, + last_paid_timestamp_result, + pending_payable_rowid_result, + pending_payable_hash_result, + ) { ( - ":referential_rowid", - &i64::try_from(rowid).expect("SQLite was wrong when choosing the rowid"), - ), - ]; - match stmt.execute(params) { - Ok(1) => Ok(()), - Ok(num) => panic!( - "Trying to decrease balance for rowid {}: {} rows changed instead of 1", - rowid, num + Ok(wallet), + Ok(high_bytes), + Ok(low_bytes), + Ok(last_paid_timestamp), + Ok(rowid_opt), + Ok(hash_opt), + ) => Ok(PayableAccount { + wallet, + balance_wei: checked_conversion::(BigIntDivider::reconstitute( + high_bytes, low_bytes, + )), + last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), + pending_payable_opt: rowid_opt.map(|rowid| { + let hash_str = + hash_opt.expect("database corrupt; missing hash but existing rowid"); + PendingPayableId { + rowid: u64::try_from(rowid).unwrap(), + hash: H256::from_str(&hash_str[2..]) + .unwrap_or_else(|_| panic!("wrong form of tx hash {}", hash_str)), + } + }), + }), + e => panic!( + "Database is corrupt: PAYABLE table columns and/or types: {:?}", + e ), - Err(e) => Err(format!("{}", e)), } } + + fn stm_assembler_of_payable_cq(feeder: AssemblerFeeder) -> String { + format!( + "select + wallet_address, + balance_high_b, + balance_low_b, + last_paid_timestamp, + pending_payable_rowid, + pending_payable.transaction_hash + from + payable + left join pending_payable on + pending_payable.rowid = payable.pending_payable_rowid + {} {} + order by + {}, + {} + {}", + feeder.main_where_clause, + feeder.where_clause_extension, + feeder.order_by_first_param, + feeder.order_by_second_param, + feeder.limit_clause + ) + } +} + +impl TableNameDAO for PayableDaoReal { + fn table_name() -> String { + String::from("payable") + } } #[cfg(test)] mod tests { use super::*; - use crate::accountant::test_utils::{account_status, make_pending_payable_fingerprint}; + use crate::accountant::dao_utils::{from_time_t, now_time_t, to_time_t}; + use crate::accountant::gwei_to_wei; + use crate::accountant::test_utils::{ + assert_account_creation_fn_fails_on_finding_wrong_columns_and_value_types, + make_pending_payable_fingerprint, + }; use crate::database::connection_wrapper::ConnectionWrapperReal; - use crate::database::dao_utils::{from_time_t, to_time_t}; - use crate::database::db_initializer::{DbInitializer, DbInitializerReal, DATABASE_FILE}; - use crate::database::db_migrations::MigratorConfig; + use crate::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, DATABASE_FILE, + }; use crate::test_utils::make_wallet; use ethereum_types::BigEndianHash; + use masq_lib::messages::TopRecordsOrdering::{Age, Balance}; use masq_lib::test_utils::utils::ensure_node_home_directory_exists; use rusqlite::Connection as RusqliteConnection; use rusqlite::{Connection, OpenFlags}; use std::path::Path; + use std::str::FromStr; use web3::types::U256; - #[test] - #[should_panic( - expected = "Trying to decrease balance for rowid 45: 0 rows changed instead of 1" - )] - fn try_decrease_balance_changed_no_rows() { - let home_dir = ensure_node_home_directory_exists( - "payable_dao", - "try_decrease_balance_changed_no_rows", - ); - let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); - let subject = PayableDaoReal::new(wrapped_conn); - - let _ = subject.try_decrease_balance(45, 1111, SystemTime::now()); - } - #[test] fn more_money_payable_works_for_new_address() { let home_dir = ensure_node_home_directory_exists( @@ -363,18 +421,17 @@ mod tests { let wallet = make_wallet("booga"); let status = { let boxed_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PayableDaoReal::new(boxed_conn); - let secondary_conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); subject.more_money_payable(now, &wallet, 1234).unwrap(); - account_status(&secondary_conn, &wallet).unwrap() + subject.account_status(&wallet).unwrap() }; assert_eq!(status.wallet, wallet); - assert_eq!(status.balance, 1234); + assert_eq!(status.balance_wei, 1234); assert_eq!(to_time_t(status.last_paid_timestamp), to_time_t(now)); } @@ -387,26 +444,28 @@ mod tests { let wallet = make_wallet("booga"); let now = SystemTime::now(); let boxed_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); - let secondary_conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); - let subject = PayableDaoReal::new(boxed_conn); - subject.more_money_payable(now, &wallet, 1234).unwrap(); - - let status = { + let subject = { + let subject = PayableDaoReal::new(boxed_conn); + subject.more_money_payable(now, &wallet, 1234).unwrap(); subject - .more_money_payable(SystemTime::UNIX_EPOCH, &wallet, 2345) - .unwrap(); - - account_status(&secondary_conn, &wallet).unwrap() }; + subject + .more_money_payable(SystemTime::UNIX_EPOCH, &wallet, 2345) + .unwrap(); + + let status = subject.account_status(&wallet).unwrap(); assert_eq!(status.wallet, wallet); - assert_eq!(status.balance, 3579); + assert_eq!(status.balance_wei, 3579); assert_eq!(to_time_t(status.last_paid_timestamp), to_time_t(now)); } #[test] + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128" + )] fn more_money_payable_works_for_overflow() { let home_dir = ensure_node_home_directory_exists( "payable_dao", @@ -415,13 +474,11 @@ mod tests { let wallet = make_wallet("booga"); let subject = PayableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); - let result = subject.more_money_payable(SystemTime::now(), &wallet, u64::MAX); - - assert_eq!(result, Err(PayableDaoError::SignConversion(u64::MAX))); + let _ = subject.more_money_payable(SystemTime::now(), &wallet, u128::MAX); } #[test] @@ -433,29 +490,26 @@ mod tests { let wallet = make_wallet("booga"); let pending_payable_rowid = 656; let boxed_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); - let secondary_conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); { - let mut stm = boxed_conn.prepare("insert into payable (wallet_address, balance, last_paid_timestamp) values (?,?,?)").unwrap(); - let params: &[&dyn ToSql] = &[&wallet, &5000, &150_000_000]; - stm.execute(params).unwrap(); + insert_record_fn(&*boxed_conn, &wallet.to_string(), 5000, 150_000_000, None); } let subject = PayableDaoReal::new(boxed_conn); - let before_account_status = account_status(&secondary_conn, &wallet).unwrap(); - let before_expected_status = PayableAccount { - wallet: wallet.clone(), - balance: 5000, - last_paid_timestamp: from_time_t(150_000_000), - pending_payable_opt: None, - }; - assert_eq!(before_account_status, before_expected_status.clone()); + let before_account_status = subject.account_status(&wallet).unwrap(); subject .mark_pending_payable_rowid(&wallet, pending_payable_rowid) .unwrap(); - let after_account_status = account_status(&secondary_conn, &wallet).unwrap(); + let before_expected_status = PayableAccount { + wallet: wallet.clone(), + balance_wei: 5000, + last_paid_timestamp: from_time_t(150_000_000), + pending_payable_opt: None, + }; + assert_eq!(before_account_status, before_expected_status); + let after_account_status = subject.account_status(&wallet).unwrap(); let mut after_expected_status = before_expected_status; after_expected_status.pending_payable_opt = Some(PendingPayableId { rowid: pending_payable_rowid, @@ -476,7 +530,7 @@ mod tests { let wallet = make_wallet("booga"); let rowid = 656; let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PayableDaoReal::new(conn); @@ -491,7 +545,7 @@ mod tests { ); let wallet = make_wallet("booga"); let rowid = 656; - let conn = how_to_trick_rusqlite_for_an_error(&home_dir); + let conn = trick_rusqlite_with_read_only_conn(&home_dir); let conn_wrapped = ConnectionWrapperReal::new(conn); let subject = PayableDaoReal::new(Box::new(conn_wrapped)); @@ -505,37 +559,13 @@ mod tests { ) } - fn create_account_with_pending_payment( - conn: &dyn ConnectionWrapper, - recipient_wallet: &Wallet, - amount: i64, - timestamp: SystemTime, - rowid: u64, - ) { - let mut stm1 = conn - .prepare( - "insert into payable (wallet_address, balance, \ - last_paid_timestamp, pending_payable_rowid) values (?,?,?,?)", - ) - .unwrap(); - let params: &[&dyn ToSql] = &[ - &recipient_wallet, - &amount, - &to_time_t(timestamp), - &unsigned_to_signed(rowid).unwrap(), - ]; - let row_changed = stm1.execute(params).unwrap(); - assert_eq!(row_changed, 1); - } - #[test] fn transaction_confirmed_works() { let home_dir = ensure_node_home_directory_exists("payable_dao", "transaction_confirmed_works"); let boxed_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); - let secondary_conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); let hash = H256::from_uint(&U256::from(12345)); let rowid = 789; let previous_timestamp = from_time_t(190_000_000); @@ -545,46 +575,46 @@ mod tests { let payment = 6666; let wallet = make_wallet("bobble"); { - create_account_with_pending_payment( - boxed_conn.as_ref(), - &wallet, + insert_record_fn( + &*boxed_conn, + &wallet.to_string(), starting_amount, - previous_timestamp, - rowid, - ) + to_time_t(previous_timestamp), + Some(sign_conversion::(rowid).unwrap()), + ); } let subject = PayableDaoReal::new(boxed_conn); - let status_before = account_status(&secondary_conn, &wallet); - assert_eq!( - status_before, - Some(PayableAccount { - wallet: wallet.clone(), - balance: starting_amount, - last_paid_timestamp: previous_timestamp, - pending_payable_opt: Some(PendingPayableId { - rowid, - hash: H256::from_uint(&U256::from(0)) - }) //hash is just garbage - }) - ); let pending_payable_fingerprint = PendingPayableFingerprint { rowid_opt: Some(rowid), timestamp: payable_timestamp, hash, attempt_opt: Some(attempt), - amount: payment as u64, + amount: payment, process_error: None, }; + let status_before = subject.account_status(&wallet); let result = subject.transaction_confirmed(&pending_payable_fingerprint); assert_eq!(result, Ok(())); - let status_after = account_status(&secondary_conn, &wallet); + assert_eq!( + status_before, + Some(PayableAccount { + wallet: wallet.clone(), + balance_wei: starting_amount as u128, + last_paid_timestamp: previous_timestamp, + pending_payable_opt: Some(PendingPayableId { + rowid, + hash: H256::from_uint(&U256::from(0)) + }) //hash is just garbage + }) + ); + let status_after = subject.account_status(&wallet); assert_eq!( status_after, Some(PayableAccount { wallet, - balance: starting_amount - payment, + balance_wei: starting_amount as u128 - payment, last_paid_timestamp: payable_timestamp, pending_payable_opt: None }) @@ -597,26 +627,31 @@ mod tests { "payable_dao", "transaction_confirmed_works_for_generic_sql_error", ); - let conn = how_to_trick_rusqlite_for_an_error(&home_dir); - let conn_wrapped = ConnectionWrapperReal::new(conn); + let conn = trick_rusqlite_with_read_only_conn(&home_dir); + let conn_wrapped = Box::new(ConnectionWrapperReal::new(conn)); let mut pending_payable_fingerprint = make_pending_payable_fingerprint(); let hash = H256::from_uint(&U256::from(12345)); let rowid = 789; pending_payable_fingerprint.hash = hash; pending_payable_fingerprint.rowid_opt = Some(rowid); - let subject = PayableDaoReal::new(Box::new(conn_wrapped)); + let subject = PayableDaoReal::new(conn_wrapped); let result = subject.transaction_confirmed(&pending_payable_fingerprint); assert_eq!( result, Err(PayableDaoError::RusqliteError( - "attempt to write a readonly database".to_string() + "Error from invalid update command for payable table and change of -12345 wei to \ + 'pending_payable_rowid = 789' with error 'attempt to write a readonly database'" + .to_string() )) ) } #[test] + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128" + )] fn transaction_confirmed_works_for_overflow_from_amount_stored_in_pending_payable_fingerprint() { let home_dir = ensure_node_home_directory_exists( @@ -625,7 +660,7 @@ mod tests { ); let subject = PayableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let mut pending_payable_fingerprint = make_pending_payable_fingerprint(); @@ -633,31 +668,27 @@ mod tests { let rowid = 789; pending_payable_fingerprint.hash = hash; pending_payable_fingerprint.rowid_opt = Some(rowid); - pending_payable_fingerprint.amount = u64::MAX; - //The overflow occurs before we start modifying the payable account so I decided not to create an example in the database - - let result = subject.transaction_confirmed(&pending_payable_fingerprint); + pending_payable_fingerprint.amount = u128::MAX; + //The overflow occurs before we start modifying the payable account so we can have the database empty - assert_eq!(result, Err(PayableDaoError::SignConversion(u64::MAX))) + let _ = subject.transaction_confirmed(&pending_payable_fingerprint); } - fn how_to_trick_rusqlite_for_an_error(path: &Path) -> Connection { + fn trick_rusqlite_with_read_only_conn(path: &Path) -> Connection { let db_path = path.join("experiment.db"); let conn = RusqliteConnection::open_with_flags(&db_path, OpenFlags::default()).unwrap(); - { - let mut stm = conn - .prepare( - "\ - create table payable (\ - wallet_address text primary key, - balance integer not null, - last_paid_timestamp integer not null, - pending_payable_rowid integer null)\ - ", - ) - .unwrap(); - stm.execute([]).unwrap(); - } + conn.prepare( + " + create table payable ( + wallet_address text primary key, + balance_high_b integer not null, + balance_low_b integer not null, + last_paid_timestamp integer not null, + pending_payable_rowid integer null)", + ) + .unwrap() + .execute([]) + .unwrap(); conn.close().unwrap(); let conn = RusqliteConnection::open_with_flags(&db_path, OpenFlags::SQLITE_OPEN_READ_ONLY) .unwrap(); @@ -670,14 +701,15 @@ mod tests { "payable_dao", "non_pending_payables_should_return_an_empty_vec_when_the_database_is_empty", ); - let subject = PayableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); - assert_eq!(subject.non_pending_payables(), vec![]); + let result = subject.non_pending_payables(); + + assert_eq!(result, vec![]); } #[test] @@ -688,25 +720,26 @@ mod tests { ); let subject = PayableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let mut flags = OpenFlags::empty(); flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE); let conn = Connection::open_with_flags(&home_dir.join(DATABASE_FILE), flags).unwrap(); - let insert = |wallet: &str, balance: i64, pending_payable_rowid: Option| { - let params: &[&dyn ToSql] = &[&wallet, &balance, &0i64, &pending_payable_rowid]; - - conn - .prepare("insert into payable (wallet_address, balance, last_paid_timestamp, pending_payable_rowid) values (?, ?, ?, ?)") - .unwrap() - .execute(params) - .unwrap(); + let conn = ConnectionWrapperReal::new(conn); + let insert = |wallet: &str, pending_payable_rowid: Option| { + insert_record_fn( + &conn, + wallet, + 1234567890123456, + 111_111_111, + pending_payable_rowid, + ); }; - insert("0x0000000000000000000000000000000000666f6f", 42, Some(15)); - insert("0x0000000000000000000000000000000000626172", 24, Some(16)); - insert(&make_wallet("foobar").to_string(), 44, None); - insert(&make_wallet("barfoo").to_string(), 22, None); + insert("0x0000000000000000000000000000000000666f6f", Some(15)); + insert(&make_wallet("foobar").to_string(), None); + insert("0x0000000000000000000000000000000000626172", Some(16)); + insert(&make_wallet("barfoo").to_string(), None); let result = subject.non_pending_payables(); @@ -715,14 +748,14 @@ mod tests { vec![ PayableAccount { wallet: make_wallet("foobar"), - balance: 44, - last_paid_timestamp: from_time_t(0), + balance_wei: 1234567890123456 as u128, + last_paid_timestamp: from_time_t(111_111_111), pending_payable_opt: None }, PayableAccount { wallet: make_wallet("barfoo"), - balance: 22, - last_paid_timestamp: from_time_t(0), + balance_wei: 1234567890123456 as u128, + last_paid_timestamp: from_time_t(111_111_111), pending_payable_opt: None }, ] @@ -730,91 +763,182 @@ mod tests { } #[test] - fn payable_amount_errors_on_insert_when_out_of_range() { + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128" + )] + fn payable_amount_panics_on_insert_with_overflow() { let home_dir = ensure_node_home_directory_exists( "payable_dao", - "payable_amount_precision_loss_panics_on_insert", + "payable_amount_panics_on_insert_with_overflow", ); let subject = PayableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); - let result = - subject.more_money_payable(SystemTime::now(), &make_wallet("foobar"), u64::MAX); - - assert_eq!(result, Err(PayableDaoError::SignConversion(u64::MAX))) + let _ = subject.more_money_payable(SystemTime::now(), &make_wallet("foobar"), u128::MAX); } #[test] - fn top_records_and_total() { - let home_dir = ensure_node_home_directory_exists("payable_dao", "top_records_and_total"); - let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); - let insert = |wallet: &str, - balance: i64, - timestamp: i64, - pending_payable_rowid: Option| { - let params: &[&dyn ToSql] = &[&wallet, &balance, ×tamp, &pending_payable_rowid]; - conn - .prepare("insert into payable (wallet_address, balance, last_paid_timestamp, pending_payable_rowid) values (?, ?, ?, ?)") - .unwrap() - .execute(params) - .unwrap(); - }; - let timestamp1 = dao_utils::now_time_t() - 80_000; - let timestamp2 = dao_utils::now_time_t() - 86_401; - let timestamp3 = dao_utils::now_time_t() - 86_000; - let timestamp4 = dao_utils::now_time_t() - 86_001; - insert( - "0x1111111111111111111111111111111111111111", - 999_999_999, // below minimum amount - reject - timestamp1, // below maximum age - None, - ); - insert( - "0x2222222222222222222222222222222222222222", - 1_000_000_000, // minimum amount - timestamp2, // above maximum age - reject - None, - ); - insert( - "0x3333333333333333333333333333333333333333", - 1_000_000_000, // minimum amount - timestamp3, // below maximum age - None, - ); - insert( - "0x4444444444444444444444444444444444444444", - 1_000_000_001, // above minimum amount - timestamp4, // below maximum age - Some(1), + fn custom_query_handles_empty_table_in_top_records_mode() { + let main_test_setup = |_conn: &dyn ConnectionWrapper, _insert: InsertPayableHelperFn| {}; + let subject = custom_query_test_body_for_payable( + "custom_query_handles_empty_table_in_top_records_mode", + main_test_setup, ); + + let result = subject.custom_query(CustomQuery::TopRecords { + count: 6, + ordered_by: Balance, + }); + + assert_eq!(result, None) + } + + type InsertPayableHelperFn<'b> = + &'b dyn for<'a> Fn(&'a dyn ConnectionWrapper, &'a str, i128, i64, Option); + + fn insert_record_fn( + conn: &dyn ConnectionWrapper, + wallet: &str, + balance: i128, + timestamp: i64, + pending_payable_rowid: Option, + ) { + let (high_bytes, low_bytes) = BigIntDivider::deconstruct(balance); let params: &[&dyn ToSql] = &[ - &String::from("0xabc4546cce78230a2312e12f3acb78747340456fe5237896666100143abcd223"), - &40, - &177777777, - &1, + &wallet, + &high_bytes, + &low_bytes, + ×tamp, + &pending_payable_rowid, ]; conn - .prepare("insert into pending_payable (transaction_hash,amount,payable_timestamp,attempt) values (?,?,?,?)") + .prepare("insert into payable (wallet_address, balance_high_b, balance_low_b, last_paid_timestamp, pending_payable_rowid) values (?, ?, ?, ?, ?)") .unwrap() .execute(params) .unwrap(); + } - let subject = PayableDaoReal::new(conn); + fn accounts_for_tests_of_top_records( + now: i64, + ) -> Box { + Box::new(move |conn, insert: InsertPayableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + 1_000_000_002, + now - 86_401, + None, + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + 7_562_000_300_000, + now - 86_001, + None, + ); + insert( + conn, + "0x3333333333333333333333333333333333333333", + 999_999_999, //balance smaller than 1 gwei + now - 86_000, + None, + ); + insert( + conn, + "0x4444444444444444444444444444444444444444", + 10_000_000_100, + now - 86_300, + None, + ); + insert( + conn, + "0x5555555555555555555555555555555555555555", + 10_000_000_100, + now - 86_401, + Some(1), + ); + }) + } + + #[test] + fn custom_query_in_top_records_mode_with_default_ordering() { + //Accounts of balances smaller than one gwei don't qualify. + //Two accounts differ only in debt's age but not balance which allows to check doubled ordering, + //here by balance and then by age. + let now = now_time_t(); + let main_test_setup = accounts_for_tests_of_top_records(now); + let subject = custom_query_test_body_for_payable( + "custom_query_in_top_records_mode_with_default_ordering", + main_test_setup, + ); + + let result = subject + .custom_query(CustomQuery::TopRecords { + count: 3, + ordered_by: Balance, + }) + .unwrap(); - let top_records = subject.top_records(1_000_000_000, 86400); - let total = subject.total(); assert_eq!( - top_records, + result, vec![ + PayableAccount { + wallet: Wallet::new("0x2222222222222222222222222222222222222222"), + balance_wei: 7_562_000_300_000, + last_paid_timestamp: from_time_t(now - 86_001), + pending_payable_opt: None + }, + PayableAccount { + wallet: Wallet::new("0x5555555555555555555555555555555555555555"), + balance_wei: 10_000_000_100, + last_paid_timestamp: from_time_t(now - 86_401), + pending_payable_opt: Some(PendingPayableId { + rowid: 1, + hash: H256::from_str( + "abc4546cce78230a2312e12f3acb78747340456fe5237896666100143abcd223" + ) + .unwrap() + }) + }, PayableAccount { wallet: Wallet::new("0x4444444444444444444444444444444444444444"), - balance: 1_000_000_001, - last_paid_timestamp: from_time_t(timestamp4), + balance_wei: 10_000_000_100, + last_paid_timestamp: from_time_t(now - 86_300), + pending_payable_opt: None + }, + ] + ); + } + + #[test] + fn custom_query_in_top_records_mode_ordered_by_age() { + //Accounts of balances smaller than one gwei don't qualify. + //Two accounts differ only in balance but not in the debt's age which allows to check doubled ordering, + //here by age and then by balance. + let now = now_time_t(); + let main_test_setup = accounts_for_tests_of_top_records(now); + let subject = custom_query_test_body_for_payable( + "custom_query_in_top_records_mode_ordered_by_age", + main_test_setup, + ); + + let result = subject + .custom_query(CustomQuery::TopRecords { + count: 3, + ordered_by: Age, + }) + .unwrap(); + + assert_eq!( + result, + vec![ + PayableAccount { + wallet: Wallet::new("0x5555555555555555555555555555555555555555"), + balance_wei: 10_000_000_100, + last_paid_timestamp: from_time_t(now - 86_401), pending_payable_opt: Some(PendingPayableId { rowid: 1, hash: H256::from_str( @@ -824,14 +948,255 @@ mod tests { }) }, PayableAccount { - wallet: Wallet::new("0x3333333333333333333333333333333333333333"), - balance: 1_000_000_000, - last_paid_timestamp: from_time_t(timestamp3), + wallet: Wallet::new("0x1111111111111111111111111111111111111111"), + balance_wei: 1_000_000_002, + last_paid_timestamp: from_time_t(now - 86_401), + pending_payable_opt: None + }, + PayableAccount { + wallet: Wallet::new("0x4444444444444444444444444444444444444444"), + balance_wei: 10_000_000_100, + last_paid_timestamp: from_time_t(now - 86_300), + pending_payable_opt: None + }, + ] + ); + } + + #[test] + fn custom_query_handles_empty_table_in_range_mode() { + let main_test_setup = |_conn: &dyn ConnectionWrapper, _insert: InsertPayableHelperFn| {}; + let subject = custom_query_test_body_for_payable( + "custom_query_handles_empty_table_in_range_mode", + main_test_setup, + ); + + let result = subject.custom_query(CustomQuery::RangeQuery { + min_age_s: 20000, + max_age_s: 200000, + min_amount_gwei: 500000000, + max_amount_gwei: 3500000000, + timestamp: SystemTime::now(), + }); + + assert_eq!(result, None) + } + + #[test] + fn custom_query_in_range_mode() { + //Two accounts differ only in debt's age but not balance which allows to check doubled ordering, + //by balance and then by age. + let now = now_time_t(); + let main_setup = |conn: &dyn ConnectionWrapper, insert: InsertPayableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + gwei_to_wei::<_, u64>(499_999_999), //too small + now - 70_000, + None, + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + gwei_to_wei::<_, u64>(1_800_456_000), + now - 55_120, + Some(1), + ); + insert( + conn, + "0x3333333333333333333333333333333333333333", + gwei_to_wei::<_, u64>(600_123_456), + now - 200_001, //too old + None, + ); + insert( + conn, + "0x4444444444444444444444444444444444444444", + gwei_to_wei::<_, u64>(1_033_456_000_u64), + now - 19_999, //too young + None, + ); + insert( + conn, + "0x5555555555555555555555555555555555555555", + gwei_to_wei::<_, u64>(35_000_000_001), //too big + now - 30_786, + None, + ); + insert( + conn, + "0x6666666666666666666666666666666666666666", + gwei_to_wei::<_, u64>(1_800_456_000u64), + now - 100_401, + None, + ); + insert( + conn, + "0x7777777777777777777777777777777777777777", + gwei_to_wei::<_, u64>(2_500_647_000u64), + now - 80_333, + None, + ); + }; + let subject = custom_query_test_body_for_payable("custom_query_in_range_mode", main_setup); + + let result = subject + .custom_query(CustomQuery::RangeQuery { + min_age_s: 20000, + max_age_s: 200000, + min_amount_gwei: 500_000_000, + max_amount_gwei: 35_000_000_000, + timestamp: from_time_t(now), + }) + .unwrap(); + + assert_eq!( + result, + vec![ + PayableAccount { + wallet: Wallet::new("0x7777777777777777777777777777777777777777"), + balance_wei: gwei_to_wei(2_500_647_000_u32), + last_paid_timestamp: from_time_t(now - 80_333), + pending_payable_opt: None + }, + PayableAccount { + wallet: Wallet::new("0x6666666666666666666666666666666666666666"), + balance_wei: gwei_to_wei(1_800_456_000_u32), + last_paid_timestamp: from_time_t(now - 100_401), pending_payable_opt: None }, + PayableAccount { + wallet: Wallet::new("0x2222222222222222222222222222222222222222"), + balance_wei: gwei_to_wei(1_800_456_000_u32), + last_paid_timestamp: from_time_t(now - 55_120), + pending_payable_opt: Some(PendingPayableId { + rowid: 1, + hash: H256::from_str( + "abc4546cce78230a2312e12f3acb78747340456fe5237896666100143abcd223" + ) + .unwrap() + }) + } ] ); - assert_eq!(total, 4_000_000_000) + } + + #[test] + fn range_query_does_not_display_values_from_below_1_gwei() { + let now = now_time_t(); + let timestamp_1 = now - 11_001; + let timestamp_2 = now - 5000; + let main_setup = |conn: &dyn ConnectionWrapper, insert: InsertPayableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + 400_005_601, + timestamp_1, + None, + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + 30_000_300_000, + timestamp_2, + None, + ); + }; + let subject = custom_query_test_body_for_payable( + "range_query_does_not_display_values_from_below_1_gwei", + main_setup, + ); + + let result = subject + .custom_query(CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: 200000, + min_amount_gwei: u64::MIN, + max_amount_gwei: 35, + timestamp: SystemTime::now(), + }) + .unwrap(); + + assert_eq!( + result, + vec![PayableAccount { + wallet: Wallet::new("0x2222222222222222222222222222222222222222"), + balance_wei: 30_000_300_000, + last_paid_timestamp: from_time_t(timestamp_2), + pending_payable_opt: None + },] + ) + } + + #[test] + fn total_works() { + let home_dir = ensure_node_home_directory_exists("payable_dao", "total_works"); + let conn = DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + let timestamp = dao_utils::now_time_t(); + insert_record_fn( + &*conn, + "0x1111111111111111111111111111111111111111", + 999_999_999, + timestamp - 1000, + None, + ); + insert_record_fn( + &*conn, + "0x2222222222222222222222222222222222222222", + 1_000_123_123, + timestamp - 2000, + None, + ); + insert_record_fn( + &*conn, + "0x3333333333333333333333333333333333333333", + 1_000_000_000, + timestamp - 3000, + None, + ); + insert_record_fn( + &*conn, + "0x4444444444444444444444444444444444444444", + 1_000_000_001, + timestamp - 4000, + Some(3), + ); + let subject = PayableDaoReal::new(conn); + + let total = subject.total(); + + assert_eq!(total, 4_000_123_123) + } + + #[test] + #[should_panic( + expected = "database corrupted: found negative value -999999 in payable table for row id 2" + )] + fn total_takes_negative_value_as_error() { + let home_dir = + ensure_node_home_directory_exists("payable_dao", "total_takes_negative_value_as_error"); + let conn = DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + insert_record_fn( + &*conn, + "0x1111111111111111111111111111111111111111", + 123_456, + 111_111_111, + None, + ); + insert_record_fn( + &*conn, + "0x2222222222222222222222222222222222222222", + -999_999, + 222_222_222, + None, + ); + let subject = PayableDaoReal::new(conn); + + let _ = subject.total(); } #[test] @@ -839,7 +1204,7 @@ mod tests { let home_dir = ensure_node_home_directory_exists("payable_dao", "correctly_totals_zero_records"); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PayableDaoReal::new(conn); @@ -847,4 +1212,44 @@ mod tests { assert_eq!(result, 0) } + + #[test] + #[should_panic( + expected = "Database is corrupt: PAYABLE table columns and/or types: (Err(FromSqlConversionFailure(0, Text, InvalidAddress)), Err(InvalidColumnIndex(1))" + )] + fn create_payable_account_panics_on_database_error() { + assert_account_creation_fn_fails_on_finding_wrong_columns_and_value_types( + PayableDaoReal::create_payable_account, + ); + } + + #[test] + fn payable_dao_implements_dao_table_identifier() { + assert_eq!(PayableDaoReal::table_name(), "payable") + } + + fn custom_query_test_body_for_payable(test_name: &str, main_setup_fn: F) -> PayableDaoReal + where + F: Fn(&dyn ConnectionWrapper, InsertPayableHelperFn), + { + let home_dir = ensure_node_home_directory_exists("payable_dao", test_name); + let conn = DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + main_setup_fn(conn.as_ref(), &insert_record_fn); + + let pending_payable_account: &[&dyn ToSql] = &[ + &String::from("0xabc4546cce78230a2312e12f3acb78747340456fe5237896666100143abcd223"), + &40, + &478945, + &177777777, + &1, + ]; + conn + .prepare("insert into pending_payable (transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt) values (?,?,?,?,?)") + .unwrap() + .execute(pending_payable_account) + .unwrap(); + PayableDaoReal::new(conn) + } } diff --git a/node/src/accountant/pending_payable_dao.rs b/node/src/accountant/pending_payable_dao.rs index 1c89b02f7..09afe61a8 100644 --- a/node/src/accountant/pending_payable_dao.rs +++ b/node/src/accountant/pending_payable_dao.rs @@ -1,9 +1,12 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. -use crate::accountant::unsigned_to_signed; +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::dao_utils::{ + from_time_t, to_time_t, DaoFactoryReal, VigilantRusqliteFlatten, +}; +use crate::accountant::{checked_conversion, sign_conversion}; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::dao_utils::{from_time_t, to_time_t, DaoFactoryReal}; use masq_lib::utils::ExpectValue; use rusqlite::types::Value::Null; use rusqlite::{Row, ToSql}; @@ -27,7 +30,7 @@ pub trait PendingPayableDao { fn insert_new_fingerprint( &self, transaction_hash: H256, - amount: u64, + amount: u128, timestamp: SystemTime, ) -> Result<(), PendingPayableDaoError>; fn delete_fingerprint(&self, id: u64) -> Result<(), PendingPayableDaoError>; @@ -52,42 +55,44 @@ impl PendingPayableDao for PendingPayableDaoReal<'_> { } fn return_all_fingerprints(&self) -> Vec { - let mut stm = self.conn.prepare("select rowid, transaction_hash, amount, payable_timestamp, attempt from pending_payable where process_error is null").expect("Internal error"); + let mut stm = self.conn.prepare("select rowid, transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt from pending_payable where process_error is null").expect("Internal error"); stm.query_map([], |row| { let rowid: u64 = Self::get_with_expect(row, 0); let transaction_hash: String = Self::get_with_expect(row, 1); - let amount: u64 = Self::get_with_expect(row, 2); - let timestamp: i64 = Self::get_with_expect(row, 3); - let attempt: u16 = Self::get_with_expect(row, 4); + let amount_high_bytes: i64 = Self::get_with_expect(row, 2); + let amount_low_bytes: i64 = Self::get_with_expect(row, 3); + let timestamp: i64 = Self::get_with_expect(row, 4); + let attempt: u16 = Self::get_with_expect(row, 5); Ok(PendingPayableFingerprint { rowid_opt: Some(rowid), timestamp: from_time_t(timestamp), hash: H256::from_str(&transaction_hash[2..]).expectv("string hash"), attempt_opt: Some(attempt), - amount, + amount: checked_conversion::(BigIntDivider::reconstitute( + amount_high_bytes, + amount_low_bytes, + )), process_error: None, }) }) .expect("rusqlite failure") - .map(|fingerprint_result| match fingerprint_result { - Ok(val) => val, - Err(e) => panic!("hitting an error: {:?}", e), - }) + .vigilant_flatten() .collect() } fn insert_new_fingerprint( &self, transaction_hash: H256, - amount: u64, + amount: u128, timestamp: SystemTime, ) -> Result<(), PendingPayableDaoError> { - let signed_amount = - unsigned_to_signed(amount).map_err(PendingPayableDaoError::SignConversionError)?; - let mut stm = self.conn.prepare("insert into pending_payable (transaction_hash, amount, payable_timestamp, attempt, process_error) values (?,?,?,?,?)").expect("Internal error"); + let (high_bytes, low_bytes) = + BigIntDivider::deconstruct(checked_conversion::(amount)); + let mut stm = self.conn.prepare("insert into pending_payable (transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt, process_error) values (?,?,?,?,?,?)").expect("Internal error"); let params: &[&dyn ToSql] = &[ &format!("{:?}", transaction_hash), - &signed_amount, + &high_bytes, + &low_bytes, &to_time_t(timestamp), &1, &Null, @@ -100,8 +105,8 @@ impl PendingPayableDao for PendingPayableDaoReal<'_> { } fn delete_fingerprint(&self, id: u64) -> Result<(), PendingPayableDaoError> { - let signed_id = - unsigned_to_signed(id).expect("SQLite counts up to i64::MAX; should never happen"); + let signed_id = sign_conversion::(id) + .expect("SQLite counts up to i64::MAX; should never happen"); let mut stm = self .conn .prepare("delete from pending_payable where rowid = ?") @@ -117,8 +122,8 @@ impl PendingPayableDao for PendingPayableDaoReal<'_> { } fn update_fingerprint(&self, id: u64) -> Result<(), PendingPayableDaoError> { - let signed_id = - unsigned_to_signed(id).expect("SQLite counts up to i64::MAX; should never happen"); + let signed_id = sign_conversion::(id) + .expect("SQLite counts up to i64::MAX; should never happen"); let mut stm = self .conn .prepare("update pending_payable set attempt = attempt + 1 where rowid = ?") @@ -134,8 +139,8 @@ impl PendingPayableDao for PendingPayableDaoReal<'_> { } fn mark_failure(&self, id: u64) -> Result<(), PendingPayableDaoError> { - let signed_id = - unsigned_to_signed(id).expect("SQLite counts up to i64::MAX; should never happen"); + let signed_id = sign_conversion::(id) + .expect("SQLite counts up to i64::MAX; should never happen"); let mut stm = self .conn .prepare("update pending_payable set process_error = 'ERROR' where rowid = ?") @@ -170,6 +175,7 @@ impl<'a> PendingPayableDaoReal<'a> { pub fn new(conn: Box) -> Self { Self { conn } } + fn get_with_expect(row: &Row, index: usize) -> T { row.get(index).expect("database is corrupt") } @@ -177,15 +183,17 @@ impl<'a> PendingPayableDaoReal<'a> { #[cfg(test)] mod tests { + use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; + use crate::accountant::dao_utils::from_time_t; use crate::accountant::pending_payable_dao::{ PendingPayableDao, PendingPayableDaoError, PendingPayableDaoReal, }; - use crate::accountant::unsigned_to_signed; + use crate::accountant::{checked_conversion, sign_conversion}; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::database::connection_wrapper::ConnectionWrapperReal; - use crate::database::dao_utils::from_time_t; - use crate::database::db_initializer::{DbInitializer, DbInitializerReal, DATABASE_FILE}; - use crate::database::db_migrations::MigratorConfig; + use crate::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, DATABASE_FILE, + }; use ethereum_types::BigEndianHash; use masq_lib::test_utils::utils::ensure_node_home_directory_exists; use rusqlite::{Connection, Error, OpenFlags, Row}; @@ -200,7 +208,7 @@ mod tests { "insert_fingerprint_happy_path", ); let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let hash = H256::from_uint(&U256::from(45466)); let amount = 55556; @@ -231,7 +239,7 @@ mod tests { ensure_node_home_directory_exists("pending_payable_dao", "insert_fingerprint_sad_path"); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); } let conn_read_only = Connection::open_with_flags( @@ -262,7 +270,7 @@ mod tests { "fingerprint_rowid_when_record_reachable", ); let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PendingPayableDaoReal::new(wrapped_conn); let timestamp = from_time_t(195_000_000); @@ -286,7 +294,7 @@ mod tests { "fingerprint_rowid_when_nonexistent_record", ); let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); { let mut stm = wrapped_conn @@ -311,7 +319,7 @@ mod tests { "return_all_fingerprints_works_when_no_records_with_errors_marks", ); let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PendingPayableDaoReal::new(wrapped_conn); let timestamp_1 = from_time_t(195_000_000); @@ -363,7 +371,7 @@ mod tests { "return_all_fingerprints_works_when_some_records_with_errors_marks", ); let wrapped_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = PendingPayableDaoReal::new(wrapped_conn); let timestamp = from_time_t(198_000_000); @@ -406,7 +414,7 @@ mod tests { "delete_fingerprint_happy_path", ); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let hash = H256::from_uint(&U256::from(666666)); let rowid = 1; @@ -422,7 +430,7 @@ mod tests { assert_eq!(result, Ok(())); let conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); - let signed_row_id = unsigned_to_signed(rowid).unwrap(); + let signed_row_id = sign_conversion::(rowid).unwrap(); let mut stm2 = conn .prepare("select * from pending_payable where rowid = ?") .unwrap(); @@ -438,7 +446,7 @@ mod tests { ensure_node_home_directory_exists("pending_payable_dao", "delete_fingerprint_sad_path"); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); } let conn_read_only = Connection::open_with_flags( @@ -467,7 +475,7 @@ mod tests { "update_fingerprint_after_scan_cycle_works", ); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let hash = H256::from_uint(&U256::from(666)); let amount = 1234; @@ -505,7 +513,7 @@ mod tests { ); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); } let conn_read_only = Connection::open_with_flags( @@ -531,7 +539,7 @@ mod tests { let home_dir = ensure_node_home_directory_exists("pending_payable_dao", "mark_failure_works"); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let hash = H256::from_uint(&U256::from(666)); let amount = 1234; @@ -544,44 +552,42 @@ mod tests { } let assert_conn = Connection::open(home_dir.join(DATABASE_FILE)).unwrap(); let mut assert_stm = assert_conn - .prepare("select * from pending_payable") + .prepare("select rowid, transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt, process_error from pending_payable") .unwrap(); - let mut assert_closure = || { - assert_stm - .query_row([], |row| { - let rowid: u64 = row.get(0).unwrap(); - let transaction_hash: String = row.get(1).unwrap(); - let amount: u64 = row.get(2).unwrap(); - let timestamp: i64 = row.get(3).unwrap(); - let attempt: u16 = row.get(4).unwrap(); - let process_error: Option = row.get(5).unwrap(); - Ok(PendingPayableFingerprint { - rowid_opt: Some(rowid), - timestamp: from_time_t(timestamp), - hash: H256::from_str(&transaction_hash[2..]).unwrap(), - attempt_opt: Some(attempt), - amount, - process_error, - }) - }) - .unwrap() - }; - let assertion_before = assert_closure(); - assert_eq!(assertion_before.hash, hash); - assert_eq!(assertion_before.rowid_opt.unwrap(), 1); - assert_eq!(assertion_before.attempt_opt.unwrap(), 1); - assert_eq!(assertion_before.process_error, None); - assert_eq!(assertion_before.timestamp, timestamp); let result = subject.mark_failure(1); assert_eq!(result, Ok(())); - let assertion_after = assert_closure(); - assert_eq!(assertion_after.hash, hash); - assert_eq!(assertion_after.rowid_opt.unwrap(), 1); - assert_eq!(assertion_after.attempt_opt.unwrap(), 1); - assert_eq!(assertion_after.process_error, Some("ERROR".to_string())); - assert_eq!(assertion_after.timestamp, timestamp); + let resulting_fingerprint = assert_stm + .query_row([], |row| { + let rowid: u64 = row.get(0).unwrap(); + let transaction_hash: String = row.get(1).unwrap(); + let amount_high_b: i64 = row.get(2).unwrap(); + let amount_low_b: i64 = row.get(3).unwrap(); + let timestamp: i64 = row.get(4).unwrap(); + let attempt: u16 = row.get(5).unwrap(); + let process_error: Option = row.get(6).unwrap(); + Ok(PendingPayableFingerprint { + rowid_opt: Some(rowid), + timestamp: from_time_t(timestamp), + hash: H256::from_str(&transaction_hash[2..]).unwrap(), + attempt_opt: Some(attempt), + amount: checked_conversion::(BigIntDivider::reconstitute( + amount_high_b, + amount_low_b, + )), + process_error, + }) + }) + .unwrap(); + assert_eq!(resulting_fingerprint.hash, hash); + assert_eq!(resulting_fingerprint.rowid_opt.unwrap(), 1); + assert_eq!(resulting_fingerprint.attempt_opt.unwrap(), 1); + assert_eq!( + resulting_fingerprint.process_error, + Some("ERROR".to_string()) + ); + assert_eq!(resulting_fingerprint.timestamp, timestamp); } #[test] @@ -590,7 +596,7 @@ mod tests { ensure_node_home_directory_exists("pending_payable_dao", "mark_failure_sad_path"); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); } let conn_read_only = Connection::open_with_flags( diff --git a/node/src/accountant/receivable_dao.rs b/node/src/accountant/receivable_dao.rs index f8b6d797d..822ab3530 100644 --- a/node/src/accountant/receivable_dao.rs +++ b/node/src/accountant/receivable_dao.rs @@ -1,23 +1,42 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +use crate::accountant::big_int_processing::big_int_db_processor::KnownKeyVariants::WalletAddress; +use crate::accountant::big_int_processing::big_int_db_processor::WeiChange::{ + Addition, Subtraction, +}; +use crate::accountant::big_int_processing::big_int_db_processor::{ + BigIntDbProcessor, BigIntSqlConfig, SQLParamsBuilder, TableNameDAO, +}; +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::dao_utils::{ + sum_i128_values_from_table, to_time_t, AssemblerFeeder, CustomQuery, DaoFactoryReal, + RangeStmConfig, TopStmConfig, VigilantRusqliteFlatten, +}; use crate::accountant::receivable_dao::ReceivableDaoError::RusqliteError; -use crate::accountant::unsigned_to_signed; +use crate::accountant::{checked_conversion, ThresholdUtils}; +use crate::accountant::{dao_utils, gwei_to_wei}; use crate::blockchain::blockchain_interface::BlockchainTransaction; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::dao_utils; -use crate::database::dao_utils::{to_time_t, DaoFactoryReal}; +use crate::database::db_initializer::{connection_or_panic, DbInitializerReal}; use crate::db_config::persistent_configuration::PersistentConfigError; use crate::sub_lib::accountant::PaymentThresholds; use crate::sub_lib::wallet::Wallet; use indoc::indoc; +use itertools::Either; +use itertools::Either::Left; +use masq_lib::constants::WEIS_OF_GWEI; use masq_lib::logger::Logger; -use rusqlite::types::{ToSql, Type}; +use masq_lib::utils::{plus, ExpectValue}; +use rusqlite::OptionalExtension; +use rusqlite::Row; use rusqlite::{named_params, Error}; -use rusqlite::{OptionalExtension, Row}; +#[cfg(test)] +use std::any::Any; use std::time::SystemTime; #[derive(Debug, PartialEq, Eq)] pub enum ReceivableDaoError { - SignConversion(u64), + SignConversion(u128), ConfigurationError(String), RusqliteError(String), } @@ -37,7 +56,7 @@ impl From for ReceivableDaoError { #[derive(Debug, Clone, PartialEq, Eq)] pub struct ReceivableAccount { pub wallet: Wallet, - pub balance: i64, + pub balance_wei: i128, pub last_received_timestamp: SystemTime, } @@ -46,15 +65,11 @@ pub trait ReceivableDao: Send { &self, now: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), ReceivableDaoError>; fn more_money_received(&mut self, now: SystemTime, transactions: Vec); - fn account_status(&self, wallet: &Wallet) -> Option; - - fn receivables(&self) -> Vec; - fn new_delinquencies( &self, now: SystemTime, @@ -63,9 +78,14 @@ pub trait ReceivableDao: Send { fn paid_delinquencies(&self, payment_thresholds: &PaymentThresholds) -> Vec; - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec; + fn custom_query(&self, custom_query: CustomQuery) -> Option>; + + fn total(&self) -> i128; - fn total(&self) -> i64; + //test only intended method but because of share with multi-node tests conditional compilation is disallowed + fn account_status(&self, wallet: &Wallet) -> Option; + + as_any_dcl!(); } pub trait ReceivableDaoFactory { @@ -74,12 +94,25 @@ pub trait ReceivableDaoFactory { impl ReceivableDaoFactory for DaoFactoryReal { fn make(&self) -> Box { - Box::new(ReceivableDaoReal::new(self.make_connection())) + let init_config = self + .init_config + .take() + .expectv("init config") + .add_special_conn_setup( + BigIntDivider::register_big_int_deconstruction_for_sqlite_connection, + ); + Box::new(ReceivableDaoReal::new(connection_or_panic( + &DbInitializerReal::default(), + self.data_directory.as_path(), + init_config, + ))) } } +#[derive(Debug)] pub struct ReceivableDaoReal { conn: Box, + big_int_db_processor: BigIntDbProcessor, logger: Logger, } @@ -88,251 +121,157 @@ impl ReceivableDao for ReceivableDaoReal { &self, timestamp: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), ReceivableDaoError> { - let signed_amount = - unsigned_to_signed(amount).map_err(ReceivableDaoError::SignConversion)?; - match self.try_add_debt(wallet, signed_amount) { - Ok(true) => Ok(()), - Ok(false) => match self.try_insert(timestamp, wallet, signed_amount) { - Ok(_) => Ok(()), - Err(e) => { - fatal!(self.logger, "Couldn't insert; database is corrupt: {}", e); - } - }, - Err(e) => { - fatal!(self.logger, "Couldn't update: database is corrupt: {}", e); - } - } + Ok(self.big_int_db_processor.execute(Left(self.conn.as_ref()), BigIntSqlConfig::new( + "insert into receivable (wallet_address, balance_high_b, balance_low_b, last_received_timestamp) values (:wallet, :balance_high_b, :balance_low_b, :last_received) on conflict (wallet_address) do \ + update set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b", + "update receivable set balance_high_b = :balance_high_b, balance_low_b = :balance_low_b", + SQLParamsBuilder::default() + .key( WalletAddress(wallet)) + .wei_change(Addition("balance",amount)) + .other(vec![(":last_received",&to_time_t(timestamp))]) + .build() + ))?) } fn more_money_received(&mut self, timestamp: SystemTime, payments: Vec) { self.try_multi_insert_payment(timestamp, &payments) - .unwrap_or_else(|e| { - let mut report_lines = - vec![format!("{:10} {:42} {:18}", "Block #", "Wallet", "Amount")]; - let mut sum = 0u64; - payments.iter().for_each(|t| { - report_lines.push(format!( - "{:10} {:42} {:18}", - t.block_number, t.from, t.gwei_amount - )); - sum += t.gwei_amount; - }); - report_lines.push(format!("{:10} {:42} {:18}", "TOTAL", "", sum)); - let report = report_lines.join("\n"); - error!( - self.logger, - "Payment reception failed, rolling back: {:?}\n{}", e, report - ); - }) - } - - fn account_status(&self, wallet: &Wallet) -> Option { - let mut stmt = self - .conn - .prepare( - "select wallet_address, balance, last_received_timestamp from receivable where wallet_address = ?", - ) - .expect("Internal error"); - match stmt.query_row(&[&wallet], Self::row_to_account).optional() { - Ok(value) => value, - Err(e) => panic!("Database is corrupt: {:?}", e), - } - } - - fn receivables(&self) -> Vec { - let mut stmt = self - .conn - .prepare("select balance, last_received_timestamp, wallet_address from receivable") - .expect("Internal error"); - - stmt.query_map([], |row| { - let balance_result = row.get::(0); - let last_received_timestamp_result = row.get::(1); - let wallet = row.get::(2); - match (balance_result, last_received_timestamp_result, wallet) { - (Ok(balance), Ok(last_received_timestamp), Ok(wallet)) => Ok(ReceivableAccount { - wallet, - balance, - last_received_timestamp: dao_utils::from_time_t(last_received_timestamp), - }), - tuple => panic!("receivables(): Database is corrupt: RECEIVABLE table columns and/or types: {:?}", tuple), - } - }) - .expect("Database is corrupt") - .flatten() - .collect() + .unwrap_or_else(|e| self.more_money_received_pretty_error_log(&payments, e)) } fn new_delinquencies( &self, - system_now: SystemTime, + now: SystemTime, payment_thresholds: &PaymentThresholds, ) -> Vec { - let now = to_time_t(system_now); - let slope = (payment_thresholds.permanent_debt_allowed_gwei as f64 - - payment_thresholds.debt_threshold_gwei as f64) - / (payment_thresholds.threshold_interval_sec as f64); + let slope = ThresholdUtils::slope(payment_thresholds) as i64; + let (permanent_debt_allowed_high_b, permanent_debt_allowed_low_b) = + BigIntDivider::deconstruct(gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei)); let sql = indoc!( r" - select r.wallet_address, r.balance, r.last_received_timestamp - from receivable r left outer join banned b on r.wallet_address = b.wallet_address - where - r.last_received_timestamp < :sugg_and_grace - and r.balance > :balance_to_decrease_from + :slope * (:sugg_and_grace - r.last_received_timestamp) - and r.balance > :permanent_debt - and b.wallet_address is null - " + select r.wallet_address, r.balance_high_b, r.balance_low_b, r.last_received_timestamp + from receivable r + left outer join banned b on r.wallet_address = b.wallet_address + where + r.last_received_timestamp < :sugg_and_grace + and ((r.balance_high_b > slope_drop_high_bytes(:debt_threshold, :slope, :sugg_and_grace - r.last_received_timestamp)) + or ((r.balance_high_b = slope_drop_high_bytes(:debt_threshold, :slope, :sugg_and_grace - r.last_received_timestamp)) + and (r.balance_low_b > slope_drop_low_bytes(:debt_threshold, :slope, :sugg_and_grace - r.last_received_timestamp)))) + and ((r.balance_high_b > :permanent_debt_allowed_high_b) or ((r.balance_high_b = 0) and (r.balance_low_b > :permanent_debt_allowed_low_b))) + and b.wallet_address is null + " ); - let mut stmt = self.conn.prepare(sql).expect("Couldn't prepare statement"); - stmt.query_map( - named_params! { - ":slope": slope, - ":sugg_and_grace": payment_thresholds.sugg_and_grace(now), - ":balance_to_decrease_from": payment_thresholds.debt_threshold_gwei, - ":permanent_debt": payment_thresholds.permanent_debt_allowed_gwei, - }, - Self::row_to_account, - ) - .expect("Couldn't retrieve new delinquencies: database corruption") - .flatten() - .collect() + self.conn + .prepare(sql) + .expect("Couldn't prepare statement") + .query_map( + named_params! { + ":debt_threshold": checked_conversion::(payment_thresholds.debt_threshold_gwei), + ":slope": slope, + ":sugg_and_grace": payment_thresholds.sugg_and_grace(to_time_t(now)), + ":permanent_debt_allowed_high_b": permanent_debt_allowed_high_b, + ":permanent_debt_allowed_low_b": permanent_debt_allowed_low_b + }, + Self::create_receivable_account, + ) + .expect("Couldn't retrieve new delinquencies: database corruption") + .vigilant_flatten() + .collect() } fn paid_delinquencies(&self, payment_thresholds: &PaymentThresholds) -> Vec { let sql = indoc!( r" - select r.wallet_address, r.balance, r.last_received_timestamp + select r.wallet_address, r.balance_high_b, r.balance_low_b, r.last_received_timestamp from receivable r inner join banned b on r.wallet_address = b.wallet_address where - r.balance <= :unban_balance + (r.balance_high_b < :unban_balance_high_b) or ((balance_high_b = :unban_balance_high_b) and (balance_low_b <= :unban_balance_low_b)) " ); let mut stmt = self.conn.prepare(sql).expect("Couldn't prepare statement"); + let (unban_balance_high_b, unban_balance_low_b) = BigIntDivider::deconstruct( + (payment_thresholds.unban_below_gwei as i128) * WEIS_OF_GWEI, + ); stmt.query_map( named_params! { - ":unban_balance": payment_thresholds.unban_below_gwei, + ":unban_balance_high_b": unban_balance_high_b, + ":unban_balance_low_b": unban_balance_low_b }, - Self::row_to_account, + Self::create_receivable_account, ) .expect("Couldn't retrieve new delinquencies: database corruption") - .flatten() + .vigilant_flatten() .collect() } - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec { - let min_amt = unsigned_to_signed(minimum_amount).unwrap_or(0x7FFF_FFFF_FFFF_FFFF); - let max_age = unsigned_to_signed(maximum_age).unwrap_or(0x7FFF_FFFF_FFFF_FFFF); - let min_timestamp = dao_utils::now_time_t() - max_age; - let mut stmt = self - .conn - .prepare( - r#" - select - balance, - last_received_timestamp, - wallet_address - from - receivable - where - balance >= ? and - last_received_timestamp >= ? - order by - balance desc, - last_received_timestamp desc - "#, - ) - .expect("Internal error"); - let params: &[&dyn ToSql] = &[&min_amt, &min_timestamp]; - stmt.query_map(params, |row| { - let balance_result = row.get(0); - let last_paid_timestamp_result = row.get(1); - let wallet_result: Result = row.get(2); - match (balance_result, last_paid_timestamp_result, wallet_result) { - (Ok(balance), Ok(last_paid_timestamp), Ok(wallet)) => Ok(ReceivableAccount { - wallet, - balance, - last_received_timestamp: dao_utils::from_time_t(last_paid_timestamp), - }), - tuple => panic!("top_records(): Database is corrupt: RECEIVABLE table columns and/or types: {:?}", tuple), - } - }) - .expect("Database is corrupt") - .flatten() - .collect() + fn custom_query(&self, custom_query: CustomQuery) -> Option> { + let variant_top = TopStmConfig{ + limit_clause: "limit :limit_count", + gwei_min_resolution_clause: "where (balance_high_b > 0) or ((balance_high_b = 0) and (balance_low_b >= 1000000000))", + age_ordering_clause: "last_received_timestamp asc", + }; + let variant_range = RangeStmConfig { + where_clause: "where ((last_received_timestamp <= :max_timestamp) and (last_received_timestamp >= :min_timestamp)) \ + and ((balance_high_b > :min_balance_high_b) or ((balance_high_b = :min_balance_high_b) and (balance_low_b >= :min_balance_low_b))) \ + and ((balance_high_b < :max_balance_high_b) or ((balance_high_b = :max_balance_high_b) and (balance_low_b <= :max_balance_low_b)))", + gwei_min_resolution_clause: "and (((balance_high_b > 0) or ((balance_high_b = 0) and (balance_low_b >= 1000000000))) \ + or ((balance_high_b < -1) or ((balance_high_b = -1) and (balance_low_b <= 9223372035854775807))))", //i64::MAX - 1*10^9 + secondary_order_param: "last_received_timestamp asc" + }; + + custom_query.query::<_, i64, _, _>( + self.conn.as_ref(), + Self::stm_assembler_of_receivable_cq, + variant_top, + variant_range, + Self::create_receivable_account, + ) + } + + fn total(&self) -> i128 { + let value_creation = |_: usize, row: &Row| { + Ok(BigIntDivider::reconstitute( + row.get::(0).expectv("high bytes"), + row.get::(1).expectv("low_bytes"), + )) + }; + sum_i128_values_from_table( + self.conn.as_ref(), + &Self::table_name(), + "balance", + value_creation, + ) } - fn total(&self) -> i64 { + fn account_status(&self, wallet: &Wallet) -> Option { let mut stmt = self .conn - .prepare("select sum(balance) from receivable") + .prepare( + "select wallet_address, balance_high_b, balance_low_b, last_received_timestamp from receivable where wallet_address = ?", + ) .expect("Internal error"); - match stmt.query_row([], |row| { - let total_balance_result: Result = row.get(0); - match total_balance_result { - Ok(total_balance) => Ok(total_balance), - Err(e) - if e == rusqlite::Error::InvalidColumnType( - 0, - "sum(balance)".to_string(), - Type::Null, - ) => - { - Ok(0) - } - Err(e) => panic!( - "total(): Database is corrupt: RECEIVABLE table columns and/or types: {:?}", - e - ), - } - }) { + match stmt + .query_row(&[&wallet], Self::create_receivable_account) + .optional() + { Ok(value) => value, Err(e) => panic!("Database is corrupt: {:?}", e), } } + + as_any_impl!(); } impl ReceivableDaoReal { pub fn new(conn: Box) -> ReceivableDaoReal { ReceivableDaoReal { conn, + big_int_db_processor: BigIntDbProcessor::default(), logger: Logger::new("ReceivableDaoReal"), } } - // Question: Why would we not update last_received_timestamp here? Is this a bug? - // Answer: No, it's not a bug. Adding more debt is different from receiving a payment. - fn try_add_debt(&self, wallet: &Wallet, amount: i64) -> Result { - let mut stmt = self - .conn - .prepare("update receivable set balance = balance + ? where wallet_address = ?") - .expect("Internal error"); - let params: &[&dyn ToSql] = &[&amount, &wallet]; - match stmt.execute(params) { - Ok(0) => Ok(false), - Ok(_) => Ok(true), - Err(e) => Err(format!("{}", e)), - } - } - - // Question: We didn't just receive a payment; why are we setting last_received_timestamp? - // Answer: All new debts should start out young so that they don't trigger immediate delinquency. - // Except for exotic tests, timestamp should be now or in the very recent past. - fn try_insert( - &self, - timestamp: SystemTime, - wallet: &Wallet, - amount: i64, - ) -> Result<(), String> { - let mut stmt = self.conn.prepare("insert into receivable (wallet_address, balance, last_received_timestamp) values (?, ?, ?)").expect("Internal error"); - let params: &[&dyn ToSql] = &[&wallet, &amount, &to_time_t(timestamp)]; - match stmt.execute(params) { - Ok(_) => Ok(()), - Err(e) => Err(format!("{}", e)), - } - } - fn try_multi_insert_payment( &mut self, timestamp: SystemTime, @@ -340,17 +279,17 @@ impl ReceivableDaoReal { ) -> Result<(), ReceivableDaoError> { let xactn = self.conn.transaction()?; { - let mut stmt = xactn.prepare("update receivable set balance = balance - ?, last_received_timestamp = ? where wallet_address = ?") - .expect ("Internal SQL error"); for transaction in payments { - let timestamp = dao_utils::to_time_t(timestamp); - let gwei_amount = match unsigned_to_signed(transaction.gwei_amount) { - Ok(amount) => amount, - Err(e) => return Err(ReceivableDaoError::SignConversion(e)), - }; - let params: &[&dyn ToSql] = &[&gwei_amount, ×tamp, &transaction.from]; - stmt.execute(params) - .map_err(|e| ReceivableDaoError::RusqliteError(e.to_string()))?; + self.big_int_db_processor.execute(Either::Right(&xactn), BigIntSqlConfig::new( + //the plus signs are correct, 'Subtraction' in the wei_change converts x of u128 to -x of i128 which leads into the high bytes integer being negative + "update receivable set balance_high_b = balance_high_b + :balance_high_b, balance_low_b = balance_low_b + :balance_low_b, last_received_timestamp = :last_received where wallet_address = :wallet", + "update receivable set balance_high_b = :balance_high_b, balance_low_b = :balance_low_b, last_received_timestamp = :last_received where wallet_address = :wallet", + SQLParamsBuilder::default() + .key( WalletAddress(&transaction.from)) + .wei_change(Subtraction("balance",transaction.wei_amount)) + .other(vec![(":last_received", &to_time_t(timestamp))]) + .build() + ))? } } match xactn.commit() { @@ -360,34 +299,116 @@ impl ReceivableDaoReal { } } - fn row_to_account(row: &Row) -> rusqlite::Result { - let wallet: Result = row.get(0); - let balance_result = row.get(1); - let last_received_timestamp_result = row.get(2); - match (wallet, balance_result, last_received_timestamp_result) { - (Ok(wallet), Ok(balance), Ok(last_received_timestamp)) => Ok(ReceivableAccount { - wallet, - balance, - last_received_timestamp: dao_utils::from_time_t(last_received_timestamp), - }), - tuple => panic!("row_to_account(): Database is corrupt: RECEIVABLE table columns and/or types: {:?}", tuple), + fn create_receivable_account(row: &Row) -> rusqlite::Result { + let wallet: Result = row.get(0); + let balance_high_b_result = row.get(1); + let balance_low_b_result = row.get(2); + let last_received_timestamp_result = row.get(3); + match ( + wallet, + balance_high_b_result, + balance_low_b_result, + last_received_timestamp_result, + ) { + (Ok(wallet), Ok(high_bytes), Ok(low_bytes), Ok(last_received_timestamp)) => { + Ok(ReceivableAccount { + wallet, + balance_wei: BigIntDivider::reconstitute(high_bytes, low_bytes), + last_received_timestamp: dao_utils::from_time_t(last_received_timestamp), + }) + } + e => panic!( + "Database is corrupt: RECEIVABLE table columns and/or types: {:?}", + e + ), } } + + fn stm_assembler_of_receivable_cq(feeder: AssemblerFeeder) -> String { + format!( + "select + wallet_address, + balance_high_b, + balance_low_b, + last_received_timestamp + from + receivable + {} {} + order by + {}, + {} + {}", + feeder.main_where_clause, + feeder.where_clause_extension, + feeder.order_by_first_param, + feeder.order_by_second_param, + feeder.limit_clause + ) + } + + fn more_money_received_pretty_error_log( + &self, + payments: &[BlockchainTransaction], + error: ReceivableDaoError, + ) { + fn finalize_report(data: (Vec, u128)) -> String { + let (report_lines, sum) = data; + plus(report_lines, format!("{:10} {:42} {:18}", "TOTAL", "", sum)).join("\n") + } + fn record_one_more_transaction( + acc: (Vec, u128), + bc_tx: &BlockchainTransaction, + ) -> (Vec, u128) { + let lines_adjusted = plus( + acc.0, + format!( + "{:10} {:42} {:18}", + bc_tx.block_number, bc_tx.from, bc_tx.wei_amount + ), + ); + let sum_so_far = acc.1 + bc_tx.wei_amount; + (lines_adjusted, sum_so_far) + } + let init = ( + vec![format!("{:10} {:42} {:18}", "Block #", "Wallet", "Amount")], + 0_u128, + ); + let aggregated = payments.iter().fold(init, record_one_more_transaction); + error!( + self.logger, + "Payment reception failed, rolling back: {:?}\n{}", + error, + finalize_report(aggregated) + ); + } +} + +impl TableNameDAO for ReceivableDaoReal { + fn table_name() -> String { + String::from("receivable") + } } #[cfg(test)] mod tests { use super::*; - use crate::accountant::test_utils::make_receivable_account; - use crate::database::dao_utils::{from_time_t, now_time_t, to_time_t}; - use crate::database::db_initializer::DbInitializer; - use crate::database::db_initializer::DbInitializerReal; - use crate::database::db_migrations::MigratorConfig; + use crate::accountant::dao_utils::{from_time_t, now_time_t, to_time_t}; + use crate::accountant::gwei_to_wei; + use crate::accountant::test_utils::{ + assert_account_creation_fn_fails_on_finding_wrong_columns_and_value_types, + make_receivable_account, + }; + use crate::database::db_initializer::{DbInitializationConfig, DbInitializer}; + use crate::database::db_initializer::{DbInitializerReal, ExternalData}; use crate::db_config::persistent_configuration::PersistentConfigError; use crate::test_utils::assert_contains; use crate::test_utils::make_wallet; + use masq_lib::messages::TopRecordsOrdering::{Age, Balance}; use masq_lib::test_utils::logging::{init_test_logging, TestLogHandler}; use masq_lib::test_utils::utils::ensure_node_home_directory_exists; + use masq_lib::utils::NeighborhoodModeLight; + use rusqlite::ToSql; + use std::path::Path; #[test] fn conversion_from_pce_works() { @@ -402,6 +423,44 @@ mod tests { } #[test] + fn factory_produces_connection_that_is_familiar_with_our_defined_sqlite_functions() { + let home_dir = ensure_node_home_directory_exists( + "receivable_dao", + "factory_produces_connection_that_is_familiar_with_our_defined_sqlite_functions", + ); + DbInitializerReal::default() + .initialize( + &home_dir, + DbInitializationConfig::create_or_migrate(ExternalData { + chain: Default::default(), + neighborhood_mode: NeighborhoodModeLight::Standard, + db_password_opt: None, + }), + ) + .unwrap(); + let subject = DaoFactoryReal::new(&home_dir, DbInitializationConfig::panic_on_migration()); + + let receivable_dao = subject.make(); + + let definite_dao = receivable_dao + .as_any() + .downcast_ref::() + .unwrap(); + definite_dao + .conn + .prepare("select slope_drop_high_bytes(4578745, -2220000000, 123456)") + .unwrap(); + definite_dao + .conn + .prepare("select slope_drop_low_bytes(787845, -2220000000, 123456)") + .unwrap(); + //we didn't blow up, all is good + } + + #[test] + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128" + )] fn try_multi_insert_payment_handles_error_of_number_sign_check() { let home_dir = ensure_node_home_directory_exists( "receivable_dao", @@ -409,21 +468,16 @@ mod tests { ); let mut subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let payments = vec![BlockchainTransaction { block_number: 42u64, from: make_wallet("some_address"), - gwei_amount: 18446744073709551615, + wei_amount: u128::MAX, }]; - let result = subject.try_multi_insert_payment(SystemTime::now(), &payments.as_slice()); - - assert_eq!( - result, - Err(ReceivableDaoError::SignConversion(18446744073709551615)) - ) + let _ = subject.try_multi_insert_payment(SystemTime::now(), &payments.as_slice()); } #[test] @@ -434,7 +488,7 @@ mod tests { "try_multi_insert_payment_handles_error_adding_receivables", ); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); { let mut stmt = conn.prepare("drop table receivable").unwrap(); @@ -445,7 +499,7 @@ mod tests { let payments = vec![BlockchainTransaction { block_number: 42u64, from: make_wallet("some_address"), - gwei_amount: 18446744073709551615, + wei_amount: 18446744073709551615, }]; let _ = subject.try_multi_insert_payment(SystemTime::now(), payments.as_slice()); @@ -459,19 +513,17 @@ mod tests { ); let now = SystemTime::now(); let wallet = make_wallet("booga"); - let status = { - let subject = ReceivableDaoReal::new( - DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(), - ); + let subject = ReceivableDaoReal::new( + DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(), + ); - subject.more_money_receivable(now, &wallet, 1234).unwrap(); - subject.account_status(&wallet).unwrap() - }; + subject.more_money_receivable(now, &wallet, 1234).unwrap(); + let status = subject.account_status(&wallet).unwrap(); assert_eq!(status.wallet, wallet); - assert_eq!(status.balance, 1234); + assert_eq!(status.balance_wei, 1234); assert_eq!(to_time_t(status.last_received_timestamp), to_time_t(now)); } @@ -484,7 +536,7 @@ mod tests { let wallet = make_wallet("booga"); let subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let now = SystemTime::now(); @@ -496,11 +548,14 @@ mod tests { let status = subject.account_status(&wallet).unwrap(); assert_eq!(status.wallet, wallet); - assert_eq!(status.balance, 3579); + assert_eq!(status.balance_wei, 3579); assert_eq!(to_time_t(status.last_received_timestamp), to_time_t(now)); } #[test] + #[should_panic( + expected = "Overflow detected with 340282366920938463463374607431768211455: cannot be converted from u128 to i128" + )] fn more_money_receivable_works_for_overflow() { let home_dir = ensure_node_home_directory_exists( "receivable_dao", @@ -508,14 +563,11 @@ mod tests { ); let subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); - let result = - subject.more_money_receivable(SystemTime::now(), &make_wallet("booga"), u64::MAX); - - assert_eq!(result, Err(ReceivableDaoError::SignConversion(u64::MAX))) + let _ = subject.more_money_receivable(SystemTime::now(), &make_wallet("booga"), u128::MAX); } #[test] @@ -530,41 +582,35 @@ mod tests { let mut subject = { let subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); subject.more_money_receivable(now, &debtor1, 1234).unwrap(); subject.more_money_receivable(now, &debtor2, 2345).unwrap(); subject }; + let transactions = vec![ + BlockchainTransaction { + from: debtor1.clone(), + wei_amount: 1200_u128, + block_number: 35_u64, + }, + BlockchainTransaction { + from: debtor2.clone(), + wei_amount: 2300_u128, + block_number: 57_u64, + }, + ]; - let (status1, status2) = { - let transactions = vec![ - BlockchainTransaction { - from: debtor1.clone(), - gwei_amount: 1200u64, - block_number: 35u64, - }, - BlockchainTransaction { - from: debtor2.clone(), - gwei_amount: 2300u64, - block_number: 57u64, - }, - ]; - - subject.more_money_received(now, transactions); - ( - subject.account_status(&debtor1).unwrap(), - subject.account_status(&debtor2).unwrap(), - ) - }; + subject.more_money_received(now, transactions); + let status1 = subject.account_status(&debtor1).unwrap(); assert_eq!(status1.wallet, debtor1); - assert_eq!(status1.balance, 34); + assert_eq!(status1.balance_wei, 34); assert_eq!(to_time_t(status1.last_received_timestamp), to_time_t(now)); - + let status2 = subject.account_status(&debtor2).unwrap(); assert_eq!(status2.wallet, debtor2); - assert_eq!(status2.balance, 45); + assert_eq!(status2.balance_wei, 45); assert_eq!(to_time_t(status2.last_received_timestamp), to_time_t(now)); } @@ -577,40 +623,37 @@ mod tests { let debtor = make_wallet("unknown_wallet"); let mut subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); + let transactions = vec![BlockchainTransaction { + from: debtor.clone(), + wei_amount: 2300_u128, + block_number: 33_u64, + }]; - let status = { - let transactions = vec![BlockchainTransaction { - from: debtor.clone(), - gwei_amount: 2300u64, - block_number: 33u64, - }]; - subject.more_money_received(SystemTime::now(), transactions); - subject.account_status(&debtor) - }; + subject.more_money_received(SystemTime::now(), transactions); + let status = subject.account_status(&debtor); assert!(status.is_none()); } #[test] fn more_money_received_logs_when_try_multi_insert_payment_fails() { init_test_logging(); - let home_dir = ensure_node_home_directory_exists( "receivable_dao", "more_money_received_logs_when_try_multi_insert_payment_fails", ); let mut subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); // Sabotage the database so there'll be an error { let mut conn = DbInitializerReal::default() - .initialize(&home_dir, false, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::panic_on_migration()) .unwrap(); let xactn = conn.transaction().unwrap(); xactn @@ -620,29 +663,31 @@ mod tests { .unwrap(); xactn.commit().unwrap(); } - let payments = vec![ BlockchainTransaction { block_number: 1234567890, from: Wallet::new("0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), - gwei_amount: 123456789123456789, + wei_amount: 123456789123456789, }, BlockchainTransaction { block_number: 2345678901, from: Wallet::new("0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"), - gwei_amount: 234567891234567891, + wei_amount: 234567891234567891, }, BlockchainTransaction { block_number: 3456789012, from: Wallet::new("0xCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC"), - gwei_amount: 345678912345678912, + wei_amount: 345678912345678912, }, ]; subject.more_money_received(SystemTime::now(), payments); TestLogHandler::new().exists_log_containing(&format!( - "ERROR: ReceivableDaoReal: Payment reception failed, rolling back: RusqliteError(\"no such table: receivable\")\n\ + "ERROR: ReceivableDaoReal: Payment reception failed, rolling back: RusqliteError(\ + \"Error from invalid update command for receivable table and change of -123456789123456789 \ + wei to 'wallet_address = 0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' with error 'no such table: receivable'\")\ + \n\ Block # Wallet Amount \n\ 1234567890 0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 123456789123456789\n\ 2345678901 0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 234567891234567891\n\ @@ -660,7 +705,7 @@ mod tests { let wallet = make_wallet("booga"); let subject = ReceivableDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); @@ -669,93 +714,60 @@ mod tests { assert_eq!(result, None); } - #[test] - fn receivables_fetches_all_receivable_accounts() { - let home_dir = ensure_node_home_directory_exists( - "receivable_dao", - "receivables_fetches_all_receivable_accounts", + fn make_connection_with_our_defined_sqlite_functions( + home_dir: &Path, + ) -> Box { + let init_config = DbInitializationConfig::test_default().add_special_conn_setup( + BigIntDivider::register_big_int_deconstruction_for_sqlite_connection, ); - let wallet1 = make_wallet("wallet1"); - let wallet2 = make_wallet("wallet2"); - let time_stub = SystemTime::now(); - let subject = ReceivableDaoReal::new( - DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(), - ); - - subject - .more_money_receivable(time_stub, &wallet1, 1234) - .unwrap(); - subject - .more_money_receivable(time_stub, &wallet2, 2345) - .unwrap(); - - let accounts = subject - .receivables() - .into_iter() - .map(|r| ReceivableAccount { - last_received_timestamp: time_stub, - ..r - }) - .collect::>(); - assert_eq!( - accounts, - vec![ - ReceivableAccount { - wallet: wallet1, - balance: 1234, - last_received_timestamp: time_stub - }, - ReceivableAccount { - wallet: wallet2, - balance: 2345, - last_received_timestamp: time_stub - }, - ] - ) + DbInitializerReal::default() + .initialize(home_dir, init_config) + .unwrap() } #[test] fn new_delinquencies_unit_slope() { - let pcs = PaymentThresholds { + let payment_thresholds = PaymentThresholds { maturity_threshold_sec: 25, payment_grace_period_sec: 50, permanent_debt_allowed_gwei: 100, debt_threshold_gwei: 200, threshold_interval_sec: 100, - unban_below_gwei: 0, // doesn't matter for this test + unban_below_gwei: 0, }; let now = now_time_t(); let mut not_delinquent_inside_grace_period = make_receivable_account(1234, false); - not_delinquent_inside_grace_period.balance = pcs.debt_threshold_gwei + 1; + not_delinquent_inside_grace_period.balance_wei = + gwei_to_wei(payment_thresholds.debt_threshold_gwei + 1); not_delinquent_inside_grace_period.last_received_timestamp = - from_time_t(pcs.sugg_and_grace(now) + 2); + from_time_t(payment_thresholds.sugg_and_grace(now) + 2); let mut not_delinquent_after_grace_below_slope = make_receivable_account(2345, false); - not_delinquent_after_grace_below_slope.balance = pcs.debt_threshold_gwei - 2; + not_delinquent_after_grace_below_slope.balance_wei = + gwei_to_wei(payment_thresholds.debt_threshold_gwei - 2); not_delinquent_after_grace_below_slope.last_received_timestamp = - from_time_t(pcs.sugg_and_grace(now) - 1); + from_time_t(payment_thresholds.sugg_and_grace(now) - 1); let mut delinquent_above_slope_after_grace = make_receivable_account(3456, true); - delinquent_above_slope_after_grace.balance = pcs.debt_threshold_gwei - 1; + delinquent_above_slope_after_grace.balance_wei = + gwei_to_wei(payment_thresholds.debt_threshold_gwei - 1); delinquent_above_slope_after_grace.last_received_timestamp = - from_time_t(pcs.sugg_and_grace(now) - 2); + from_time_t(payment_thresholds.sugg_and_grace(now) - 2); let mut not_delinquent_below_slope_before_stop = make_receivable_account(4567, false); - not_delinquent_below_slope_before_stop.balance = pcs.permanent_debt_allowed_gwei + 1; + not_delinquent_below_slope_before_stop.balance_wei = + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei + 1); not_delinquent_below_slope_before_stop.last_received_timestamp = - from_time_t(pcs.sugg_thru_decreasing(now) + 2); + from_time_t(payment_thresholds.sugg_thru_decreasing(now) + 2); let mut delinquent_above_slope_before_stop = make_receivable_account(5678, true); - delinquent_above_slope_before_stop.balance = pcs.permanent_debt_allowed_gwei + 2; + delinquent_above_slope_before_stop.balance_wei = + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei + 2); delinquent_above_slope_before_stop.last_received_timestamp = - from_time_t(pcs.sugg_thru_decreasing(now) + 1); + from_time_t(payment_thresholds.sugg_thru_decreasing(now) + 1); let mut not_delinquent_above_slope_after_stop = make_receivable_account(6789, false); - not_delinquent_above_slope_after_stop.balance = pcs.permanent_debt_allowed_gwei - 1; + not_delinquent_above_slope_after_stop.balance_wei = + gwei_to_wei(payment_thresholds.permanent_debt_allowed_gwei - 1); not_delinquent_above_slope_after_stop.last_received_timestamp = - from_time_t(pcs.sugg_thru_decreasing(now) - 2); + from_time_t(payment_thresholds.sugg_thru_decreasing(now) - 2); let home_dir = ensure_node_home_directory_exists("accountant", "new_delinquencies"); - let db_initializer = DbInitializerReal::default(); - let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); add_receivable_account(&conn, ¬_delinquent_inside_grace_period); add_receivable_account(&conn, ¬_delinquent_after_grace_below_slope); add_receivable_account(&conn, &delinquent_above_slope_after_grace); @@ -764,7 +776,7 @@ mod tests { add_receivable_account(&conn, ¬_delinquent_above_slope_after_stop); let subject = ReceivableDaoReal::new(conn); - let result = subject.new_delinquencies(from_time_t(now), &pcs); + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); assert_contains(&result, &delinquent_above_slope_after_grace); assert_contains(&result, &delinquent_above_slope_before_stop); @@ -773,125 +785,181 @@ mod tests { #[test] fn new_delinquencies_shallow_slope() { - let pcs = PaymentThresholds { + let payment_thresholds = PaymentThresholds { maturity_threshold_sec: 100, payment_grace_period_sec: 100, permanent_debt_allowed_gwei: 100, debt_threshold_gwei: 110, threshold_interval_sec: 100, - unban_below_gwei: 0, // doesn't matter for this test + unban_below_gwei: 0, }; let now = now_time_t(); let mut not_delinquent = make_receivable_account(1234, false); - not_delinquent.balance = 105; - not_delinquent.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 25); + not_delinquent.balance_wei = gwei_to_wei(105); + not_delinquent.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 25); let mut delinquent = make_receivable_account(2345, true); - delinquent.balance = 105; - delinquent.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 75); + delinquent.balance_wei = gwei_to_wei(105); + delinquent.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 75); let home_dir = ensure_node_home_directory_exists("accountant", "new_delinquencies_shallow_slope"); - let db_initializer = DbInitializerReal::default(); - let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); add_receivable_account(&conn, ¬_delinquent); add_receivable_account(&conn, &delinquent); let subject = ReceivableDaoReal::new(conn); - let result = subject.new_delinquencies(from_time_t(now), &pcs); + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); assert_contains(&result, &delinquent); - assert_eq!(1, result.len()); + assert_eq!(result.len(), 1); } #[test] fn new_delinquencies_steep_slope() { - let pcs = PaymentThresholds { + let payment_thresholds = PaymentThresholds { maturity_threshold_sec: 100, payment_grace_period_sec: 100, permanent_debt_allowed_gwei: 100, debt_threshold_gwei: 1100, threshold_interval_sec: 100, - unban_below_gwei: 0, // doesn't matter for this test + unban_below_gwei: 0, }; let now = now_time_t(); let mut not_delinquent = make_receivable_account(1234, false); - not_delinquent.balance = 600; - not_delinquent.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 25); + not_delinquent.balance_wei = gwei_to_wei(600); + not_delinquent.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 25); let mut delinquent = make_receivable_account(2345, true); - delinquent.balance = 600; - delinquent.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 75); + delinquent.balance_wei = gwei_to_wei(600); + delinquent.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 75); let home_dir = ensure_node_home_directory_exists("accountant", "new_delinquencies_steep_slope"); - let db_initializer = DbInitializerReal::default(); - let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); add_receivable_account(&conn, ¬_delinquent); add_receivable_account(&conn, &delinquent); let subject = ReceivableDaoReal::new(conn); - let result = subject.new_delinquencies(from_time_t(now), &pcs); + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); assert_contains(&result, &delinquent); - assert_eq!(1, result.len()); + assert_eq!(result.len(), 1); } #[test] fn new_delinquencies_does_not_find_existing_delinquencies() { - let pcs = PaymentThresholds { + let payment_thresholds = PaymentThresholds { maturity_threshold_sec: 25, payment_grace_period_sec: 50, permanent_debt_allowed_gwei: 100, debt_threshold_gwei: 200, threshold_interval_sec: 100, - unban_below_gwei: 0, // doesn't matter for this test + unban_below_gwei: 0, }; let now = now_time_t(); let mut existing_delinquency = make_receivable_account(1234, true); - existing_delinquency.balance = 250; - existing_delinquency.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 1); + existing_delinquency.balance_wei = gwei_to_wei(250); + existing_delinquency.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 1); let mut new_delinquency = make_receivable_account(2345, true); - new_delinquency.balance = 250; - new_delinquency.last_received_timestamp = from_time_t(pcs.sugg_and_grace(now) - 1); - + new_delinquency.balance_wei = gwei_to_wei(250); + new_delinquency.last_received_timestamp = + from_time_t(payment_thresholds.sugg_and_grace(now) - 1); let home_dir = ensure_node_home_directory_exists( "receivable_dao", "new_delinquencies_does_not_find_existing_delinquencies", ); - let db_initializer = DbInitializerReal::default(); - let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) - .unwrap(); + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); add_receivable_account(&conn, &existing_delinquency); add_receivable_account(&conn, &new_delinquency); add_banned_account(&conn, &existing_delinquency); let subject = ReceivableDaoReal::new(conn); - let result = subject.new_delinquencies(from_time_t(now), &pcs); + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); assert_contains(&result, &new_delinquency); - assert_eq!(1, result.len()); + assert_eq!(result.len(), 1); + } + + #[test] + fn new_delinquencies_works_for_still_empty_tables() { + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 25, + payment_grace_period_sec: 50, + permanent_debt_allowed_gwei: 100, + debt_threshold_gwei: 200, + threshold_interval_sec: 100, + unban_below_gwei: 0, + }; + let now = now_time_t(); + let home_dir = ensure_node_home_directory_exists( + "receivable_dao", + "new_delinquencies_work_for_still_empty_tables", + ); + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); + let subject = ReceivableDaoReal::new(conn); + + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); + + assert!(result.is_empty()) + } + + #[test] + fn new_delinquencies_handles_too_young_debts_causing_slope_parameter_to_be_negative() { + //situation where sugg_and_grace makes more time than the age of the debt + let home_dir = ensure_node_home_directory_exists( + "receivable_dao", + "new_delinquencies_handles_too_young_debts_causing_slope_parameter_to_be_negative", + ); + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 25, + payment_grace_period_sec: 50, + permanent_debt_allowed_gwei: 100, + debt_threshold_gwei: 123, + threshold_interval_sec: 100, + unban_below_gwei: 0, + }; + let now = to_time_t(SystemTime::now()); + let sugg_and_grace = payment_thresholds.sugg_and_grace(now); + let too_young_new_delinquency = ReceivableAccount { + wallet: make_wallet("abc123"), + balance_wei: 123_456_789_101_112, + last_received_timestamp: from_time_t(sugg_and_grace + 1), + }; + let ok_new_delinquency = ReceivableAccount { + wallet: make_wallet("aaa999"), + balance_wei: 123_456_789_101_112, + last_received_timestamp: from_time_t(sugg_and_grace - 1), + }; + let conn = make_connection_with_our_defined_sqlite_functions(&home_dir); + add_receivable_account(&conn, &too_young_new_delinquency); + add_receivable_account(&conn, &ok_new_delinquency.clone()); + let subject = ReceivableDaoReal::new(conn); + + let result = subject.new_delinquencies(from_time_t(now), &payment_thresholds); + + assert_eq!(result, vec![ok_new_delinquency]) } #[test] fn paid_delinquencies() { - let pcs = PaymentThresholds { - maturity_threshold_sec: 0, // doesn't matter for this test - payment_grace_period_sec: 0, // doesn't matter for this test - permanent_debt_allowed_gwei: 0, // doesn't matter for this test - debt_threshold_gwei: 0, // doesn't matter for this test - threshold_interval_sec: 0, // doesn't matter for this test + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 0, + payment_grace_period_sec: 0, + permanent_debt_allowed_gwei: 0, + debt_threshold_gwei: 0, + threshold_interval_sec: 0, unban_below_gwei: 50, }; let mut paid_delinquent = make_receivable_account(1234, true); - paid_delinquent.balance = 50; + paid_delinquent.balance_wei = 50_000_000_000; let mut unpaid_delinquent = make_receivable_account(2345, true); - unpaid_delinquent.balance = 51; + unpaid_delinquent.balance_wei = 50_000_000_001; let home_dir = ensure_node_home_directory_exists("accountant", "paid_delinquencies"); let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); add_receivable_account(&conn, &paid_delinquent); add_receivable_account(&conn, &unpaid_delinquent); @@ -899,26 +967,26 @@ mod tests { add_banned_account(&conn, &unpaid_delinquent); let subject = ReceivableDaoReal::new(conn); - let result = subject.paid_delinquencies(&pcs); + let result = subject.paid_delinquencies(&payment_thresholds); assert_contains(&result, &paid_delinquent); - assert_eq!(1, result.len()); + assert_eq!(result.len(), 1); } #[test] fn paid_delinquencies_does_not_find_existing_nondelinquencies() { - let pcs = PaymentThresholds { - maturity_threshold_sec: 0, // doesn't matter for this test - payment_grace_period_sec: 0, // doesn't matter for this test - permanent_debt_allowed_gwei: 0, // doesn't matter for this test - debt_threshold_gwei: 0, // doesn't matter for this test - threshold_interval_sec: 0, // doesn't matter for this test + let payment_thresholds = PaymentThresholds { + maturity_threshold_sec: 0, + payment_grace_period_sec: 0, + permanent_debt_allowed_gwei: 0, + debt_threshold_gwei: 0, + threshold_interval_sec: 0, unban_below_gwei: 50, }; let mut newly_non_delinquent = make_receivable_account(1234, false); - newly_non_delinquent.balance = 25; + newly_non_delinquent.balance_wei = gwei_to_wei(25); let mut old_non_delinquent = make_receivable_account(2345, false); - old_non_delinquent.balance = 25; + old_non_delinquent.balance_wei = gwei_to_wei(25); let home_dir = ensure_node_home_directory_exists( "receivable_dao", @@ -926,78 +994,362 @@ mod tests { ); let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); add_receivable_account(&conn, &newly_non_delinquent); add_receivable_account(&conn, &old_non_delinquent); add_banned_account(&conn, &newly_non_delinquent); let subject = ReceivableDaoReal::new(conn); - let result = subject.paid_delinquencies(&pcs); + let result = subject.paid_delinquencies(&payment_thresholds); assert_contains(&result, &newly_non_delinquent); - assert_eq!(1, result.len()); + assert_eq!(result.len(), 1); } #[test] - fn top_records_and_total() { - let home_dir = ensure_node_home_directory_exists("receivable_dao", "top_records_and_total"); - let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + fn custom_query_handles_empty_table_in_top_records_mode() { + let main_test_setup = |_conn: &dyn ConnectionWrapper, _insert: InsertReceivableHelperFn| {}; + let subject = custom_query_test_body_for_receivable( + "custom_query_handles_empty_table_in_top_records_mode", + main_test_setup, + ); + + let result = subject.custom_query(CustomQuery::TopRecords { + count: 6, + ordered_by: Balance, + }); + + assert_eq!(result, None) + } + + type InsertReceivableHelperFn<'b> = + &'b dyn for<'a> Fn(&'a dyn ConnectionWrapper, &'a str, i128, i64); + + fn common_setup_of_accounts_for_tests_of_top_records( + now: i64, + ) -> Box { + //Accounts of balances smaller than one gwei don't qualify. + //Two accounts differ only in balance but not the debt's age, two other in debt's age but are same at balance. + //That setup allows a check of doubled ordering + Box::new(move |conn, insert: InsertReceivableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + 1_000_000_001, + now - 86_480, + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + 1_000_000_001, + now - 222_000, + ); + insert( + conn, + "0x3333333333333333333333333333333333333333", + 990_000_000, //below 1 gwei + now - 86_000, + ); + insert( + conn, + "0x4444444444444444444444444444444444444444", + 1_000_000_000, + now - 86_111, + ); + insert( + conn, + "0x5555555555555555555555555555555555555555", + 32_000_000_200, + now - 86_480, + ) + }) + } + + #[test] + fn custom_query_in_top_records_mode_default_ordering() { + let now = now_time_t(); + let main_test_setup = common_setup_of_accounts_for_tests_of_top_records(now); + let subject = custom_query_test_body_for_receivable( + "custom_query_in_top_records_mode_default_ordering", + main_test_setup, + ); + + let result = subject + .custom_query(CustomQuery::TopRecords { + count: 3, + ordered_by: Balance, + }) .unwrap(); - let insert = |wallet: &str, balance: i64, timestamp: i64| { - let params: &[&dyn ToSql] = &[&wallet, &balance, ×tamp]; - conn - .prepare("insert into receivable (wallet_address, balance, last_received_timestamp) values (?, ?, ?)") - .unwrap() - .execute(params) - .unwrap(); - }; - let timestamp1 = dao_utils::now_time_t() - 80_000; - let timestamp2 = dao_utils::now_time_t() - 86_401; - let timestamp3 = dao_utils::now_time_t() - 86_000; - let timestamp4 = dao_utils::now_time_t() - 86_001; - insert( - "0x1111111111111111111111111111111111111111", - 999_999_999, // below minimum amount - reject - timestamp1, // below maximum age + + assert_eq!( + result, + vec![ + ReceivableAccount { + wallet: Wallet::new("0x5555555555555555555555555555555555555555"), + balance_wei: 32_000_000_200, + last_received_timestamp: from_time_t(now - 86_480), + }, + ReceivableAccount { + wallet: Wallet::new("0x2222222222222222222222222222222222222222"), + balance_wei: 1_000_000_001, + last_received_timestamp: from_time_t(now - 222_000), + }, + ReceivableAccount { + wallet: Wallet::new("0x1111111111111111111111111111111111111111"), + balance_wei: 1_000_000_001, + last_received_timestamp: from_time_t(now - 86_480), + }, + ] ); - insert( - "0x2222222222222222222222222222222222222222", - 1_000_000_000, // minimum amount - timestamp2, // above maximum age - reject + } + + #[test] + fn custom_query_in_top_records_mode_ordered_by_age() { + let now = now_time_t(); + let main_test_setup = common_setup_of_accounts_for_tests_of_top_records(now); + let subject = custom_query_test_body_for_receivable( + "custom_query_in_top_records_mode_ordered_by_age", + main_test_setup, ); - insert( - "0x3333333333333333333333333333333333333333", - 1_000_000_000, // minimum amount - timestamp3, // below maximum age + + let result = subject + .custom_query(CustomQuery::TopRecords { + count: 3, + ordered_by: Age, + }) + .unwrap(); + + assert_eq!( + result, + vec![ + ReceivableAccount { + wallet: Wallet::new("0x2222222222222222222222222222222222222222"), + balance_wei: 1_000_000_001, + last_received_timestamp: from_time_t(now - 222_000), + }, + ReceivableAccount { + wallet: Wallet::new("0x5555555555555555555555555555555555555555"), + balance_wei: 32_000_000_200, + last_received_timestamp: from_time_t(now - 86_480), + }, + ReceivableAccount { + wallet: Wallet::new("0x1111111111111111111111111111111111111111"), + balance_wei: 1_000_000_001, + last_received_timestamp: from_time_t(now - 86_480), + }, + ] ); - insert( - "0x4444444444444444444444444444444444444444", - 1_000_000_001, // above minimum amount - timestamp4, // below maximum age + } + + #[test] + fn custom_query_handles_empty_table_in_range_mode() { + let main_test_setup = |_conn: &dyn ConnectionWrapper, _insert: InsertReceivableHelperFn| {}; + let subject = custom_query_test_body_for_receivable( + "custom_query_handles_empty_table_in_range_mode", + main_test_setup, ); - let subject = ReceivableDaoReal::new(conn); - let top_records = subject.top_records(1_000_000_000, 86400); - let total = subject.total(); + let result = subject.custom_query(CustomQuery::RangeQuery { + min_age_s: 20000, + max_age_s: 200000, + min_amount_gwei: 500000000, + max_amount_gwei: 3500000000, + timestamp: SystemTime::now(), + }); + + assert_eq!(result, None) + } + + #[test] + fn custom_query_in_range_mode() { + //Two accounts differ only in debt's age but not balance which allows to check doubled ordering, + //by balance and then by age. + let now = now_time_t(); + let main_test_setup = |conn: &dyn ConnectionWrapper, insert: InsertReceivableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + gwei_to_wei(999_454_656), + now - 99_001, //too old + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + gwei_to_wei(-560_001), //too small + now - 86_401, + ); + insert( + conn, + "0x3333333333333333333333333333333333333333", + gwei_to_wei(1_000_000_230), + now - 70_000, + ); + insert( + conn, + "0x4444444444444444444444444444444444444444", + gwei_to_wei(1_100_000_001), //too big + now - 69_000, + ); + insert( + conn, + "0x5555555555555555555555555555555555555555", + gwei_to_wei(1_000_000_230), + now - 86_000, + ); + insert( + conn, + "0x6666666666666666666666666666666666666666", + gwei_to_wei(1_050_444_230), + now - 66_244, + ); + insert( + conn, + "0x7777777777777777777777777777777777777777", + gwei_to_wei(900_000_000), + now - 59_999, //too young + ); + insert( + conn, + "0x8888888888888888888888888888888888888888", + gwei_to_wei(-90), + now - 66000, + ); + }; + let subject = + custom_query_test_body_for_receivable("custom_query_in_range_mode", main_test_setup); + + let result = subject + .custom_query(CustomQuery::RangeQuery { + min_age_s: 60000, + max_age_s: 99000, + min_amount_gwei: -560000, + max_amount_gwei: 1_100_000_000, + timestamp: from_time_t(now), + }) + .unwrap(); assert_eq!( - top_records, + result, vec![ ReceivableAccount { - wallet: Wallet::new("0x4444444444444444444444444444444444444444"), - balance: 1_000_000_001, - last_received_timestamp: dao_utils::from_time_t(timestamp4), + wallet: Wallet::new("0x6666666666666666666666666666666666666666"), + balance_wei: gwei_to_wei(1_050_444_230), + last_received_timestamp: from_time_t(now - 66_244), + }, + ReceivableAccount { + wallet: Wallet::new("0x5555555555555555555555555555555555555555"), + balance_wei: gwei_to_wei(1_000_000_230), + last_received_timestamp: from_time_t(now - 86_000), }, ReceivableAccount { wallet: Wallet::new("0x3333333333333333333333333333333333333333"), - balance: 1_000_000_000, - last_received_timestamp: dao_utils::from_time_t(timestamp3), + balance_wei: gwei_to_wei(1_000_000_230), + last_received_timestamp: from_time_t(now - 70_000), }, + ReceivableAccount { + wallet: Wallet::new("0x8888888888888888888888888888888888888888"), + balance_wei: gwei_to_wei(-90), + last_received_timestamp: from_time_t(now - 66_000), + } ] ); - assert_eq!(total, 4_000_000_000) + } + + #[test] + fn range_query_does_not_display_values_from_below_1_gwei() { + let timestamp1 = now_time_t() - 5000; + let timestamp2 = now_time_t() - 3232; + let main_setup = |conn: &dyn ConnectionWrapper, insert: InsertReceivableHelperFn| { + insert( + conn, + "0x1111111111111111111111111111111111111111", + 999_999_999, //smaller than 1 gwei + now_time_t() - 11_001, + ); + insert( + conn, + "0x2222222222222222222222222222222222222222", + -999_999_999, //smaller than -1 gwei + now_time_t() - 5_606, + ); + insert( + conn, + "0x3333333333333333333333333333333333333333", + 30_000_300_000, + timestamp1, + ); + insert( + conn, + "0x4444444444444444444444444444444444444444", + -2_000_300_000, + timestamp2, + ); + }; + let subject = custom_query_test_body_for_receivable( + "range_query_does_not_display_values_from_below_1_gwei", + main_setup, + ); + + let result = subject + .custom_query(CustomQuery::RangeQuery { + min_age_s: 0, + max_age_s: 200000, + min_amount_gwei: i64::MIN, + max_amount_gwei: 35_000_000_000, + timestamp: SystemTime::now(), + }) + .unwrap(); + + assert_eq!( + result, + vec![ + ReceivableAccount { + wallet: Wallet::new("0x3333333333333333333333333333333333333333"), + balance_wei: 30_000_300_000, + last_received_timestamp: from_time_t(timestamp1), + }, + ReceivableAccount { + wallet: Wallet::new("0x4444444444444444444444444444444444444444"), + balance_wei: -2_000_300_000, + last_received_timestamp: from_time_t(timestamp2), + } + ] + ) + } + + #[test] + fn total_works() { + let home_dir = ensure_node_home_directory_exists("receivable_dao", "total_works"); + let conn = DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + + let insert = insert_account_by_separate_values; + let timestamp = dao_utils::now_time_t(); + insert( + &*conn, + "0x1111111111111111111111111111111111111111", + 999_999_800, + timestamp - 1000, + ); + insert( + &*conn, + "0x2222222222222222222222222222222222222222", + 1_000_000_070, + timestamp - 3333, + ); + insert( + &*conn, + "0x3333333333333333333333333333333333333333", + 1_000_000_130, + timestamp - 4567, + ); + let subject = ReceivableDaoReal::new(conn); + + let total = subject.total(); + + assert_eq!(total, 3_000_000_000) } #[test] @@ -1005,7 +1357,7 @@ mod tests { let home_dir = ensure_node_home_directory_exists("receivable_dao", "correctly_totals_zero_records"); let conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = ReceivableDaoReal::new(conn); @@ -1014,20 +1366,67 @@ mod tests { assert_eq!(result, 0) } + #[test] + #[should_panic( + expected = "Database is corrupt: RECEIVABLE table columns and/or types: (Err(FromSqlConversionFailure(0, Text, InvalidAddress)), Err(InvalidColumnIndex(1))" + )] + fn create_receivable_account_panics_on_database_error() { + assert_account_creation_fn_fails_on_finding_wrong_columns_and_value_types( + ReceivableDaoReal::create_receivable_account, + ); + } + + #[test] + fn receivable_dao_implements_dao_table_identifier() { + assert_eq!(ReceivableDaoReal::table_name(), "receivable") + } + fn add_receivable_account(conn: &Box, account: &ReceivableAccount) { - let mut stmt = conn.prepare ("insert into receivable (wallet_address, balance, last_received_timestamp) values (?, ?, ?)").unwrap(); + let mut stmt = conn.prepare ("insert into receivable (wallet_address, balance_high_b, balance_low_b, last_received_timestamp) values (?, ?, ?, ?)").unwrap(); + let (high_bytes, low_bytes) = BigIntDivider::deconstruct(account.balance_wei); let params: &[&dyn ToSql] = &[ &account.wallet, - &account.balance, + &high_bytes, + &low_bytes, &to_time_t(account.last_received_timestamp), ]; stmt.execute(params).unwrap(); } + fn insert_account_by_separate_values( + conn: &dyn ConnectionWrapper, + wallet: &str, + balance: i128, + timestamp: i64, + ) { + let (high_bytes, low_bytes) = BigIntDivider::deconstruct(balance); + let params: &[&dyn ToSql] = &[&wallet, &high_bytes, &low_bytes, ×tamp]; + conn + .prepare("insert into receivable (wallet_address, balance_high_b, balance_low_b, last_received_timestamp) values (?, ?, ?, ?)") + .unwrap() + .execute(params) + .unwrap(); + } + fn add_banned_account(conn: &Box, account: &ReceivableAccount) { let mut stmt = conn .prepare("insert into banned (wallet_address) values (?)") .unwrap(); stmt.execute(&[&account.wallet]).unwrap(); } + + fn custom_query_test_body_for_receivable( + test_name: &str, + main_test_setup: F, + ) -> ReceivableDaoReal + where + F: Fn(&dyn ConnectionWrapper, InsertReceivableHelperFn), + { + let home_dir = ensure_node_home_directory_exists("receivable_dao", test_name); + let conn = DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + main_test_setup(conn.as_ref(), &insert_account_by_separate_values); + ReceivableDaoReal::new(conn) + } } diff --git a/node/src/accountant/test_utils.rs b/node/src/accountant/test_utils.rs index 33255d35b..736bbf005 100644 --- a/node/src/accountant/test_utils.rs +++ b/node/src/accountant/test_utils.rs @@ -2,6 +2,7 @@ #![cfg(test)] +use crate::accountant::dao_utils::{from_time_t, to_time_t, CustomQuery}; use crate::accountant::payable_dao::{ PayableAccount, PayableDao, PayableDaoError, PayableDaoFactory, }; @@ -11,13 +12,11 @@ use crate::accountant::pending_payable_dao::{ use crate::accountant::receivable_dao::{ ReceivableAccount, ReceivableDao, ReceivableDaoError, ReceivableDaoFactory, }; -use crate::accountant::{Accountant, PendingPayableId}; +use crate::accountant::{gwei_to_wei, Accountant, PendingPayableId}; use crate::banned_dao::{BannedDao, BannedDaoFactory}; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::blockchain::blockchain_interface::BlockchainTransaction; use crate::bootstrapper::BootstrapperConfig; -use crate::database::dao_utils; -use crate::database::dao_utils::{from_time_t, to_time_t}; use crate::db_config::config_dao::{ConfigDao, ConfigDaoFactory}; use crate::db_config::mocks::ConfigDaoMock; use crate::sub_lib::accountant::{AccountantConfig, MessageIdGenerator, PaymentThresholds}; @@ -26,8 +25,9 @@ use crate::test_utils::make_wallet; use crate::test_utils::unshared_test_utils::make_populated_accountant_config_with_defaults; use actix::System; use ethereum_types::{BigEndianHash, H256, U256}; -use rusqlite::{Connection, Error, OptionalExtension}; +use rusqlite::{Connection, Row}; use std::cell::RefCell; +use std::fmt::Debug; use std::rc::Rc; use std::sync::{Arc, Mutex}; use std::time::SystemTime; @@ -40,7 +40,7 @@ pub fn make_receivable_account(n: u64, expected_delinquent: bool) -> ReceivableA n, if expected_delinquent { "d" } else { "n" } )), - balance: (n * 1_000_000_000) as i64, + balance_wei: gwei_to_wei(n), last_received_timestamp: from_time_t(now - (n as i64)), } } @@ -50,19 +50,19 @@ pub fn make_payable_account(n: u64) -> PayableAccount { let timestamp = from_time_t(now - (n as i64)); make_payable_account_with_recipient_and_balance_and_timestamp_opt( make_wallet(&format!("wallet{}", n)), - (n * 1_000_000_000) as i64, + gwei_to_wei(n), Some(timestamp), ) } pub fn make_payable_account_with_recipient_and_balance_and_timestamp_opt( recipient: Wallet, - balance: i64, + balance: u128, timestamp_opt: Option, ) -> PayableAccount { PayableAccount { wallet: recipient, - balance, + balance_wei: balance, last_paid_timestamp: timestamp_opt.unwrap_or(SystemTime::now()), pending_payable_opt: None, } @@ -260,20 +260,20 @@ impl ConfigDaoFactoryMock { #[derive(Debug, Default)] pub struct PayableDaoMock { - more_money_payable_parameters: Arc>>, + more_money_payable_parameters: Arc>>, more_money_payable_results: RefCell>>, - non_pending_payables_params: Arc>>, - non_pending_payables_results: RefCell>>, + non_pending_payable_params: Arc>>, + non_pending_payable_results: RefCell>>, mark_pending_payable_rowid_parameters: Arc>>, mark_pending_payable_rowid_results: RefCell>>, transaction_confirmed_params: Arc>>, transaction_confirmed_results: RefCell>>, transaction_canceled_params: Arc>>, transaction_canceled_results: RefCell>>, - top_records_parameters: Arc>>, - top_records_results: RefCell>>, - total_results: RefCell>, - pub have_non_pending_payables_shut_down_the_system: bool, + custom_query_params: Arc>>>, + custom_query_result: RefCell>>>, + total_results: RefCell>, + pub have_non_pending_payable_shut_down_the_system: bool, } impl PayableDao for PayableDaoMock { @@ -281,7 +281,7 @@ impl PayableDao for PayableDaoMock { &self, now: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), PayableDaoError> { self.more_money_payable_parameters .lock() @@ -316,27 +316,29 @@ impl PayableDao for PayableDaoMock { } fn non_pending_payables(&self) -> Vec { - self.non_pending_payables_params.lock().unwrap().push(()); - if self.have_non_pending_payables_shut_down_the_system - && self.non_pending_payables_results.borrow().is_empty() + self.non_pending_payable_params.lock().unwrap().push(()); + if self.have_non_pending_payable_shut_down_the_system + && self.non_pending_payable_results.borrow().is_empty() { System::current().stop(); return vec![]; } - self.non_pending_payables_results.borrow_mut().remove(0) + self.non_pending_payable_results.borrow_mut().remove(0) } - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec { - self.top_records_parameters - .lock() - .unwrap() - .push((minimum_amount, maximum_age)); - self.top_records_results.borrow_mut().remove(0) + fn custom_query(&self, custom_query: CustomQuery) -> Option> { + self.custom_query_params.lock().unwrap().push(custom_query); + self.custom_query_result.borrow_mut().remove(0) } - fn total(&self) -> u64 { + fn total(&self) -> u128 { self.total_results.borrow_mut().remove(0) } + + fn account_status(&self, _wallet: &Wallet) -> Option { + //test-only trait member + intentionally_blank!() + } } impl PayableDaoMock { @@ -346,7 +348,7 @@ impl PayableDaoMock { pub fn more_money_payable_params( mut self, - parameters: Arc>>, + parameters: Arc>>, ) -> Self { self.more_money_payable_parameters = parameters; self @@ -358,12 +360,12 @@ impl PayableDaoMock { } pub fn non_pending_payables_params(mut self, params: &Arc>>) -> Self { - self.non_pending_payables_params = params.clone(); + self.non_pending_payable_params = params.clone(); self } pub fn non_pending_payables_result(self, result: Vec) -> Self { - self.non_pending_payables_results.borrow_mut().push(result); + self.non_pending_payable_results.borrow_mut().push(result); self } @@ -408,17 +410,17 @@ impl PayableDaoMock { self } - pub fn top_records_parameters(mut self, parameters: &Arc>>) -> Self { - self.top_records_parameters = parameters.clone(); + pub fn custom_query_params(mut self, params: &Arc>>>) -> Self { + self.custom_query_params = params.clone(); self } - pub fn top_records_result(self, result: Vec) -> Self { - self.top_records_results.borrow_mut().push(result); + pub fn custom_query_result(self, result: Option>) -> Self { + self.custom_query_result.borrow_mut().push(result); self } - pub fn total_result(self, result: u64) -> Self { + pub fn total_result(self, result: u128) -> Self { self.total_results.borrow_mut().push(result); self } @@ -426,20 +428,17 @@ impl PayableDaoMock { #[derive(Debug, Default)] pub struct ReceivableDaoMock { - account_status_parameters: Arc>>, - account_status_results: RefCell>>, - more_money_receivable_parameters: Arc>>, + more_money_receivable_parameters: Arc>>, more_money_receivable_results: RefCell>>, more_money_received_parameters: Arc)>>>, more_money_received_results: RefCell>>, - receivables_results: RefCell>>, new_delinquencies_parameters: Arc>>, new_delinquencies_results: RefCell>>, paid_delinquencies_parameters: Arc>>, paid_delinquencies_results: RefCell>>, - top_records_parameters: Arc>>, - top_records_results: RefCell>>, - total_results: RefCell>, + custom_query_params: Arc>>>, + custom_query_result: RefCell>>>, + total_results: RefCell>, pub have_new_delinquencies_shutdown_the_system: bool, } @@ -448,7 +447,7 @@ impl ReceivableDao for ReceivableDaoMock { &self, now: SystemTime, wallet: &Wallet, - amount: u64, + amount: u128, ) -> Result<(), ReceivableDaoError> { self.more_money_receivable_parameters .lock() @@ -464,19 +463,6 @@ impl ReceivableDao for ReceivableDaoMock { .push((now, transactions)); } - fn account_status(&self, wallet: &Wallet) -> Option { - self.account_status_parameters - .lock() - .unwrap() - .push(wallet.clone()); - - self.account_status_results.borrow_mut().remove(0) - } - - fn receivables(&self) -> Vec { - self.receivables_results.borrow_mut().remove(0) - } - fn new_delinquencies( &self, now: SystemTime, @@ -503,17 +489,19 @@ impl ReceivableDao for ReceivableDaoMock { self.paid_delinquencies_results.borrow_mut().remove(0) } - fn top_records(&self, minimum_amount: u64, maximum_age: u64) -> Vec { - self.top_records_parameters - .lock() - .unwrap() - .push((minimum_amount, maximum_age)); - self.top_records_results.borrow_mut().remove(0) + fn custom_query(&self, custom_query: CustomQuery) -> Option> { + self.custom_query_params.lock().unwrap().push(custom_query); + self.custom_query_result.borrow_mut().remove(0) } - fn total(&self) -> i64 { + fn total(&self) -> i128 { self.total_results.borrow_mut().remove(0) } + + fn account_status(&self, _wallet: &Wallet) -> Option { + //test-only trait member + intentionally_blank!() + } } impl ReceivableDaoMock { @@ -523,7 +511,7 @@ impl ReceivableDaoMock { pub fn more_money_receivable_parameters( mut self, - parameters: &Arc>>, + parameters: &Arc>>, ) -> Self { self.more_money_receivable_parameters = parameters.clone(); self @@ -573,17 +561,17 @@ impl ReceivableDaoMock { self } - pub fn top_records_parameters(mut self, parameters: &Arc>>) -> Self { - self.top_records_parameters = parameters.clone(); + pub fn custom_query_params(mut self, params: &Arc>>>) -> Self { + self.custom_query_params = params.clone(); self } - pub fn top_records_result(self, result: Vec) -> Self { - self.top_records_results.borrow_mut().push(result); + pub fn custom_query_result(self, result: Option>) -> Self { + self.custom_query_result.borrow_mut().push(result); self } - pub fn total_result(self, result: i64) -> Self { + pub fn total_result(self, result: i128) -> Self { self.total_results.borrow_mut().push(result); self } @@ -666,7 +654,7 @@ pub struct PendingPayableDaoMock { fingerprint_rowid_results: RefCell>>, delete_fingerprint_params: Arc>>, delete_fingerprint_results: RefCell>>, - insert_fingerprint_params: Arc>>, + insert_fingerprint_params: Arc>>, insert_fingerprint_results: RefCell>>, update_fingerprint_params: Arc>>, update_fingerprint_results: RefCell>>, @@ -700,7 +688,7 @@ impl PendingPayableDao for PendingPayableDaoMock { fn insert_new_fingerprint( &self, transaction_hash: H256, - amount: u64, + amount: u128, timestamp: SystemTime, ) -> Result<(), PendingPayableDaoError> { self.insert_fingerprint_params @@ -739,7 +727,7 @@ impl PendingPayableDaoMock { pub fn insert_fingerprint_params( mut self, - params: &Arc>>, + params: &Arc>>, ) -> Self { self.insert_fingerprint_params = params.clone(); self @@ -830,6 +818,13 @@ pub fn make_pending_payable_fingerprint() -> PendingPayableFingerprint { } } +pub fn convert_to_all_string_values(str_args: Vec<(&str, &str)>) -> Vec<(String, String)> { + str_args + .into_iter() + .map(|(a, b)| (a.to_string(), b.to_string())) + .collect() +} + #[derive(Default)] pub struct MessageIdGeneratorMock { ids: RefCell>, @@ -848,35 +843,16 @@ impl MessageIdGeneratorMock { } } -//warning: this test function will not tell you anything about the transaction record in the pending_payable table -pub fn account_status(conn: &Connection, wallet: &Wallet) -> Option { - let mut stmt = conn - .prepare("select balance, last_paid_timestamp, pending_payable_rowid from payable where wallet_address = ?") +pub fn assert_account_creation_fn_fails_on_finding_wrong_columns_and_value_types(tested_fn: F) +where + F: Fn(&Row) -> rusqlite::Result, +{ + let conn = Connection::open_in_memory().unwrap(); + conn.execute("create table whatever (exclamations text)", []) + .unwrap(); + conn.execute("insert into whatever (exclamations) values ('Gosh')", []) + .unwrap(); + + conn.query_row("select exclamations from whatever", [], tested_fn) .unwrap(); - stmt.query_row(&[&wallet], |row| { - let balance_result = row.get(0); - let last_paid_timestamp_result = row.get(1); - let pending_payable_rowid_result: Result, Error> = row.get(2); - match ( - balance_result, - last_paid_timestamp_result, - pending_payable_rowid_result, - ) { - (Ok(balance), Ok(last_paid_timestamp), Ok(rowid)) => Ok(PayableAccount { - wallet: wallet.clone(), - balance, - last_paid_timestamp: dao_utils::from_time_t(last_paid_timestamp), - pending_payable_opt: match rowid { - Some(rowid) => Some(PendingPayableId { - rowid: u64::try_from(rowid).unwrap(), - hash: H256::from_uint(&U256::from(0)), //garbage - }), - None => None, - }, - }), - _ => panic!("Database is corrupt: PAYABLE table columns and/or types"), - } - }) - .optional() - .unwrap() } diff --git a/node/src/actor_system_factory.rs b/node/src/actor_system_factory.rs index a8d1b0b36..06f980069 100644 --- a/node/src/actor_system_factory.rs +++ b/node/src/actor_system_factory.rs @@ -13,11 +13,13 @@ use super::ui_gateway::UiGateway; use crate::banned_dao::{BannedCacheLoader, BannedCacheLoaderReal}; use crate::blockchain::blockchain_bridge::BlockchainBridge; use crate::bootstrapper::CryptDEPair; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{connection_or_panic, DbInitializer, DbInitializerReal}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::persistent_configuration::PersistentConfiguration; use crate::node_configurator::configurator::Configurator; -use crate::sub_lib::accountant::AccountantSubs; +use crate::sub_lib::accountant::{ + AccountantSubs, AccountantSubsFactory, AccountantSubsFactoryReal, +}; use crate::sub_lib::blockchain_bridge::BlockchainBridgeSubs; use crate::sub_lib::configurator::ConfiguratorSubs; use crate::sub_lib::cryptde::CryptDE; @@ -47,7 +49,6 @@ use masq_lib::logger::Logger; use masq_lib::ui_gateway::{NodeFromUiMessage, NodeToUiMessage}; use masq_lib::utils::{exit_process, AutomapProtocol}; use std::net::{IpAddr, Ipv4Addr}; -use std::path::Path; pub trait ActorSystemFactory { fn make_and_start_actors( @@ -152,9 +153,9 @@ impl ActorSystemFactoryTools for ActorSystemFactoryToolsReal { let neighborhood_subs = actor_factory.make_and_start_neighborhood(cryptdes.main, &config); let accountant_subs = actor_factory.make_and_start_accountant( &config, - &config.data_directory.clone(), &db_initializer, &BannedCacheLoaderReal {}, + &AccountantSubsFactoryReal {}, ); let ui_gateway_subs = actor_factory.make_and_start_ui_gateway(&config); let stream_handler_pool_subs = actor_factory.make_and_start_stream_handler_pool(&config); @@ -359,9 +360,9 @@ pub trait ActorFactory { fn make_and_start_accountant( &self, config: &BootstrapperConfig, - data_directory: &Path, db_initializer: &dyn DbInitializer, banned_cache_loader: &dyn BannedCacheLoader, + accountant_subs_factory: &dyn AccountantSubsFactory, ) -> AccountantSubs; fn make_and_start_ui_gateway(&self, config: &BootstrapperConfig) -> UiGatewaySubs; fn make_and_start_stream_handler_pool( @@ -438,11 +439,11 @@ impl ActorFactory for ActorFactoryReal { fn make_and_start_accountant( &self, config: &BootstrapperConfig, - data_directory: &Path, db_initializer: &dyn DbInitializer, banned_cache_loader: &dyn BannedCacheLoader, + accountant_subs_factory: &dyn AccountantSubsFactory, ) -> AccountantSubs { - let cloned_config = config.clone(); + let data_directory = config.data_directory.as_path(); let payable_dao_factory = Accountant::dao_factory(data_directory); let receivable_dao_factory = Accountant::dao_factory(data_directory); let pending_payable_dao_factory = Accountant::dao_factory(data_directory); @@ -450,9 +451,9 @@ impl ActorFactory for ActorFactoryReal { banned_cache_loader.load(connection_or_panic( db_initializer, data_directory, - false, - MigratorConfig::panic_on_migration(), + DbInitializationConfig::panic_on_migration(), )); + let cloned_config = config.clone(); let arbiter = Arbiter::builder().stop_system_on_panic(true); let addr: Addr = arbiter.start(move |_| { Accountant::new( @@ -463,7 +464,7 @@ impl ActorFactory for ActorFactoryReal { Box::new(banned_dao_factory), ) }); - Accountant::make_subs_from(&addr) + accountant_subs_factory.make(&addr) } fn make_and_start_ui_gateway(&self, config: &BootstrapperConfig) -> UiGatewaySubs { @@ -599,6 +600,9 @@ impl LogRecipientSetter for LogRecipientSetterReal { #[cfg(test)] mod tests { use super::*; + use crate::accountant::check_sqlite_fns::TestUserDefinedSqliteFnsForNewDelinquencies; + use crate::accountant::test_utils::bc_from_ac_plus_earning_wallet; + use crate::actor_system_factory::tests::ShouldWeRunTheTest::{GoAhead, Skip}; use crate::bootstrapper::{Bootstrapper, RealUser}; use crate::database::connection_wrapper::ConnectionWrapper; use crate::node_test_utils::{ @@ -638,20 +642,27 @@ mod tests { use automap_lib::mocks::{ parameterizable_automap_control, TransactorMock, PUBLIC_IP, ROUTER_IP, }; - use crossbeam_channel::unbounded; + use crossbeam_channel::{bounded, unbounded, Sender}; use log::LevelFilter; use masq_lib::constants::DEFAULT_CHAIN; use masq_lib::crash_point::CrashPoint; #[cfg(feature = "log_recipient_test")] use masq_lib::logger::INITIALIZATION_COUNTER; use masq_lib::messages::{ToMessageBody, UiCrashRequest, UiDescriptorRequest}; - use masq_lib::test_utils::utils::TEST_DEFAULT_CHAIN; + use masq_lib::test_utils::utils::{ + check_if_source_code_is_attached, ensure_node_home_directory_exists, ShouldWeRunTheTest, + TEST_DEFAULT_CHAIN, + }; use masq_lib::ui_gateway::NodeFromUiMessage; use masq_lib::utils::running_test; use masq_lib::utils::AutomapProtocol::Igdp; + use regex::Regex; use std::cell::RefCell; use std::collections::HashMap; use std::convert::TryFrom; + use std::env::current_dir; + use std::fs::File; + use std::io::{BufRead, BufReader}; use std::net::Ipv4Addr; use std::net::{IpAddr, SocketAddr, SocketAddrV4}; use std::path::PathBuf; @@ -854,15 +865,15 @@ mod tests { fn make_and_start_accountant( &self, config: &BootstrapperConfig, - data_directory: &Path, _db_initializer: &dyn DbInitializer, _banned_cache_loader: &dyn BannedCacheLoader, + _accountant_subs_factory: &dyn AccountantSubsFactory, ) -> AccountantSubs { self.parameters .accountant_params .lock() .unwrap() - .get_or_insert((config.clone(), data_directory.to_path_buf())); + .get_or_insert(config.clone()); let addr: Addr = start_recorder_refcell_opt(&self.accountant); make_accountant_subs_from_recorder(&addr) } @@ -939,7 +950,7 @@ mod tests { proxy_server_params: Arc>>, hopper_params: Arc>>, neighborhood_params: Arc>>, - accountant_params: Arc>>, + accountant_params: Arc>>, ui_gateway_params: Arc>>, blockchain_bridge_params: Arc>>, configurator_params: Arc>>, @@ -1812,6 +1823,152 @@ mod tests { ); } + struct AccountantSubsFactoryTestOnly { + address_leaker: Sender>, + } + + impl AccountantSubsFactory for AccountantSubsFactoryTestOnly { + fn make(&self, addr: &Addr) -> AccountantSubs { + self.address_leaker.try_send(addr.clone()).unwrap(); + let nonsensical_addr = Recorder::new().start(); + make_accountant_subs_from_recorder(&nonsensical_addr) + } + } + + fn check_ongoing_usage_of_user_defined_fns_within_new_delinquencies_for_receivable_dao( + ) -> ShouldWeRunTheTest { + fn skip_down_to_first_line_saying_new_delinquencies( + previous: impl Iterator, + ) -> impl Iterator { + previous + .skip_while(|line| { + let adjusted_line: String = line + .chars() + .skip_while(|char| char.is_whitespace()) + .collect(); + !adjusted_line.starts_with("fn new_delinquencies(") + }) + .skip(1) + } + fn user_defined_functions_detected(line_undivided_fn_body: &str) -> bool { + line_undivided_fn_body.contains(" slope_drop_high_bytes(") + && line_undivided_fn_body.contains(" slope_drop_low_bytes(") + } + fn assert_is_not_trait_definition(body_lines: impl Iterator) -> String { + fn yield_if_contains_semicolon(line: &str) -> Option { + line.contains(';').then(|| line.to_string()) + } + let mut semicolon_line_opt = None; + let line_undivided_fn_body = body_lines + .map(|line| { + if semicolon_line_opt.is_none() { + if let Some(result) = yield_if_contains_semicolon(&line) { + semicolon_line_opt = Some(result) + } + } + line + }) + .collect::(); + if let Some(line) = semicolon_line_opt { + let regex = Regex::new(r"Vec<\w+>;").unwrap(); + if regex.is_match(&line) { + // The important part of the regex is the ending semicolon. Trait implementations don't use it; + // they just go on with an opening bracket of the function body. Its presence therefore signifies + // we have to do with a trait definition + panic!("the second parsed chunk of code is a trait definition and the implementation lies first") + } + } else { + () //means is a clean function body without semicolon + } + line_undivided_fn_body + } + + let current_dir = current_dir().unwrap(); + let file_path = current_dir + .join("src") + .join("accountant") + .join("receivable_dao.rs"); + let file = match File::open(file_path) { + Ok(file) => file, + Err(_) => { + if Skip == check_if_source_code_is_attached(¤t_dir) { + return Skip; + } else { + panic!( + "if panics, the file receivable_dao.rs probably doesn't exist or \ + has been moved to an unexpected location" + ) + } + } + }; + let reader = BufReader::new(file); + let lines_without_fn_trait_definition = + skip_down_to_first_line_saying_new_delinquencies(reader.lines().flatten()); + let function_body_ready_for_final_check = { + let assumed_implemented_function_body = + skip_down_to_first_line_saying_new_delinquencies(lines_without_fn_trait_definition) + .take_while(|line| { + let adjusted_line: String = line + .chars() + .skip_while(|char| char.is_whitespace()) + .collect(); + !adjusted_line.starts_with("fn") + }); + assert_is_not_trait_definition(assumed_implemented_function_body) + }; + if user_defined_functions_detected(&function_body_ready_for_final_check) { + GoAhead + } else { + panic!("was about to test user-defined SQLite functions (slope_drop_high_bytes and slope_drop_low_bytes) + in new_delinquencies() but found out those are absent at the expected place and would leave falsely positive results") + } + } + + #[test] + fn our_big_int_sqlite_functions_are_linked_to_receivable_dao_within_accountant() { + //condition: .new_delinquencies() still encompasses our user defined functions (that's why a formal check opens this test) + if let Skip = + check_ongoing_usage_of_user_defined_fns_within_new_delinquencies_for_receivable_dao() + { + eprintln!("skipping test our_big_int_sqlite_functions_are_linked_to_receivable_dao_within_accountant; + was unable to find receivable_dao.rs"); + return; + }; + let data_dir = ensure_node_home_directory_exists( + "actor_system_factory", + "our_big_int_sqlite_functions_are_linked_to_receivable_dao_within_accountant", + ); + let accountant_config = make_populated_accountant_config_with_defaults(); + let _ = DbInitializerReal::default() + .initialize(data_dir.as_ref(), DbInitializationConfig::test_default()) + .unwrap(); + let mut b_config = bc_from_ac_plus_earning_wallet(accountant_config, make_wallet("mine")); + b_config.data_directory = data_dir; + let system = System::new( + "our_big_int_sqlite_functions_are_linked_to_receivable_dao_within_accountant", + ); + let (addr_tx, addr_rv) = bounded(1); + let subject = ActorFactoryReal {}; + + subject.make_and_start_accountant( + &b_config, + &DbInitializerReal::default(), + &BannedCacheLoaderMock::default(), + &AccountantSubsFactoryTestOnly { + address_leaker: addr_tx, + }, + ); + + let accountant_addr = addr_rv.try_recv().unwrap(); + //this message also stops the system after the check + accountant_addr + .try_send(TestUserDefinedSqliteFnsForNewDelinquencies {}) + .unwrap(); + assert_eq!(system.run(), 0); + //we didn't blow up, it recognized the functions + //this is an example of the error: "no such function: slope_drop_high_bytes" + } + #[test] fn make_and_start_actors_happy_path() { let validate_database_chain_params_arc = Arc::new(Mutex::new(vec![])); @@ -1829,10 +1986,8 @@ mod tests { let alias_cryptde_public_key_before = public_key_for_dyn_cryptde_being_null(alias_cryptde); let actor_factory = Box::new(ActorFactoryReal {}) as Box; let actor_factory_before_raw_address = addr_of!(*actor_factory); - let persistent_config_id = ArbitraryIdStamp::new(); - let persistent_config = Box::new( - PersistentConfigurationMock::default().set_arbitrary_id_stamp(persistent_config_id), - ); + let persistent_config = Box::new(PersistentConfigurationMock::default()); + let persistent_config_id = persistent_config.set_arbitrary_id_stamp(); let persistent_config_before_raw = addr_of!(*persistent_config); let tools = ActorSystemFactoryToolsMock::default() .cryptdes_result(CryptDEPair { diff --git a/node/src/banned_dao.rs b/node/src/banned_dao.rs index f730320cf..e17bd0b7e 100644 --- a/node/src/banned_dao.rs +++ b/node/src/banned_dao.rs @@ -1,6 +1,6 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. +use crate::accountant::dao_utils::{DaoFactoryReal, VigilantRusqliteFlatten}; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::dao_utils::DaoFactoryReal; use crate::sub_lib::wallet::Wallet; use lazy_static::lazy_static; use rusqlite::{Error, ErrorCode, ToSql}; @@ -96,7 +96,7 @@ impl BannedDao for BannedDaoReal { .expect("Failed to prepare a statement"); stmt.query_map([], |row| row.get(0)) .expect("Couldn't retrieve delinquency-ban list: database corrupt") - .flatten() + .vigilant_flatten() .collect() } @@ -147,8 +147,8 @@ impl BannedDao for BannedDaoReal { #[cfg(test)] mod tests { use super::*; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; - use crate::database::db_migrations::MigratorConfig; use crate::test_utils::make_paying_wallet; use crate::test_utils::make_wallet; use masq_lib::test_utils::utils::{ @@ -164,7 +164,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let subject = { let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); BannedDaoReal::new(conn) }; @@ -172,7 +172,7 @@ mod tests { subject.ban(&make_wallet("donalddrumph")); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut stmt = conn.prepare("select wallet_address from banned").unwrap(); let mut banned_addresses = stmt.query([]).unwrap(); @@ -192,7 +192,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let subject = { let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); BannedDaoReal::new(conn) }; @@ -211,7 +211,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let subject = { let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); BannedDaoReal::new(conn) }; @@ -228,7 +228,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let wallet = &make_wallet("booga"); conn.prepare("insert into banned (wallet_address) values (?)") @@ -241,7 +241,7 @@ mod tests { subject.unban(wallet); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut stmt = conn .prepare("select wallet_address from banned where wallet_address = ?") @@ -258,7 +258,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = BannedDaoReal::new(conn); @@ -276,7 +276,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); conn.prepare("insert into banned (wallet_address) values ('0x000000000000000000495f414d5f42414e4e4544')") .unwrap() @@ -303,7 +303,7 @@ mod tests { let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let subject = BannedDaoReal::new(conn); @@ -319,7 +319,7 @@ mod tests { ensure_node_home_directory_does_not_exist("banned_dao", "unban_removes_from_ban_cache"); let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let unban_me_baby = make_wallet("UNBAN_ME_BABY"); conn.prepare("insert into banned (wallet_address) values ('UNBAN_ME_BABY')") diff --git a/node/src/blockchain/blockchain_bridge.rs b/node/src/blockchain/blockchain_bridge.rs index fb2892168..cd59bc4e5 100644 --- a/node/src/blockchain/blockchain_bridge.rs +++ b/node/src/blockchain/blockchain_bridge.rs @@ -2,15 +2,15 @@ use crate::accountant::payable_dao::{Payable, PayableAccount}; use crate::accountant::{ - ReceivedPayments, ResponseSkeleton, ScanError, SentPayable, SkeletonOptHolder, + ReceivedPayments, ResponseSkeleton, ScanError, SentPayables, SkeletonOptHolder, }; use crate::accountant::{ReportTransactionReceipts, RequestTransactionReceipts}; use crate::blockchain::blockchain_interface::{ BlockchainError, BlockchainInterface, BlockchainInterfaceClandestine, - BlockchainInterfaceNonClandestine, BlockchainResult, SendTransactionInputs, + BlockchainInterfaceNonClandestine, BlockchainResult, BlockchainTxnInputs, }; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DATABASE_FILE}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::ConfigDaoReal; use crate::db_config::persistent_configuration::{ PersistentConfiguration, PersistentConfigurationReal, @@ -32,7 +32,6 @@ use masq_lib::logger::Logger; use masq_lib::messages::ScanType; use masq_lib::ui_gateway::NodeFromUiMessage; use masq_lib::utils::plus; -use std::convert::TryFrom; use std::path::PathBuf; use std::time::SystemTime; use web3::transports::Http; @@ -46,7 +45,7 @@ pub struct BlockchainBridge { logger: Logger, persistent_config: Box, set_consuming_wallet_subs_opt: Option>>, - sent_payable_subs_opt: Option>, + sent_payable_subs_opt: Option>, received_payments_subs_opt: Option>, scan_error_subs_opt: Option>, crashable: bool, @@ -149,7 +148,7 @@ pub struct PendingPayableFingerprint { pub timestamp: SystemTime, pub hash: H256, pub attempt_opt: Option, //None when initialized - pub amount: u64, + pub amount: u128, pub process_error: Option, } @@ -207,7 +206,10 @@ impl BlockchainBridge { }; let config_dao = Box::new(ConfigDaoReal::new( db_initializer - .initialize(&data_directory, true, MigratorConfig::panic_on_migration()) + .initialize( + &data_directory, + DbInitializationConfig::panic_on_migration(), + ) .unwrap_or_else(|_| { panic!( "Failed to connect to database at {:?}", @@ -236,12 +238,12 @@ impl BlockchainBridge { creditors_msg: &ReportAccountsPayable, ) -> Result<(), String> { let skeleton = creditors_msg.response_skeleton_opt; - let processed_payments = self.handle_report_accounts_payable_inner(creditors_msg); + let processed_payments = self.preprocess_payments(creditors_msg); processed_payments.map(|payments| { self.sent_payable_subs_opt .as_ref() .expect("Accountant is unbound") - .try_send(SentPayable { + .try_send(SentPayables { timestamp: SystemTime::now(), payable: payments, response_skeleton_opt: skeleton, @@ -250,15 +252,17 @@ impl BlockchainBridge { }) } - fn handle_report_accounts_payable_inner( + fn preprocess_payments( &self, creditors_msg: &ReportAccountsPayable, ) -> Result>, String> { match self.consuming_wallet_opt.as_ref() { Some(consuming_wallet) => match self.persistent_config.gas_price() { - Ok(gas_price) => { - Ok(self.process_payments(creditors_msg, gas_price, consuming_wallet)) - } + Ok(gas_price) => Ok(creditors_msg + .accounts + .iter() + .map(|payable| self.process_payments(payable, gas_price, consuming_wallet)) + .collect::>>()), Err(err) => Err(format!("ReportAccountPayable: gas-price: {:?}", err)), }, None => Err(String::from("No consuming wallet specified")), @@ -380,19 +384,6 @@ impl BlockchainBridge { } fn process_payments( - &self, - creditors_msg: &ReportAccountsPayable, - gas_price: u64, - consuming_wallet: &Wallet, - ) -> Vec> { - creditors_msg - .accounts - .iter() - .map(|payable| self.process_payments_inner_body(payable, gas_price, consuming_wallet)) - .collect::>>() - } - - fn process_payments_inner_body( &self, payable: &PayableAccount, gas_price: u64, @@ -401,9 +392,7 @@ impl BlockchainBridge { let nonce = self .blockchain_interface .get_transaction_count(consuming_wallet)?; - let unsigned_amount = u64::try_from(payable.balance) - .expect("negative balance for qualified payable is nonsense"); - let send_tools = self.blockchain_interface.send_transaction_tools( + let send_tx_tools = self.blockchain_interface.send_transaction_tools( self.payment_confirmation .transaction_fingerprint_subs_opt .as_ref() @@ -411,16 +400,16 @@ impl BlockchainBridge { ); match self .blockchain_interface - .send_transaction(SendTransactionInputs::new( + .send_transaction(BlockchainTxnInputs::new( payable, consuming_wallet, nonce, gas_price, - send_tools.as_ref(), - )?) { + send_tx_tools.as_ref(), + )) { Ok((hash, timestamp)) => Ok(Payable::new( payable.wallet.clone(), - unsigned_amount, + payable.balance_wei, hash, timestamp, )), @@ -440,6 +429,7 @@ struct PendingTxInfo { #[cfg(test)] mod tests { use super::*; + use crate::accountant::dao_utils::from_time_t; use crate::accountant::payable_dao::PayableAccount; use crate::accountant::test_utils::make_pending_payable_fingerprint; use crate::blockchain::bip32::Bip32ECKeyProvider; @@ -450,7 +440,6 @@ mod tests { }; use crate::blockchain::test_utils::BlockchainInterfaceMock; use crate::blockchain::tool_wrappers::SendTransactionToolsWrapperNull; - use crate::database::dao_utils::from_time_t; use crate::database::db_initializer::test_utils::DbInitializerMock; use crate::db_config::persistent_configuration::PersistentConfigError; use crate::node_test_utils::check_timestamp; @@ -571,7 +560,7 @@ mod tests { let request = ReportAccountsPayable { accounts: vec![PayableAccount { wallet: make_wallet("blah"), - balance: 42, + balance_wei: 42, last_paid_timestamp: SystemTime::now(), pending_payable_opt: None, }], @@ -583,7 +572,7 @@ mod tests { .payment_confirmation .transaction_fingerprint_subs_opt = Some(fingerprint_recipient); - let result = subject.handle_report_accounts_payable_inner(&request); + let result = subject.preprocess_payments(&request); assert_eq!( result, @@ -609,14 +598,14 @@ mod tests { let request = ReportAccountsPayable { accounts: vec![PayableAccount { wallet: make_wallet("blah"), - balance: 42, + balance_wei: 42, last_paid_timestamp: SystemTime::now(), pending_payable_opt: None, }], response_skeleton_opt: None, }; - let result = subject.handle_report_accounts_payable_inner(&request); + let result = subject.preprocess_payments(&request); assert_eq!(result, Err("No consuming wallet specified".to_string())); } @@ -664,13 +653,13 @@ mod tests { accounts: vec![ PayableAccount { wallet: make_wallet("blah"), - balance: 420, + balance_wei: 420, last_paid_timestamp: from_time_t(150_000_000), pending_payable_opt: None, }, PayableAccount { wallet: make_wallet("foo"), - balance: 210, + balance_wei: 210, last_paid_timestamp: from_time_t(160_000_000), pending_payable_opt: None, }, @@ -712,11 +701,11 @@ mod tests { ); let accountant_received_payment = accountant_recording_arc.lock().unwrap(); assert_eq!(accountant_received_payment.len(), 1); - let sent_payments_msg = accountant_received_payment.get_record::(0); + let sent_payments_msg = accountant_received_payment.get_record::(0); check_timestamp(before, sent_payments_msg.timestamp, after); assert_eq!( *sent_payments_msg, - SentPayable { + SentPayables { timestamp: sent_payments_msg.timestamp, payable: vec![ Ok(Payable { @@ -757,7 +746,7 @@ mod tests { let request = ReportAccountsPayable { accounts: vec![PayableAccount { wallet: make_wallet("blah"), - balance: 42, + balance_wei: 42, last_paid_timestamp: SystemTime::now(), pending_payable_opt: None, }], @@ -1054,12 +1043,12 @@ mod tests { BlockchainTransaction { block_number: 7, from: earning_wallet.clone(), - gwei_amount: amount, + wei_amount: amount, }, BlockchainTransaction { block_number: 9, from: earning_wallet.clone(), - gwei_amount: amount2, + wei_amount: amount2, }, ], }; @@ -1212,7 +1201,7 @@ mod tests { transactions: vec![BlockchainTransaction { block_number: 1000, from: make_wallet("somewallet"), - gwei_amount: 2345, + wei_amount: 2345, }], }), ); diff --git a/node/src/blockchain/blockchain_interface.rs b/node/src/blockchain/blockchain_interface.rs index b46937c74..bef563193 100644 --- a/node/src/blockchain/blockchain_interface.rs +++ b/node/src/blockchain/blockchain_interface.rs @@ -15,6 +15,7 @@ use std::convert::{From, TryFrom, TryInto}; use std::fmt; use std::fmt::{Debug, Display, Formatter}; use std::time::SystemTime; +use thousands::Separable; use web3::contract::{Contract, Options}; use web3::transports::EventLoopHandle; use web3::types::{ @@ -38,7 +39,7 @@ const TRANSFER_METHOD_ID: [u8; 4] = [0xa9, 0x05, 0x9c, 0xbb]; pub struct BlockchainTransaction { pub block_number: u64, pub from: Wallet, - pub gwei_amount: u64, + pub wei_amount: u128, } impl fmt::Display for BlockchainTransaction { @@ -46,7 +47,7 @@ impl fmt::Display for BlockchainTransaction { write!( f, "{}gw from {} ({})", - self.gwei_amount, self.from, self.block_number + self.wei_amount, self.from, self.block_number ) } } @@ -57,7 +58,7 @@ pub enum BlockchainError { InvalidAddress, InvalidResponse, QueryFailed(String), - SignedValueConversion(i64), + SignedValueConversion(i128), TransactionFailed { msg: String, hash_opt: Option }, } @@ -96,7 +97,7 @@ pub trait BlockchainInterface { fn send_transaction( &self, - inputs: SendTransactionInputs, + inputs: BlockchainTxnInputs, ) -> Result<(H256, SystemTime), BlockchainTransactionError>; fn get_eth_balance(&self, address: &Wallet) -> Balance; @@ -158,7 +159,7 @@ impl BlockchainInterface for BlockchainInterfaceClandestine { fn send_transaction<'a>( &self, - _inputs: SendTransactionInputs, + _inputs: BlockchainTxnInputs, ) -> Result<(H256, SystemTime), BlockchainTransactionError> { let msg = "Can't send transactions clandestinely yet".to_string(); error!(self.logger, "{}", &msg); @@ -212,10 +213,6 @@ pub struct BlockchainInterfaceNonClandestine { const GWEI: U256 = U256([1_000_000_000u64, 0, 0, 0]); -pub fn to_gwei(wei: U256) -> Option { - u64::try_from(wei / GWEI).ok() -} - pub fn to_wei(gwub: u64) -> U256 { let subgwei = U256::from(gwub); subgwei.full_mul(GWEI).try_into().expect("Internal Error") @@ -277,12 +274,14 @@ where .filter_map(|log: &Log| match log.block_number { Some(block_number) => { let amount: U256 = U256::from(log.data.0.as_slice()); - let gwei_amount = to_gwei(amount); - gwei_amount.map(|gwei_amount| BlockchainTransaction { - block_number: u64::try_from(block_number) - .expect("Internal Error"), - from: Wallet::from(log.topics[1]), - gwei_amount, + let wei_amount_result = u128::try_from(amount); + wei_amount_result.ok().map(|wei_amount| { + BlockchainTransaction { + block_number: u64::try_from(block_number) + .expect("Internal Error"), + from: Wallet::from(log.topics[1]), + wei_amount, + } }) } None => None, @@ -313,10 +312,10 @@ where fn send_transaction<'a>( &self, - inputs: SendTransactionInputs, + inputs: BlockchainTxnInputs, ) -> Result<(H256, SystemTime), BlockchainTransactionError> { self.logger.debug(|| self.preparation_log(&inputs)); - let signed_transaction = self.prepare_signed_transaction(&inputs, inputs.tools)?; + let signed_transaction = self.prepare_signed_transaction(&inputs)?; let payable_timestamp = inputs .tools .request_new_payable_fingerprint(signed_transaction.transaction_hash, inputs.amount); @@ -400,15 +399,16 @@ where } } - fn prepare_signed_transaction<'a>( + fn prepare_signed_transaction( &self, - inputs: &SendTransactionInputs, - send_transaction_tools: &'a dyn SendTransactionToolsWrapper, + inputs: &BlockchainTxnInputs, ) -> Result { let mut data = [0u8; 4 + 32 + 32]; data[0..4].copy_from_slice(&TRANSFER_METHOD_ID); data[16..36].copy_from_slice(&inputs.recipient.address().0[..]); - to_wei(inputs.amount).to_big_endian(&mut data[36..68]); + U256::try_from(inputs.amount) + .expect("shouldn't overflow") + .to_big_endian(&mut data[36..68]); let base_gas_limit = Self::base_gas_limit(self.chain); let gas_limit = ethereum_types::U256::try_from(data.iter().fold(base_gas_limit, |acc, v| { @@ -439,30 +439,30 @@ where Err(e) => return Err(BlockchainTransactionError::UnusableWallet(e.to_string())), }; - match send_transaction_tools.sign_transaction(transaction_parameters, &key) { + match inputs.tools.sign_transaction(transaction_parameters, &key) { Ok(tx) => Ok(tx), Err(e) => Err(BlockchainTransactionError::Signing(e.to_string())), } } - fn preparation_log(&self, inputs: &SendTransactionInputs) -> String { - format!("Preparing transaction for {} Gwei to {} from {} (chain_id: {}, contract: {:#x}, gas price: {})", //TODO fix this later to Wei - inputs.amount, + fn preparation_log(&self, inputs: &BlockchainTxnInputs) -> String { + format!("Preparing transaction for {} wei to {} from {} (chain: {}, contract: {:#x}, gas price: {})", + inputs.amount.separate_with_commas(), inputs.recipient, inputs.consuming_wallet, - self.chain.rec().num_chain_id, + self.chain.rec().literal_identifier, self.contract_address(), inputs.gas_price) } - fn transmission_log(&self, recipient: &Wallet, amount: u64) -> String { + fn transmission_log(&self, recipient: &Wallet, amount: u128) -> String { format!( "About to send transaction:\n\ recipient: {},\n\ - amount: {},\n\ + amount: {} wei,\n\ (chain: {}, contract: {:#x})", recipient, - amount, + amount.separate_with_commas(), self.chain.rec().literal_identifier, self.contract_address() ) @@ -483,36 +483,35 @@ where } #[derive(Debug, Clone)] -pub struct SendTransactionInputs<'a> { +pub struct BlockchainTxnInputs<'a> { tools: &'a dyn SendTransactionToolsWrapper, recipient: &'a Wallet, consuming_wallet: &'a Wallet, - amount: u64, + amount: u128, nonce: U256, gas_price: u64, } -impl<'a> SendTransactionInputs<'a> { +impl<'a> BlockchainTxnInputs<'a> { pub fn new( account: &'a PayableAccount, consuming_wallet: &'a Wallet, nonce: U256, gas_price: u64, tools: &'a dyn SendTransactionToolsWrapper, - ) -> Result { - Ok(Self { + ) -> Self { + Self { tools, recipient: &account.wallet, consuming_wallet, - amount: u64::try_from(account.balance) - .map_err(|_| BlockchainError::SignedValueConversion(account.balance))?, + amount: account.balance_wei, nonce, gas_price, - }) + } } #[cfg(test)] - pub fn abstract_for_assertions(self) -> (Wallet, Wallet, u64, U256, u64) { + pub fn abstract_for_assertions(self) -> (Wallet, Wallet, u128, U256, u64) { ( self.consuming_wallet.clone(), self.recipient.clone(), @@ -812,13 +811,13 @@ mod tests { block_number: 0x4be663, from: Wallet::from_str("0x3ab28ecedea6cdb6feed398e93ae8c7b316b1182") .unwrap(), - gwei_amount: 4_503_599, + wei_amount: 4_503_599_627_370_496, }, BlockchainTransaction { block_number: 0x4be662, from: Wallet::from_str("0x3f69f9efd4f2592fd70be8c32ecd9dce71c472fc") .unwrap(), - gwei_amount: 4_503_599, + wei_amount: 4_503_599_627_370_496, }, ] } @@ -1156,7 +1155,7 @@ mod tests { make_fake_event_loop_handle(), TEST_DEFAULT_CHAIN, ); - let amount = 9000; + let amount = 9_000_000_000_000; let gas_price = 120; let account = make_payable_account_with_recipient_and_balance_and_timestamp_opt( make_wallet("blah123"), @@ -1165,14 +1164,13 @@ mod tests { ); let consuming_wallet = make_paying_wallet(b"gdasgsa"); let tools = subject.send_transaction_tools(&recipient_of_pending_payable_fingerprint); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &account, &consuming_wallet, U256::from(1), gas_price, tools.as_ref(), - ) - .unwrap(); + ); let test_timestamp_before = SystemTime::now(); let result = subject.send_transaction(inputs).unwrap(); @@ -1197,16 +1195,16 @@ mod tests { timestamp, hash, attempt_opt: None, - amount: amount as u64, + amount, process_error: None, }; assert_eq!(sent_backup, &expected_pending_payable_fingerprint); let log_handler = TestLogHandler::new(); - log_handler.exists_log_containing("DEBUG: BlockchainInterface: Preparing transaction for 9000 Gwei to 0x00000000000000000000000000626c6168313233 from 0x5c361ba8d82fcf0e5538b2a823e9d457a2296725 (chain_id: 3, contract: 0x384dec25e03f94931767ce4c3556168468ba24c3, gas price: 120)" ); + log_handler.exists_log_containing("DEBUG: BlockchainInterface: Preparing transaction for 9,000,000,000,000 wei to 0x00000000000000000000000000626c6168313233 from 0x5c361ba8d82fcf0e5538b2a823e9d457a2296725 (chain: eth-ropsten, contract: 0x384dec25e03f94931767ce4c3556168468ba24c3, gas price: 120)" ); log_handler.exists_log_containing( "INFO: BlockchainInterface: About to send transaction:\n\ recipient: 0x00000000000000000000000000626c6168313233,\n\ - amount: 9000,\n\ + amount: 9,000,000,000,000 wei,\n\ (chain: eth-ropsten, contract: 0x384dec25e03f94931767ce4c3556168468ba24c3)", ); } @@ -1257,21 +1255,20 @@ mod tests { .request_new_pending_payable_fingerprint_result(payable_timestamp) .send_raw_transaction_params(&send_raw_transaction_params_arc) .send_raw_transaction_result(Ok(hash)); - let amount = 50_000; + let amount_of_wei = 50_000_000_000_000; let account = make_payable_account_with_recipient_and_balance_and_timestamp_opt( make_wallet("blah123"), - amount, + amount_of_wei, None, ); let consuming_wallet = make_paying_wallet(consuming_wallet_secret_raw_bytes); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &account, &consuming_wallet, nonce, 123, send_transaction_tools, - ) - .unwrap(); + ); let result = subject.send_transaction(inputs); @@ -1292,7 +1289,7 @@ mod tests { .unwrap(); assert_eq!( *request_new_pending_payable_fingerprint_params, - vec![(hash, amount as u64)] + vec![(hash, amount_of_wei)] ); let send_raw_transaction = send_raw_transaction_params_arc.lock().unwrap(); assert_eq!( @@ -1374,14 +1371,13 @@ mod tests { .sign_transaction_result(Err(Web3Error::Internal)); let payable_account = make_payable_account(1); let consuming_wallet = make_paying_wallet(consuming_wallet_secret_raw_bytes); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &payable_account, &consuming_wallet, U256::from(5), 123, send_transaction_tools, - ) - .unwrap(); + ); let _ = subject.send_transaction(inputs); @@ -1418,14 +1414,13 @@ mod tests { None, ); let tools = subject.send_transaction_tools(&recipient); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &account, &address_only_wallet, U256::from(1), 123, tools.as_ref(), - ) - .unwrap(); + ); let result = subject.send_transaction(inputs); @@ -1459,14 +1454,13 @@ mod tests { None, ); let consuming_wallet = make_paying_wallet(consuming_wallet_secret_raw_bytes); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &account, &consuming_wallet, U256::from(1), 123, send_transaction_tools, - ) - .unwrap(); + ); let result = subject.send_transaction(inputs); @@ -1500,14 +1494,13 @@ mod tests { None, ); let consuming_wallet = make_paying_wallet(consuming_wallet_secret_raw_bytes); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &account, &consuming_wallet, U256::from(1), 123, send_transaction_tools, - ) - .unwrap(); + ); let result = subject.send_transaction(inputs); @@ -1536,7 +1529,7 @@ mod tests { Wallet::from(address) } - const TEST_PAYMENT_AMOUNT: u64 = 1000; + const TEST_PAYMENT_AMOUNT: u128 = 1_000_000_000_000; const TEST_GAS_PRICE_ETH: u64 = 110; const TEST_GAS_PRICE_POLYGON: u64 = 50; @@ -1546,7 +1539,6 @@ mod tests { template: &[u8], ) { let recipient = { - //the place where this recipient would've been really used cannot be found in this test; we just need to supply some let (accountant, _, _) = make_recorder(); let account_addr = accountant.start(); recipient!(account_addr, PendingPayableFingerprint) @@ -1565,21 +1557,18 @@ mod tests { }; let payable_account = make_payable_account_with_recipient_and_balance_and_timestamp_opt( recipient_wallet, - i64::try_from(TEST_PAYMENT_AMOUNT).unwrap(), + TEST_PAYMENT_AMOUNT, None, ); - let inputs = SendTransactionInputs::new( + let inputs = BlockchainTxnInputs::new( &payable_account, &consuming_wallet, nonce_correct_type, gas_price, send_transaction_tools.as_ref(), - ) - .unwrap(); + ); - let signed_transaction = subject - .prepare_signed_transaction(&inputs, send_transaction_tools.as_ref()) - .unwrap(); + let signed_transaction = subject.prepare_signed_transaction(&inputs).unwrap(); let byte_set_to_compare = signed_transaction.raw_transaction.0; assert_eq!(&byte_set_to_compare, template) @@ -1900,11 +1889,6 @@ mod tests { }; } - #[test] - fn to_gwei_truncates_units_smaller_than_gwei() { - assert_eq!(Some(1), to_gwei(U256::from(1_999_999_999))); - } - #[test] fn to_wei_converts_units_properly_for_max_value() { let converted_wei = to_wei(u64::MAX); @@ -1944,25 +1928,6 @@ mod tests { ); } - #[test] - fn constructor_for_send_transaction_inputs_handles_value_out_of_range() { - let mut payable_account = make_payable_account(5); - payable_account.balance = -100; - let consuming_wallet = make_wallet("blah"); - let send_transaction_tools = SendTransactionToolsWrapperNull; - - let error = SendTransactionInputs::new( - &payable_account, - &consuming_wallet, - U256::from(4545), - 130, - &send_transaction_tools, - ) - .unwrap_err(); - - assert_eq!(error, BlockchainError::SignedValueConversion(-100)) - } - #[test] fn conversion_between_errors_work() { let hash = H256::from_uint(&U256::from(4555)); diff --git a/node/src/blockchain/test_utils.rs b/node/src/blockchain/test_utils.rs index f1ce0f8b7..7479908f4 100644 --- a/node/src/blockchain/test_utils.rs +++ b/node/src/blockchain/test_utils.rs @@ -5,7 +5,7 @@ use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::blockchain::blockchain_interface::{ Balance, BlockchainError, BlockchainInterface, BlockchainResult, BlockchainTransactionError, - Nonce, Receipt, SendTransactionInputs, REQUESTS_IN_PARALLEL, + BlockchainTxnInputs, Nonce, Receipt, REQUESTS_IN_PARALLEL, }; use crate::blockchain::tool_wrappers::SendTransactionToolsWrapper; use crate::sub_lib::wallet::Wallet; @@ -56,7 +56,7 @@ pub struct BlockchainInterfaceMock { retrieve_transactions_parameters: Arc>>, retrieve_transactions_results: RefCell>>, - send_transaction_parameters: Arc>>, + send_transaction_parameters: Arc>>, send_transaction_results: RefCell>>, get_transaction_receipt_params: Arc>>, get_transaction_receipt_results: RefCell>, @@ -82,7 +82,7 @@ impl BlockchainInterfaceMock { pub fn send_transaction_params( mut self, - params: &Arc>>, + params: &Arc>>, ) -> Self { self.send_transaction_parameters = params.clone(); self @@ -153,7 +153,7 @@ impl BlockchainInterface for BlockchainInterfaceMock { fn send_transaction<'b>( &self, - inputs: SendTransactionInputs, + inputs: BlockchainTxnInputs, ) -> Result<(H256, SystemTime), BlockchainTransactionError> { self.send_transaction_parameters .lock() @@ -278,7 +278,7 @@ pub struct SendTransactionToolsWrapperMock { sign_transaction_params: Arc>>, sign_transaction_results: RefCell>>, - request_new_pending_payable_fingerprint_params: Arc>>, + request_new_pending_payable_fingerprint_params: Arc>>, request_new_pending_payable_fingerprint_results: RefCell>, send_raw_transaction_params: Arc>>, send_raw_transaction_results: RefCell>>, @@ -297,7 +297,7 @@ impl SendTransactionToolsWrapper for SendTransactionToolsWrapperMock { self.sign_transaction_results.borrow_mut().remove(0) } - fn request_new_payable_fingerprint(&self, transaction_hash: H256, amount: u64) -> SystemTime { + fn request_new_payable_fingerprint(&self, transaction_hash: H256, amount: u128) -> SystemTime { self.request_new_pending_payable_fingerprint_params .lock() .unwrap() @@ -328,7 +328,7 @@ impl SendTransactionToolsWrapperMock { pub fn request_new_pending_payable_fingerprint_params( mut self, - params: &Arc>>, + params: &Arc>>, ) -> Self { self.request_new_pending_payable_fingerprint_params = params.clone(); self diff --git a/node/src/blockchain/tool_wrappers.rs b/node/src/blockchain/tool_wrappers.rs index 0dbffae75..edd08e159 100644 --- a/node/src/blockchain/tool_wrappers.rs +++ b/node/src/blockchain/tool_wrappers.rs @@ -16,7 +16,7 @@ pub trait SendTransactionToolsWrapper: Debug { transaction_params: TransactionParameters, key: &secp256k1secrets::key::SecretKey, ) -> Result; - fn request_new_payable_fingerprint(&self, transaction_hash: H256, amount: u64) -> SystemTime; + fn request_new_payable_fingerprint(&self, transaction_hash: H256, amount: u128) -> SystemTime; fn send_raw_transaction(&self, rlp: Bytes) -> Result; } @@ -57,7 +57,7 @@ impl<'a, T: Transport + Debug> SendTransactionToolsWrapper .wait() } - fn request_new_payable_fingerprint(&self, hash: H256, amount: u64) -> SystemTime { + fn request_new_payable_fingerprint(&self, hash: H256, amount: u128) -> SystemTime { let now = SystemTime::now(); self.pending_payable_fingerprint_sub .try_send(PendingPayableFingerprint { @@ -89,7 +89,11 @@ impl SendTransactionToolsWrapper for SendTransactionToolsWrapperNull { panic!("sign_transaction() should never be called on the null object") } - fn request_new_payable_fingerprint(&self, _transaction_hash: H256, _amount: u64) -> SystemTime { + fn request_new_payable_fingerprint( + &self, + _transaction_hash: H256, + _amount: u128, + ) -> SystemTime { panic!( "request_new_pending_payable_fingerprint() should never be called on the null object" ) diff --git a/node/src/bootstrapper.rs b/node/src/bootstrapper.rs index 47b0ad42d..bcb71cd26 100644 --- a/node/src/bootstrapper.rs +++ b/node/src/bootstrapper.rs @@ -3,8 +3,8 @@ use crate::actor_system_factory::ActorSystemFactory; use crate::actor_system_factory::ActorSystemFactoryReal; use crate::actor_system_factory::{ActorFactoryReal, ActorSystemFactoryToolsReal}; use crate::crash_test_dummy::CrashTestDummy; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::ConfigDaoReal; use crate::db_config::persistent_configuration::{ PersistentConfiguration, PersistentConfigurationReal, @@ -505,8 +505,7 @@ impl ConfiguredByPrivilege for Bootstrapper { Box::new(ActorFactoryReal {}), initialize_database( &self.config.data_directory, - false, - MigratorConfig::panic_on_migration(), + DbInitializationConfig::panic_on_migration(), ), ); @@ -614,8 +613,7 @@ impl Bootstrapper { let conn = DbInitializerReal::default() .initialize( &self.config.data_directory, - false, - MigratorConfig::panic_on_migration(), + DbInitializationConfig::panic_on_migration(), ) .expect("Cannot initialize database"); let config_dao = ConfigDaoReal::new(conn); @@ -694,8 +692,8 @@ mod tests { main_cryptde_ref, Bootstrapper, BootstrapperConfig, EnvironmentWrapper, PortConfiguration, RealUser, }; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; - use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::ConfigDaoReal; use crate::db_config::persistent_configuration::{ PersistentConfigError, PersistentConfiguration, PersistentConfigurationReal, @@ -1784,7 +1782,7 @@ mod tests { "set_up_clandestine_port_handles_specified_port_in_standard_mode", ); let conn = DbInitializerReal::default() - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(); let cryptde_actual = CryptDENull::from(&PublicKey::new(&[1, 2, 3, 4]), TEST_DEFAULT_CHAIN); let cryptde: &dyn CryptDE = &cryptde_actual; @@ -1858,7 +1856,7 @@ mod tests { "set_up_clandestine_port_handles_unspecified_port_in_standard_mode", ); let conn = DbInitializerReal::default() - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(); let mut config = BootstrapperConfig::new(); config.neighborhood_config = NeighborhoodConfig { diff --git a/node/src/daemon/mod.rs b/node/src/daemon/mod.rs index f64d97e13..35052e26f 100644 --- a/node/src/daemon/mod.rs +++ b/node/src/daemon/mod.rs @@ -1542,7 +1542,12 @@ mod tests { subject_addr .try_send(make_daemon_bind_message(ui_gateway)) .unwrap(); - let body: MessageBody = UiFinancialsRequest {}.tmb(4321); + let body: MessageBody = UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None, + } + .tmb(4321); subject_addr .try_send(NodeFromUiMessage { diff --git a/node/src/daemon/setup_reporter.rs b/node/src/daemon/setup_reporter.rs index 1174c6ac2..2ac51e43f 100644 --- a/node/src/daemon/setup_reporter.rs +++ b/node/src/daemon/setup_reporter.rs @@ -5,8 +5,8 @@ use crate::bootstrapper::BootstrapperConfig; use crate::daemon::dns_inspector::dns_inspector_factory::{ DnsInspectorFactory, DnsInspectorFactoryReal, }; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal, InitializationError}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao_null::ConfigDaoNull; use crate::db_config::persistent_configuration::{ PersistentConfiguration, PersistentConfigurationReal, @@ -417,8 +417,7 @@ impl SetupReporterReal { let initializer = DbInitializerReal::default(); match initializer.initialize( data_directory, - false, - MigratorConfig::migration_suppressed_with_error(), + DbInitializationConfig::migration_suppressed_with_error(), ) { Ok(conn) => { let parse_args_configuration = UnprivilegedParseArgsConfigurationDaoReal {}; @@ -1156,7 +1155,7 @@ mod tests { ); let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut config = PersistentConfigurationReal::from(conn); config.change_password(None, "password").unwrap(); @@ -1438,7 +1437,7 @@ mod tests { ("MASQ_MAPPING_PROTOCOL", "pmp"), ("MASQ_NEIGHBORHOOD_MODE", "originate-only"), ("MASQ_NEIGHBORS", "masq://eth-ropsten:MTIzNDU2Nzg5MTEyMzQ1Njc4OTIxMjM0NTY3ODkzMTI@1.2.3.4:1234,masq://eth-ropsten:MTIzNDU2Nzg5MTEyMzQ1Njc4OTIxMjM0NTY3ODkzMTI@5.6.7.8:5678"), - ("MASQ_PAYMENT_THRESHOLDS","1234|50000|1000|1234|19000|20000"), + ("MASQ_PAYMENT_THRESHOLDS","12345|50000|1000|1234|19000|20000"), ("MASQ_RATE_PACK","1|3|3|8"), #[cfg(not(target_os = "windows"))] ("MASQ_REAL_USER", "9999:9999:booga"), @@ -1469,7 +1468,7 @@ mod tests { ("mapping-protocol", "pmp", Configured), ("neighborhood-mode", "originate-only", Configured), ("neighbors", "masq://eth-ropsten:MTIzNDU2Nzg5MTEyMzQ1Njc4OTIxMjM0NTY3ODkzMTI@1.2.3.4:1234,masq://eth-ropsten:MTIzNDU2Nzg5MTEyMzQ1Njc4OTIxMjM0NTY3ODkzMTI@5.6.7.8:5678", Configured), - ("payment-thresholds","1234|50000|1000|1234|19000|20000",Configured), + ("payment-thresholds","12345|50000|1000|1234|19000|20000",Configured), ("rate-pack","1|3|3|8",Configured), #[cfg(not(target_os = "windows"))] ("real-user", "9999:9999:booga", Configured), @@ -1528,7 +1527,7 @@ mod tests { config_file.write_all(b"scans = \"off\"\n").unwrap(); config_file.write_all(b"rate-pack = \"2|2|2|2\"\n").unwrap(); config_file - .write_all(b"payment-thresholds = \"33|55|33|646|999|999\"\n") + .write_all(b"payment-thresholds = \"3333|55|33|646|999|999\"\n") .unwrap(); config_file .write_all(b"scan-intervals = \"111|100|99\"\n") @@ -1572,7 +1571,7 @@ mod tests { .write_all(b"rate-pack = \"55|50|60|61\"\n") .unwrap(); config_file - .write_all(b"payment-thresholds = \"1000|1000|3000|3333|10000|20000\"\n") + .write_all(b"payment-thresholds = \"4000|1000|3000|3333|10000|20000\"\n") .unwrap(); config_file .write_all(b"scan-intervals = \"555|555|555\"\n") @@ -1630,7 +1629,7 @@ mod tests { ("neighbors", "", Blank), ( "payment-thresholds", - "1000|1000|3000|3333|10000|20000", + "4000|1000|3000|3333|10000|20000", Configured, ), ("rate-pack", "55|50|60|61", Configured), @@ -2060,11 +2059,11 @@ mod tests { } #[test] - fn run_configuration_without_existing_database_implies_config_dao_null_to_use() { + fn run_configuration_without_existing_database_implies_config_dao_null_to_be_used() { let _guard = EnvironmentGuard::new(); let home_dir = ensure_node_home_directory_exists( "setup_reporter", - "run_configuration_without_existing_database_implies_config_dao_null_to_use", + "run_configuration_without_existing_database_implies_config_dao_null_to_be_used", ); let current_default_gas_price = DEFAULT_GAS_PRICE; let gas_price_for_set_attempt = current_default_gas_price + 78; @@ -2077,7 +2076,7 @@ mod tests { subject.run_configuration(&multi_config, &home_dir); let error = DbInitializerReal::default() - .initialize(&home_dir, false, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::panic_on_migration()) .unwrap_err(); assert_eq!(error, InitializationError::Nonexistent); assert_eq!( @@ -3115,7 +3114,7 @@ mod tests { fn payment_thresholds_computed_default_persistent_config_unequal_to_default() { let mut payment_thresholds = *DEFAULT_PAYMENT_THRESHOLDS; payment_thresholds.maturity_threshold_sec += 12; - payment_thresholds.unban_below_gwei -= 11; + payment_thresholds.unban_below_gwei -= 12; payment_thresholds.debt_threshold_gwei += 1111; assert_computed_default_when_persistent_config_unequal_to_default( diff --git a/node/src/database/config_dumper.rs b/node/src/database/config_dumper.rs index f26efe306..369de9003 100644 --- a/node/src/database/config_dumper.rs +++ b/node/src/database/config_dumper.rs @@ -3,10 +3,10 @@ use crate::apps::app_config_dumper; use crate::blockchain::bip39::Bip39; use crate::bootstrapper::RealUser; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{ DbInitializer, DbInitializerReal, InitializationError, DATABASE_FILE, }; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::{ConfigDao, ConfigDaoReal, ConfigDaoRecord}; use crate::db_config::typed_config_layer::{decode_bytes, encode_bytes}; use crate::node_configurator::DirsWrapperReal; @@ -41,7 +41,10 @@ impl DumpConfigRunner for DumpConfigRunnerReal { distill_args(&DirsWrapperReal {}, args)?; let cryptde = CryptDEReal::new(chain); PrivilegeDropperReal::new().drop_privileges(&real_user); - let config_dao = make_config_dao(&data_directory, MigratorConfig::migration_suppressed()); //dump config is not supposed to migrate db + let config_dao = make_config_dao( + &data_directory, + DbInitializationConfig::migration_suppressed(), + ); //dump config is not supposed to migrate db let configuration = config_dao.get_all().expect("Couldn't fetch configuration"); let json = configuration_to_json(configuration, password_opt, &cryptde); write_string(streams, json); @@ -118,10 +121,12 @@ fn translate_bytes(json_name: &str, input: PlainData, cryptde: &dyn CryptDE) -> } } -fn make_config_dao(data_directory: &Path, migrator_config: MigratorConfig) -> ConfigDaoReal { +fn make_config_dao(data_directory: &Path, init_config: DbInitializationConfig) -> ConfigDaoReal { let conn = DbInitializerReal::default() - .initialize(data_directory, false, migrator_config) - .unwrap_or_else(|e| if e == InitializationError::Nonexistent {panic!("Could not find database at: {}. Would be created when the Node firstly operates. Running --dump-config before has no effect",data_directory.to_string_lossy())} else { + .initialize(data_directory,init_config) + .unwrap_or_else(|e| if e == InitializationError::Nonexistent {panic!("\ + Could not find database at: {}. It is created when the Node operates the first time. Running \ + --dump-config before that has no effect",data_directory.to_string_lossy())} else { panic!( "Can't initialize database at {:?}: {:?}", data_directory.join(DATABASE_FILE), @@ -154,8 +159,7 @@ mod tests { use super::*; use crate::blockchain::bip39::Bip39; use crate::database::connection_wrapper::ConnectionWrapperReal; - use crate::database::db_initializer::CURRENT_SCHEMA_VERSION; - use crate::database::db_migrations::ExternalData; + use crate::database::db_initializer::{ExternalData, CURRENT_SCHEMA_VERSION}; use crate::db_config::config_dao::ConfigDao; use crate::db_config::persistent_configuration::{ PersistentConfiguration, PersistentConfigurationReal, @@ -196,7 +200,14 @@ mod tests { .unwrap_err(); let string_panic = caught_panic.downcast_ref::().unwrap(); - assert_eq!(string_panic,&format!("Could not find database at: {}. Would be created when the Node firstly operates. Running --dump-config before has no effect",data_dir.to_str().unwrap())); + assert_eq!( + string_panic, + &format!( + "Could not find database at: {}. It is created when the Node \ + operates the first time. Running --dump-config before that has no effect", + data_dir.to_str().unwrap() + ) + ); let err = File::open(&data_dir.join(DATABASE_FILE)).unwrap_err(); assert_eq!(err.kind(), ErrorKind::NotFound) } @@ -224,7 +235,8 @@ mod tests { assert!(result.is_ok()); let schema_version_after = dao.get("schema_version").unwrap().value_opt.unwrap(); - assert_eq!(schema_version_before, schema_version_after) + assert_eq!(schema_version_before, schema_version_after); + assert_eq!(holder.stderr.get_bytes().is_empty(), true); } #[test] @@ -241,8 +253,7 @@ mod tests { let conn = DbInitializerReal::default() .initialize( &data_dir, - true, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( TEST_DEFAULT_CHAIN, NeighborhoodModeLight::ZeroHop, None, @@ -297,7 +308,7 @@ mod tests { x => panic!("Expected JSON object; found {:?}", x), }; let conn = DbInitializerReal::default() - .initialize(&data_dir, false, MigratorConfig::panic_on_migration()) + .initialize(&data_dir, DbInitializationConfig::panic_on_migration()) .unwrap(); let dao = ConfigDaoReal::new(conn); assert_value("blockchainServiceUrl", "https://infura.io/ID", &map); @@ -360,8 +371,7 @@ mod tests { let conn = DbInitializerReal::default() .initialize( &data_dir, - true, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( TEST_DEFAULT_CHAIN, NeighborhoodModeLight::ConsumeOnly, None, @@ -417,7 +427,7 @@ mod tests { x => panic!("Expected JSON object; found {:?}", x), }; let conn = DbInitializerReal::default() - .initialize(&data_dir, false, MigratorConfig::panic_on_migration()) + .initialize(&data_dir, DbInitializationConfig::panic_on_migration()) .unwrap(); let dao = Box::new(ConfigDaoReal::new(conn)); assert_value("blockchainServiceUrl", "https://infura.io/ID", &map); @@ -469,8 +479,7 @@ mod tests { let conn = DbInitializerReal::default() .initialize( &data_dir, - true, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( TEST_DEFAULT_CHAIN, NeighborhoodModeLight::Standard, None, @@ -526,7 +535,7 @@ mod tests { x => panic!("Expected JSON object; found {:?}", x), }; let conn = DbInitializerReal::default() - .initialize(&data_dir, false, MigratorConfig::panic_on_migration()) + .initialize(&data_dir, DbInitializationConfig::panic_on_migration()) .unwrap(); let dao = Box::new(ConfigDaoReal::new(conn)); assert_value("blockchainServiceUrl", "https://infura.io/ID", &map); diff --git a/node/src/database/connection_wrapper.rs b/node/src/database/connection_wrapper.rs index ca612c64b..2e9dd9da9 100644 --- a/node/src/database/connection_wrapper.rs +++ b/node/src/database/connection_wrapper.rs @@ -1,11 +1,18 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. +#[cfg(test)] +use crate::arbitrary_id_stamp_in_trait; +#[cfg(test)] +use crate::test_utils::unshared_test_utils::ArbitraryIdStamp; use rusqlite::{Connection, Error, Statement, Transaction}; use std::fmt::Debug; pub trait ConnectionWrapper: Debug + Send { fn prepare(&self, query: &str) -> Result; fn transaction<'a: 'b, 'b>(&'a mut self) -> Result, rusqlite::Error>; + + #[cfg(test)] + arbitrary_id_stamp_in_trait!(); } #[derive(Debug)] diff --git a/node/src/database/dao_utils.rs b/node/src/database/dao_utils.rs deleted file mode 100644 index 1251ced4b..000000000 --- a/node/src/database/dao_utils.rs +++ /dev/null @@ -1,73 +0,0 @@ -// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. -use crate::accountant::unsigned_to_signed; -use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::db_initializer::{connection_or_panic, DbInitializerReal}; -use crate::database::db_migrations::MigratorConfig; -use masq_lib::utils::ExpectValue; -use std::cell::RefCell; -use std::path::{Path, PathBuf}; -use std::time::Duration; -use std::time::SystemTime; - -pub fn to_time_t(system_time: SystemTime) -> i64 { - match system_time.duration_since(SystemTime::UNIX_EPOCH) { - Err(e) => unimplemented!("{}", e), - Ok(d) => unsigned_to_signed(d.as_secs()).expect("MASQNode has expired"), - } -} - -pub fn now_time_t() -> i64 { - to_time_t(SystemTime::now()) -} - -pub fn from_time_t(time_t: i64) -> SystemTime { - let interval = Duration::from_secs(time_t as u64); - SystemTime::UNIX_EPOCH + interval -} - -pub struct DaoFactoryReal { - pub data_directory: PathBuf, - pub create_if_necessary: bool, - pub migrator_config: RefCell>, -} - -impl DaoFactoryReal { - pub fn new( - data_directory: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, - ) -> Self { - Self { - data_directory: data_directory.to_path_buf(), - create_if_necessary, - migrator_config: RefCell::new(Some(migrator_config)), - } - } - - pub fn make_connection(&self) -> Box { - connection_or_panic( - &DbInitializerReal::default(), - &self.data_directory, - self.create_if_necessary, - self.migrator_config.take().expectv("MigratorConfig"), - ) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use std::str::FromStr; - - #[test] - #[should_panic(expected = "Failed to connect to database at \"nonexistent")] - fn connection_panics_if_connection_cannot_be_made() { - let subject = DaoFactoryReal::new( - &PathBuf::from_str("nonexistent").unwrap(), - false, - MigratorConfig::test_default(), - ); - - let _ = subject.make_connection(); - } -} diff --git a/node/src/database/db_initializer.rs b/node/src/database/db_initializer.rs index 0b3ea5747..2ca099259 100644 --- a/node/src/database/db_initializer.rs +++ b/node/src/database/db_initializer.rs @@ -1,28 +1,28 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use crate::database::connection_wrapper::{ConnectionWrapper, ConnectionWrapperReal}; -use crate::database::db_migrations::{ - DbMigrator, DbMigratorReal, ExternalData, MigratorConfig, Suppression, -}; +use crate::database::db_migrations::{DbMigrator, DbMigratorReal}; use crate::db_config::secure_config_layer::EXAMPLE_ENCRYPTED; use crate::sub_lib::accountant::{DEFAULT_PAYMENT_THRESHOLDS, DEFAULT_SCAN_INTERVALS}; use crate::sub_lib::neighborhood::DEFAULT_RATE_PACK; +use masq_lib::blockchains::chains::Chain; use masq_lib::constants::{ DEFAULT_GAS_PRICE, HIGHEST_RANDOM_CLANDESTINE_PORT, LOWEST_USABLE_INSECURE_PORT, }; use masq_lib::logger::Logger; +#[cfg(test)] +use masq_lib::test_utils::utils::TEST_DEFAULT_CHAIN; +use masq_lib::utils::NeighborhoodModeLight; use rand::prelude::*; -use rusqlite::Error::InvalidColumnType; use rusqlite::{Connection, OpenFlags}; -use std::collections::HashMap; -use std::fmt::Debug; -use std::fs; +use std::fmt::{Debug, Formatter}; use std::io::ErrorKind; use std::net::{Ipv4Addr, SocketAddr, SocketAddrV4}; use std::path::Path; +use std::{fs, vec}; use tokio::net::TcpListener; pub const DATABASE_FILE: &str = "node-data.db"; -pub const CURRENT_SCHEMA_VERSION: usize = 6; +pub const CURRENT_SCHEMA_VERSION: usize = 7; #[derive(Debug, PartialEq)] pub enum InitializationError { @@ -37,16 +37,14 @@ pub trait DbInitializer { fn initialize( &self, path: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError>; fn initialize_to_version( &self, path: &Path, target_version: usize, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError>; } @@ -57,74 +55,51 @@ impl DbInitializer for DbInitializerReal { fn initialize( &self, path: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError> { - self.initialize_to_version( - path, - CURRENT_SCHEMA_VERSION, - create_if_necessary, - migrator_config, - ) + self.initialize_to_version(path, CURRENT_SCHEMA_VERSION, init_config) } fn initialize_to_version( &self, path: &Path, target_version: usize, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError> { let is_creation_necessary = Self::is_creation_necessary(path); - if !create_if_necessary && is_creation_necessary { + if !matches!( + init_config.mode, + InitializationMode::CreationAndMigration { .. } + ) && is_creation_necessary + { return Err(InitializationError::Nonexistent); } Self::create_data_directory_if_necessary(path); - let flags = OpenFlags::SQLITE_OPEN_READ_WRITE; - let database_file_path = &path.join(DATABASE_FILE); - match Connection::open_with_flags(database_file_path, flags) { + let db_file_path = &path.join(DATABASE_FILE); + match Connection::open_with_flags(db_file_path, OpenFlags::SQLITE_OPEN_READ_WRITE) { Ok(conn) => { - eprintln!("Opened existing database at {:?}", database_file_path); - let config = self.extract_configurations(&conn); - match ( - Self::is_migration_required(config.get("schema_version"))?, - &migrator_config.should_be_suppressed, - ) { - (None, _) => Ok(Box::new(ConnectionWrapperReal::new(conn))), - (Some(mismatched_version), &Suppression::No) => { - let external_params = ExternalData::from((migrator_config, false)); - let migrator = Box::new(DbMigratorReal::new(external_params)); - self.migrate_and_return_connection( - conn, - mismatched_version, - target_version, - database_file_path, - flags, - migrator, - ) - } - (Some(_), &Suppression::Yes) => Ok(Box::new(ConnectionWrapperReal::new(conn))), - (Some(_), &Suppression::WithErr) => { - Err(InitializationError::SuppressedMigration) - } - } + eprintln!("Opened existing database at {:?}", db_file_path); + Self::extra_configuration(&conn, &init_config)?; + self.check_migrations_and_return_connection( + conn, + init_config, + db_file_path, + target_version, + OpenFlags::SQLITE_OPEN_READ_WRITE, + ) } - Err(_) => { - let mut flags = OpenFlags::empty(); - flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE); - flags.insert(OpenFlags::SQLITE_OPEN_CREATE); - match Connection::open_with_flags(database_file_path, flags) { - Ok(conn) => { - eprintln!("Created new database at {:?}", database_file_path); - self.create_database_tables( - &conn, - ExternalData::from((migrator_config, true)), - ); - Ok(Box::new(ConnectionWrapperReal::new(conn))) - } - Err(e) => Err(InitializationError::SqliteError(e)), + Err(_) => match Connection::open_with_flags( + db_file_path, + OpenFlags::SQLITE_OPEN_CREATE | OpenFlags::SQLITE_OPEN_READ_WRITE, + ) { + Ok(conn) => { + eprintln!("Created new database at {:?}", db_file_path); + Self::extra_configuration(&conn, &init_config)?; + self.create_database_tables(&conn, ExternalData::from(init_config)); + Ok(Box::new(ConnectionWrapperReal::new(conn))) } - } + Err(e) => Err(InitializationError::SqliteError(e)), + }, } } } @@ -284,7 +259,8 @@ impl DbInitializerReal { "create table if not exists pending_payable ( rowid integer primary key, transaction_hash text not null, - amount integer not null, + amount_high_b integer not null, + amount_low_b integer not null, payable_timestamp integer not null, attempt integer not null, process_error text null @@ -303,10 +279,11 @@ impl DbInitializerReal { conn.execute( "create table if not exists payable ( wallet_address text primary key, - balance integer not null, + balance_high_b integer not null, + balance_low_b integer not null, last_paid_timestamp integer not null, pending_payable_rowid integer null - )", + ) strict", [], ) .expect("Can't create payable table"); @@ -316,9 +293,10 @@ impl DbInitializerReal { conn.execute( "create table if not exists receivable ( wallet_address text primary key, - balance integer not null, + balance_high_b integer not null, + balance_low_b integer not null, last_received_timestamp integer not null - )", + ) strict", [], ) .expect("Can't create receivable table"); @@ -332,51 +310,79 @@ impl DbInitializerReal { .expect("Can't create banned table"); } - fn extract_configurations(&self, conn: &Connection) -> HashMap, bool)> { - let mut stmt = conn - .prepare("select name, value, encrypted from config") - .unwrap(); - let query_result = stmt.query_map([], |row| { - Ok(( - row.get(0), - row.get(1), - row.get(2).map(|encrypted: i64| encrypted > 0), - )) - }); - match query_result { - Ok(rows) => rows, - Err(e) => panic!("Error retrieving configuration: {}", e), - } - .map(|row| match row { - Ok((Ok(name), Ok(value), Ok(encrypted))) => (name, (Some(value), encrypted)), - Ok((Ok(name), Err(InvalidColumnType(1, _, _)), Ok(encrypted))) => { - (name, (None, encrypted)) + fn extra_configuration( + conn: &Connection, + init_config: &DbInitializationConfig, + ) -> Result<(), InitializationError> { + if init_config.special_conn_configuration.is_empty() { + Ok(()) + } else { + match init_config + .special_conn_configuration + .iter() + .try_for_each(|setup_fn| setup_fn(conn)) + { + Ok(()) => Ok(()), + Err(e) => Err(InitializationError::SqliteError(e)), } - e => panic!("Error retrieving configuration: {:?}", e), - }) - .collect::, bool)>>() + } } - fn is_migration_required( - version_found: Option<&(Option, bool)>, - ) -> Result, InitializationError> { - match version_found { - None => Err(InitializationError::UndetectableVersion(format!( - "Need {}, found nothing", - CURRENT_SCHEMA_VERSION - ))), - Some((None, _)) => Err(InitializationError::UndetectableVersion(format!( - "Need {}, found nothing", - CURRENT_SCHEMA_VERSION - ))), - Some((Some(v_from_db), _)) => { - let v_from_db = Self::validate_schema_version(v_from_db); - if v_from_db == CURRENT_SCHEMA_VERSION { - Ok(None) - } else { - Ok(Some(v_from_db)) - } + fn check_migrations_and_return_connection( + &self, + conn: Connection, + init_config: DbInitializationConfig, + db_file_path: &Path, + target_version: usize, + flags: OpenFlags, + ) -> Result, InitializationError> { + let str_sv = Self::read_current_schema_version(&conn)?; + match (Self::is_migration_required(&str_sv)?, init_config.mode) { + (None, _) => Ok(Box::new(ConnectionWrapperReal::new(conn))), + (Some(_), InitializationMode::CreationBannedMigrationPanics) => { + panic!("Broken code: Migrating database at inappropriate place") + } + ( + Some(mismatched_version), + InitializationMode::CreationAndMigration { external_data }, + ) => { + let migrator = Box::new(DbMigratorReal::new(external_data)); + self.migrate_and_return_connection( + conn, + mismatched_version, + target_version, + db_file_path, + flags, + migrator, + ) } + (Some(_), InitializationMode::CreationBannedMigrationSuppressed) => { + Ok(Box::new(ConnectionWrapperReal::new(conn))) + } + (Some(_), InitializationMode::CreationBannedMigrationRaisesErr) => { + Err(InitializationError::SuppressedMigration) + } + } + } + + fn read_current_schema_version(conn: &Connection) -> Result { + conn.prepare("select value from config where name = 'schema_version'") + .expect("select failed") + .query_row([], |row| row.get::(0)) + .map_err(|e| { + InitializationError::UndetectableVersion(format!( + "Need {}, found nothing (err: {})", + CURRENT_SCHEMA_VERSION, e + )) + }) + } + + fn is_migration_required(version_read_str: &str) -> Result, InitializationError> { + let v_from_db = Self::validate_schema_version(version_read_str); + if v_from_db == CURRENT_SCHEMA_VERSION { + Ok(None) + } else { + Ok(Some(v_from_db)) } } @@ -386,7 +392,7 @@ impl DbInitializerReal { mismatched_version: usize, target_version: usize, db_file_path: &Path, - opening_flags: OpenFlags, + flags: OpenFlags, migrator: Box, ) -> Result, InitializationError> { warning!( @@ -401,7 +407,7 @@ impl DbInitializerReal { ) { Ok(_) => { let wrapped_conn = - self.double_check_migration_result(db_file_path, opening_flags, target_version); + self.double_check_migration_result(db_file_path, flags, target_version); Ok(wrapped_conn) } Err(e) => Err(InitializationError::MigrationError(e)), @@ -416,19 +422,13 @@ impl DbInitializerReal { ) -> Box { let conn = Connection::open_with_flags(db_file_path, opening_flags) .unwrap_or_else(|e| panic!("The database undoubtedly exists, but: {}", e)); - let config_table_content = self.extract_configurations(&conn); - let schema_version_entry = config_table_content.get("schema_version"); - let found_schema = Self::validate_schema_version( - schema_version_entry - .expect("Db migration failed; cannot find a row with the schema version") - .0 - .as_ref() - .expect("Db migration failed; the value for the schema version is missing"), - ); - if found_schema.eq(&target_version) { + let str_schema = Self::read_current_schema_version(&conn) + .expect("Db migration failed; cannot find the row with the schema version"); + let numeric_schema = Self::validate_schema_version(&str_schema); + if numeric_schema == target_version { Box::new(ConnectionWrapperReal::new(conn)) } else { - panic!("DB migration failed, the resulting records are still incorrect; found schema {} but expecting {}", found_schema,target_version) + panic!("DB migration failed, the resulting records are still incorrect; found schema {} but expecting {}", numeric_schema, target_version) } } @@ -483,11 +483,10 @@ impl DbInitializerReal { pub fn connection_or_panic( db_initializer: &dyn DbInitializer, path: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Box { db_initializer - .initialize(path, create_if_necessary, migrator_config) + .initialize(path, init_config) .unwrap_or_else(|_| { panic!( "Failed to connect to database at {:?}", @@ -496,11 +495,137 @@ pub fn connection_or_panic( }) } +#[derive(Clone)] +pub struct DbInitializationConfig { + pub mode: InitializationMode, + pub special_conn_configuration: Vec rusqlite::Result<()>>, +} + +#[derive(PartialEq, Eq, Debug, Clone)] +pub enum InitializationMode { + CreationAndMigration { external_data: ExternalData }, + CreationBannedMigrationPanics, + CreationBannedMigrationSuppressed, + CreationBannedMigrationRaisesErr, +} + +impl DbInitializationConfig { + pub fn add_special_conn_setup( + mut self, + setter: fn(&Connection) -> rusqlite::Result<()>, + ) -> Self { + self.special_conn_configuration.push(setter); + self + } + + pub fn panic_on_migration() -> Self { + Self { + mode: InitializationMode::CreationBannedMigrationPanics, + special_conn_configuration: vec![], + } + } + + //standard way of Node to create a new db, possibly only one real occurrence ever + pub fn create_or_migrate(external_data: ExternalData) -> Self { + Self { + mode: InitializationMode::CreationAndMigration { external_data }, + special_conn_configuration: vec![], + } + } + + //used in the config dumper + pub fn migration_suppressed() -> Self { + Self { + mode: InitializationMode::CreationBannedMigrationSuppressed, + special_conn_configuration: vec![], + } + } + + //it makes Daemon ignore db configuration until the Node + //starts up and manage the migration on its own + pub fn migration_suppressed_with_error() -> Self { + Self { + mode: InitializationMode::CreationBannedMigrationRaisesErr, + special_conn_configuration: vec![], + } + } + + #[cfg(test)] + pub fn test_default() -> Self { + Self { + mode: InitializationMode::CreationAndMigration { + external_data: ExternalData { + chain: TEST_DEFAULT_CHAIN, + neighborhood_mode: NeighborhoodModeLight::Standard, + db_password_opt: None, + }, + }, + special_conn_configuration: vec![], + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct ExternalData { + pub chain: Chain, + pub neighborhood_mode: NeighborhoodModeLight, + pub db_password_opt: Option, +} + +impl ExternalData { + pub fn new( + chain: Chain, + neighborhood_mode: NeighborhoodModeLight, + db_password_opt: Option, + ) -> Self { + Self { + chain, + neighborhood_mode, + db_password_opt, + } + } +} + +impl From for ExternalData { + fn from(init_config: DbInitializationConfig) -> Self { + match init_config.mode { + InitializationMode::CreationAndMigration { external_data } => external_data, + _ => panic!("Attempt to create new database without proper configuration"), + } + } +} + +impl PartialEq for DbInitializationConfig { + fn eq(&self, other: &Self) -> bool { + (self.mode == other.mode) + //I'm making most of this, fn pointers cannot be compared in Rust by August 2022 + && (self.special_conn_configuration.len() == other.special_conn_configuration.len()) + } +} + +impl Debug for DbInitializationConfig { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!( + f, + "DbInitializationConfig{{init_config: {:?}, special_conn_setup: Addresses{:?}}}", + self.mode, + self.special_conn_configuration + .iter() + //reportedly, there is no guarantee the number varies by different functions, + //so it rather shows how many items are in than anything else + .map(|pointer| *pointer as usize) + .collect::>(), + ) + } +} + #[cfg(test)] pub mod test_utils { use crate::database::connection_wrapper::ConnectionWrapper; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, InitializationError}; - use crate::database::db_migrations::MigratorConfig; + use crate::test_utils::unshared_test_utils::ArbitraryIdStamp; + use crate::{arbitrary_id_stamp, set_arbitrary_id_stamp}; use rusqlite::Transaction; use rusqlite::{Error, Statement}; use std::cell::RefCell; @@ -512,6 +637,7 @@ pub mod test_utils { prepare_params: Arc>>, prepare_results: RefCell, Error>>>, transaction_results: RefCell, Error>>>, + arbitrary_id_stamp_opt: RefCell>, } unsafe impl<'a: 'b, 'b> Send for ConnectionWrapperMock<'a, 'b> {} @@ -530,6 +656,8 @@ pub mod test_utils { self.transaction_results.borrow_mut().push(result); self } + + set_arbitrary_id_stamp!(); } impl<'a: 'b, 'b> ConnectionWrapper for ConnectionWrapperMock<'a, 'b> { @@ -544,11 +672,13 @@ pub mod test_utils { fn transaction<'_a: '_b, '_b>(&'_a mut self) -> Result, Error> { self.transaction_results.borrow_mut().remove(0) } + + arbitrary_id_stamp!(); } #[derive(Default)] pub struct DbInitializerMock { - pub initialize_params: Arc>>, + pub initialize_params: Arc>>, pub initialize_results: RefCell, InitializationError>>>, } @@ -557,14 +687,12 @@ pub mod test_utils { fn initialize( &self, path: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError> { - self.initialize_params.lock().unwrap().push(( - path.to_path_buf(), - create_if_necessary, - migrator_config, - )); + self.initialize_params + .lock() + .unwrap() + .push((path.to_path_buf(), init_config)); self.initialize_results.borrow_mut().remove(0) } @@ -573,8 +701,7 @@ pub mod test_utils { &self, path: &Path, target_version: usize, - create_if_necessary: bool, - migrator_config: MigratorConfig, + init_config: DbInitializationConfig, ) -> Result, InitializationError> { intentionally_blank!() /*all existing test calls only initialize() in the mocked version, @@ -590,7 +717,7 @@ pub mod test_utils { pub fn initialize_parameters( mut self, - parameters: Arc>>, + parameters: Arc>>, ) -> DbInitializerMock { self.initialize_params = parameters; self @@ -609,13 +736,16 @@ pub mod test_utils { #[cfg(test)] mod tests { use super::*; + use crate::database::db_initializer::InitializationError::SqliteError; use crate::db_config::config_dao::{ConfigDao, ConfigDaoReal}; use crate::test_utils::database_utils::{ - assert_create_table_statement_contains_all_important_parts, - assert_index_statement_is_coupled_with_right_parameter, assert_no_index_exists_for_table, - bring_db_0_back_to_life_and_return_connection, retrieve_config_row, DbMigratorMock, + assert_create_table_stm_contains_all_parts, + assert_index_stm_is_coupled_with_right_parameter, assert_no_index_exists_for_table, + assert_table_created_as_strict, bring_db_0_back_to_life_and_return_connection, + make_external_data, retrieve_config_row, DbMigratorMock, }; - use itertools::Itertools; + use itertools::Either::{Left, Right}; + use itertools::{Either, Itertools}; use masq_lib::blockchains::chains::Chain; use masq_lib::test_utils::logging::{init_test_logging, TestLogHandler}; use masq_lib::test_utils::utils::{ @@ -623,7 +753,10 @@ mod tests { TEST_DEFAULT_CHAIN, }; use masq_lib::utils::NeighborhoodModeLight; + use regex::Regex; + use rusqlite::Error::InvalidColumnType; use rusqlite::{Error, OpenFlags}; + use std::collections::HashMap; use std::fs::File; use std::io::{Read, Write}; use std::net::{Ipv4Addr, SocketAddr, SocketAddrV4}; @@ -635,45 +768,7 @@ mod tests { #[test] fn constants_have_correct_values() { assert_eq!(DATABASE_FILE, "node-data.db"); - assert_eq!(CURRENT_SCHEMA_VERSION, 6); - } - - #[test] - fn db_initialize_does_not_create_if_directed_not_to_and_directory_does_not_exist() { - let home_dir = ensure_node_home_directory_does_not_exist( - "db_initializer", - "db_initialize_does_not_create_if_directed_not_to_and_directory_does_not_exist", - ); - let subject = DbInitializerReal::default(); - - let result = subject.initialize(&home_dir, false, MigratorConfig::test_default()); - - assert_eq!(result.err().unwrap(), InitializationError::Nonexistent); - let result = Connection::open(&home_dir.join(DATABASE_FILE)); - match result.err().unwrap() { - Error::SqliteFailure(_, _) => (), - x => panic!("Expected SqliteFailure, got {:?}", x), - } - } - - #[test] - fn db_initialize_does_not_create_if_directed_not_to_and_database_file_does_not_exist() { - let home_dir = ensure_node_home_directory_exists( - "db_initializer", - "db_initialize_does_not_create_if_directed_not_to_and_database_file_does_not_exist", - ); - let subject = DbInitializerReal::default(); - - let result = subject.initialize(&home_dir, false, MigratorConfig::test_default()); - - assert_eq!(result.err().unwrap(), InitializationError::Nonexistent); - let mut flags = OpenFlags::empty(); - flags.insert(OpenFlags::SQLITE_OPEN_READ_ONLY); - let result = Connection::open_with_flags(&home_dir.join(DATABASE_FILE), flags); - match result.err().unwrap() { - Error::SqliteFailure(_, _) => (), - x => panic!("Expected SqliteFailure, got {:?}", x), - } + assert_eq!(CURRENT_SCHEMA_VERSION, 7); } #[test] @@ -685,23 +780,19 @@ mod tests { let subject = DbInitializerReal::default(); let conn = subject - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut stmt = conn .prepare("select name, value, encrypted from config") .unwrap(); let _ = stmt.query_map([], |_| Ok(42)).unwrap(); - let expected_key_words = [ - ["name", "text", "primary", "key"].as_slice(), - ["value", "text"].as_slice(), - ["encrypted", "integer", "not", "null"].as_slice(), + let expected_key_words: &[&[&str]] = &[ + &["name", "text", "primary", "key"], + &["value", "text"], + &["encrypted", "integer", "not", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn.as_ref(), - "config", - expected_key_words.as_slice(), - ); + assert_create_table_stm_contains_all_parts(conn.as_ref(), "config", expected_key_words); assert_no_index_exists_for_table(conn.as_ref(), "config") } @@ -714,30 +805,27 @@ mod tests { let subject = DbInitializerReal::default(); let conn = subject - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); - let mut stmt = conn.prepare("select rowid, transaction_hash, amount, payable_timestamp, attempt, process_error from pending_payable").unwrap(); + let mut stmt = conn.prepare("select rowid, transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt, process_error from pending_payable").unwrap(); let mut payable_contents = stmt.query_map([], |_| Ok(42)).unwrap(); assert!(payable_contents.next().is_none()); - let expected_key_words = [ - ["rowid", "integer", "primary", "key"].as_slice(), - ["transaction_hash", "text", "not", "null"].as_slice(), - ["amount", "integer", "not", "null"].as_slice(), - ["payable_timestamp", "integer", "not", "null"].as_slice(), - ["attempt", "integer", "not", "null"].as_slice(), - ["process_error", "text", "null"].as_slice(), + let expected_key_words: &[&[&str]] = &[ + &["rowid", "integer", "primary", "key"], + &["transaction_hash", "text", "not", "null"], + &["amount_high_b", "integer", "not", "null"], + &["amount_low_b", "integer", "not", "null"], + &["payable_timestamp", "integer", "not", "null"], + &["attempt", "integer", "not", "null"], + &["process_error", "text", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn.as_ref(), - "pending_payable", - expected_key_words.as_slice(), - ); - let expected_key_words = [["transaction_hash"].as_slice()]; - assert_index_statement_is_coupled_with_right_parameter( + assert_create_table_stm_contains_all_parts(&*conn, "pending_payable", expected_key_words); + let expected_key_words: &[&[&str]] = &[&["transaction_hash"]]; + assert_index_stm_is_coupled_with_right_parameter( conn.as_ref(), "pending_payable_hash_idx", - expected_key_words.as_slice(), + expected_key_words, ) } @@ -750,23 +838,21 @@ mod tests { let subject = DbInitializerReal::default(); let conn = subject - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); - let mut stmt = conn.prepare ("select wallet_address, balance, last_paid_timestamp, pending_payable_rowid from payable").unwrap (); + let mut stmt = conn.prepare ("select wallet_address, balance_high_b, balance_low_b, last_paid_timestamp, pending_payable_rowid from payable").unwrap (); let mut payable_contents = stmt.query_map([], |_| Ok(42)).unwrap(); assert!(payable_contents.next().is_none()); - let expected_key_words = [ - ["wallet_address", "text", "primary", "key"].as_slice(), - ["balance", "integer", "not", "null"].as_slice(), - ["last_paid_timestamp", "integer", "not", "null"].as_slice(), - ["pending_payable_rowid", "integer", "null"].as_slice(), + assert_table_created_as_strict(&*conn, "payable"); + let expected_key_words: &[&[&str]] = &[ + &["wallet_address", "text", "primary", "key"], + &["balance_high_b", "integer", "not", "null"], + &["balance_low_b", "integer", "not", "null"], + &["last_paid_timestamp", "integer", "not", "null"], + &["pending_payable_rowid", "integer", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn.as_ref(), - "payable", - expected_key_words.as_slice(), - ); + assert_create_table_stm_contains_all_parts(&*conn, "payable", expected_key_words); assert_no_index_exists_for_table(conn.as_ref(), "payable") } @@ -779,24 +865,22 @@ mod tests { let subject = DbInitializerReal::default(); let conn = subject - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut stmt = conn - .prepare("select wallet_address, balance, last_received_timestamp from receivable") + .prepare("select wallet_address, balance_high_b, balance_low_b, last_received_timestamp from receivable") .unwrap(); let mut receivable_contents = stmt.query_map([], |_| Ok(())).unwrap(); assert!(receivable_contents.next().is_none()); - let expected_key_words = [ - ["wallet_address", "text", "primary", "key"].as_slice(), - ["balance", "integer", "not", "null"].as_slice(), - ["last_received_timestamp", "integer", "not", "null"].as_slice(), + assert_table_created_as_strict(&*conn, "receivable"); + let expected_key_words: &[&[&str]] = &[ + &["wallet_address", "text", "primary", "key"], + &["balance_high_b", "integer", "not", "null"], + &["balance_low_b", "integer", "not", "null"], + &["last_received_timestamp", "integer", "not", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn.as_ref(), - "receivable", - expected_key_words.as_slice(), - ); + assert_create_table_stm_contains_all_parts(conn.as_ref(), "receivable", expected_key_words); assert_no_index_exists_for_table(conn.as_ref(), "receivable") } @@ -810,18 +894,14 @@ mod tests { let subject = DbInitializerReal::default(); let conn = subject - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut stmt = conn.prepare("select wallet_address from banned").unwrap(); let mut banned_contents = stmt.query_map([], |_| Ok(42)).unwrap(); assert!(banned_contents.next().is_none()); - let expected_key_words = [["wallet_address", "text", "primary", "key"].as_slice()]; - assert_create_table_statement_contains_all_important_parts( - conn.as_ref(), - "banned", - expected_key_words.as_slice(), - ); + let expected_key_words: &[&[&str]] = &[&["wallet_address", "text", "primary", "key"]]; + assert_create_table_stm_contains_all_parts(conn.as_ref(), "banned", expected_key_words); assert_no_index_exists_for_table(conn.as_ref(), "banned") } @@ -867,6 +947,31 @@ mod tests { ); } + fn extract_configurations(conn: &Connection) -> HashMap, bool)> { + let mut stmt = conn + .prepare("select name, value, encrypted from config") + .unwrap(); + let query_result = stmt.query_map([], |row| { + Ok(( + row.get(0), + row.get(1), + row.get(2).map(|encrypted: i64| encrypted > 0), + )) + }); + match query_result { + Ok(rows) => rows, + Err(e) => panic!("Error retrieving configuration: {}", e), + } + .map(|row| match row { + Ok((Ok(name), Ok(value), Ok(encrypted))) => (name, (Some(value), encrypted)), + Ok((Ok(name), Err(InvalidColumnType(1, _, _)), Ok(encrypted))) => { + (name, (None, encrypted)) + } + e => panic!("Error retrieving configuration: {:?}", e), + }) + .collect::, bool)>>() + } + #[test] fn existing_database_with_correct_version_is_accepted_without_changes() { let home_dir = ensure_node_home_directory_exists( @@ -876,7 +981,7 @@ mod tests { let subject = DbInitializerReal::default(); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); } { @@ -891,13 +996,13 @@ mod tests { } subject - .initialize(&home_dir, true, MigratorConfig::panic_on_migration()) + .initialize(&home_dir, DbInitializationConfig::panic_on_migration()) .unwrap(); let mut flags = OpenFlags::empty(); flags.insert(OpenFlags::SQLITE_OPEN_READ_ONLY); let conn = Connection::open_with_flags(&home_dir.join(DATABASE_FILE), flags).unwrap(); - let config_map = subject.extract_configurations(&conn); + let config_map = extract_configurations(&conn); let mut config_vec: Vec<(String, (Option, bool))> = config_map.into_iter().collect(); config_vec.sort_by_key(|(name, _)| name.clone()); @@ -990,7 +1095,7 @@ mod tests { ); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut flags = OpenFlags::empty(); flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE); @@ -1000,12 +1105,12 @@ mod tests { } let subject = DbInitializerReal::default(); - let result = subject.initialize(&home_dir, true, MigratorConfig::panic_on_migration()); + let result = subject.initialize(&home_dir, DbInitializationConfig::panic_on_migration()); assert_eq!( result.err().unwrap(), InitializationError::UndetectableVersion(format!( - "Need {}, found nothing", + "Need {}, found nothing (err: Query returned no rows)", CURRENT_SCHEMA_VERSION )), ); @@ -1020,7 +1125,7 @@ mod tests { ); { DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut flags = OpenFlags::empty(); flags.insert(OpenFlags::SQLITE_OPEN_READ_WRITE); @@ -1033,7 +1138,136 @@ mod tests { } let subject = DbInitializerReal::default(); - let _ = subject.initialize(&home_dir, true, MigratorConfig::panic_on_migration()); + let _ = subject.initialize(&home_dir, DbInitializationConfig::panic_on_migration()); + } + + const PRAGMA_CASE_SENSITIVE: &str = "case_sensitive_like"; + + fn assert_case_sensitivity_has_been_turned_on( + conn: Either<&Connection, &dyn ConnectionWrapper>, + ) { + let sql = "select 'a' like 'A'"; + let mut stm = match conn { + Left(conn) => conn.prepare(sql).unwrap(), + Right(wrapped_conn) => wrapped_conn.prepare(sql).unwrap(), + }; + let is_considered_the_same = stm + .query_row([], |row| Ok(row.get::(0))) + .unwrap(); + assert_eq!(is_considered_the_same, Ok(false)); + } + + #[test] + fn add_special_setup_works() { + let subject = DbInitializationConfig::test_default(); + let setup_fn = move |conn: &Connection| { + conn.pragma_update(None, PRAGMA_CASE_SENSITIVE, true) + .unwrap(); + Ok(()) + }; + let conn = Connection::open_in_memory().unwrap(); + + let result = subject.add_special_conn_setup(setup_fn); + + result.special_conn_configuration[0](&conn).unwrap(); + assert_case_sensitivity_has_been_turned_on(Left(&conn)) + } + + #[test] + fn extra_configuration_retrieves_first_error_encountered() { + let fn_one = |_: &_| Ok(()); + let fn_two = |_: &_| Err(Error::ExecuteReturnedResults); + let fn_three = |_: &_| Err(Error::GetAuxWrongType); + let conn = Connection::open_in_memory().unwrap(); + let init_config = + make_default_config_with_different_pointers(vec![fn_one, fn_two, fn_three]); + + let result = DbInitializerReal::extra_configuration(&conn, &init_config); + + assert_eq!(result, Err(SqliteError(Error::ExecuteReturnedResults))) + } + + #[test] + fn add_conn_special_setup_works_at_database_creation() { + let home_dir = ensure_node_home_directory_exists( + "db_initializer", + "add_conn_special_setup_works_at_database_creation", + ); + let example_function = |conn: &Connection| { + conn.pragma_update(None, PRAGMA_CASE_SENSITIVE, true) + .unwrap(); + Ok(()) + }; + let init_config = + DbInitializationConfig::test_default().add_special_conn_setup(example_function); + + let assert_conn = DbInitializerReal::default() + .initialize(&home_dir, init_config) + .unwrap(); + + assert_case_sensitivity_has_been_turned_on(Right(assert_conn.as_ref())) + } + + #[test] + fn conn_special_setup_works_at_opening_existing_database() { + let home_dir = ensure_node_home_directory_exists( + "db_initializer", + "conn_special_setup_works_at_opening_existing_database", + ); + DbInitializerReal::default() + .initialize(&home_dir, DbInitializationConfig::test_default()) + .unwrap(); + let init_config = + DbInitializationConfig::test_default().add_special_conn_setup(|conn: &Connection| { + conn.pragma_update(None, PRAGMA_CASE_SENSITIVE, true) + .unwrap(); + Ok(()) + }); + + let assert_conn = DbInitializerReal::default() + .initialize(&home_dir, init_config) + .unwrap(); + + assert_case_sensitivity_has_been_turned_on(Right(assert_conn.as_ref())) + } + + fn processing_special_setup_test_body(test_name: &str, pre_initialization: fn(&Path)) { + let home_dir = ensure_node_home_directory_exists("db_initializer", test_name); + { + pre_initialization(home_dir.as_path()) + } + let malformed_setup_function = |_: &_| Err(Error::GetAuxWrongType); + let init_config = + DbInitializationConfig::test_default().add_special_conn_setup(malformed_setup_function); + + let error = DbInitializerReal::default() + .initialize(home_dir.as_path(), init_config) + .unwrap_err(); + + assert_eq!( + error, + InitializationError::SqliteError(Error::GetAuxWrongType) + ) + } + + #[test] + fn processing_special_setup_to_the_connection_goes_wrong_on_new_database() { + processing_special_setup_test_body( + "processing_special_setup_to_the_connection_goes_wrong_on_new_database", + |path| { + DbInitializerReal::default() + .initialize(path, DbInitializationConfig::test_default()) + .unwrap(); + }, + ) + } + + #[test] + fn processing_special_setup_to_the_connection_goes_wrong_on_existing_database() { + processing_special_setup_test_body( + "processing_special_setup_to_the_connection_goes_wrong_on_existing_database", + |_path| {}, + ) } #[test] @@ -1055,8 +1289,7 @@ mod tests { let _ = subject .initialize( &updated_db_path_dir, - true, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( Chain::EthRopsten, NeighborhoodModeLight::Standard, Some("password".to_string()), @@ -1066,8 +1299,7 @@ mod tests { let _ = subject .initialize( &from_scratch_db_path_dir, - true, - MigratorConfig::test_default(), + DbInitializationConfig::test_default(), ) .unwrap(); @@ -1084,8 +1316,8 @@ mod tests { OpenFlags::SQLITE_OPEN_READ_ONLY, ) .unwrap(); - let extract_from_updated = subject.extract_configurations(&conn_updated); - let extract_from_from_scratch = subject.extract_configurations(&conn_from_scratch); + let extract_from_updated = extract_configurations(&conn_updated); + let extract_from_from_scratch = extract_configurations(&conn_from_scratch); //please, write all rows with unpredictable values here let sieve = |updated_parameter: &String| updated_parameter != "clandestine_port"; let zipped_iterators = extract_from_updated @@ -1162,7 +1394,7 @@ mod tests { let schema_version_before = dao.get("schema_version").unwrap().value_opt.unwrap(); let subject = DbInitializerReal::default(); - let result = subject.initialize(&data_dir, false, MigratorConfig::migration_suppressed()); + let result = subject.initialize(&data_dir, DbInitializationConfig::migration_suppressed()); let wrapped_connection = result.unwrap(); let (schema_version_after, _) = @@ -1171,7 +1403,7 @@ mod tests { } #[test] - #[should_panic(expected = "Attempt to migrate the database at an inappropriate place")] + #[should_panic(expected = "Broken code: Migrating database at inappropriate place")] fn database_migration_causes_panic_if_not_allowed() { let data_dir = ensure_node_home_directory_exists( "db_initializer", @@ -1180,23 +1412,56 @@ mod tests { let _ = bring_db_0_back_to_life_and_return_connection(&data_dir.join(DATABASE_FILE)); let subject = DbInitializerReal::default(); - let _ = subject.initialize(&data_dir, false, MigratorConfig::panic_on_migration()); + let _ = subject.initialize(&data_dir, DbInitializationConfig::panic_on_migration()); + } + + fn assert_new_database_was_not_created(home_dir: &Path) { + let mut flags = OpenFlags::empty(); + flags.insert(OpenFlags::SQLITE_OPEN_READ_ONLY); + let result = Connection::open_with_flags(home_dir.join(DATABASE_FILE), flags); + match result.err().unwrap() { + Error::SqliteFailure(_, _) => (), + x => panic!("Expected SqliteFailure, got {:?}", x), + } + } + + fn assert_that_database_is_not_created_by_certain_initialization_configs(data_dir: &Path) { + let subject = DbInitializerReal::default(); + + [ + DbInitializationConfig::panic_on_migration(), + DbInitializationConfig::migration_suppressed(), + DbInitializationConfig::migration_suppressed_with_error(), + ] + .into_iter() + .for_each(|init_config| { + let result = subject.initialize(data_dir, init_config); + + assert_eq!(result.err().unwrap(), InitializationError::Nonexistent); + assert_new_database_was_not_created(data_dir) + }) } #[test] - #[should_panic(expected = "Attempt to create a new database without proper configuration")] - fn database_creation_panics_if_config_is_missing() { - let data_dir = ensure_node_home_directory_exists( + fn db_initialize_does_not_create_if_directed_not_to_via_initialization_config_and_directory_does_not_exist( + ) { + let data_dir = ensure_node_home_directory_does_not_exist( "db_initializer", - "database_creation_panics_if_config_is_missing", + "db_initialize_does_not_create_if_directed_not_to_via_initialization_config_and_directory_does_not_exist", ); - let subject = DbInitializerReal::default(); - let _ = subject.initialize( - &data_dir, - true, - MigratorConfig::migration_suppressed(), //suppressed doesn't contain a populated config; only 'create_or_migrate()' does + assert_that_database_is_not_created_by_certain_initialization_configs(&data_dir) + } + + #[test] + fn db_initialize_does_not_create_if_directed_not_to_via_initialization_config_and_database_file_does_not_exist( + ) { + let data_dir = ensure_node_home_directory_exists( + "db_initializer", + "db_initialize_does_not_create_if_directed_not_to_via_initialization_config_and_database_file_does_not_exist", ); + + assert_that_database_is_not_created_by_certain_initialization_configs(&data_dir) } #[test] @@ -1212,8 +1477,7 @@ mod tests { let result = subject.initialize( &data_dir, - false, - MigratorConfig::migration_suppressed_with_error(), + DbInitializationConfig::migration_suppressed_with_error(), ); let err = match result { @@ -1225,6 +1489,131 @@ mod tests { assert_eq!(schema_version_after, schema_version_before) } + #[test] + fn panic_on_migration_properly_set() { + assert_eq!( + DbInitializationConfig::panic_on_migration(), + DbInitializationConfig { + mode: InitializationMode::CreationBannedMigrationPanics, + special_conn_configuration: vec![], + } + ) + } + + #[test] + fn create_or_migrate_properly_set() { + assert_eq!( + DbInitializationConfig::create_or_migrate(make_external_data()), + DbInitializationConfig { + mode: InitializationMode::CreationAndMigration { + external_data: make_external_data() + }, + special_conn_configuration: vec![], + } + ) + } + + #[test] + fn migration_suppressed_properly_set() { + assert_eq!( + DbInitializationConfig::migration_suppressed(), + DbInitializationConfig { + mode: InitializationMode::CreationBannedMigrationSuppressed, + special_conn_configuration: vec![], + } + ) + } + + #[test] + fn suppressed_with_error_properly_set() { + assert_eq!( + DbInitializationConfig::migration_suppressed_with_error(), + DbInitializationConfig { + mode: InitializationMode::CreationBannedMigrationRaisesErr, + special_conn_configuration: vec![], + } + ) + } + + fn make_default_config_with_different_pointers( + pointers: Vec rusqlite::Result<()>>, + ) -> DbInitializationConfig { + DbInitializationConfig { + mode: InitializationMode::CreationBannedMigrationPanics, + special_conn_configuration: pointers, + } + } + + fn make_config_filled_with_external_data() -> DbInitializationConfig { + DbInitializationConfig { + mode: InitializationMode::CreationAndMigration { + external_data: ExternalData { + chain: Default::default(), + neighborhood_mode: NeighborhoodModeLight::Standard, + db_password_opt: None, + }, + }, + special_conn_configuration: vec![|_: &_| Ok(())], + } + } + + #[test] + fn partial_eq_is_implemented_for_db_initialization_config() { + let fn_one = |_: &_| Ok(()); + let fn_two = |_: &_| Err(rusqlite::Error::GetAuxWrongType); + let config_one = make_default_config_with_different_pointers(vec![fn_one]); + //Rust doesn't allow differentiate between fn pointers + let config_two = make_default_config_with_different_pointers(vec![fn_two]); + let config_three = make_default_config_with_different_pointers(vec![]); + let config_four = make_default_config_with_different_pointers(vec![fn_one, fn_one]); + let config_five = DbInitializationConfig { + mode: InitializationMode::CreationBannedMigrationSuppressed, + special_conn_configuration: vec![fn_one], + }; + let config_six = make_config_filled_with_external_data(); + + assert_eq!(config_one, config_one); + assert_eq!(config_one, config_two); + //down from here, only inequality + assert_ne!(config_one, config_three); + assert_ne!(config_one, config_four); + assert_ne!(config_one, config_five); + assert_ne!(config_one, config_six) + } + + #[test] + fn debug_is_implemented_for_db_initialization_config() { + let fn_one = |_: &_| Ok(()); + let fn_two = |_: &_| Err(rusqlite::Error::GetAuxWrongType); + let config_one = make_config_filled_with_external_data(); + let config_two = make_default_config_with_different_pointers(vec![fn_one, fn_two]); + + let config_one_debug = format!("{:?}", config_one); + let config_two_debug = format!("{:?}", config_two); + + let regex_one = Regex::new("special_conn_setup: Addresses\\[\\d+\\]").unwrap(); + let regex_two = Regex::new("special_conn_setup: Addresses\\[\\d+(, \\d+)*\\]").unwrap(); + assert!( + config_one_debug.contains( + "DbInitializationConfig{init_config: CreationAndMigration { external_data: \ + ExternalData { chain: PolyMainnet, neighborhood_mode: Standard, db_password_opt: \ + None } }, special_conn_setup: Addresses[" + ), + "instead, the first printed message contained: {}", + config_one_debug + ); + assert!( + config_two_debug.contains( + "DbInitializationConfig{init_config: \ + CreationBannedMigrationPanics, special_conn_setup: Addresses[" + ), + "instead, the second printed message contained: {}", + config_two_debug + ); + assert!(regex_one.is_match(&config_one_debug)); + assert!(regex_two.is_match(&config_two_debug)) + } + #[test] fn choose_clandestine_port_chooses_different_unused_ports_each_time() { let _listeners = (0..10) diff --git a/node/src/database/db_migrations.rs b/node/src/database/db_migrations.rs index 2437ba9ec..6ed5dcb6f 100644 --- a/node/src/database/db_migrations.rs +++ b/node/src/database/db_migrations.rs @@ -1,21 +1,21 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. +use crate::accountant::big_int_processing::big_int_divider::BigIntDivider; +use crate::accountant::dao_utils::VigilantRusqliteFlatten; +use crate::accountant::gwei_to_wei; use crate::blockchain::bip39::Bip39; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::db_initializer::CURRENT_SCHEMA_VERSION; +use crate::database::db_initializer::{ExternalData, CURRENT_SCHEMA_VERSION}; use crate::db_config::db_encryption_layer::DbEncryptionLayer; use crate::db_config::typed_config_layer::decode_bytes; use crate::sub_lib::accountant::{DEFAULT_PAYMENT_THRESHOLDS, DEFAULT_SCAN_INTERVALS}; use crate::sub_lib::cryptde::PlainData; -use crate::sub_lib::neighborhood::DEFAULT_RATE_PACK; +use crate::sub_lib::neighborhood::{RatePack, DEFAULT_RATE_PACK}; use itertools::Itertools; -use masq_lib::blockchains::chains::Chain; use masq_lib::logger::Logger; -#[cfg(test)] -use masq_lib::test_utils::utils::TEST_DEFAULT_CHAIN; -use masq_lib::utils::{ExpectValue, NeighborhoodModeLight, WrapResult}; -use rusqlite::{Error, Transaction}; -use std::fmt::Debug; +use masq_lib::utils::{ExpectValue, WrapResult}; +use rusqlite::{params_from_iter, Error, Row, ToSql, Transaction}; +use std::fmt::{Debug, Display, Formatter}; use tiny_hderive::bip32::ExtendedPrivKey; pub trait DbMigrator { @@ -53,7 +53,7 @@ impl DbMigrator for DbMigratorReal { } } -trait DatabaseMigration: Debug { +trait DatabaseMigration { fn migrate<'a>( &self, mig_declaration_utilities: Box, @@ -64,7 +64,10 @@ trait DatabaseMigration: Debug { trait MigDeclarationUtilities { fn db_password(&self) -> Option; fn transaction(&self) -> &Transaction; - fn execute_upon_transaction<'a>(&self, sql_statements: &[&'a str]) -> rusqlite::Result<()>; + fn execute_upon_transaction<'a>( + &self, + sql_statements: &[&'a dyn StatementObject], + ) -> rusqlite::Result<()>; fn external_parameters(&self) -> &ExternalData; fn logger(&self) -> &Logger; } @@ -176,11 +179,14 @@ impl MigDeclarationUtilities for MigDeclarationUtilitiesReal<'_> { self.root_transaction_ref } - fn execute_upon_transaction<'a>(&self, sql_statements: &[&'a str]) -> rusqlite::Result<()> { + fn execute_upon_transaction<'a>( + &self, + sql_statements: &[&dyn StatementObject], + ) -> rusqlite::Result<()> { let transaction = self.root_transaction_ref; sql_statements.iter().fold(Ok(()), |so_far, stm| { if so_far.is_ok() { - match transaction.execute(stm, []) { + match stm.execute(transaction) { Ok(_) => Ok(()), Err(e) if e == Error::ExecuteReturnedResults => Ok(()), Err(e) => Err(e), @@ -200,6 +206,41 @@ impl MigDeclarationUtilities for MigDeclarationUtilitiesReal<'_> { } } +trait StatementObject: Display { + fn execute(&self, transaction: &Transaction) -> rusqlite::Result<()>; +} + +impl StatementObject for &str { + fn execute(&self, transaction: &Transaction) -> rusqlite::Result<()> { + transaction.execute(self, []).map(|_| ()) + } +} + +impl StatementObject for String { + fn execute(&self, transaction: &Transaction) -> rusqlite::Result<()> { + self.as_str().execute(transaction) + } +} + +struct StatementWithRusqliteParams { + sql_stm: String, + params: Vec>, +} + +impl StatementObject for StatementWithRusqliteParams { + fn execute(&self, transaction: &Transaction) -> rusqlite::Result<()> { + transaction + .execute(&self.sql_stm, params_from_iter(self.params.iter())) + .map(|_| ()) + } +} + +impl Display for StatementWithRusqliteParams { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!(f, "{}", self.sql_stm) + } +} + struct DBMigratorInnerConfiguration { db_configuration_table: String, current_schema_version: usize, @@ -214,7 +255,6 @@ impl DBMigratorInnerConfiguration { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_0_to_1; @@ -224,7 +264,7 @@ impl DatabaseMigration for Migrate_0_to_1 { declaration_utils: Box, ) -> rusqlite::Result<()> { declaration_utils.execute_upon_transaction(&[ - "INSERT INTO config (name, value, encrypted) VALUES ('mapping_protocol', null, 0)", + &"INSERT INTO config (name, value, encrypted) VALUES ('mapping_protocol', null, 0)", ]) } @@ -233,7 +273,6 @@ impl DatabaseMigration for Migrate_0_to_1 { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_1_to_2; @@ -250,7 +289,7 @@ impl DatabaseMigration for Migrate_1_to_2 { .rec() .literal_identifier ); - declaration_utils.execute_upon_transaction(&[statement.as_str()]) + declaration_utils.execute_upon_transaction(&[&statement]) } fn old_version(&self) -> usize { @@ -258,7 +297,6 @@ impl DatabaseMigration for Migrate_1_to_2 { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_2_to_3; @@ -273,7 +311,7 @@ impl DatabaseMigration for Migrate_2_to_3 { "INSERT INTO config (name, value, encrypted) VALUES ('neighborhood_mode', '{}', 0)", declaration_utils.external_parameters().neighborhood_mode ); - declaration_utils.execute_upon_transaction(&[statement_1, statement_2.as_str()]) + declaration_utils.execute_upon_transaction(&[&statement_1, &statement_2]) } fn old_version(&self) -> usize { @@ -281,7 +319,6 @@ impl DatabaseMigration for Migrate_2_to_3 { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_3_to_4; @@ -346,9 +383,9 @@ impl DatabaseMigration for Migrate_3_to_4 { "null".to_string() }; utils.execute_upon_transaction(&[ - format! ("insert into config (name, value, encrypted) values ('consuming_wallet_private_key', {}, 1)", - private_key_column).as_str(), - "delete from config where name in ('seed', 'consuming_wallet_derivation_path', 'consuming_wallet_public_key')", + &format! ("insert into config (name, value, encrypted) values ('consuming_wallet_private_key', {}, 1)", + private_key_column), + &"delete from config where name in ('seed', 'consuming_wallet_derivation_path', 'consuming_wallet_public_key')", ]) } @@ -357,16 +394,12 @@ impl DatabaseMigration for Migrate_3_to_4 { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_4_to_5; impl DatabaseMigration for Migrate_4_to_5 { - fn migrate<'a>( - &self, - declaration_utils: Box, - ) -> rusqlite::Result<()> { - let mut select_statement = declaration_utils + fn migrate<'a>(&self, utils: Box) -> rusqlite::Result<()> { + let mut select_statement = utils .transaction() .prepare("select pending_payment_transaction from payable where pending_payment_transaction is not null")?; let unresolved_pending_transactions: Vec = select_statement @@ -375,15 +408,16 @@ impl DatabaseMigration for Migrate_4_to_5 { .get::(0) .expect("select statement was badly prepared")) })? - .flatten() + .vigilant_flatten() .collect(); if !unresolved_pending_transactions.is_empty() { - warning!(declaration_utils.logger(),"Migration from 4 to 5: database belonging to the chain '{}'; \ - we discovered possibly abandoned transactions that are said yet to be pending, these are: '{}'; \ - continuing",declaration_utils.external_parameters().chain.rec().literal_identifier,unresolved_pending_transactions.join("', '") ) + warning!(utils.logger(), + "Migration from 4 to 5: database belonging to the chain '{}'; \ + we discovered possibly abandoned transactions that are said yet to be pending, these are: '{}'; continuing", + utils.external_parameters().chain.rec().literal_identifier,unresolved_pending_transactions.join("', '") ) } else { debug!( - declaration_utils.logger(), + utils.logger(), "Migration from 4 to 5: no previous pending transactions found; continuing" ) }; @@ -410,18 +444,18 @@ impl DatabaseMigration for Migrate_4_to_5 { )"; let statement_10 = "insert into config (name, value, encrypted) select name, value, encrypted from _config_old"; let statement_11 = "drop table _config_old"; - declaration_utils.execute_upon_transaction(&[ - statement_1, - statement_2, - statement_3, - statement_4, - statement_5, - statement_6, - statement_7, - statement_8, - statement_9, - statement_10, - statement_11, + utils.execute_upon_transaction(&[ + &statement_1, + &statement_2, + &statement_3, + &statement_4, + &statement_5, + &statement_6, + &statement_7, + &statement_8, + &statement_9, + &statement_10, + &statement_11, ]) } @@ -430,7 +464,6 @@ impl DatabaseMigration for Migrate_4_to_5 { } } -#[derive(Debug)] #[allow(non_camel_case_types)] struct Migrate_5_to_6; @@ -449,11 +482,7 @@ impl DatabaseMigration for Migrate_5_to_6 { "scan_intervals", &DEFAULT_SCAN_INTERVALS.to_string(), ); - declaration_utils.execute_upon_transaction(&[ - statement_1.as_str(), - statement_2.as_str(), - statement_3.as_str(), - ]) + declaration_utils.execute_upon_transaction(&[&statement_1, &statement_2, &statement_3]) } fn old_version(&self) -> usize { @@ -470,6 +499,221 @@ impl Migrate_5_to_6 { } } +#[allow(non_camel_case_types)] +struct Migrate_6_to_7; + +#[allow(non_camel_case_types)] +struct Migrate_6_to_7_carrier<'a> { + utils: &'a (dyn MigDeclarationUtilities + 'a), + statements: Vec>, +} + +impl DatabaseMigration for Migrate_6_to_7 { + fn migrate<'a>(&self, utils: Box) -> rusqlite::Result<()> { + let mut migration_carrier = Migrate_6_to_7_carrier::new(utils.as_ref()); + migration_carrier.retype_table( + "payable", + "balance", + "wallet_address text primary key, + balance_high_b integer not null, + balance_low_b integer not null, + last_paid_timestamp integer not null, + pending_payable_rowid integer null", + )?; + migration_carrier.retype_table( + "receivable", + "balance", + "wallet_address text primary key, + balance_high_b integer not null, + balance_low_b integer not null, + last_received_timestamp integer not null", + )?; + migration_carrier.retype_table( + "pending_payable", + "amount", + "rowid integer primary key, + transaction_hash text not null, + amount_high_b integer not null, + amount_low_b integer not null, + payable_timestamp integer not null, + attempt integer not null, + process_error text null", + )?; + + migration_carrier.update_rate_pack(); + + migration_carrier.utils.execute_upon_transaction( + &migration_carrier + .statements + .iter() + .map(|boxed| boxed.as_ref()) + .collect_vec(), + ) + } + + fn old_version(&self) -> usize { + 6 + } +} + +impl<'a> Migrate_6_to_7_carrier<'a> { + fn new(utils: &'a (dyn MigDeclarationUtilities + 'a)) -> Self { + Self { + utils, + statements: vec![], + } + } + + fn retype_table( + &mut self, + table: &str, + old_param_name_of_future_big_int: &str, + create_new_table_stm: &str, + ) -> rusqlite::Result<()> { + self.utils.execute_upon_transaction(&[ + &format!("alter table {table} rename to _{table}_old"), + &format!( + "create table compensatory_{table} (old_rowid integer, high_bytes integer null, low_bytes integer null)" + ), + &format!("create table {table} ({create_new_table_stm}) strict"), + ])?; + let param_names = Self::extract_param_names(create_new_table_stm); + self.maybe_compose_insert_stm_with_auxiliary_table_to_handle_new_big_int_data( + table, + old_param_name_of_future_big_int, + param_names, + ); + self.statements + .push(Box::new(format!("drop table _{table}_old"))); + Ok(()) + } + + fn maybe_compose_insert_stm_with_auxiliary_table_to_handle_new_big_int_data( + &mut self, + table: &str, + big_int_param_old_name: &str, + param_names: Vec, + ) { + let big_int_params_new_names = param_names + .iter() + .filter(|segment| segment.contains(big_int_param_old_name)) + .map(|name| name.to_owned()) + .collect::>(); + let (easy_params, normal_params_prepared_for_inner_join) = + Self::prepare_unchanged_params(param_names, &big_int_params_new_names); + let future_big_int_values_including_old_rowids = self + .utils + .transaction() + .prepare(&format!( + "select rowid, {big_int_param_old_name} from _{table}_old", + )) + .expect("rusqlite internal error") + .query_map([], |row: &Row| { + let old_rowid = row.get(0).expect("rowid fetching error"); + let balance = row.get(1).expect("old param fetching error"); + Ok((old_rowid, balance)) + }) + .expect("map failed") + .vigilant_flatten() + .collect::>(); + if !future_big_int_values_including_old_rowids.is_empty() { + self.fill_compensatory_table(future_big_int_values_including_old_rowids, table); + let new_big_int_params = big_int_params_new_names.join(", "); + let final_insert_statement = format!( + "insert into {table} ({easy_params}, {new_big_int_params}) select {normal_params_prepared_for_inner_join}, \ + R.high_bytes, R.low_bytes from _{table}_old L inner join compensatory_{table} R where L.rowid = R.old_rowid", + ); + self.statements.push(Box::new(final_insert_statement)) + } else { + debug!( + self.utils.logger(), + "Migration from 6 to 7: no data to migrate in {}", table + ) + }; + } + + fn prepare_unchanged_params( + param_names_for_select_stm: Vec, + big_int_params_names: &[String], + ) -> (String, String) { + let easy_params_vec = param_names_for_select_stm + .into_iter() + .filter(|name| !big_int_params_names.contains(name)) + .collect_vec(); + let easy_params = easy_params_vec.iter().join(", "); + let easy_params_preformatted_for_inner_join = easy_params_vec + .into_iter() + .map(|word| format!("L.{}", word.trim())) + .join(", "); + (easy_params, easy_params_preformatted_for_inner_join) + } + + fn fill_compensatory_table(&mut self, all_big_int_values_found: Vec<(i64, i64)>, table: &str) { + let sql_stm = format!( + "insert into compensatory_{} (old_rowid, high_bytes, low_bytes) values {}", + table, + (0..all_big_int_values_found.len()) + .map(|_| "(?, ?, ?)") + .collect::() + ); + let params = all_big_int_values_found + .into_iter() + .flat_map(|(old_rowid, i64_balance)| { + let (high, low) = BigIntDivider::deconstruct(gwei_to_wei(i64_balance)); + vec![ + Box::new(old_rowid) as Box, + Box::new(high), + Box::new(low), + ] + }) + .collect::>>(); + let statement = StatementWithRusqliteParams { sql_stm, params }; + self.statements.push(Box::new(statement)); + } + + fn extract_param_names(table_creation_lines: &str) -> Vec { + table_creation_lines + .split(',') + .map(|line| { + let line = line.trim_start(); + line.chars() + .take_while(|char| !char.is_whitespace()) + .collect::() + }) + .collect() + } + + fn update_rate_pack(&mut self) { + let transaction = self.utils.transaction(); + let mut stm = transaction + .prepare("select value from config where name = 'rate_pack'") + .expect("stm preparation failed"); + let old_rate_pack = stm + .query_row([], |row| row.get::(0)) + .expect("row query failed"); + let old_rate_pack_as_native = + RatePack::try_from(old_rate_pack.as_str()).unwrap_or_else(|_| { + panic!( + "rate pack conversion failed with value: {}; database corrupt!", + old_rate_pack + ) + }); + let new_rate_pack = RatePack { + routing_byte_rate: gwei_to_wei(old_rate_pack_as_native.routing_byte_rate), + routing_service_rate: gwei_to_wei(old_rate_pack_as_native.routing_service_rate), + exit_byte_rate: gwei_to_wei(old_rate_pack_as_native.exit_byte_rate), + exit_service_rate: gwei_to_wei(old_rate_pack_as_native.exit_service_rate), + }; + let serialized_rate_pack = new_rate_pack.to_string(); + let params: Vec> = vec![Box::new(serialized_rate_pack)]; + + self.statements.push(Box::new(StatementWithRusqliteParams { + sql_stm: "update config set value = ? where name = 'rate_pack'".to_string(), + params, + })) + } +} + //////////////////////////////////////////////////////////////////////////////////////////////////// impl DbMigratorReal { @@ -480,7 +724,7 @@ impl DbMigratorReal { } } - fn list_of_migrations<'a>() -> &'a [&'a dyn DatabaseMigration] { + const fn list_of_migrations<'a>() -> &'a [&'a dyn DatabaseMigration] { &[ &Migrate_0_to_1, &Migrate_1_to_2, @@ -488,6 +732,7 @@ impl DbMigratorReal { &Migrate_3_to_4, &Migrate_4_to_5, &Migrate_5_to_6, + &Migrate_6_to_7, ] } @@ -591,99 +836,6 @@ impl DbMigratorReal { } } -#[derive(PartialEq, Eq, Debug, Clone)] -pub struct MigratorConfig { - pub should_be_suppressed: Suppression, - pub external_dataset: Option, -} - -#[derive(PartialEq, Eq, Debug, Clone, Copy)] -pub enum Suppression { - No, - Yes, - WithErr, -} - -impl MigratorConfig { - pub fn panic_on_migration() -> Self { - Self { - should_be_suppressed: Suppression::No, - external_dataset: None, - } - } - - pub fn create_or_migrate(external_params: ExternalData) -> Self { - //is used also if a fresh db is being created - Self { - should_be_suppressed: Suppression::No, - external_dataset: Some(external_params), - } - } - - pub fn migration_suppressed() -> Self { - Self { - should_be_suppressed: Suppression::Yes, - external_dataset: None, - } - } - - pub fn migration_suppressed_with_error() -> Self { - Self { - should_be_suppressed: Suppression::WithErr, - external_dataset: None, - } - } - - #[cfg(test)] - pub fn test_default() -> Self { - Self { - should_be_suppressed: Suppression::Yes, - external_dataset: Some(ExternalData { - chain: TEST_DEFAULT_CHAIN, - neighborhood_mode: NeighborhoodModeLight::Standard, - db_password_opt: None, - }), - } - } -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct ExternalData { - pub chain: Chain, - pub neighborhood_mode: NeighborhoodModeLight, - pub db_password_opt: Option, -} - -impl ExternalData { - pub fn new( - chain: Chain, - neighborhood_mode: NeighborhoodModeLight, - db_password_opt: Option, - ) -> Self { - Self { - chain, - neighborhood_mode, - db_password_opt, - } - } -} - -impl From<(MigratorConfig, bool)> for ExternalData { - fn from(tuple: (MigratorConfig, bool)) -> Self { - let (migrator_config, db_newly_created) = tuple; - migrator_config.external_dataset.unwrap_or_else(|| { - panic!( - "{}", - if db_newly_created { - "Attempt to create a new database without proper configuration" - } else { - "Attempt to migrate the database at an inappropriate place" - } - ) - }) - } -} - #[derive(Debug)] struct InterimMigrationPlaceholder(usize); @@ -702,30 +854,35 @@ impl DatabaseMigration for InterimMigrationPlaceholder { #[cfg(test)] mod tests { + use crate::accountant::dao_utils::{from_time_t, to_time_t}; use crate::blockchain::bip39::Bip39; use crate::database::connection_wrapper::{ConnectionWrapper, ConnectionWrapperReal}; - use crate::database::dao_utils::{from_time_t, to_time_t}; use crate::database::db_initializer::test_utils::ConnectionWrapperMock; use crate::database::db_initializer::{ - DbInitializer, DbInitializerReal, CURRENT_SCHEMA_VERSION, DATABASE_FILE, + DbInitializationConfig, DbInitializer, DbInitializerReal, ExternalData, + CURRENT_SCHEMA_VERSION, DATABASE_FILE, }; use crate::database::db_migrations::{ DBMigrationUtilities, DBMigrationUtilitiesReal, DatabaseMigration, DbMigrator, - ExternalData, MigDeclarationUtilities, Migrate_0_to_1, MigratorConfig, Suppression, + MigDeclarationUtilities, Migrate_0_to_1, StatementObject, StatementWithRusqliteParams, }; use crate::database::db_migrations::{DBMigratorInnerConfiguration, DbMigratorReal}; use crate::db_config::db_encryption_layer::DbEncryptionLayer; + use crate::db_config::persistent_configuration::{ + PersistentConfiguration, PersistentConfigurationReal, + }; use crate::db_config::typed_config_layer::encode_bytes; use crate::sub_lib::accountant::{DEFAULT_PAYMENT_THRESHOLDS, DEFAULT_SCAN_INTERVALS}; use crate::sub_lib::cryptde::PlainData; use crate::sub_lib::neighborhood::DEFAULT_RATE_PACK; use crate::sub_lib::wallet::Wallet; - use crate::test_utils::database_utils::retrieve_config_row; use crate::test_utils::database_utils::{ - assert_create_table_statement_contains_all_important_parts, - assert_index_statement_is_coupled_with_right_parameter, assert_no_index_exists_for_table, - bring_db_0_back_to_life_and_return_connection, + assert_create_table_stm_contains_all_parts, + assert_index_stm_is_coupled_with_right_parameter, assert_no_index_exists_for_table, + assert_table_does_not_exist, bring_db_0_back_to_life_and_return_connection, + make_external_data, }; + use crate::test_utils::database_utils::{assert_table_created_as_strict, retrieve_config_row}; use crate::test_utils::make_wallet; use bip39::{Language, Mnemonic, MnemonicType, Seed}; use ethereum_types::BigEndianHash; @@ -737,13 +894,14 @@ mod tests { use masq_lib::utils::{derivation_path, NeighborhoodModeLight}; use rand::Rng; use rusqlite::types::Value::Null; - use rusqlite::{Connection, Error, OptionalExtension, ToSql, Transaction}; + use rusqlite::{Connection, Error, OptionalExtension, Row, ToSql, Transaction}; use std::cell::RefCell; use std::collections::HashMap; use std::fmt::Debug; use std::fs::create_dir_all; use std::iter::once; use std::panic::{catch_unwind, AssertUnwindSafe}; + use std::str::FromStr; use std::sync::{Arc, Mutex}; use std::time::SystemTime; use tiny_hderive::bip32::ExtendedPrivKey; @@ -868,11 +1026,14 @@ mod tests { unimplemented!("Not needed so far") } - fn execute_upon_transaction<'a>(&self, sql_statements: &[&'a str]) -> rusqlite::Result<()> { + fn execute_upon_transaction<'a>( + &self, + sql_statements: &[&'a dyn StatementObject], + ) -> rusqlite::Result<()> { self.execute_upon_transaction_params.lock().unwrap().push( sql_statements .iter() - .map(|str| str.to_string()) + .map(|stm_obj| stm_obj.to_string()) .collect::>(), ); self.execute_upon_transaction_results.borrow_mut().remove(0) @@ -887,61 +1048,33 @@ mod tests { } } - fn make_external_migration_parameters() -> ExternalData { - ExternalData { - chain: TEST_DEFAULT_CHAIN, - neighborhood_mode: NeighborhoodModeLight::Standard, - db_password_opt: None, - } - } - #[test] - fn panic_on_migration_properly_set() { - assert_eq!( - MigratorConfig::panic_on_migration(), - MigratorConfig { - should_be_suppressed: Suppression::No, - external_dataset: None - } - ) - } + fn statement_with_rusqlite_params_can_display_its_stm() { + let subject = StatementWithRusqliteParams { + sql_stm: "insert into table2 (column) values (?)".to_string(), + params: vec![Box::new(12345)], + }; - #[test] - fn create_or_migrate_properly_set() { - assert_eq!( - MigratorConfig::create_or_migrate(make_external_migration_parameters()), - MigratorConfig { - should_be_suppressed: Suppression::No, - external_dataset: Some(make_external_migration_parameters()) - } - ) - } + let stm = subject.to_string(); - #[test] - fn migration_suppressed_properly_set() { - assert_eq!( - MigratorConfig::migration_suppressed(), - MigratorConfig { - should_be_suppressed: Suppression::Yes, - external_dataset: None - } - ) + assert_eq!(stm, "insert into table2 (column) values (?)".to_string()) } - #[test] - fn suppressed_with_error_properly_set() { - assert_eq!( - MigratorConfig::migration_suppressed_with_error(), - MigratorConfig { - should_be_suppressed: Suppression::WithErr, - external_dataset: None - } - ) + const _REMINDER_FROM_COMPILATION_TIME: () = check_schema_version_continuity(); + + #[allow(dead_code)] + const fn check_schema_version_continuity() { + if DbMigratorReal::list_of_migrations().len() != CURRENT_SCHEMA_VERSION { + panic!( + "It appears you need to increment the current schema version to have DbMigrator \ + work correctly if any new migration added" + ) + }; } #[test] fn migrate_database_handles_an_error_from_creating_the_root_transaction() { - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let mismatched_schema = 0; let target_version = 5; //irrelevant let connection = ConnectionWrapperMock::default() @@ -965,7 +1098,7 @@ mod tests { let mig_config = DBMigratorInnerConfiguration::new(); let migration_utilities = DBMigrationUtilitiesReal::new(&mut conn_wrapper, mig_config).unwrap(); - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let captured_panic = catch_unwind(AssertUnwindSafe(|| { subject.initiate_migrations( @@ -1106,7 +1239,7 @@ mod tests { )) .update_schema_version_result(Err(Error::InvalidQuery)) .update_schema_version_params(&update_schema_version_params_arc); - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let result = subject.migrate_semi_automated( &mut migration_record, @@ -1131,7 +1264,7 @@ mod tests { .make_mig_declaration_utils_result(Box::new(migrate_declaration_utils)); let mismatched_schema = 0; let target_version = 5; //not relevant - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let result = subject.initiate_migrations( mismatched_schema, @@ -1166,7 +1299,7 @@ mod tests { }, ) .unwrap(); - let mut external_parameters = make_external_migration_parameters(); + let mut external_parameters = make_external_data(); external_parameters.db_password_opt = Some("booga".to_string()); let logger = Logger::new("test_logger"); let subject = utils.make_mig_declaration_utils(&external_parameters, &logger); @@ -1190,7 +1323,7 @@ mod tests { }, ) .unwrap(); - let external_parameters = make_external_migration_parameters(); + let external_parameters = make_external_data(); let logger = Logger::new("test_logger"); let subject = utils.make_mig_declaration_utils(&external_parameters, &logger); @@ -1221,14 +1354,14 @@ mod tests { let erroneous_statement_1 = "INSERT INTO botanic_garden (name, count) VALUES (sunflowers, 100)"; let erroneous_statement_2 = "INSERT INTO milky_way (star) VALUES (just_discovered)"; - let set_of_sql_statements = &[ - correct_statement_1, - erroneous_statement_1, - erroneous_statement_2, + let set_of_sql_statements: &[&dyn StatementObject] = &[ + &correct_statement_1, + &erroneous_statement_1, + &erroneous_statement_2, ]; let mut connection_wrapper = ConnectionWrapperReal::new(connection); let config = DBMigratorInnerConfiguration::new(); - let external_parameters = make_external_migration_parameters(); + let external_parameters = make_external_data(); let subject = DBMigrationUtilitiesReal::new(&mut connection_wrapper, config).unwrap(); let result = subject @@ -1270,10 +1403,11 @@ mod tests { let statement_1 = "INSERT INTO botanic_garden (name,count) VALUES ('sun_flowers', 100)"; let statement_2 = "ALTER TABLE botanic_garden RENAME TO just_garden"; //this statement returns an overview of the new table on its execution let statement_3 = "COMMIT"; - let set_of_sql_statements = &[statement_1, statement_2, statement_3]; + let set_of_sql_statements: &[&dyn StatementObject] = + &[&statement_1, &statement_2, &statement_3]; let mut connection_wrapper = ConnectionWrapperReal::new(connection); let config = DBMigratorInnerConfiguration::new(); - let external_parameters = make_external_migration_parameters(); + let external_parameters = make_external_data(); let subject = DBMigrationUtilitiesReal::new(&mut connection_wrapper, config).unwrap(); let result = subject @@ -1283,7 +1417,7 @@ mod tests { assert_eq!(result, Ok(())); let connection = Connection::open(&db_path).unwrap(); let assertion: Option<(String, i64)> = connection - .query_row("SELECT * FROM just_garden", [], |row| { + .query_row("SELECT name, count FROM just_garden", [], |row| { Ok((row.get(0).unwrap(), row.get(1).unwrap())) }) .optional() @@ -1291,6 +1425,72 @@ mod tests { assert!(assertion.is_some()) //means there is a table named 'just_garden' now } + #[test] + fn execute_upon_transaction_handles_also_error_from_stm_with_params() { + let dir_path = ensure_node_home_directory_exists( + "db_migrations", + "execute_upon_transaction_handles_also_error_from_stm_with_params", + ); + let db_path = dir_path.join("test_database.db"); + let conn = Connection::open(&db_path).unwrap(); + conn.execute( + "CREATE TABLE botanic_garden ( + name TEXT, + count integer + )", + [], + ) + .unwrap(); + let statement_1_simple = + "INSERT INTO botanic_garden (name,count) VALUES ('sun_flowers', 100)"; + let statement_2_good = StatementWithRusqliteParams { + sql_stm: "select * from botanic_garden".to_string(), + params: { + let params: Vec> = vec![]; + params + }, + }; + let statement_3_bad = StatementWithRusqliteParams { + sql_stm: "select name, count from foo".to_string(), + params: vec![Box::new("another_whatever")], + }; + //we expect not to get down to this statement, the error from statement_3 immediately terminates the circuit + let statement_4_demonstrative = StatementWithRusqliteParams { + sql_stm: "select name, count from bar".to_string(), + params: vec![Box::new("also_whatever")], + }; + let set_of_sql_statements: &[&dyn StatementObject] = &[ + &statement_1_simple, + &statement_2_good, + &statement_3_bad, + &statement_4_demonstrative, + ]; + let mut conn_wrapper = ConnectionWrapperReal::new(conn); + let config = DBMigratorInnerConfiguration::new(); + let external_params = make_external_data(); + let subject = DBMigrationUtilitiesReal::new(&mut conn_wrapper, config).unwrap(); + + let result = subject + .make_mig_declaration_utils(&external_params, &Logger::new("test logger")) + .execute_upon_transaction(set_of_sql_statements); + + match result { + Err(Error::SqliteFailure(_, err_msg_opt)) => { + assert_eq!(err_msg_opt, Some("no such table: foo".to_string())) + } + x => panic!("we expected SqliteFailure(..) but got: {:?}", x), + } + let assert_conn = Connection::open(&db_path).unwrap(); + let assertion: Option<(String, i64)> = assert_conn + .query_row("SELECT * FROM botanic_garden", [], |row| { + Ok((row.get(0).unwrap(), row.get(1).unwrap())) + }) + .optional() + .unwrap(); + assert_eq!(assertion, None) + //the table remained empty because an error causes the whole transaction to abort + } + fn make_success_mig_record( old_version: usize, empty_params_arc: &Arc>>, @@ -1345,7 +1545,7 @@ mod tests { db_configuration_table: "test".to_string(), current_schema_version: 5, }; - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let mismatched_schema = 2; let target_version = 5; @@ -1413,7 +1613,7 @@ mod tests { db_configuration_table: "test".to_string(), current_schema_version: 5, }; - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let mismatched_schema = 0; let target_version = 3; @@ -1455,7 +1655,7 @@ mod tests { .update_schema_version_result(Ok(())) .commit_result(Ok(())); let target_version = 5; //not relevant - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let result = subject.initiate_migrations( outdated_schema, @@ -1515,7 +1715,7 @@ mod tests { .update_schema_version_result(Ok(())) .update_schema_version_result(Ok(())) .commit_result(Err("Committing transaction failed".to_string())); - let subject = DbMigratorReal::new(make_external_migration_parameters()); + let subject = DbMigratorReal::new(make_external_data()); let result = subject.initiate_migrations(0, 2, Box::new(migration_utils), list_of_migrations); @@ -1541,8 +1741,7 @@ mod tests { let result = subject.initialize_to_version( &dir_path, 1, - false, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ); let connection = result.unwrap(); let (mp_value, mp_encrypted) = retrieve_config_row(connection.as_ref(), "mapping_protocol"); @@ -1569,8 +1768,7 @@ mod tests { .initialize_to_version( &dir_path, start_at, - true, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ) .unwrap(); } @@ -1578,8 +1776,7 @@ mod tests { let result = subject.initialize_to_version( &dir_path, start_at + 1, - false, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ); let connection = result.unwrap(); @@ -1609,8 +1806,7 @@ mod tests { .initialize_to_version( &dir_path, start_at, - true, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ) .unwrap(); } @@ -1618,8 +1814,7 @@ mod tests { let result = subject.initialize_to_version( &dir_path, start_at + 1, - false, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( DEFAULT_CHAIN, NeighborhoodModeLight::ConsumeOnly, None, @@ -1650,16 +1845,12 @@ mod tests { let _ = bring_db_0_back_to_life_and_return_connection(&db_path); let password_opt = &Some("password".to_string()); let subject = DbInitializerReal::default(); - let mut migrator_config = - MigratorConfig::create_or_migrate(make_external_migration_parameters()); - migrator_config - .external_dataset - .as_mut() - .unwrap() - .db_password_opt = password_opt.clone(); + let mut external_data = make_external_data(); + external_data.db_password_opt = password_opt.as_ref().cloned(); + let init_config = DbInitializationConfig::create_or_migrate(external_data); let original_private_key = { let schema3_conn = subject - .initialize_to_version(&data_path, 3, true, migrator_config.clone()) + .initialize_to_version(&data_path, 3, init_config.clone()) .unwrap(); let mnemonic = Mnemonic::new(MnemonicType::Words24, Language::English); let seed = Seed::new(&mnemonic, "booga"); @@ -1707,7 +1898,7 @@ mod tests { let migrated_private_key = { let mut schema4_conn = subject - .initialize_to_version(&data_path, 4, false, migrator_config) + .initialize_to_version(&data_path, 4, init_config) .unwrap(); { let mut stmt = schema4_conn.prepare("select count(*) from config where name in ('consuming_wallet_derivation_path', 'consuming_wallet_public_key', 'seed')").unwrap(); @@ -1741,16 +1932,17 @@ mod tests { let _ = bring_db_0_back_to_life_and_return_connection(&db_path); let password_opt = &Some("password".to_string()); let subject = DbInitializerReal::default(); - let mut migrator_config = - MigratorConfig::create_or_migrate(make_external_migration_parameters()); - migrator_config - .external_dataset - .as_mut() - .unwrap() - .db_password_opt = password_opt.clone(); + let mut external_data = make_external_data(); + external_data.db_password_opt = password_opt.as_ref().cloned(); + let init_config = DbInitializationConfig::create_or_migrate(external_data); + { + subject + .initialize_to_version(&data_path, 3, init_config.clone()) + .unwrap(); + }; let mut schema4_conn = subject - .initialize_to_version(&data_path, 4, false, migrator_config) + .initialize_to_version(&data_path, 4, init_config) .unwrap(); { @@ -1782,8 +1974,7 @@ mod tests { .initialize_to_version( &dir_path, start_at, - true, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ) .unwrap(); let wallet_1 = make_wallet("scotland_yard"); @@ -1800,8 +1991,7 @@ mod tests { .initialize_to_version( &dir_path, start_at + 1, - false, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( TEST_DEFAULT_CHAIN, NeighborhoodModeLight::ConsumeOnly, Some("password".to_string()), @@ -1861,8 +2051,7 @@ mod tests { .initialize_to_version( &dir_path, start_at, - true, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + DbInitializationConfig::create_or_migrate(make_external_data()), ) .unwrap(); } @@ -1886,8 +2075,7 @@ mod tests { .initialize_to_version( &dir_path, start_at + 1, - false, - MigratorConfig::create_or_migrate(ExternalData::new( + DbInitializationConfig::create_or_migrate(ExternalData::new( TEST_DEFAULT_CHAIN, NeighborhoodModeLight::ConsumeOnly, Some("password".to_string()), @@ -1905,58 +2093,43 @@ mod tests { } fn assert_on_schema_5_was_adopted(conn_schema5: &dyn ConnectionWrapper) { - let expected_key_words = [ - ["rowid", "integer", "primary", "key"].as_slice(), - ["transaction_hash", "text", "not", "null"].as_slice(), - ["amount", "integer", "not", "null"].as_slice(), - ["payable_timestamp", "integer", "not", "null"].as_slice(), - ["attempt", "integer", "not", "null"].as_slice(), - ["process_error", "text", "null"].as_slice(), + let expected_key_words: &[&[&str]] = &[ + &["rowid", "integer", "primary", "key"], + &["transaction_hash", "text", "not", "null"], + &["amount", "integer", "not", "null"], + &["payable_timestamp", "integer", "not", "null"], + &["attempt", "integer", "not", "null"], + &["process_error", "text", "null"], ]; - assert_create_table_statement_contains_all_important_parts( + assert_create_table_stm_contains_all_parts( conn_schema5, "pending_payable", - expected_key_words.as_slice(), + expected_key_words, ); - let expected_key_words = [["transaction_hash"].as_slice()]; - assert_index_statement_is_coupled_with_right_parameter( + let expected_key_words: &[&[&str]] = &[&["transaction_hash"]]; + assert_index_stm_is_coupled_with_right_parameter( conn_schema5, "pending_payable_hash_idx", - expected_key_words.as_slice(), + expected_key_words, ); - let expected_key_words = [ - ["wallet_address", "text", "primary", "key"].as_slice(), - ["balance", "integer", "not", "null"].as_slice(), - ["last_paid_timestamp", "integer", "not", "null"].as_slice(), - ["pending_payable_rowid", "integer", "null"].as_slice(), + let expected_key_words: &[&[&str]] = &[ + &["wallet_address", "text", "primary", "key"], + &["balance", "integer", "not", "null"], + &["last_paid_timestamp", "integer", "not", "null"], + &["pending_payable_rowid", "integer", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn_schema5, - "payable", - expected_key_words.as_slice(), - ); - let expected_key_words = [ - ["name", "text", "primary", "key"].as_slice(), - ["value", "text"].as_slice(), - ["encrypted", "integer", "not", "null"].as_slice(), + assert_create_table_stm_contains_all_parts(conn_schema5, "payable", expected_key_words); + let expected_key_words: &[&[&str]] = &[ + &["name", "text", "primary", "key"], + &["value", "text"], + &["encrypted", "integer", "not", "null"], ]; - assert_create_table_statement_contains_all_important_parts( - conn_schema5, - "config", - expected_key_words.as_slice(), - ); + assert_create_table_stm_contains_all_parts(conn_schema5, "config", expected_key_words); assert_no_index_exists_for_table(conn_schema5, "config"); assert_no_index_exists_for_table(conn_schema5, "payable"); assert_no_index_exists_for_table(conn_schema5, "receivable"); assert_no_index_exists_for_table(conn_schema5, "banned"); - let error_stm = conn_schema5 - .prepare("select * from _config_old") - .unwrap_err(); - let error_msg = match error_stm { - rusqlite::Error::SqliteFailure(_, Some(msg)) => msg, - x => panic!("we expected SqliteFailure but we got: {:?}", x), - }; - assert_eq!(error_msg, "no such table: _config_old".to_string()) + assert_table_does_not_exist(conn_schema5, "_config_old") } fn fetch_all_from_config_table( @@ -1997,9 +2170,8 @@ mod tests { subject .initialize_to_version( &dir_path, - 6, - true, - MigratorConfig::create_or_migrate(make_external_migration_parameters()), + 5, + DbInitializationConfig::create_or_migrate(make_external_data()), ) .unwrap(); } @@ -2007,12 +2179,7 @@ mod tests { let result = subject.initialize_to_version( &dir_path, 6, - true, - MigratorConfig::create_or_migrate(ExternalData::new( - DEFAULT_CHAIN, - NeighborhoodModeLight::ConsumeOnly, - None, - )), + DbInitializationConfig::create_or_migrate(make_external_data()), ); let connection = result.unwrap(); @@ -2031,4 +2198,132 @@ mod tests { assert_eq!(scan_intervals, Some(DEFAULT_SCAN_INTERVALS.to_string())); assert_eq!(encrypted, false); } + + #[test] + fn migration_from_6_to_7_works() { + let dir_path = + ensure_node_home_directory_exists("db_migrations", "migration_from_6_to_7_works"); + let db_path = dir_path.join(DATABASE_FILE); + let _ = bring_db_0_back_to_life_and_return_connection(&db_path); + let subject = DbInitializerReal::default(); + let pre_db_conn = subject + .initialize_to_version( + &dir_path, + 6, + DbInitializationConfig::create_or_migrate(make_external_data()), + ) + .unwrap(); + insert_value(&*pre_db_conn,"insert into payable (wallet_address, balance, last_paid_timestamp, pending_payable_rowid) \ + values (\"0xD7d1b2cF58f6500c7CB22fCA42B8512d06813a03\", 56784545484899, 11111, null)"); + insert_value( + &*pre_db_conn, + "insert into receivable (wallet_address, balance, last_received_timestamp) \ + values (\"0xD2d1b2eF58f6500c7ae22fCA42B8512d06813a03\",-56784,22222)", + ); + insert_value(&*pre_db_conn,"insert into pending_payable (rowid, transaction_hash, amount, payable_timestamp,attempt, process_error) \ + values (5, \"0xb5c8bd9430b6cc87a0e2fe110ece6bf527fa4f222a4bc8cd032f768fc5219838\" ,9123 ,33333 ,1 ,null)"); + let mut persistent_config = PersistentConfigurationReal::from(pre_db_conn); + let old_rate_pack_in_gwei = "44|50|20|32".to_string(); + persistent_config + .set_rate_pack(old_rate_pack_in_gwei.clone()) + .unwrap(); + + let conn = subject + .initialize_to_version( + &dir_path, + 7, + DbInitializationConfig::create_or_migrate(make_external_data()), + ) + .unwrap(); + + assert_table_created_as_strict(&*conn, "payable"); + assert_table_created_as_strict(&*conn, "receivable"); + let select_sql = "select wallet_address, balance_high_b, balance_low_b, last_paid_timestamp, pending_payable_rowid from payable"; + query_rows_helper(&*conn, select_sql, |row| { + assert_eq!( + row.get::(0).unwrap(), + Wallet::from_str("0xD7d1b2cF58f6500c7CB22fCA42B8512d06813a03").unwrap() + ); + assert_eq!(row.get::(1).unwrap(), 6156); + assert_eq!(row.get::(2).unwrap(), 5467226021000125952); + assert_eq!(row.get::(3).unwrap(), 11111); + assert_eq!(row.get::>(4).unwrap(), None); + Ok(()) + }); + let select_sql = "select wallet_address, balance_high_b, balance_low_b, last_received_timestamp from receivable"; + query_rows_helper(&*conn, select_sql, |row| { + assert_eq!( + row.get::(0).unwrap(), + Wallet::from_str("0xD2d1b2eF58f6500c7ae22fCA42B8512d06813a03").unwrap() + ); + assert_eq!(row.get::(1).unwrap(), -1); + assert_eq!(row.get::(2).unwrap(), 9223315252854775808); + assert_eq!(row.get::(3).unwrap(), 22222); + Ok(()) + }); + let select_sql = "select rowid, transaction_hash, amount_high_b, amount_low_b, payable_timestamp, attempt, process_error from pending_payable"; + query_rows_helper(&*conn, select_sql, |row| { + assert_eq!(row.get::(0).unwrap(), 5); + assert_eq!( + row.get::(1).unwrap(), + "0xb5c8bd9430b6cc87a0e2fe110ece6bf527fa4f222a4bc8cd032f768fc5219838".to_string() + ); + assert_eq!(row.get::(2).unwrap(), 0); + assert_eq!(row.get::(3).unwrap(), 9123000000000); + assert_eq!(row.get::(4).unwrap(), 33333); + assert_eq!(row.get::(5).unwrap(), 1); + assert_eq!(row.get::>(6).unwrap(), None); + Ok(()) + }); + let (rate_pack, encrypted) = retrieve_config_row(&*conn, "rate_pack"); + assert_eq!( + rate_pack, + Some("44000000000|50000000000|20000000000|32000000000".to_string()) + ); + assert_eq!(encrypted, false); + } + + #[test] + fn migration_from_6_to_7_without_any_data() { + init_test_logging(); + let dir_path = ensure_node_home_directory_exists( + "db_migrations", + "migration_from_6_to_7_without_any_data", + ); + let db_path = dir_path.join(DATABASE_FILE); + let _ = bring_db_0_back_to_life_and_return_connection(&db_path); + let subject = DbInitializerReal::default(); + let conn = subject + .initialize_to_version( + &dir_path, + 6, + DbInitializationConfig::create_or_migrate(make_external_data()), + ) + .unwrap(); + let mut subject = DbMigratorReal::new(make_external_data()); + subject.logger = Logger::new("migration_from_6_to_7_without_any_data"); + + subject.migrate_database(6, 7, conn).unwrap(); + + let test_log_handler = TestLogHandler::new(); + ["payable", "receivable", "pending_payable"] + .iter() + .for_each(|table_name| { + test_log_handler.exists_log_containing(&format!("DEBUG: migration_from_6_to_7_without_any_data: Migration from 6 to 7: no data to migrate in {table_name}")); + }) + } + + fn insert_value(conn: &dyn ConnectionWrapper, insert_stm: &str) { + let mut statement = conn.prepare(insert_stm).unwrap(); + statement.execute([]).unwrap(); + } + + fn query_rows_helper( + conn: &dyn ConnectionWrapper, + sql: &str, + expected_typed_values: fn(&Row) -> rusqlite::Result<()>, + ) { + let mut statement = conn.prepare(sql).unwrap(); + statement.query_row([], expected_typed_values).unwrap(); + } } diff --git a/node/src/database/mod.rs b/node/src/database/mod.rs index 1eb1b08ea..3ba58912e 100644 --- a/node/src/database/mod.rs +++ b/node/src/database/mod.rs @@ -2,6 +2,5 @@ pub mod config_dumper; pub mod connection_wrapper; -pub mod dao_utils; pub mod db_initializer; pub mod db_migrations; diff --git a/node/src/db_config/config_dao.rs b/node/src/db_config/config_dao.rs index 3d9727204..90aa3c283 100644 --- a/node/src/db_config/config_dao.rs +++ b/node/src/db_config/config_dao.rs @@ -1,6 +1,6 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. +use crate::accountant::dao_utils::DaoFactoryReal; use crate::database::connection_wrapper::ConnectionWrapper; -use crate::database::dao_utils::DaoFactoryReal; use rusqlite::types::ToSql; use rusqlite::{Row, Rows, Statement}; @@ -151,10 +151,10 @@ fn row_to_config_dao_record(row: &Row) -> ConfigDaoRecord { #[cfg(test)] mod tests { use super::*; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{ DbInitializer, DbInitializerReal, CURRENT_SCHEMA_VERSION, }; - use crate::database::db_migrations::MigratorConfig; use crate::test_utils::assert_contains; use masq_lib::constants::ROPSTEN_TESTNET_CONTRACT_CREATION_BLOCK; use masq_lib::test_utils::utils::ensure_node_home_directory_exists; @@ -165,7 +165,7 @@ mod tests { ensure_node_home_directory_exists("config_dao", "get_all_returns_multiple_results"); let subject = ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); @@ -201,7 +201,7 @@ mod tests { ); let subject = ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); @@ -215,7 +215,7 @@ mod tests { let home_dir = ensure_node_home_directory_exists("config_dao", "set_and_get_work"); let subject = ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let modified_value = ConfigDaoRecord::new( @@ -244,7 +244,7 @@ mod tests { ); let subject = ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); @@ -261,7 +261,7 @@ mod tests { ); let subject = ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(), ); let _ = subject.set("schema_version", None).unwrap(); diff --git a/node/src/db_config/config_dao_null.rs b/node/src/db_config/config_dao_null.rs index 245961d56..9a0942c77 100644 --- a/node/src/db_config/config_dao_null.rs +++ b/node/src/db_config/config_dao_null.rs @@ -132,8 +132,8 @@ impl Default for ConfigDaoNull { #[cfg(test)] mod tests { use super::*; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::DbInitializer; - use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::ConfigDaoReal; use masq_lib::blockchains::chains::Chain; use masq_lib::constants::{DEFAULT_CHAIN, ETH_MAINNET_CONTRACT_CREATION_BLOCK}; @@ -207,7 +207,7 @@ mod tests { ); let db_initializer = DbInitializerReal::default(); let conn = db_initializer - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(); let real_config_dao = ConfigDaoReal::new(conn); let subject = ConfigDaoNull::default(); diff --git a/node/src/db_config/persistent_configuration.rs b/node/src/db_config/persistent_configuration.rs index d54713644..30b311c39 100644 --- a/node/src/db_config/persistent_configuration.rs +++ b/node/src/db_config/persistent_configuration.rs @@ -1,7 +1,7 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. #[cfg(test)] -use crate::arbitrary_id_stamp; +use crate::arbitrary_id_stamp_in_trait; use crate::blockchain::bip32::Bip32ECKeyProvider; use crate::blockchain::bip39::{Bip39, Bip39Error}; use crate::database::connection_wrapper::ConnectionWrapper; @@ -148,7 +148,7 @@ pub trait PersistentConfiguration { fn set_scan_intervals(&mut self, intervals: String) -> Result<(), PersistentConfigError>; #[cfg(test)] - arbitrary_id_stamp!(); + arbitrary_id_stamp_in_trait!(); } pub struct PersistentConfigurationReal { @@ -258,9 +258,9 @@ impl PersistentConfiguration for PersistentConfigurationReal { || (unchecked_port > u64::from(HIGHEST_USABLE_PORT)) { panic!("Can't continue; clandestine port configuration is incorrect. Must be between {} and {}, not {}. Specify --clandestine-port

where

is an unused port.", - LOWEST_USABLE_INSECURE_PORT, - HIGHEST_USABLE_PORT, - unchecked_port + LOWEST_USABLE_INSECURE_PORT, + HIGHEST_USABLE_PORT, + unchecked_port ); } let port = unchecked_port as u16; @@ -538,7 +538,7 @@ impl PersistentConfigurationReal { ) -> Result { match decoder(self.get(parameter)?)? { None => Self::missing_value_panic(parameter), - Some(rate) => Ok(rate), + Some(value) => Ok(value), } } @@ -552,7 +552,7 @@ impl PersistentConfigurationReal { { match decode_combined_params(values_parser, self.get(parameter)?)? { None => Self::missing_value_panic(parameter), - Some(rate) => Ok(rate), + Some(value) => Ok(value), } } @@ -568,8 +568,9 @@ impl PersistentConfigurationReal { mod tests { use super::*; use crate::blockchain::bip39::Bip39; - use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; - use crate::database::db_migrations::MigratorConfig; + use crate::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, + }; use crate::db_config::config_dao::ConfigDaoRecord; use crate::db_config::mocks::ConfigDaoMock; use crate::db_config::secure_config_layer::EXAMPLE_ENCRYPTED; @@ -1888,7 +1889,7 @@ mod tests { "current_config_table_schema", ); let db_conn = DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::test_default()) + .initialize(&home_dir, DbInitializationConfig::test_default()) .unwrap(); let mut statement = db_conn.prepare("select name from config").unwrap(); statement diff --git a/node/src/neighborhood/mod.rs b/node/src/neighborhood/mod.rs index ecd6e98a5..9d7ee2078 100644 --- a/node/src/neighborhood/mod.rs +++ b/node/src/neighborhood/mod.rs @@ -28,8 +28,8 @@ use masq_lib::ui_gateway::{MessageTarget, NodeFromUiMessage, NodeToUiMessage}; use masq_lib::utils::{exit_process, ExpectValue}; use crate::bootstrapper::BootstrapperConfig; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::persistent_configuration::{ PersistentConfigError, PersistentConfiguration, PersistentConfigurationReal, }; @@ -551,8 +551,7 @@ impl Neighborhood { let conn = db_initializer .initialize( &self.data_directory, - false, - MigratorConfig::panic_on_migration(), + DbInitializationConfig::panic_on_migration(), ) .expect("Neighborhood could not connect to database"); self.persistent_config_opt = Some(Box::new(PersistentConfigurationReal::from(conn))); @@ -1716,7 +1715,7 @@ mod tests { ); { let _ = DbInitializerReal::default() - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(); } let cryptde = main_cryptde(); @@ -4377,7 +4376,7 @@ mod tests { ); { let _ = DbInitializerReal::default() - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(); } let cryptde: &dyn CryptDE = main_cryptde(); diff --git a/node/src/node_configurator/configurator.rs b/node/src/node_configurator/configurator.rs index e315e225f..ce38d3b45 100644 --- a/node/src/node_configurator/configurator.rs +++ b/node/src/node_configurator/configurator.rs @@ -19,8 +19,8 @@ use masq_lib::ui_gateway::{ use crate::blockchain::bip32::Bip32ECKeyProvider; use crate::blockchain::bip39::Bip39; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::ConfigDaoReal; use crate::db_config::persistent_configuration::{ PersistentConfigError, PersistentConfiguration, PersistentConfigurationReal, @@ -97,7 +97,10 @@ impl Configurator { pub fn new(data_directory: PathBuf, crashable: bool) -> Self { let initializer = DbInitializerReal::default(); let conn = initializer - .initialize(&data_directory, false, MigratorConfig::panic_on_migration()) + .initialize( + &data_directory, + DbInitializationConfig::panic_on_migration(), + ) .expect("Couldn't initialize database"); let config_dao = ConfigDaoReal::new(conn); let persistent_config: Box = @@ -855,7 +858,7 @@ mod tests { ensure_node_home_directory_exists("configurator", "constructor_connects_with_database"); let verifier = PersistentConfigurationReal::new(Box::new(ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&data_dir, true, MigratorConfig::test_default()) + .initialize(&data_dir, DbInitializationConfig::test_default()) .unwrap(), ))); let (recorder, _, _) = make_recorder(); diff --git a/node/src/node_configurator/mod.rs b/node/src/node_configurator/mod.rs index a5f00cb7c..aab9cd293 100644 --- a/node/src/node_configurator/mod.rs +++ b/node/src/node_configurator/mod.rs @@ -6,8 +6,8 @@ pub mod node_configurator_standard; pub mod unprivileged_parse_args_configuration; use crate::bootstrapper::RealUser; +use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal, DATABASE_FILE}; -use crate::database::db_migrations::MigratorConfig; use crate::db_config::persistent_configuration::{ PersistentConfiguration, PersistentConfigurationReal, }; @@ -66,11 +66,10 @@ pub fn determine_config_file_path( pub fn initialize_database( data_directory: &Path, - create_if_necessary: bool, - migrator_config: MigratorConfig, + migrator_config: DbInitializationConfig, ) -> Box { let conn = DbInitializerReal::default() - .initialize(data_directory, create_if_necessary, migrator_config) + .initialize(data_directory, migrator_config) .unwrap_or_else(|e| { panic!( "Can't initialize database at {:?}: {:?}", diff --git a/node/src/node_configurator/node_configurator_standard.rs b/node/src/node_configurator/node_configurator_standard.rs index 67e3c51f8..71958ef1d 100644 --- a/node/src/node_configurator/node_configurator_standard.rs +++ b/node/src/node_configurator/node_configurator_standard.rs @@ -16,7 +16,7 @@ use log::LevelFilter; use crate::apps::app_node; use crate::bootstrapper::PortConfiguration; -use crate::database::db_migrations::{ExternalData, MigratorConfig}; +use crate::database::db_initializer::{DbInitializationConfig, ExternalData}; use crate::db_config::persistent_configuration::PersistentConfiguration; use crate::http_request_start_finder::HttpRequestDiscriminatorFactory; use crate::node_configurator::unprivileged_parse_args_configuration::{ @@ -83,8 +83,7 @@ impl NodeConfigurator for NodeConfiguratorStandardUnprivileg ) -> Result { let mut persistent_config = initialize_database( &self.privileged_config.data_directory, - true, - MigratorConfig::create_or_migrate(self.wrap_up_db_externals(multi_config)), + DbInitializationConfig::create_or_migrate(ExternalData::from((self, multi_config))), ); let mut unprivileged_config = BootstrapperConfig::new(); let parse_args_configurator = UnprivilegedParseArgsConfigurationDaoReal {}; @@ -99,25 +98,36 @@ impl NodeConfigurator for NodeConfiguratorStandardUnprivileg } } -impl NodeConfiguratorStandardUnprivileged { - pub fn new(privileged_config: &BootstrapperConfig) -> Self { - Self { - privileged_config: privileged_config.clone(), - logger: Logger::new("NodeConfiguratorStandardUnprivileged"), - } - } - - fn wrap_up_db_externals(&self, multi_config: &MultiConfig) -> ExternalData { +impl<'a> + From<( + &'a NodeConfiguratorStandardUnprivileged, + &'a MultiConfig<'a>, + )> for ExternalData +{ + fn from(tuple: (&'a NodeConfiguratorStandardUnprivileged, &'a MultiConfig)) -> ExternalData { + let (node_configurator_standard, multi_config) = tuple; let (neighborhood_mode, db_password_opt) = collect_externals_from_multi_config(multi_config); ExternalData::new( - self.privileged_config.blockchain_bridge_config.chain, + node_configurator_standard + .privileged_config + .blockchain_bridge_config + .chain, neighborhood_mode, db_password_opt, ) } } +impl NodeConfiguratorStandardUnprivileged { + pub fn new(privileged_config: &BootstrapperConfig) -> Self { + Self { + privileged_config: privileged_config.clone(), + logger: Logger::new("NodeConfiguratorStandardUnprivileged"), + } + } +} + fn collect_externals_from_multi_config( multi_config: &MultiConfig, ) -> (NeighborhoodModeLight, Option) { @@ -311,7 +321,7 @@ mod tests { .unwrap()]; { let conn = DbInitializerReal::default() - .initialize(home_dir.as_path(), true, MigratorConfig::test_default()) + .initialize(home_dir.as_path(), DbInitializationConfig::test_default()) .unwrap(); let mut persistent_config = PersistentConfigurationReal::from(conn); persistent_config.change_password(None, "password").unwrap(); @@ -456,7 +466,7 @@ mod tests { ); let mut persistent_config = PersistentConfigurationReal::new(Box::new(ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir.clone(), true, MigratorConfig::test_default()) + .initialize(&home_dir.clone(), DbInitializationConfig::test_default()) .unwrap(), ))); let consuming_private_key = @@ -937,9 +947,13 @@ mod tests { } #[test] - fn wrap_up_db_externals_is_properly_set_when_password_is_provided() { - let mut subject = NodeConfiguratorStandardUnprivileged::new(&BootstrapperConfig::new()); - subject.privileged_config.blockchain_bridge_config.chain = DEFAULT_CHAIN; + fn external_data_is_properly_created_when_password_is_provided() { + let mut configurator_standard = + NodeConfiguratorStandardUnprivileged::new(&BootstrapperConfig::new()); + configurator_standard + .privileged_config + .blockchain_bridge_config + .chain = DEFAULT_CHAIN; let multi_config = make_simplified_multi_config([ "--neighborhood-mode", "zero-hop", @@ -947,7 +961,7 @@ mod tests { "password", ]); - let result = subject.wrap_up_db_externals(&multi_config); + let result = ExternalData::from((&configurator_standard, &multi_config)); let expected = ExternalData::new( DEFAULT_CHAIN, @@ -958,12 +972,16 @@ mod tests { } #[test] - fn wrap_up_db_externals_is_properly_set_when_no_password_is_provided() { - let mut subject = NodeConfiguratorStandardUnprivileged::new(&BootstrapperConfig::new()); - subject.privileged_config.blockchain_bridge_config.chain = DEFAULT_CHAIN; + fn external_data_is_properly_created_when_no_password_is_provided() { + let mut configurator_standard = + NodeConfiguratorStandardUnprivileged::new(&BootstrapperConfig::new()); + configurator_standard + .privileged_config + .blockchain_bridge_config + .chain = DEFAULT_CHAIN; let multi_config = make_simplified_multi_config(["--neighborhood-mode", "zero-hop"]); - let result = subject.wrap_up_db_externals(&multi_config); + let result = ExternalData::from((&configurator_standard, &multi_config)); let expected = ExternalData::new(DEFAULT_CHAIN, NeighborhoodModeLight::ZeroHop, None); assert_eq!(result, expected) diff --git a/node/src/node_configurator/unprivileged_parse_args_configuration.rs b/node/src/node_configurator/unprivileged_parse_args_configuration.rs index ab30645e5..eee796c22 100644 --- a/node/src/node_configurator/unprivileged_parse_args_configuration.rs +++ b/node/src/node_configurator/unprivileged_parse_args_configuration.rs @@ -482,23 +482,28 @@ fn configure_accountant_config( let suppress_initial_scans = value_m!(multi_config, "scans", String).unwrap_or_else(|| "on".to_string()) == *"off"; + let scan_intervals = process_combined_params( + "scan-intervals", + multi_config, + persist_config, + |str: &str| ScanIntervals::try_from(str), + |pc: &dyn PersistentConfiguration| pc.scan_intervals(), + |pc: &mut dyn PersistentConfiguration, intervals| pc.set_scan_intervals(intervals), + )?; + let payment_thresholds = process_combined_params( + "payment-thresholds", + multi_config, + persist_config, + |str: &str| PaymentThresholds::try_from(str), + |pc: &dyn PersistentConfiguration| pc.payment_thresholds(), + |pc: &mut dyn PersistentConfiguration, curves| pc.set_payment_thresholds(curves), + )?; + + check_payment_thresholds(&payment_thresholds)?; + let accountant_config = AccountantConfig { - scan_intervals: process_combined_params( - "scan-intervals", - multi_config, - persist_config, - |str: &str| ScanIntervals::try_from(str), - |pc: &dyn PersistentConfiguration| pc.scan_intervals(), - |pc: &mut dyn PersistentConfiguration, intervals| pc.set_scan_intervals(intervals), - )?, - payment_thresholds: process_combined_params( - "payment-thresholds", - multi_config, - persist_config, - |str: &str| PaymentThresholds::try_from(str), - |pc: &dyn PersistentConfiguration| pc.payment_thresholds(), - |pc: &mut dyn PersistentConfiguration, curves| pc.set_payment_thresholds(curves), - )?, + scan_intervals, + payment_thresholds, suppress_initial_scans, when_pending_too_long_sec: DEFAULT_PENDING_TOO_LONG_SEC, }; @@ -506,6 +511,25 @@ fn configure_accountant_config( Ok(()) } +fn check_payment_thresholds( + payment_thresholds: &PaymentThresholds, +) -> Result<(), ConfiguratorError> { + if payment_thresholds.debt_threshold_gwei <= payment_thresholds.permanent_debt_allowed_gwei { + let msg = format!( + "Value of DebtThresholdGwei ({}) must be bigger than PermanentDebtAllowedGwei ({})", + payment_thresholds.debt_threshold_gwei, payment_thresholds.permanent_debt_allowed_gwei + ); + return Err(ConfiguratorError::required("payment-thresholds", &msg)); + } + if payment_thresholds.threshold_interval_sec > 10_u64.pow(9) { + return Err(ConfiguratorError::required( + "payment-thresholds", + "Value of ThresholdIntervalSec must not exceed 1,000,000,000 s", + )); + } + Ok(()) +} + fn configure_rate_pack( multi_config: &MultiConfig, persist_config: &mut dyn PersistentConfiguration, @@ -594,13 +618,15 @@ fn is_user_specified(multi_config: &MultiConfig, parameter: &str) -> bool { #[cfg(test)] mod tests { use super::*; + use crate::accountant::ThresholdUtils; use crate::apps::app_node; use crate::blockchain::bip32::Bip32ECKeyProvider; + use crate::database::db_initializer::DbInitializationConfig; use crate::database::db_initializer::{DbInitializer, DbInitializerReal}; - use crate::database::db_migrations::MigratorConfig; use crate::db_config::config_dao::{ConfigDao, ConfigDaoReal}; use crate::db_config::persistent_configuration::PersistentConfigError::NotPresent; use crate::db_config::persistent_configuration::PersistentConfigurationReal; + use crate::sub_lib::accountant::DEFAULT_PAYMENT_THRESHOLDS; use crate::sub_lib::cryptde::{PlainData, PublicKey}; use crate::sub_lib::neighborhood::DEFAULT_RATE_PACK; use crate::sub_lib::utils::make_new_test_multi_config; @@ -1376,7 +1402,7 @@ mod tests { running_test(); let config_dao: Box = Box::new(ConfigDaoReal::new( DbInitializerReal::default() - .initialize(&home_dir.clone(), true, MigratorConfig::test_default()) + .initialize(&home_dir.clone(), DbInitializationConfig::test_default()) .unwrap(), )); let consuming_private_key_text = @@ -1693,7 +1719,7 @@ mod tests { } #[test] - fn unprivileged_parse_args_accountant_config_aggregated_params_command_line_values_different_from_database( + fn unprivileged_parse_args_accountant_config_with_combined_params_from_command_line_different_from_database( ) { running_test(); let set_scan_intervals_params_arc = Arc::new(Mutex::new(vec![])); @@ -1704,7 +1730,7 @@ mod tests { "--scan-intervals", "180|150|130", "--payment-thresholds", - "10000|10000|1000|20000|1000|20000", + "100000|10000|1000|20000|1000|20000", ]; let mut config = BootstrapperConfig::new(); let multi_config = make_simplified_multi_config(args); @@ -1747,7 +1773,7 @@ mod tests { }, payment_thresholds: PaymentThresholds { threshold_interval_sec: 1000, - debt_threshold_gwei: 10000, + debt_threshold_gwei: 100000, payment_grace_period_sec: 1000, maturity_threshold_sec: 10000, permanent_debt_allowed_gwei: 20000, @@ -1762,7 +1788,7 @@ mod tests { let set_payment_thresholds_params = set_payment_thresholds_params_arc.lock().unwrap(); assert_eq!( *set_payment_thresholds_params, - vec!["10000|10000|1000|20000|1000|20000".to_string()] + vec!["100000|10000|1000|20000|1000|20000".to_string()] ) } @@ -1968,6 +1994,94 @@ mod tests { //no prepared results for the setter methods, that is they're uncalled } + #[test] + fn configure_accountant_config_discovers_invalid_payment_thresholds_params_combination_given_from_users_input( + ) { + let multi_config = make_simplified_multi_config([ + "--payment-thresholds", + "19999|10000|1000|20000|1000|20000", + ]); + let mut bootstrapper_config = BootstrapperConfig::new(); + let mut persistent_config = + configure_default_persistent_config(ACCOUNTANT_CONFIG_PARAMS | MAPPING_PROTOCOL) + .set_payment_thresholds_result(Ok(())); + + let result = configure_accountant_config( + &multi_config, + &mut bootstrapper_config, + &mut persistent_config, + ); + + let expected_msg = "Value of DebtThresholdGwei (19999) must be bigger than PermanentDebtAllowedGwei (20000)"; + assert_eq!( + result, + Err(ConfiguratorError::required( + "payment-thresholds", + expected_msg + )) + ) + } + + #[test] + fn check_payment_thresholds_works_for_equal_debt_parameters() { + let mut payment_thresholds = *DEFAULT_PAYMENT_THRESHOLDS; + payment_thresholds.permanent_debt_allowed_gwei = 10000; + payment_thresholds.debt_threshold_gwei = 10000; + + let result = check_payment_thresholds(&payment_thresholds); + + let expected_msg = "Value of DebtThresholdGwei (10000) must be bigger than PermanentDebtAllowedGwei (10000)"; + assert_eq!( + result, + Err(ConfiguratorError::required( + "payment-thresholds", + expected_msg + )) + ) + } + + #[test] + fn check_payment_thresholds_works_for_too_small_debt_threshold() { + let mut payment_thresholds = *DEFAULT_PAYMENT_THRESHOLDS; + payment_thresholds.permanent_debt_allowed_gwei = 10000; + payment_thresholds.debt_threshold_gwei = 9999; + + let result = check_payment_thresholds(&payment_thresholds); + + let expected_msg = "Value of DebtThresholdGwei (9999) must be bigger than PermanentDebtAllowedGwei (10000)"; + assert_eq!( + result, + Err(ConfiguratorError::required( + "payment-thresholds", + expected_msg + )) + ) + } + + #[test] + fn check_payment_thresholds_does_not_permit_threshold_interval_longer_than_1_000_000_000_s() { + //this goes to the furthest extreme where the delta of debt limits is just 1 gwei, which, + //if divided by the slope interval equal or longer 10^9 and rounded, gives 0 + let mut payment_thresholds = *DEFAULT_PAYMENT_THRESHOLDS; + payment_thresholds.permanent_debt_allowed_gwei = 100; + payment_thresholds.debt_threshold_gwei = 101; + payment_thresholds.threshold_interval_sec = 1_000_000_001; + + let result = check_payment_thresholds(&payment_thresholds); + + let expected_msg = "Value of ThresholdIntervalSec must not exceed 1,000,000,000 s"; + assert_eq!( + result, + Err(ConfiguratorError::required( + "payment-thresholds", + expected_msg + )) + ); + payment_thresholds.threshold_interval_sec -= 1; + let last_value_possible = ThresholdUtils::slope(&payment_thresholds); + assert_eq!(last_value_possible, -1) + } + #[test] fn unprivileged_parse_args_with_invalid_consuming_wallet_private_key_reacts_correctly() { running_test(); diff --git a/node/src/sub_lib/accountant.rs b/node/src/sub_lib/accountant.rs index 965a1a828..f6b0ac41a 100644 --- a/node/src/sub_lib/accountant.rs +++ b/node/src/sub_lib/accountant.rs @@ -1,10 +1,13 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. -use crate::accountant::{ReceivedPayments, ReportTransactionReceipts, ScanError, SentPayable}; +use crate::accountant::{ + checked_conversion, Accountant, ReceivedPayments, ReportTransactionReceipts, ScanError, + SentPayables, +}; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::sub_lib::peer_actors::{BindMessage, StartMessage}; use crate::sub_lib::wallet::Wallet; -use actix::Message; use actix::Recipient; +use actix::{Addr, Message}; use lazy_static::lazy_static; use masq_lib::ui_gateway::NodeFromUiMessage; #[cfg(test)] @@ -38,22 +41,18 @@ lazy_static! { //please, alphabetical order #[derive(PartialEq, Eq, Debug, Clone, Copy, Default)] pub struct PaymentThresholds { - pub debt_threshold_gwei: i64, - pub maturity_threshold_sec: i64, - pub payment_grace_period_sec: i64, - pub permanent_debt_allowed_gwei: i64, - pub threshold_interval_sec: i64, - pub unban_below_gwei: i64, + pub debt_threshold_gwei: u64, + pub maturity_threshold_sec: u64, + pub payment_grace_period_sec: u64, + pub permanent_debt_allowed_gwei: u64, + pub threshold_interval_sec: u64, + pub unban_below_gwei: u64, } -//this code is used in tests in Accountant impl PaymentThresholds { pub fn sugg_and_grace(&self, now: i64) -> i64 { - now - self.maturity_threshold_sec - self.payment_grace_period_sec - } - - pub fn sugg_thru_decreasing(&self, now: i64) -> i64 { - self.sugg_and_grace(now) - self.threshold_interval_sec + now - checked_conversion::(self.maturity_threshold_sec) + - checked_conversion::(self.payment_grace_period_sec) } } @@ -72,7 +71,7 @@ pub struct AccountantConfig { pub when_pending_too_long_sec: u64, } -#[derive(Clone)] +#[derive(Clone, PartialEq, Eq)] pub struct AccountantSubs { pub bind: Recipient, pub start: Recipient, @@ -82,7 +81,7 @@ pub struct AccountantSubs { pub report_new_payments: Recipient, pub pending_payable_fingerprint: Recipient, pub report_transaction_receipts: Recipient, - pub report_sent_payments: Recipient, + pub report_sent_payments: Recipient, pub scan_errors: Recipient, pub ui_message_sub: Recipient, } @@ -93,6 +92,18 @@ impl Debug for AccountantSubs { } } +pub trait AccountantSubsFactory { + fn make(&self, addr: &Addr) -> AccountantSubs; +} + +pub struct AccountantSubsFactoryReal {} + +impl AccountantSubsFactory for AccountantSubsFactoryReal { + fn make(&self, addr: &Addr) -> AccountantSubs { + Accountant::make_subs_from(addr) + } +} + // TODO: These four structures all consist of exactly the same five fields. They could be factored out. #[derive(Clone, PartialEq, Eq, Debug, Message)] pub struct ReportRoutingServiceProvidedMessage { @@ -137,8 +148,15 @@ pub struct ExitServiceConsumed { #[derive(Clone, PartialEq, Eq, Debug, Default)] pub struct FinancialStatistics { - pub total_paid_payable: u64, - pub total_paid_receivable: u64, + pub total_paid_payable_wei: u128, + pub total_paid_receivable_wei: u128, +} + +#[derive(PartialEq, Eq, Debug)] +pub enum SignConversionError { + U64(String), + U128(String), + I128(String), } pub trait MessageIdGenerator { @@ -158,10 +176,13 @@ impl MessageIdGenerator for MessageIdGeneratorReal { #[cfg(test)] mod tests { + use crate::accountant::test_utils::AccountantBuilder; + use crate::accountant::{checked_conversion, Accountant}; use crate::sub_lib::accountant::{ - MessageIdGenerator, MessageIdGeneratorReal, PaymentThresholds, ScanIntervals, - DEFAULT_EARNING_WALLET, DEFAULT_PAYMENT_THRESHOLDS, DEFAULT_SCAN_INTERVALS, - MSG_ID_INCREMENTER, TEMPORARY_CONSUMING_WALLET, + AccountantSubsFactory, AccountantSubsFactoryReal, MessageIdGenerator, + MessageIdGeneratorReal, PaymentThresholds, ScanIntervals, DEFAULT_EARNING_WALLET, + DEFAULT_PAYMENT_THRESHOLDS, DEFAULT_SCAN_INTERVALS, MSG_ID_INCREMENTER, + TEMPORARY_CONSUMING_WALLET, }; use crate::sub_lib::wallet::Wallet; use crate::test_utils::recorder::{make_accountant_subs_from_recorder, Recorder}; @@ -173,6 +194,12 @@ mod tests { static MSG_ID_GENERATOR_TEST_GUARD: Mutex<()> = Mutex::new(()); + impl PaymentThresholds { + pub fn sugg_thru_decreasing(&self, now: i64) -> i64 { + self.sugg_and_grace(now) - checked_conversion::(self.threshold_interval_sec) + } + } + #[test] fn constants_have_correct_values() { let default_earning_wallet_expected: Wallet = @@ -210,6 +237,17 @@ mod tests { assert_eq!(format!("{:?}", subject), "AccountantSubs"); } + #[test] + fn accountant_subs_factory_produces_proper_subs() { + let subject = AccountantSubsFactoryReal {}; + let accountant = AccountantBuilder::default().build(); + let addr = accountant.start(); + + let subs = subject.make(&addr); + + assert_eq!(subs, Accountant::make_subs_from(&addr)) + } + #[test] fn msg_id_generator_increments_by_one_with_every_call() { let _guard = MSG_ID_GENERATOR_TEST_GUARD.lock().unwrap(); diff --git a/node/src/sub_lib/combined_parameters.rs b/node/src/sub_lib/combined_parameters.rs index 4d71805bd..a00210dd0 100644 --- a/node/src/sub_lib/combined_parameters.rs +++ b/node/src/sub_lib/combined_parameters.rs @@ -1,7 +1,8 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. use crate::sub_lib::accountant::{PaymentThresholds, ScanIntervals}; -use crate::sub_lib::combined_parameters::CombinedParamsDataTypes::{I64, U64}; +use crate::sub_lib::combined_parameters::CombinedParamsDataTypes::U64; +use crate::sub_lib::combined_parameters::InitializationState::{Initialized, Uninitialized}; use crate::sub_lib::neighborhood::RatePack; use masq_lib::constants::COMBINED_PARAMETERS_DELIMITER; use masq_lib::utils::ExpectValue; @@ -80,17 +81,27 @@ impl CombinedParamsValueRetriever { CombinedParamsValueRetriever::I64(num) => num, CombinedParamsValueRetriever::U128(num) => num, }; - *dynamic - .downcast_ref::() - .unwrap_or_else(|| panic!("expected Some() of {}", std::any::type_name::())) + *dynamic.downcast_ref::().unwrap_or_else(|| { + panic!( + "expected Some() of {} for {}", + std::any::type_name::(), + parameter_name + ) + }) } } #[derive(Debug)] enum CombinedParams { - RatePack(Option), - PaymentThresholds(Option), - ScanIntervals(Option), + RatePack(InitializationState), + PaymentThresholds(InitializationState), + ScanIntervals(InitializationState), +} + +#[derive(Debug)] +enum InitializationState { + Uninitialized, + Initialized(T), } impl CombinedParams { @@ -100,7 +111,7 @@ impl CombinedParams { COMBINED_PARAMETERS_DELIMITER, self.into(), )?; - Ok(self.initiate_objects(parsed_values)) + Ok(self.initialize_objects(parsed_values)) } fn parse_combined_params( @@ -136,12 +147,12 @@ impl CombinedParams { .collect()) } - fn initiate_objects( + fn initialize_objects( &self, parsed_values: HashMap, ) -> Self { match self { - Self::RatePack(None) => Self::RatePack(Some(initiate_struct!( + Self::RatePack(Uninitialized) => Self::RatePack(Initialized(initiate_struct!( RatePack, &parsed_values, "routing_byte_rate", @@ -149,24 +160,28 @@ impl CombinedParams { "exit_byte_rate", "exit_service_rate" ))), - Self::PaymentThresholds(None) => Self::PaymentThresholds(Some(initiate_struct!( - PaymentThresholds, - &parsed_values, - "maturity_threshold_sec", - "payment_grace_period_sec", - "permanent_debt_allowed_gwei", - "debt_threshold_gwei", - "threshold_interval_sec", - "unban_below_gwei" - ))), - Self::ScanIntervals(None) => Self::ScanIntervals(Some(initiate_struct!( - ScanIntervals, - &parsed_values, - Duration::from_secs, - "pending_payable_scan_interval", - "payable_scan_interval", - "receivable_scan_interval" - ))), + Self::PaymentThresholds(Uninitialized) => { + Self::PaymentThresholds(Initialized(initiate_struct!( + PaymentThresholds, + &parsed_values, + "maturity_threshold_sec", + "payment_grace_period_sec", + "permanent_debt_allowed_gwei", + "debt_threshold_gwei", + "threshold_interval_sec", + "unban_below_gwei" + ))) + } + Self::ScanIntervals(Uninitialized) => { + Self::ScanIntervals(Initialized(initiate_struct!( + ScanIntervals, + &parsed_values, + Duration::from_secs, + "pending_payable_scan_interval", + "payable_scan_interval", + "receivable_scan_interval" + ))) + } _ => panic!( "should be called only on uninitialized object, not: {:?}", self @@ -178,21 +193,21 @@ impl CombinedParams { impl From<&CombinedParams> for &[(&str, CombinedParamsDataTypes)] { fn from(params: &CombinedParams) -> &'static [(&'static str, CombinedParamsDataTypes)] { match params { - CombinedParams::RatePack(None) => &[ + CombinedParams::RatePack(Uninitialized) => &[ ("routing_byte_rate", U64), ("routing_service_rate", U64), ("exit_byte_rate", U64), ("exit_service_rate", U64), ], - CombinedParams::PaymentThresholds(None) => &[ - ("debt_threshold_gwei", I64), - ("maturity_threshold_sec", I64), - ("payment_grace_period_sec", I64), - ("permanent_debt_allowed_gwei", I64), - ("threshold_interval_sec", I64), - ("unban_below_gwei", I64), + CombinedParams::PaymentThresholds(Uninitialized) => &[ + ("debt_threshold_gwei", U64), + ("maturity_threshold_sec", U64), + ("payment_grace_period_sec", U64), + ("permanent_debt_allowed_gwei", U64), + ("threshold_interval_sec", U64), + ("unban_below_gwei", U64), ], - CombinedParams::ScanIntervals(None) => &[ + CombinedParams::ScanIntervals(Uninitialized) => &[ ("pending_payable_scan_interval", U64), ("payable_scan_interval", U64), ("receivable_scan_interval", U64), @@ -221,8 +236,8 @@ impl TryFrom<&str> for ScanIntervals { type Error = String; fn try_from(parameters: &str) -> Result { - match CombinedParams::ScanIntervals(None).parse(parameters) { - Ok(CombinedParams::ScanIntervals(Some(scan_intervals))) => Ok(scan_intervals), + match CombinedParams::ScanIntervals(Uninitialized).parse(parameters) { + Ok(CombinedParams::ScanIntervals(Initialized(scan_intervals))) => Ok(scan_intervals), Err(e) => Err(e), _ => unreachable(), } @@ -248,8 +263,8 @@ impl TryFrom<&str> for PaymentThresholds { type Error = String; fn try_from(parameters: &str) -> Result { - match CombinedParams::PaymentThresholds(None).parse(parameters) { - Ok(CombinedParams::PaymentThresholds(Some(payment_thresholds))) => { + match CombinedParams::PaymentThresholds(Uninitialized).parse(parameters) { + Ok(CombinedParams::PaymentThresholds(Initialized(payment_thresholds))) => { Ok(payment_thresholds) } Err(e) => Err(e), @@ -275,8 +290,8 @@ impl TryFrom<&str> for RatePack { type Error = String; fn try_from(parameters: &str) -> Result { - match CombinedParams::RatePack(None).parse(parameters) { - Ok(CombinedParams::RatePack(Some(rate_pack))) => Ok(rate_pack), + match CombinedParams::RatePack(Uninitialized).parse(parameters) { + Ok(CombinedParams::RatePack(Initialized(rate_pack))) => Ok(rate_pack), Err(e) => Err(e), _ => unreachable(), } @@ -371,7 +386,7 @@ mod tests { #[test] fn combined_params_can_be_converted_to_collection_of_typed_parametres() { let rate_pack: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::RatePack(None)).into(); + (&CombinedParams::RatePack(Uninitialized)).into(); assert_eq!( rate_pack, &[ @@ -382,7 +397,7 @@ mod tests { ] ); let scan_interval: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::ScanIntervals(None)).into(); + (&CombinedParams::ScanIntervals(Uninitialized)).into(); assert_eq!( scan_interval, &[ @@ -392,16 +407,16 @@ mod tests { ] ); let payment_thresholds: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::PaymentThresholds(None)).into(); + (&CombinedParams::PaymentThresholds(Uninitialized)).into(); assert_eq!( payment_thresholds, &[ - ("debt_threshold_gwei", I64), - ("maturity_threshold_sec", I64), - ("payment_grace_period_sec", I64), - ("permanent_debt_allowed_gwei", I64), - ("threshold_interval_sec", I64), - ("unban_below_gwei", I64) + ("debt_threshold_gwei", U64), + ("maturity_threshold_sec", U64), + ("payment_grace_period_sec", U64), + ("permanent_debt_allowed_gwei", U64), + ("threshold_interval_sec", U64), + ("unban_below_gwei", U64) ] ); } @@ -410,7 +425,7 @@ mod tests { fn array_type_conversion_should_use_uninitialized_instances_only() { let panic_1 = catch_unwind(|| { let _: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::RatePack(Some(DEFAULT_RATE_PACK))).into(); + (&CombinedParams::RatePack(Initialized(DEFAULT_RATE_PACK))).into(); }) .unwrap_err(); let panic_1_msg = panic_1.downcast_ref::().unwrap(); @@ -418,14 +433,15 @@ mod tests { assert_eq!( panic_1_msg, &format!( - "should be called only on uninitialized object, not: RatePack(Some({:?}))", + "should be called only on uninitialized object, not: RatePack(Initialized({:?}))", DEFAULT_RATE_PACK ) ); let panic_2 = catch_unwind(|| { let _: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::PaymentThresholds(Some(*DEFAULT_PAYMENT_THRESHOLDS))).into(); + (&CombinedParams::PaymentThresholds(Initialized(*DEFAULT_PAYMENT_THRESHOLDS))) + .into(); }) .unwrap_err(); let panic_2_msg = panic_2.downcast_ref::().unwrap(); @@ -433,14 +449,14 @@ mod tests { assert_eq!( panic_2_msg, &format!( - "should be called only on uninitialized object, not: PaymentThresholds(Some({:?}))", + "should be called only on uninitialized object, not: PaymentThresholds(Initialized({:?}))", *DEFAULT_PAYMENT_THRESHOLDS ) ); let panic_3 = catch_unwind(|| { let _: &[(&str, CombinedParamsDataTypes)] = - (&CombinedParams::ScanIntervals(Some(*DEFAULT_SCAN_INTERVALS))).into(); + (&CombinedParams::ScanIntervals(Initialized(*DEFAULT_SCAN_INTERVALS))).into(); }) .unwrap_err(); let panic_3_msg = panic_3.downcast_ref::().unwrap(); @@ -448,7 +464,7 @@ mod tests { assert_eq!( panic_3_msg, &format!( - "should be called only on uninitialized object, not: ScanIntervals(Some({:?}))", + "should be called only on uninitialized object, not: ScanIntervals(Initialized({:?}))", *DEFAULT_SCAN_INTERVALS ) ); @@ -457,7 +473,8 @@ mod tests { #[test] fn initiate_objects_should_use_uninitialized_instances_only() { let panic_1 = catch_unwind(|| { - (&CombinedParams::RatePack(Some(DEFAULT_RATE_PACK))).initiate_objects(HashMap::new()); + (&CombinedParams::RatePack(Initialized(DEFAULT_RATE_PACK))) + .initialize_objects(HashMap::new()); }) .unwrap_err(); let panic_1_msg = panic_1.downcast_ref::().unwrap(); @@ -465,14 +482,14 @@ mod tests { assert_eq!( panic_1_msg, &format!( - "should be called only on uninitialized object, not: RatePack(Some({:?}))", + "should be called only on uninitialized object, not: RatePack(Initialized({:?}))", DEFAULT_RATE_PACK ) ); let panic_2 = catch_unwind(|| { - (&CombinedParams::PaymentThresholds(Some(*DEFAULT_PAYMENT_THRESHOLDS))) - .initiate_objects(HashMap::new()); + (&CombinedParams::PaymentThresholds(Initialized(*DEFAULT_PAYMENT_THRESHOLDS))) + .initialize_objects(HashMap::new()); }) .unwrap_err(); let panic_2_msg = panic_2.downcast_ref::().unwrap(); @@ -480,14 +497,14 @@ mod tests { assert_eq!( panic_2_msg, &format!( - "should be called only on uninitialized object, not: PaymentThresholds(Some({:?}))", + "should be called only on uninitialized object, not: PaymentThresholds(Initialized({:?}))", *DEFAULT_PAYMENT_THRESHOLDS ) ); let panic_3 = catch_unwind(|| { - (&CombinedParams::ScanIntervals(Some(*DEFAULT_SCAN_INTERVALS))) - .initiate_objects(HashMap::new()); + (&CombinedParams::ScanIntervals(Initialized(*DEFAULT_SCAN_INTERVALS))) + .initialize_objects(HashMap::new()); }) .unwrap_err(); let panic_3_msg = panic_3.downcast_ref::().unwrap(); @@ -495,7 +512,7 @@ mod tests { assert_eq!( panic_3_msg, &format!( - "should be called only on uninitialized object, not: ScanIntervals(Some({:?}))", + "should be called only on uninitialized object, not: ScanIntervals(Initialized({:?}))", *DEFAULT_SCAN_INTERVALS ) ); diff --git a/node/src/sub_lib/neighborhood.rs b/node/src/sub_lib/neighborhood.rs index d412ccfbc..eecb3bb4f 100644 --- a/node/src/sub_lib/neighborhood.rs +++ b/node/src/sub_lib/neighborhood.rs @@ -37,10 +37,10 @@ use std::time::Duration; const ASK_ABOUT_GOSSIP_INTERVAL: Duration = Duration::from_secs(10); pub const DEFAULT_RATE_PACK: RatePack = RatePack { - routing_byte_rate: 1, - routing_service_rate: 10, - exit_byte_rate: 2, - exit_service_rate: 20, + routing_byte_rate: 172_300_000, + routing_service_rate: 1_723_000_000, + exit_byte_rate: 344_600_000, + exit_service_rate: 3_446_000_000, }; pub const ZERO_RATE_PACK: RatePack = RatePack { @@ -330,6 +330,13 @@ enum DescriptorParsingError<'a> { impl Display for DescriptorParsingError<'_> { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + fn only_user_intended() -> String { + CHAINS + .iter() + .map(|record| record.literal_identifier) + .filter(|identifier| *identifier != "dev") + .join("', '") + } match self{ Self::CentralDelimiterProbablyMissing(descriptor) => write!(f, "Delimiter '@' probably missing. Should be 'masq://:@', not '{}'", descriptor), @@ -343,8 +350,8 @@ impl Display for DescriptorParsingError<'_> { write!(f,"Prefix or more missing. Should be 'masq://:@', not '{}'",descriptor), Self::WrongChainIdentifier(identifier) => write!(f, "Chain identifier '{}' is not valid; possible values are '{}' while formatted as 'masq://:@'", - identifier, - CHAINS.iter().map(|record|record.literal_identifier).filter(|identifier|*identifier != "dev").join("', '") + identifier, only_user_intended() + ) } } @@ -405,15 +412,6 @@ impl NodeQueryResponseMetadata { } } -//TODO probably dead code? -#[derive(Clone, Debug, Message, PartialEq, Eq)] -pub struct BootstrapNeighborhoodNowMessage {} - -#[derive(Clone, Debug, Message, PartialEq, Eq)] -pub struct NeighborhoodDotGraphRequest { - pub client_id: u64, -} - #[derive(Clone, Debug, PartialEq, Eq)] pub enum NodeQueryMessage { IpAddress(IpAddr), @@ -566,10 +564,10 @@ mod tests { assert_eq!( DEFAULT_RATE_PACK, RatePack { - routing_byte_rate: 1, - routing_service_rate: 10, - exit_byte_rate: 2, - exit_service_rate: 20, + routing_byte_rate: 172_300_000, + routing_service_rate: 1_723_000_000, + exit_byte_rate: 344_600_000, + exit_service_rate: 3_446_000_000, } ); assert_eq!( diff --git a/node/src/test_utils/database_utils.rs b/node/src/test_utils/database_utils.rs index 7d72f2562..43594962c 100644 --- a/node/src/test_utils/database_utils.rs +++ b/node/src/test_utils/database_utils.rs @@ -2,16 +2,19 @@ #![cfg(test)] +use crate::accountant::dao_utils::VigilantRusqliteFlatten; use crate::database::connection_wrapper::ConnectionWrapper; +use crate::database::db_initializer::ExternalData; use crate::database::db_migrations::DbMigrator; -use itertools::Itertools; use masq_lib::logger::Logger; -use rusqlite::Connection; +use masq_lib::test_utils::utils::TEST_DEFAULT_CHAIN; +use masq_lib::utils::NeighborhoodModeLight; +use rusqlite::{Connection, Error}; use std::cell::RefCell; -use std::collections::HashSet; use std::env::current_dir; use std::fs::{remove_file, File}; use std::io::Read; +use std::iter::once; use std::path::PathBuf; use std::sync::{Arc, Mutex}; @@ -99,82 +102,124 @@ pub fn retrieve_config_row(conn: &dyn ConnectionWrapper, name: &str) -> (Option< }) } -pub fn query_specific_schema_information( - conn: &dyn ConnectionWrapper, - query_object: &str, -) -> Vec { - let mut table_stm = conn - .prepare(&format!( - "SELECT sql FROM sqlite_master WHERE type='{}'", - query_object - )) - .unwrap(); - table_stm - .query_map([], |row| Ok(row.get::>(0).unwrap())) - .unwrap() - .flatten() - .flatten() - .collect() +pub fn assert_table_does_not_exist(conn: &dyn ConnectionWrapper, table_name: &str) { + let error_stm = conn + .prepare(&format!("select * from {}", table_name)) + .unwrap_err(); + let error_msg = match error_stm { + Error::SqliteFailure(_, Some(msg)) => msg, + x => panic!("we expected SqliteFailure but we got: {:?}", x), + }; + assert_eq!(error_msg, format!("no such table: {}", table_name)) } -pub fn assert_create_table_statement_contains_all_important_parts( +pub fn assert_create_table_stm_contains_all_parts( conn: &dyn ConnectionWrapper, table_name: &str, - expected_sql_chopped: &[&[&str]], + expected_sql_chopped: ExpectedLinesOfSQLChoppedIntoWords, ) { - assert_sql_statements_contain_important_parts( + assert_sql_lines_contain_parts_exhaustive( parse_sql_to_pieces(&fetch_table_sql(conn, table_name)), expected_sql_chopped, ) } -pub fn assert_index_statement_is_coupled_with_right_parameter( +pub fn assert_index_stm_is_coupled_with_right_parameter( conn: &dyn ConnectionWrapper, index_name: &str, - expected_sql_chopped: &[&[&str]], + expected_sql_chopped: ExpectedLinesOfSQLChoppedIntoWords, ) { - assert_sql_statements_contain_important_parts( + assert_sql_lines_contain_parts_exhaustive( parse_sql_to_pieces(&fetch_index_sql(conn, index_name)), expected_sql_chopped, ) } pub fn assert_no_index_exists_for_table(conn: &dyn ConnectionWrapper, table_name: &str) { - let found_indexes = query_specific_schema_information(conn, "index"); - let isolated_table_name = format!(" {} ", table_name); - found_indexes.iter().for_each(|index_stm| { - assert!( - !index_stm.contains(&isolated_table_name), - "unexpected index on this table: {}", - index_stm - ) - }) + let table_name = isolated_name(table_name); + query_specific_schema_information(conn, "index") + .iter() + .for_each(|index_stm| { + assert!( + !index_stm.contains(&table_name), + "unexpected index on this table: {}", + index_stm + ) + }) } -fn assert_sql_statements_contain_important_parts( - actual: Vec>, - expected: &[&[&str]], +pub fn assert_table_created_as_strict(conn: &dyn ConnectionWrapper, table_name: &str) { + let payable_creation_sql = fetch_table_sql(conn, table_name); + let last_word_reversed = payable_creation_sql + .chars() + .rev() + .skip_while(|char| char.is_whitespace()) + .take_while(|char| !char.is_whitespace()) + .collect::(); + let last_word = last_word_reversed.chars().rev().collect::(); + assert_eq!( + last_word, "strict", + "we expected the 'strict' key word but got: {}", + last_word + ) +} + +fn zippered<'a>( + actual: &'a Vec>, + expected: ExpectedLinesOfSQLChoppedIntoWords, +) -> Vec<(SQLLinesChoppedIntoWords, &'a Vec)> { + once(prepare_expected_vectors_of_words_including_sorting( + expected, + )) + .cycle() + .zip(actual.iter()) + .collect() +} + +fn assert_sql_lines_contain_parts_exhaustive( + actual: SQLLinesChoppedIntoWords, + expected: ExpectedLinesOfSQLChoppedIntoWords, ) { - let mut prepared_expected = expected.into_iter().map(|slice_of_strs| { - HashSet::from_iter(slice_of_strs.into_iter().map(|str| str.to_string())) + zippered(&actual, expected) + .iter() + .for_each(|(left, right)| contains_particular_list_of_key_words(left, right)) +} + +fn contains_particular_list_of_key_words( + expected_words: &SQLLinesChoppedIntoWords, + single_actual_line_of_words: &[String], +) { + let mut found = 0_u16; + expected_words.iter().for_each(|vec_of_words_expected| { + if single_actual_line_of_words == vec_of_words_expected { + found += 1 + } }); - actual.into_iter().for_each(|hash_set| { - assert!( - prepared_expected - .find(|hash_set_expected| hash_set - .symmetric_difference(&hash_set_expected) - .collect_vec() - .is_empty()) - .is_some(), - "part of the fetched statement (one line) that cannot \ - be found in the template (key words unsorted): {:?}", - hash_set - ) - }) + assert_eq!(found,1, "We found {} occurrences of the searched line in the tested sql although only a one is considered correct", found) } +fn prepare_expected_vectors_of_words_including_sorting( + expected: ExpectedLinesOfSQLChoppedIntoWords, +) -> SQLLinesChoppedIntoWords { + expected + .into_iter() + .map(|slice_of_strs| { + let mut one_line = slice_of_strs + .into_iter() + .map(|word| word.to_string()) + .collect::>(); + one_line.sort(); + one_line + }) + .collect() +} + +type SQLLinesChoppedIntoWords = Vec>; + +type ExpectedLinesOfSQLChoppedIntoWords<'a> = &'a [&'a [&'a str]]; + //prepares collections of isolated key words from a column declaration, by lines -fn parse_sql_to_pieces(sql: &str) -> Vec> { +fn parse_sql_to_pieces(sql: &str) -> SQLLinesChoppedIntoWords { let body: String = sql .chars() .skip_while(|char| char != &'(') @@ -183,30 +228,52 @@ fn parse_sql_to_pieces(sql: &str) -> Vec> { .collect(); let lines = body.split(','); lines - .map(|line| { - HashSet::from_iter( - line.split(|char: char| char.is_whitespace()) - .filter(|chunk| !chunk.is_empty()) - .map(|chunk| chunk.to_string()), - ) + .map(|sql_line| { + let mut vec_of_words = sql_line + .split(|char: char| char.is_whitespace()) + .filter(|chunk| !chunk.is_empty()) + .map(|chunk| chunk.to_string()) + .collect::>(); + vec_of_words.sort(); + vec_of_words }) .collect() } fn fetch_table_sql(conn: &dyn ConnectionWrapper, specific_table: &str) -> String { let found_table_sqls = query_specific_schema_information(conn, "table"); - let specific_table_isolated = format!(" {} ", specific_table); - select_desired_sql_element(found_table_sqls, &specific_table_isolated) + select_desired_sql_element(found_table_sqls, &isolated_name(specific_table)) } fn fetch_index_sql(conn: &dyn ConnectionWrapper, index_name: &str) -> String { let found_indexes = query_specific_schema_information(conn, "index"); - let index_name_isolated = format!(" {} ", index_name); - select_desired_sql_element(found_indexes, &index_name_isolated) + select_desired_sql_element(found_indexes, &isolated_name(index_name)) +} + +fn isolated_name(name: &str) -> String { + format!(" {} ", name) +} + +fn query_specific_schema_information( + conn: &dyn ConnectionWrapper, + query_object: &str, +) -> Vec { + let mut table_stm = conn + .prepare(&format!( + "SELECT sql FROM sqlite_master WHERE type='{}'", + query_object + )) + .unwrap(); + table_stm + .query_map([], |row| Ok(row.get::>(0).unwrap())) + .unwrap() + .vigilant_flatten() + .flatten() + .collect() } fn select_desired_sql_element(found_elements: Vec, searched_element_name: &str) -> String { - let mut wanted_element_lowercase: Vec = found_elements + let mut searched_element: Vec = found_elements .into_iter() .flat_map(|element| { let introducing_part: String = @@ -218,8 +285,19 @@ fn select_desired_sql_element(found_elements: Vec, searched_element_name } }) .collect(); - if wanted_element_lowercase.len() != 1 { - panic!("search failed, we should've found any matching element") + if searched_element.len() != 1 { + panic!( + "search failed, we should've found one matching element but got {}", + searched_element.len() + ) + } + searched_element.remove(0) +} + +pub fn make_external_data() -> ExternalData { + ExternalData { + chain: TEST_DEFAULT_CHAIN, + neighborhood_mode: NeighborhoodModeLight::Standard, + db_password_opt: None, } - wanted_element_lowercase.remove(0) } diff --git a/node/src/test_utils/mod.rs b/node/src/test_utils/mod.rs index 256f66985..b9c7b4849 100644 --- a/node/src/test_utils/mod.rs +++ b/node/src/test_utils/mod.rs @@ -913,16 +913,40 @@ pub mod unshared_test_utils { } } + //to be put among the methods in your trait #[macro_export] - macro_rules! arbitrary_id_stamp { + macro_rules! arbitrary_id_stamp_in_trait { () => { #[cfg(test)] fn arbitrary_id_stamp(&self) -> ArbitraryIdStamp { - //no necessity to implemented it for all impls of the trait this becomes a member of + //no necessity to implement it for all impls of the trait this is to be a member of intentionally_blank!() } }; } + + //the following macros might be handy but your object must contain exactly this field: + //arbitrary_id_stamp_opt: RefCell> + + #[macro_export] + macro_rules! arbitrary_id_stamp { + () => { + fn arbitrary_id_stamp(&self) -> ArbitraryIdStamp { + *self.arbitrary_id_stamp_opt.borrow().as_ref().unwrap() + } + }; + } + + #[macro_export] + macro_rules! set_arbitrary_id_stamp { + () => { + pub fn set_arbitrary_id_stamp(&self) -> ArbitraryIdStamp { + let id_stamp = ArbitraryIdStamp::new(); + self.arbitrary_id_stamp_opt.borrow_mut().replace(id_stamp); + id_stamp + } + }; + } } #[cfg(test)] diff --git a/node/src/test_utils/persistent_configuration_mock.rs b/node/src/test_utils/persistent_configuration_mock.rs index f465f2ac6..36bbe613c 100644 --- a/node/src/test_utils/persistent_configuration_mock.rs +++ b/node/src/test_utils/persistent_configuration_mock.rs @@ -7,6 +7,7 @@ use crate::sub_lib::accountant::{PaymentThresholds, ScanIntervals}; use crate::sub_lib::neighborhood::{NodeDescriptor, RatePack}; use crate::sub_lib::wallet::Wallet; use crate::test_utils::unshared_test_utils::ArbitraryIdStamp; +use crate::{arbitrary_id_stamp, set_arbitrary_id_stamp}; use masq_lib::utils::AutomapProtocol; use masq_lib::utils::NeighborhoodModeLight; use std::cell::RefCell; @@ -64,7 +65,7 @@ pub struct PersistentConfigurationMock { scan_intervals_results: RefCell>>, set_scan_intervals_params: Arc>>, set_scan_intervals_results: RefCell>>, - arbitrary_id_stamp_opt: Option, + arbitrary_id_stamp_opt: RefCell>, } impl PersistentConfiguration for PersistentConfigurationMock { @@ -266,9 +267,7 @@ impl PersistentConfiguration for PersistentConfigurationMock { self.set_scan_intervals_results.borrow_mut().remove(0) } - fn arbitrary_id_stamp(&self) -> ArbitraryIdStamp { - *self.arbitrary_id_stamp_opt.as_ref().unwrap() - } + arbitrary_id_stamp!(); } impl PersistentConfigurationMock { @@ -613,10 +612,7 @@ impl PersistentConfigurationMock { self } - pub fn set_arbitrary_id_stamp(mut self, stamp: ArbitraryIdStamp) -> Self { - self.arbitrary_id_stamp_opt = Some(stamp); - self - } + set_arbitrary_id_stamp!(); fn result_from(results: &RefCell>) -> T { let mut borrowed = results.borrow_mut(); diff --git a/node/src/test_utils/recorder.rs b/node/src/test_utils/recorder.rs index 0091ef7f4..2498c1d59 100644 --- a/node/src/test_utils/recorder.rs +++ b/node/src/test_utils/recorder.rs @@ -3,7 +3,7 @@ use crate::accountant::ReportTransactionReceipts; use crate::accountant::{ ReceivedPayments, RequestTransactionReceipts, ScanError, ScanForPayables, - ScanForPendingPayables, ScanForReceivables, SentPayable, + ScanForPendingPayables, ScanForReceivables, SentPayables, }; use crate::blockchain::blockchain_bridge::PendingPayableFingerprint; use crate::blockchain::blockchain_bridge::RetrieveTransactions; @@ -23,6 +23,7 @@ use crate::sub_lib::dispatcher::{DispatcherSubs, StreamShutdownMsg}; use crate::sub_lib::hopper::IncipientCoresPackage; use crate::sub_lib::hopper::{ExpiredCoresPackage, NoLookupIncipientCoresPackage}; use crate::sub_lib::hopper::{HopperSubs, MessageType}; +use crate::sub_lib::neighborhood::ConnectionProgressMessage; use crate::sub_lib::neighborhood::NeighborhoodSubs; use crate::sub_lib::neighborhood::NodeQueryMessage; use crate::sub_lib::neighborhood::NodeQueryResponseMetadata; @@ -30,7 +31,6 @@ use crate::sub_lib::neighborhood::NodeRecordMetadataMessage; use crate::sub_lib::neighborhood::RemoveNeighborMessage; use crate::sub_lib::neighborhood::RouteQueryMessage; use crate::sub_lib::neighborhood::RouteQueryResponse; -use crate::sub_lib::neighborhood::{ConnectionProgressMessage, NeighborhoodDotGraphRequest}; use crate::sub_lib::neighborhood::{DispatcherNodeQueryMessage, GossipFailure_0v1}; use crate::sub_lib::peer_actors::PeerActors; use crate::sub_lib::peer_actors::{BindMessage, NewPublicIp, StartMessage}; @@ -126,7 +126,6 @@ recorder_message_handler!(ExpiredCoresPackage); recorder_message_handler!(InboundClientData); recorder_message_handler!(InboundServerData); recorder_message_handler!(IncipientCoresPackage); -recorder_message_handler!(NeighborhoodDotGraphRequest); recorder_message_handler!(NewPasswordMessage); recorder_message_handler!(NewPublicIp); recorder_message_handler!(NodeFromUiMessage); @@ -141,7 +140,7 @@ recorder_message_handler!(ReportServicesConsumedMessage); recorder_message_handler!(ReportExitServiceProvidedMessage); recorder_message_handler!(ReportRoutingServiceProvidedMessage); recorder_message_handler!(ScanError); -recorder_message_handler!(SentPayable); +recorder_message_handler!(SentPayables); recorder_message_handler!(SetConsumingWalletMessage); recorder_message_handler!(SetDbPasswordMsg); recorder_message_handler!(SetGasPriceMsg); @@ -417,7 +416,7 @@ pub fn make_accountant_subs_from_recorder(addr: &Addr) -> AccountantSu report_new_payments: recipient!(addr, ReceivedPayments), pending_payable_fingerprint: recipient!(addr, PendingPayableFingerprint), report_transaction_receipts: recipient!(addr, ReportTransactionReceipts), - report_sent_payments: recipient!(addr, SentPayable), + report_sent_payments: recipient!(addr, SentPayables), scan_errors: recipient!(addr, ScanError), ui_message_sub: recipient!(addr, NodeFromUiMessage), } diff --git a/node/tests/configuration_mode_test.rs b/node/tests/configuration_mode_test.rs index bca056aa4..6294a7a78 100644 --- a/node/tests/configuration_mode_test.rs +++ b/node/tests/configuration_mode_test.rs @@ -73,7 +73,7 @@ fn dump_configuration_and_no_preexisting_database_integration() { let stderr = String::from_utf8_lossy(&output.stderr); assert_string_contains(stderr.as_ref(), "Could not find database at:"); assert_string_contains(stderr.as_ref(), - "Would be created when the Node firstly operates. Running --dump-config before has no effect" + "It is created when the Node operates the first time. Running --dump-config before that has no effect" ) } }; diff --git a/node/tests/contract_test.rs b/node/tests/contract_test.rs index 31ee00229..684240d0a 100644 --- a/node/tests/contract_test.rs +++ b/node/tests/contract_test.rs @@ -3,19 +3,20 @@ use ethabi; use futures::Future; use masq_lib::blockchains::chains::Chain; +use masq_lib::constants::MASQ_TOTAL_SUPPLY; use web3; -use web3::contract::Options; +use web3::contract::{Contract, Options}; +use web3::transports::Http; +use web3::types::U256; -fn assert_contract_body( +fn assert_contract_existence( blockchain_service_url: &str, chain: &Chain, expected_token_name: &str, expected_decimals: u32, ) -> Result<(), ()> { eprintln!("Starting a new attempt with: '{}'", blockchain_service_url); - let (_event_loop, transport) = web3::transports::Http::new(blockchain_service_url).unwrap(); - let web3 = web3::Web3::new(transport); - let address = chain.rec().contract; + let min_abi_json = r#"[{ "constant": true, "inputs": [], @@ -44,8 +45,9 @@ fn assert_contract_body( "stateMutability": "view", "type": "function" }]"#; - let abi = ethabi::Contract::load(min_abi_json.as_bytes()).unwrap(); - let contract = web3::contract::Contract::new(web3.eth(), address, abi); + + let (_event_loop, transport) = web3::transports::Http::new(blockchain_service_url).unwrap(); + let contract = create_contract_interface(transport, chain, min_abi_json); let token_name: String = match contract .query("name", (), None, Options::default(), None) .wait() @@ -79,22 +81,25 @@ fn assert_contract_body( } } -fn assert_contract( - blockchain_urls: Vec<&str>, - chain: &Chain, - expected_token_name: &str, - expected_decimals: u32, -) { - if !blockchain_urls.iter().fold(false, |acc, url| { - match ( - acc, - assert_contract_body(url, chain, expected_token_name, expected_decimals), - ) { +fn create_contract_interface(transport: Http, chain: &Chain, min_abi_json: &str) -> Contract { + let web3 = web3::Web3::new(transport); + let address = chain.rec().contract; + let abi = ethabi::Contract::load(min_abi_json.as_bytes()).unwrap(); + web3::contract::Contract::new(web3.eth(), address, abi) +} + +fn assert_contract<'a, F>(blockchain_urls: Vec<&'static str>, chain: &'a Chain, test_performer: F) +where + F: FnOnce(&'static str, &'a Chain) -> Result<(), ()> + Copy, +{ + if !blockchain_urls + .iter() + .fold(false, |acc, url| match (acc, test_performer(url, chain)) { (true, _) => true, (false, Ok(_)) => true, (false, Err(_)) => false, - } - }) { + }) + { panic!("Test failed on all blockchain services") } } @@ -108,7 +113,8 @@ fn masq_erc20_contract_exists_on_polygon_mumbai_integration() { ]; let chain = Chain::PolyMumbai; - assert_contract(blockchain_urls, &chain, "tMASQ", 18) + let assertion_body = |url, chain| assert_contract_existence(url, chain, "tMASQ", 18); + assert_contract(blockchain_urls, &chain, assertion_body) } #[test] @@ -121,7 +127,8 @@ fn masq_erc20_contract_exists_on_polygon_mainnet_integration() { ]; let chain = Chain::PolyMainnet; - assert_contract(blockchain_urls, &chain, "MASQ (PoS)", 18) + let assertion_body = |url, chain| assert_contract_existence(url, chain, "MASQ (PoS)", 18); + assert_contract(blockchain_urls, &chain, assertion_body) } #[test] @@ -129,5 +136,55 @@ fn masq_erc20_contract_exists_on_ethereum_mainnet_integration() { let blockchain_urls = vec!["https://mainnet.infura.io/v3/0ead23143b174f6983c76f69ddcf4026"]; let chain = Chain::EthMainnet; - assert_contract(blockchain_urls, &chain, "MASQ", 18) + let assertion_body = |url, chain| assert_contract_existence(url, chain, "MASQ", 18); + assert_contract(blockchain_urls, &chain, assertion_body) +} + +fn assert_total_supply( + blockchain_service_url: &str, + chain: &Chain, + expected_total_supply: u64, +) -> Result<(), ()> { + let min_abi_json = r#"[{ + "constant": true, + "inputs": [], + "name": "totalSupply", + "outputs": [ + { + "internalType": "uint256", + "name": "", + "type": "uint256" + } + ], + "payable": false, + "stateMutability": "view", + "type": "function" + }]"#; + + let (_event_loop, transport) = web3::transports::Http::new(blockchain_service_url).unwrap(); + let contract = create_contract_interface(transport, chain, min_abi_json); + let total_supply: U256 = match contract + .query("totalSupply", (), None, Options::default(), None) + .wait() + { + Ok(ts) => ts, + Err(e) => { + eprintln!("Total supply query failed due to: {:?}", e); + return Err(()); + } + }; + assert_eq!( + total_supply, + U256::from(expected_total_supply) * U256::from(10_u64.pow(18)) + ); + Ok(()) +} + +#[test] +fn max_token_supply_matches_corresponding_constant_integration() { + let blockchain_urls = vec!["https://mainnet.infura.io/v3/0ead23143b174f6983c76f69ddcf4026"]; + let chain = Chain::EthMainnet; + + let assertion_body = |url, chain| assert_total_supply(url, chain, MASQ_TOTAL_SUPPLY); + assert_contract(blockchain_urls, &chain, assertion_body) } diff --git a/node/tests/dns_round_trip_test.rs b/node/tests/dns_round_trip_test.rs index 2375c614d..c3c82c297 100644 --- a/node/tests/dns_round_trip_test.rs +++ b/node/tests/dns_round_trip_test.rs @@ -1,4 +1,5 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + pub mod utils; use node_lib::entry_dns::packet_facade::PacketFacade; use serial_test_derive::serial; diff --git a/node/tests/financials_test.rs b/node/tests/financials_test.rs new file mode 100644 index 000000000..f37949a08 --- /dev/null +++ b/node/tests/financials_test.rs @@ -0,0 +1,100 @@ +// Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. + +pub mod utils; + +use crate::utils::{make_conn, CommandConfig}; +use masq_lib::messages::{ + TopRecordsConfig, TopRecordsOrdering, UiFinancialsRequest, UiFinancialsResponse, + UiShutdownRequest, NODE_UI_PROTOCOL, +}; +use masq_lib::test_utils::ui_connection::UiConnection; +use masq_lib::test_utils::utils::ensure_node_home_directory_exists; +use masq_lib::utils::find_free_port; +use node_lib::accountant::dao_utils::{from_time_t, to_time_t}; +use node_lib::accountant::gwei_to_wei; +use node_lib::accountant::payable_dao::{PayableDao, PayableDaoReal}; +use node_lib::accountant::receivable_dao::{ReceivableDao, ReceivableDaoReal}; +use node_lib::test_utils::make_wallet; +use std::time::SystemTime; +use utils::MASQNode; + +#[test] +fn financials_command_retrieves_payable_and_receivable_records() { + fdlimit::raise_fd_limit(); + let port = find_free_port(); + let home_dir = ensure_node_home_directory_exists( + "financials_test", + "financials_command_retrieves_payable_and_receivable_records", + ); + let now = SystemTime::now(); + let timestamp_payable = from_time_t(to_time_t(now) - 678); + let timestamp_receivable_1 = from_time_t(to_time_t(now) - 10000); + let timestamp_receivable_2 = from_time_t(to_time_t(now) - 1111); + let wallet_payable = make_wallet("efef"); + let wallet_receivable_1 = make_wallet("abcde"); + let wallet_receivable_2 = make_wallet("ccccc"); + let amount_payable = gwei_to_wei(45678357_u64); + let amount_receivable_1 = gwei_to_wei(9000_u64); + let amount_receivable_2 = gwei_to_wei(345678_u64); + PayableDaoReal::new(make_conn(&home_dir)) + .more_money_payable(timestamp_payable, &wallet_payable, amount_payable) + .unwrap(); + let receivable_dao = ReceivableDaoReal::new(make_conn(&home_dir)); + receivable_dao + .more_money_receivable( + timestamp_receivable_1, + &wallet_receivable_1, + amount_receivable_1, + ) + .unwrap(); + receivable_dao + .more_money_receivable( + timestamp_receivable_2, + &wallet_receivable_2, + amount_receivable_2, + ) + .unwrap(); + let mut node = MASQNode::start_standard( + "financials_command_retrieves_payable_and_receivable_records", + Some(CommandConfig::new().pair("--ui-port", &port.to_string())), + false, + true, + false, + true, + ); + let financials_request = UiFinancialsRequest { + stats_required: false, + top_records_opt: Some(TopRecordsConfig { + count: 10, + ordered_by: TopRecordsOrdering::Balance, + }), + custom_queries_opt: None, + }; + let mut client = UiConnection::new(port, NODE_UI_PROTOCOL); + let before = SystemTime::now(); + + client.send(financials_request); + let response: UiFinancialsResponse = client.skip_until_received().unwrap(); + + let after = SystemTime::now(); + assert_eq!(response.stats_opt, None); + let query_results = response.query_results_opt.unwrap(); + let payable = query_results.payable_opt.unwrap(); + let receivable = query_results.receivable_opt.unwrap(); + assert_eq!(payable[0].wallet, wallet_payable.to_string()); + assert_eq!(payable[0].balance_gwei, 45678357_u64); + assert_eq!(payable[0].pending_payable_hash_opt, None); + assert_eq!(receivable[0].wallet, wallet_receivable_1.to_string()); + assert_eq!(receivable[0].balance_gwei, 9000_i64); + assert_eq!(receivable[1].wallet, wallet_receivable_2.to_string()); + assert_eq!(receivable[1].balance_gwei, 345678_i64); + let act_period = after.duration_since(before).unwrap().as_secs(); + let age_payable = payable[0].age_s; + assert!(age_payable >= 678 && age_payable <= (age_payable + act_period)); + let age_receivable_1 = receivable[0].age_s; + assert!(age_receivable_1 >= 10000 && age_receivable_1 <= (age_receivable_1 + act_period)); + let age_receivable_2 = receivable[1].age_s; + assert!(age_receivable_2 >= 1111 && age_receivable_2 <= (age_receivable_2 + act_period)); + client.send(UiShutdownRequest {}); + node.wait_for_exit(); +} diff --git a/node/tests/http_through_node_test.rs b/node/tests/http_through_node_test.rs index c016a40dc..f49cb25b9 100644 --- a/node/tests/http_through_node_test.rs +++ b/node/tests/http_through_node_test.rs @@ -1,6 +1,5 @@ // Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved. -#[cfg(test)] pub mod utils; use node_lib::test_utils::read_until_timeout; diff --git a/node/tests/initialization_test.rs b/node/tests/initialization_test.rs index 1eb8f6b7e..a7eaef02e 100644 --- a/node/tests/initialization_test.rs +++ b/node/tests/initialization_test.rs @@ -79,7 +79,11 @@ fn initialization_sequence_integration() { ("data-directory", Some(&data_directory.to_str().unwrap())), ])) .unwrap(); - let financials_request = UiFinancialsRequest {}; + let financials_request = UiFinancialsRequest { + stats_required: true, + top_records_opt: None, + custom_queries_opt: None, + }; let context_id = 1234; // diff --git a/node/tests/ui_gateway_test.rs b/node/tests/ui_gateway_test.rs index 3c60787d9..4dfbadfba 100644 --- a/node/tests/ui_gateway_test.rs +++ b/node/tests/ui_gateway_test.rs @@ -5,19 +5,13 @@ pub mod utils; use crate::utils::MASQNode; use masq_lib::messages::SerializableLogLevel::Warn; use masq_lib::messages::{ - UiChangePasswordRequest, UiFinancialsRequest, UiFinancialsResponse, UiLogBroadcast, UiRedirect, - UiSetupRequest, UiSetupResponse, UiShutdownRequest, UiStartOrder, UiStartResponse, + UiChangePasswordRequest, UiCheckPasswordRequest, UiCheckPasswordResponse, UiLogBroadcast, + UiRedirect, UiSetupRequest, UiSetupResponse, UiShutdownRequest, UiStartOrder, UiStartResponse, UiWalletAddressesRequest, NODE_UI_PROTOCOL, }; use masq_lib::test_utils::ui_connection::UiConnection; use masq_lib::test_utils::utils::ensure_node_home_directory_exists; use masq_lib::utils::find_free_port; -use node_lib::accountant::payable_dao::{PayableDao, PayableDaoReal}; -use node_lib::accountant::receivable_dao::{ReceivableDao, ReceivableDaoReal}; -use node_lib::database::db_initializer::{DbInitializer, DbInitializerReal}; -use node_lib::database::db_migrations::MigratorConfig; -use node_lib::test_utils::make_wallet; -use std::time::SystemTime; use utils::CommandConfig; #[test] @@ -28,17 +22,6 @@ fn ui_requests_something_and_gets_corresponding_response() { "ui_gateway_test", "ui_requests_something_and_gets_corresponding_response", ); - let make_conn = || { - DbInitializerReal::default() - .initialize(&home_dir, true, MigratorConfig::panic_on_migration()) - .unwrap() - }; - PayableDaoReal::new(make_conn()) - .more_money_payable(SystemTime::now(), &make_wallet("abc"), 45678) - .unwrap(); - ReceivableDaoReal::new(make_conn()) - .more_money_receivable(SystemTime::now(), &make_wallet("xyz"), 65432) - .unwrap(); let mut node = utils::MASQNode::start_standard( "ui_requests_something_and_gets_corresponding_response", Some( @@ -49,27 +32,21 @@ fn ui_requests_something_and_gets_corresponding_response() { home_dir.into_os_string().to_str().unwrap(), ), ), - false, + true, true, false, true, ); node.wait_for_log("UIGateway bound", Some(5000)); - let financials_request = UiFinancialsRequest {}; + let check_password_request = UiCheckPasswordRequest { + db_password_opt: None, + }; let mut client = UiConnection::new(port, NODE_UI_PROTOCOL); - client.send(financials_request); - let response: UiFinancialsResponse = client.skip_until_received().unwrap(); + client.send(check_password_request); + let response: UiCheckPasswordResponse = client.skip_until_received().unwrap(); - assert_eq!( - response, - UiFinancialsResponse { - total_unpaid_and_pending_payable: 45678, - total_paid_payable: 0, - total_unpaid_receivable: 65432, - total_paid_receivable: 0 - } - ); + assert_eq!(response, UiCheckPasswordResponse { matches: true }); client.send(UiShutdownRequest {}); node.wait_for_exit(); } diff --git a/node/tests/utils.rs b/node/tests/utils.rs index 2b081c8bc..00bf72f72 100644 --- a/node/tests/utils.rs +++ b/node/tests/utils.rs @@ -4,6 +4,10 @@ use masq_lib::constants::{CURRENT_LOGFILE_NAME, DEFAULT_UI_PORT}; use masq_lib::test_utils::utils::node_home_directory; use masq_lib::test_utils::utils::{ensure_node_home_directory_exists, TEST_DEFAULT_CHAIN}; use masq_lib::utils::localhost; +use node_lib::database::connection_wrapper::ConnectionWrapper; +use node_lib::database::db_initializer::{ + DbInitializationConfig, DbInitializer, DbInitializerReal, +}; use node_lib::test_utils::await_value; use regex::{Captures, Regex}; use std::env; @@ -185,6 +189,7 @@ impl MASQNode { limit_ms: Option, ) -> Vec> { let logfile_path = Self::path_to_logfile(logfile_dir); + let do_with_log_output = |log_output: &String, regex: &Regex| -> Option>> { let captures = regex .captures_iter(&log_output[..]) @@ -207,6 +212,7 @@ impl MASQNode { .collect::>>(); Some(structured_captures) }; + Self::wait_for_log_at_directory( pattern, logfile_path.as_ref(), @@ -535,3 +541,9 @@ fn node_command() -> String { let bin_dir = &format!("target\\{}", debug_or_release); format!("{}\\MASQNode.exe", bin_dir) } + +pub fn make_conn(home_dir: &Path) -> Box { + DbInitializerReal::default() + .initialize(home_dir, DbInitializationConfig::panic_on_migration()) + .unwrap() +} diff --git a/port_exposer/Cargo.toml b/port_exposer/Cargo.toml index 4dc44a605..6a6ec87f0 100644 --- a/port_exposer/Cargo.toml +++ b/port_exposer/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "port_exposer" -version = "0.6.3" +version = "0.7.0" authors = ["Dan Wiebe ", "MASQ"] license = "GPL-3.0-only" copyright = "Copyright (c) 2019, MASQ (https://masq.ai) and/or its affiliates. All rights reserved."