logging neighbors, devp2p hello, eth status info into mysql #16

SimonSK · 2018-02-12T01:34:07Z

All commits so far. I was going to submit multiple PRs, but branches kept diverging, so here it is. I will try my best to summarize them below.

Minor changes and other changes related to the last PRs (#13, #14):

Log messages related to sql statements are now moved to Debug (if err) and Trace (if ok)
mysql/mysqldb.py: some changes to sql table schemes
p2p/discover: sql related functions in udp.go are now moved to sql_stmt.go. addNodeMetaInfo is removed as we don't actually care about the node addresses from Neighbors packets.

DEVp2p Hello (p2p package):

p2p/node_info.go
- The structs used to manage node information and relevant functions are added here.
- Info struct contains a node's DEVp2p Hello and Ethereum Status info and also keeps track of the most recent db entry's row ID.
- knownNodeInfos is a struct that wraps a map of Infos. It contains a mutex lock.
- storeNodeInfo does the following: If a new node (not in the in-memory database): 1. add a new entry to the sql db, 2. get the entry's row ID, 3. store the information in-memory, 4. add the node as a new static node. Else, check if any information (except timestamp and remote_port) changed. If changed, treat it as a new node; otherwise, update the existing db entry.
- KnownNodes is called by the Javascript Console to print out all node information. I implemented it for debugging purpose.
p2p/rlpx.go
- doProtoHandshake and readProtocolHandshake in p2p/rlpx.go are modified to return the message received time.
- doProtoHandshake behavior changed to attempt to read the peer's Hello message before we send out ours. This change is to avoid situations where the peers reject us based on our information before their information reach us.
p2p/rlpx_test.go
- minor changes to remain consistent with changes made in p2p/rlpx.go
p2p/server.go
- Line 160: unlike other prepared statements, GetRowIDStmt is made public because the Ethereum protocol manager also uses it.
- Line 385: p2p/sql_stmt.go:initSql sets up db connection and prepares statements when the p2p server starts.
- Line 370: p2p/sql_stmt.go:CloseSql closes the statements and db connection when the server stops. It is made public because node package also needs to call it when the p2p server fails to start properly.
- Line 799: p2p/node_info.go:storeNodeInfo is called inside SetupConn. This is where storing to mysql db and updating in-memory database occur.
- Line 785: p2p/sql_stmt.go:addNodeMetaInfo is manually called if the handshake fails because the peer said "TooManyPeers".
p2p/sql_stmt.go: sql related functions
- Line 38: during the init, loadKnownNodeInfos is called to fill the in-memory database with the known information from our sql db. While loading, each node address is added to StaticNodes so that they are re-dialed automatically (currently at the default hardcoded frequency 30s).

Ethereum Status (eth package):

eth/backend.go
- The protocol manager shares the p2p server's GetRowIDStmt and KnownNodeInfos (in-memory database). It directly accesses the map of node infos (not the knownNodeInfos wrapper) because it won't need the mutex lock (it does not change/add anything new to the map).
eth/handler.go
- Line 93: Forgot to add this back while reverting the change in the PR log neighbors to database #13
- Line 164: eth/sql_stmt.go:prepareSqlStmts prepares statements when the eth protocol manager starts.
- Line 188: eth/sql_stmt.go:closeSqlStmts closes the statements when the manager stops.
- Line 209: statusDataWrapper is used to pass the peer's status info (with received timestamp and handshake failure error) to the protocol manager.
- Line 221: eth/node_info.go:storeEthNodeInfo is called. This is where storing to mysql db and updating in-memory database occur.
- Line 289: eth/node_info.go:storeEthNodeInfo is called when unexpected Status messages arrive.
eth/node_info.go
- storeEthNodeInfo only considers nodes that are present in the in-memory database. If the current information contains no Eth Status information, add them to the existing node entry. If eth information changed: 1. add a new node entry to the sql db, 2. get the rowid; else, just update the eth information of the existing entry.
eth/peer.go
- Line 238: validStatus returns true if the peer's status resulted in error because of the specified mismatches.
- Handshake and readStatus modified to handle statusDataWrapper
eth/sql_stmt.go: sql related functions

internal/web3ext/web3ext.go and node/api.go: to allow calling KnownNodes through the Javascript console

TODOs (added to #15):

ORM
~~make re-dial frequency as an configurable option~~
~~update daofork check~~

interaction with sql db yet).

address, should allow default NULL value

reduced to MaxDial/3

previously recorded tcpPort is not 0

table rowid

fixed some logging messages variables renamed

SimonSK · 2018-02-12T03:52:19Z

p2p/server.go

+			&sqlObj.p2pVersion, &sqlObj.clientId, &sqlObj.caps, &sqlObj.listenPort,
+			&sqlObj.firstHelloAt, &sqlObj.lastHelloAt, &sqlObj.protocolVersion, &sqlObj.networkId,
+			&sqlObj.firstReceivedTd, &sqlObj.lastReceivedTd, &sqlObj.bestHash, &sqlObj.genesisHash,
+			&sqlObj.daoForkSupport, &sqlObj.firstStatusAt, &sqlObj.lastStatusAt)


Just found out I screwed up the order here. This does not match the order used in line 987. &sqlObj.daoForkSupport should be the last column. I will push this change shortly.

…ate files function names renamed bug fix

them

SimonSK · 2018-02-16T01:33:20Z

After performing DAO fork check through BlockHeaders/GetBlockHeaders exchange, dao_fork field on the database and DAOForkSupport field in the Info struct are updated.

The field is changed from bool to a signed int. 1 = true (mainnet), 0 (or NULL on db) = unknown, -1 = false (classic).
eth/node_info.go
- Line 70~ (storeDAOForkSupportInfo): if current DAOForkSupport is 0, then simply update the existing entry as it is the first time we learned about the node's DAOForkSupport status. Otherwise, if the DAOForkSupport status changed for some reason, add a new entry, copying all other information from the previous entry.

SimonSK · 2018-02-16T01:54:56Z

If any part of the initial sql preparation (p2p/server.go:385) returns error, the client goes through proper shutdown procedures.
If the db handle exists but a prepared statement variable is null, the client immediately exits using log.Crit.
If the client fails to execute a prepared statement, it logs the error as well as the values that were supposed to be added to the db.

After the changes,

Trace: successful event that doesn't involve changing data, like open/closing database handle or preparing statements
Debug: successful event that involves changing data, like executing prepared statements
Error: failure of those mentioned above 2
Crit: failing to initialize/prepare sql related variables or prepared statements missing when expected

SimonSK · 2018-02-16T02:03:28Z

eth/peer.go

@@ -285,13 +296,19 @@ func (p *peer) readStatus(network uint64, status *statusData, genesis common.Has

 	p.Log().Proto("<<"+ethCodeToString[msg.Code], "receivedAt", unixTime, "obj", status, "size", int(msg.Size), "peer", p.ID())

+	statusWrapper.ReceivedAt = &msg.ReceivedAt
+	statusWrapper.Status = &status
+
 	if status.GenesisBlock != genesis {


@zzma I decided to keep this behavior (as well as checking the error code again in eth/handler.go:216) for now. If these 3 mismatch checks are removed, our node will assume the peer is compatible (e.g. on the same blockchain) and resume other behaviors, such as DAO fork check through BlockHeader exchange. I assume the peer will do the same checks and disconnect us anyways, but I am not sure if that assumption is enough to remove the checks.

SimonSK · 2018-02-16T02:09:41Z

Another TODO added to #15 is removing case-study style log messages from node-finder.
That was the first thing I brought from case-study thinking it would be useful, but now I find it too much.
I plan to remove them in the next PR, which will also include the following two:

make re-dial frequency as an configurable option
call all sql execution functions as go routine

prepared stmt checking done in each stmt exec function

… to the sql db

and total difficulty values total difficulty value is limited to 65 digits, max allowed in the field on the mysql table

SimonSK added 16 commits February 11, 2018 18:21

table scheme changes

a768df4

in-memory KnownNodeInfo data structure and p2p info update routine (no

433ad32

interaction with sql db yet).

KnowNodes function for Javascript Console

f3f17aa

both neighbors and node_meta_info tables store nodeid hashes

b2a860f

node_info table schema change: all fields, except those related to

b65a60d

address, should allow default NULL value

record received timestamp for devp2p hello message

5947dfe

add new node information to node_meta_info and node_info tables

af6bd2c

update timestamp of the most recent existing node_info entry

6131496

made p2p_version bigint to remain consistent with geth data structure

c19a186

load node info from sql db

b9910db

check db handle and prepared statements in right places

54602ba

add reachable nodes as static nodes for automatic re-dialing. dynPeers

498439e

reduced to MaxDial/3

node address of an incoming connection should be resolved iff its

9651aae

previously recorded tcpPort is not 0

better log messages, and properly closing sql connections upon failure

2d33c4a

data structure changes: mutex, hello/status received times, most recent

078d872

table rowid

logging ethereum status info to mysql db

eb72a54

SimonSK requested a review from a team February 12, 2018 01:34

SimonSK mentioned this pull request Feb 12, 2018

TODOs #15

Open

21 tasks

SimonSK added 2 commits February 11, 2018 20:21

moved sql related functions to a separate file

895ed59

moved sql and node info related functions to separate files

305ae58

fixed some logging messages variables renamed

SimonSK commented Feb 12, 2018

View reviewed changes

SimonSK added 2 commits February 11, 2018 22:01

moved sql and node info related functions for ethereum layer to separ…

276a24d

…ate files function names renamed bug fix

some changes i missed from last manual merge conflict

a4e152d

SimonSK force-pushed the simonsk/mysql-connect-option branch from 8da924f to 078d872 Compare February 12, 2018 04:41

SimonSK changed the title ~~logging neighbors and devp2p hello info into mysql~~ logging neighbors, devp2p hello, eth status info into mysql Feb 12, 2018

SimonSK force-pushed the simonsk/mysql-connect-option branch from 10d954a to a4e152d Compare February 12, 2018 06:13

bug fix: cases flipped

a10c7ec

SimonSK force-pushed the simonsk/mysql-connect-option branch from 8b0aa09 to a10c7ec Compare February 12, 2018 10:30

SimonSK added 2 commits February 15, 2018 11:16

more descriptive function name

72af2bb

unused field removed

ece4d18

SimonSK added 12 commits February 15, 2018 15:59

p2p sql stmts error message log level changed from Debug to Error

28778ad

bug fix

46e3d63

p2p hard quit when prepared stmts missing

1efdeb7

p2p server hard quit when prepared stmt missing

74a3650

blacklist log message changed

be3090a

udp server hard quit if prepared statement missing

aaf52dd

if -> else if

4a43ef1

keeping log levels consistent

d1d949c

hard quit if eth handler has no prepared statements or fails to prepare

f5c7e84

them

removed unnecessary condition check

426c7a4

info struct and sql table scheme changed

9e290c9

add/update DAO fork support flag

4d73fa9

SimonSK commented Feb 16, 2018

View reviewed changes

SimonSK added 8 commits February 16, 2018 02:01

proper db/stmt checking for AddNeighbors

d5578f1

prepared stmt checking done in each stmt exec function

log message prints the entire node id

47a65cc

timestamp and td proper update

e8e44c5

function renamed

76cfb67

sort caps

cb4b667

replace the in-memory info with new info when adding a new node entry…

9266f4a

… to the sql db

function renamed

a05b3ef

data structure change to allow more consistent log format of timestamp

7df0b04

and total difficulty values total difficulty value is limited to 65 digits, max allowed in the field on the mysql table

SimonSK merged commit 9854e53 into node-finder-master Feb 16, 2018

SimonSK deleted the simonsk/mysql-connect-option branch February 16, 2018 23:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

logging neighbors, devp2p hello, eth status info into mysql #16

logging neighbors, devp2p hello, eth status info into mysql #16

SimonSK commented Feb 12, 2018 •

edited

Loading

SimonSK Feb 12, 2018

SimonSK commented Feb 16, 2018

SimonSK commented Feb 16, 2018

SimonSK Feb 16, 2018

SimonSK commented Feb 16, 2018

logging neighbors, devp2p hello, eth status info into mysql #16

logging neighbors, devp2p hello, eth status info into mysql #16

Conversation

SimonSK commented Feb 12, 2018 • edited Loading

SimonSK Feb 12, 2018

Choose a reason for hiding this comment

SimonSK commented Feb 16, 2018

SimonSK commented Feb 16, 2018

SimonSK Feb 16, 2018

Choose a reason for hiding this comment

SimonSK commented Feb 16, 2018

SimonSK commented Feb 12, 2018 •

edited

Loading