Spike: On chain ledger

[Luka-Loncar]

Problem

We currently gossip nodeData from all nodes and all node self report, the validator then does not check to see if the self-reported data has already been received or if the data is valid. This leads to overwriting of nodeData and volatile data in the nodeData object and corruption of nodeData on the protocol. This has been an ongoing issue for some time and is blocking us from moving the protocol forward. We need to find a final solution to the problem that scales.

Specification

• Only validators can write data to the on-chain ledger (leveldb)
• Every node can get a copy of the database (leveldb)
• The nodeData object when reported by a node to a validator is resistant to tampering and attack, i.e. data altering
• All workers can report the work they have done honestly along with their uptimes and it will persist on the network, Validators are the only actors who can write to the ledger.
• A complete data request is synonymous to a transaction on the network that has a shape, size (in bytes), and metadata associated with it.

[teslashibe]

@jdutchak this is @mudler 's boilerplate blockchain implementation in EdgeVPN: https://github.com/mudler/edgevpn/tree/master/pkg/blockchain

It stores data and maintains it consistency across all nodes on the network

Take a look at it and kets talk more about this tomorrow

[teslashibe]

@jdutchak in some of the data I sampled here: https://github.com/masa-finance/metrics/blob/main/data/node_data_v_0_0_7.json

There was duplication of cid's.

Question: Are cid's unique for each worker request?

          {
            "cid": "bafkreied4fftjnp4pn3w7q3qy7bemh23d6fiiz5enk4c6ek3r3viqfqwcq",
            "duration": 60.308573292,
            "timestamp": "2024-06-13T10:56:41.094958-07:00"
          },

          {
            "cid": "bafkreied4fftjnp4pn3w7q3qy7bemh23d6fiiz5enk4c6ek3r3viqfqwcq",
            "duration": 0.255457708,
            "timestamp": "2024-06-13T10:57:43.200429-07:00"
          },

          {
            "cid": "bafkreied4fftjnp4pn3w7q3qy7bemh23d6fiiz5enk4c6ek3r3viqfqwcq",
            "duration": 0.000085958,
            "timestamp": "2024-06-13T10:59:43.785958-07:00"
          },

          {
            "cid": "bafkreied4fftjnp4pn3w7q3qy7bemh23d6fiiz5enk4c6ek3r3viqfqwcq",
            "duration": 0.000084417,
            "timestamp": "2024-06-13T11:00:43.789969-07:00"
          }

[teslashibe]

@jdutchak this ticket should fix this ticket https://github.com/masa-finance/issues/issues/191 so we should have the time fix taken care of by completing the ledger ticket.

@mudler putting https://github.com/masa-finance/issues/issues/191 on hold and marking it as blocked by this ticket.

[mudler]

in regard to this, few notes (on how edgevpn works, and how a v0 could be "based" from some of the concepts in there):

• In EdgeVPN all nodes are talking via gossip each other - and the blockchain ( a block is marshaled into JSON, and it's a k/v structure) is sent raw over there by the nodes that wishes to update content on the blockchain. The blockchain mechanism is well known - there is a PoW and as such a node can generate a new block of the chain by appending to the next one if can be proven to be correct (and thus, accepted by the nodes). In the EdgeVPN case the PoW is simplified as such there is no real difficulty (the edgevpn scope is not to have an incentive, but rather have a global state). It works like "bully" algorithms - the first that shouts louder and convince the other that his block is newer wins, and the other have to build on top of the block that has been announced.
• For production the above should be slightly changed to say at least:
  • The blockchain blocks shouldn't be sent raw in the gossip channels - but use something among the lines of bitswap https://github.com/ipfs/boxo/tree/main/bitswap to offload the blockchain content . for instance, CIDs are shared as gossip messages with part of the blockchain metadata, but the real content is delegated to bitswap.
  • I'm pretty sure there are other libraries from ipfs that we can use, but at the moment I can't recall them ( will keep you posted )
  • There probably should be a consensus protocol instead of "play bully" to write on the ledger to make it more performant, or at least to access sensible parts of it - https://github.com/libp2p/go-libp2p-raft/tree/master can be used for the purpose

Some additional points and consideration on the EdgeVPN architecture:

• In EdgeVPN the nodes in a network are supposed to be all trusted. There is no "global" network, but it's all about creating smaller, private networks among peers. Bootstrap peers from ipfs are used, but of course can use other peers as bootstrap as well. The segmentation of the network is given by the fact that communication over gossip is not only e2e encrypted, but it's symmetrically encrypted, too.
• EdgeVPN by default does not need to persist keypair of nodes - keypairs are supposed to be throw-away and the symmetric key is the only guarantee of accessing the blockchain. However, there is a "two" layer of symmetric key in play here - one used to discover the nodes (discovery rendezvous points are derived from symmetric keys), and the blockchain is sealed with AES
• Given the above implication, writing access to the blockchain is uncontrolled by whose having the symmetric key (as it is enough to have the symmetric keys) - however - there is a way to protect the network by having trusted peers with the peerguardian feature: https://mudler.github.io/edgevpn/docs/concepts/overview/peerguardian/ in case the symmetric key has been compromised . That works by having a set of trusted nodes which have a list of keypairs that are considered to be trusted. nodes will accept messages from the blockchain that "edits" a certain part of it (a trusted section) only and only if these nodes are part of the trusted list in their current blockchain view. The trusted list indeed exists also on the blockchain, so trusted peers can be extended during runtime: only trusted peers's blockchain messages will be accepted editing that section of the blockchain.

[mudler]

As from planning we decided that we should have a call/pairing session to understand the direction of this @jdutchak / @H34D

cc @teslashibe

[jdutchak]

Solution 1: Implement a Validation Layer

Overview:
Introduce a validation layer in the protocol that ensures the integrity and validity of nodeData before it is written to the on-chain ledger. This layer will be responsible for checking the data received from nodes and only allowing validated data to be written by validators.

Steps:

1. Validation Rules:

   • Develop a set of rules that nodeData must adhere to. These rules should include checks for data format, completeness, and consistency with previously reported data.
   • Implement cryptographic techniques, such as digital signatures, to ensure data integrity and authenticity.

2. Data Verification:

   • When a node reports data, the validator will first verify the digital signature to ensure the data has not been tampered with.
   • The validator will then check if the data has already been received by comparing the new data with existing data entries.

3. Data Deduplication:

   • Implement a deduplication mechanism that prevents overwriting of nodeData. This can be achieved by maintaining a hash of previously received data and comparing new entries against this hash.

4. Tamper Resistance:

   • Use cryptographic hashing to create a unique identifier for each data entry. This ensures that any alteration to the data will result in a different hash, making tampering evident.

5. Scalability:

   • Design the validation layer to be efficient and scalable, capable of handling large volumes of data without significant performance degradation.

6. Reporting and Transparency:

   • Nodes continue to report their work and uptime honestly. Validators validate and only write verified data to the ledger.
   • Provide a mechanism for nodes to query their reported data and any actions taken by validators, ensuring transparency and trust in the system.

Solution 2: Implement a Consensus Mechanism for Data Validation

Overview:
Introduce a consensus mechanism where multiple validators must agree on the validity of nodeData before it is written to the on-chain ledger. This ensures that no single validator can corrupt the data and that data integrity is maintained through collective agreement.

Steps:

1. Consensus Algorithm:

   • Implement a consensus algorithm, such as Practical Byzantine Fault Tolerance (PBFT) or a variant of Proof of Stake (PoS), where multiple validators participate in the validation process.

2. Data Reporting:

   • Nodes report their data to a subset of validators (e.g., a committee). Each validator in the committee independently verifies the data.

3. Agreement Process:

   • Validators within the committee compare their verification results. If a predefined threshold (e.g., 2/3 majority) agrees on the data's validity, the data is considered valid.
   • Only data that achieves consensus is written to the on-chain ledger.

4. Fault Tolerance:

   • The consensus mechanism provides fault tolerance by ensuring that a small number of malicious or faulty validators cannot corrupt the data.
   • This approach also mitigates the risk of overwriting and data corruption.

5. Data Integrity:

   • Use digital signatures and cryptographic hashing to ensure that data reported by nodes is tamper-resistant and can be independently verified by validators.

6. Scalability:

   • Optimize the consensus mechanism for scalability, ensuring it can handle a large number of transactions and validators without significant performance bottlenecks.

7. Transparency and Auditing:

   • Provide a mechanism for nodes and other stakeholders to audit the consensus process and verify that data validation was performed correctly.
   • Ensure that the process is transparent and that any discrepancies can be traced and resolved.

Both solutions aim to address the problem of overwriting and volatile data in the nodeData object by introducing robust validation mechanisms. The first solution focuses on implementing a validation layer with strict rules and tamper resistance, while the second solution leverages a consensus mechanism to ensure collective agreement on data validity. Both approaches ensure that only validated data is written to the on-chain ledger by validators, thereby maintaining data integrity and trust in the protocol.

[theMultitude]

From a data perspective these questions come to mind. Perhaps simplistic but trying to hit bedrock/common understanding for the train of thought:

1. What is the scale and type of the data we expect to handle?
   a. How does that impact how we think about where is stored?
   1. We will have overlapping data layers, which layer is responsible for which types of data?
      a. What are the performance/scalability/complexity implications to these solutions?
   2. How do these layers overlap so as to communicate effectively?

Also, just a ping to keep it in mind, from a data science perspective failure is as important to understanding the network as success. On a practical economic level this also relates to slashing for misdeeds.

Working through a couple other things at the moment but will try and add context to the above later this week.

[jdutchak]

Proof of Work (PoW) Implementation

Overview:
In a Proof of Work (PoW) system, validators (also known as miners) compete to solve complex cryptographic puzzles to validate transactions and add them to the blockchain. The first validator to solve the puzzle earns the right to write data to the on-chain ledger.

Implementation Steps:

1. Data Reporting by Nodes:

   • Nodes report their nodeData to all validators.
   • Each piece of nodeData includes a cryptographic hash and a digital signature to ensure integrity and authenticity.

2. Puzzle Generation:

   • For each new piece of nodeData, a cryptographic puzzle is generated. This puzzle is computationally intensive to solve but easy to verify.

3. Mining Process:

   • Validators (miners) compete to solve the puzzle. The solution involves finding a nonce (a random value) that, when hashed with the nodeData, produces a hash with specific properties (e.g., a certain number of leading zeros).

4. Validation and Consensus:

   • The first validator to solve the puzzle broadcasts the solution to the network.
   • Other validators verify the solution and check that the nodeData is valid and has not been tampered with.

5. Writing to the Ledger:

   • Once the solution is verified, the nodeData is added to the on-chain ledger (leveldb).
   • The validator who solved the puzzle is rewarded (e.g., with cryptocurrency or tokens).

6. Tamper Resistance:

   • The cryptographic nature of the puzzle ensures that any alteration to the nodeData would require re-solving the puzzle, making tampering evident and costly.

7. Scalability Considerations:

   • While PoW is secure, it is also resource-intensive and may not scale well with increasing data volumes. Optimizations and efficient algorithms are necessary to maintain performance.

Proof of Stake (PoS) Implementation

Overview:
In a Proof of Stake (PoS) system, validators are chosen to validate transactions and write data to the on-chain ledger based on the amount of stake (e.g., cryptocurrency or tokens) they hold. Validators with more stake have a higher probability of being chosen to validate data.

Implementation Steps:

1. Data Reporting by Nodes:

   • Nodes report their nodeData to all validators.
   • Each piece of nodeData includes a cryptographic hash and a digital signature to ensure integrity and authenticity.

2. Stake Allocation:

   • Validators must hold a certain amount of stake to participate in the validation process.
   • The stake acts as collateral to incentivize honest behavior and penalize malicious actions.

3. Validator Selection:

   • A validator is selected to validate the nodeData and write it to the on-chain ledger based on a combination of factors, including the amount of stake held and a random selection process.
   • The selection algorithm ensures that validators with more stake have a higher probability of being chosen, but randomness prevents any single validator from monopolizing the process.

4. Validation Process:

   • The selected validator verifies the nodeData, checking for integrity and authenticity.
   • The validator ensures that the nodeData has not been previously received and that it meets all validation rules.

5. Consensus and Writing to the Ledger:

   • Once validated, the nodeData is written to the on-chain ledger (leveldb) by the selected validator.
   • Other validators in the network can verify the validity of the nodeData and the correctness of the selection process.

6. Rewards and Penalties:

   • The selected validator is rewarded for their work (e.g., with transaction fees or newly minted tokens).
   • Validators who attempt to add invalid data can lose their stake as a penalty, ensuring that only honest validators participate.

7. Tamper Resistance:

   • The use of cryptographic signatures and hashes ensures that any tampering with nodeData can be detected.
   • Validators are incentivized to behave honestly to avoid losing their stake.

8. Scalability Considerations:

   • PoS is generally more energy-efficient and scalable than PoW, making it suitable for handling large volumes of data and transactions.
   • Efficient selection algorithms and stake management are essential to maintain performance and security.

Both PoW and PoS provide mechanisms to ensure the integrity and validity of nodeData before it is written to the on-chain ledger. PoW relies on computational effort and puzzle-solving, while PoS leverages stake and probabilistic selection to achieve consensus and maintain data integrity.

[mudler]

cross-referencing #350

The PR looks promising, at least for a v0 implementation that would be directly in the oracle codebase. However we have to still decide if we want a blockchain implementation in the oracle or having it offloaded to a separate chain (e.g. Avalanche).

As we discussed in the last call with @H34D @theMultitude @jdutchak - we want to have a list of all the options and implications with associated pro/cons to evaluate what's the best path forward.

@jdutchak is going to present the options to the team in a call with some slides/agenda to encourage knowledge sharing on the topic as part of the spike.

[jdutchak]

here are the main elements and their relationships

1. Chain: Represents the blockchain itself.
2. Block: Represents a single block in the blockchain.
3. ProofOfWork: Handles the Proof of Work consensus mechanism.
4. ProofOfStake: Handles the Proof of Stake consensus mechanism.
5. Persistance: Manages the storage of blocks.

Key Components and Interactions

1. Chain:

Initializes the blockchain.
Adds blocks to the blockchain.
Updates the last hash.
Iterates through the blockchain.

2. Block:

Contains data, hash, link to the previous block, nonce, and consensus mechanism.
Can be serialized and deserialized.

3. ProofOfWork: (option 1)

Runs the Proof of Work algorithm to find a valid nonce.
Validates blocks using the Proof of Work mechanism.

4. ProofOfStake: (option 2)

Runs the Proof of Stake algorithm to find a valid nonce.
Validates blocks using the Proof of Stake mechanism.

5. Persistance:

Manages the storage and retrieval of blocks.

1. Chain:

Init()
AddBlock()
UpdateLastHash()
IterateLink()

2. Block:

Build()
Serialize()
Deserialize()

3. ProofOfWork:

Run()
IsValidPoW()

4. ProofOfStake:

Run()
IsValidPoS()

5. Persistance:

Init()
SaveBlock()
GetLastHash()
Get()

Code References

Chain: pkg/chain/chain.go (startLine: 1, endLine: 114)
Block: pkg/chain/block.go (startLine: 1, endLine: 61)
ProofOfWork: pkg/chain/pow.go (startLine: 1, endLine: 50)
ProofOfStake: pkg/chain/pos.go (startLine: 1, endLine: 72)

Here is a textual representation of the diagram structure:

[teslashibe]

What is the time boxing and status on this @jdutchak?

[mudler]

We just discussed this today in a specific call and we have decided to go with the option to implement a ledger and blockchain by ourselves in the oracle, mainly for the following reasons:

• We want to have control of the network and do not depend on an external services that might fail (either in implementations, or going to be deprecated) to store our data in the long run
• We will have a hyrbid approach that will allow us to have a "light" ledger across the node that (I am oversimplifying here) stores small content of data (similarly to the EdgeVPN implementation in Spike: On chain ledger #345 (comment) ) a CID (content-addressable storage id) that allows other nodes to co-ordinate to pull data from (with a different protocol in the same network). Now, this can be even replaced with IPFS + a pinning service for each node to speed up development, but eventually we do not want to depend on IPFS (for the reason above)
• This approach will also allow us to have copies of the persistent data across different networks (ipfs, avalanache, etc) or services (s3, etc) available for disaster recovery and syncronize them with the ledger state

[mudler]

@jdutchak closing as completed. Follow up in #380.