Feature/dev network #31

victorshevtsov · 2023-06-11T05:45:28Z

No description provided.

…in the broker

…ch for the KYVE pool config

…e query

…hing

…work

vercel · 2023-06-11T05:45:32Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
logstore	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 12, 2023 0:05am

gitguardian · 2023-06-11T05:45:32Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	Secret	Commit	Filename
6857130	Generic Private Key	`f75a8f0`	dev-network/assets/nginx/certs/arweave.net.key	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

…acts

rsoury

All looks very good. Great job - left some comments on logic that stood out to me.

packages/broker/src/config/config.schema.json

rsoury · 2023-06-12T13:35:52Z

packages/validator/src/core/item.ts

-		const toKey = parseInt(key, 10);
-		const fromKey = toKey - config.itemTimeRange; // First iteration over the cache, will use the first nextKey -- ie. 1000
+		const toTimestamp = parseInt(key, 10) * 1000;
+		const fromTimestamp = toTimestamp - 1000;


Can we keep this itemTimeRange - ie. Enabling the Pool's configuration to determine the range that the Validator Network queries for?
This way we can manipulate the way the Validator Network works without needing to change code.

The current implementation means that the key is a time in UNIX epoch format (number of seconds since Jan 1 1970). Each data item has messages published within a time range between key seconds and (key+1) seconds.
If we we will manipulate the multiplier eventually, each key will have meaning of different number of seconds stored by a data item identified by that key.
Perhaps i misunderstood the idea of itemTimeRange option.

The goal of itemTimeRange is to enable each data item to be made up of X time range, rather than just a fixed 1 second time range.

ie. In the case we have low network usage - this way be wise to configure as itemTimeRange = 5 for 5 second time ranges for each data item.

I think this can give us a configurable option to manage the load on the network.

The query interface will still be indexed by time, as it's the Broker layer is completely separate.

However, we can leave this as is for now - and add this option back at a later date if required.

I suggest to get back to this discussion when we will implement bundle downloading and unpacking mechanism when a client queries for a data that not in a broker's cache.

Agreed - however, I don't expect the Client to ever be responsible for querying directly from Arweave, but rather Broker Nodes will hydrate their database with data from Arweave relevant to the query request.

Yes, sure this is a Broker's responsibility. I just mean this hydration will be initialized by a query sent by a client.

packages/validator/src/runtime.ts

rsoury · 2023-06-12T14:11:35Z

packages/validator/src/runtime.ts

@@ -113,60 +105,42 @@ export default class Runtime implements IRuntimeExtended {
 		validationDataItem: DataItem
 	): Promise<boolean> {
 		const proposedDataItemHash = sha256(
-			Buffer.from(JSON.stringify(proposedDataItem))
+			Buffer.from(JSON.stringify(proposedDataItem.value.messages))


This is going to skip the validation of the Report inside of the Last message all the time?

I think we need to revert to the original here - no?

We add the report to the last item in the bundle inside summarizeDataBundle function which called by KYVE runtime when it produces a proposal and when it validates a proposal. BUT, when KYVE runtime validates a data item (by calling validateDataItem) it calls summarizeDataBundle after it. So, at the moment of item validation the item does not have a report data.

Damn, I was confused in that case.

I thought validateDataItem was called after summarizeDataBundle.

This means we need to include the sha256(Buffer.from(JSON.stringify(report))) inside of bundle summary - so that there's validation of the report too.

ie. In the Runtime's summarizeDataBundle, we'll need to return lastItem.key + "_" + report_hash;

This way the report is validated here https://github.com/KYVENetwork/kyvejs/blob/7eae47e8310262b636b9fd78ff585b1a22089234/common/protocol/src/methods/validate/validateBundleProposal.ts#L218

Good option. I will modify it as proposed.

rsoury · 2023-06-12T14:12:25Z

packages/validator/src/types/index.ts

@@ -13,7 +13,6 @@ export interface IRuntimeExtended extends IRuntime {
 export interface IConfig {
 	systemStreamId: string;
 	sources: string[];
-	itemTimeRange: number; // Some range in unix time between each data item


See above note.

see the comment above

victorshevtsov added 25 commits May 27, 2023 02:43

reorganized dev-network and add KYVE chain and UI

bd2211f

update dev-network/README.md with ports used

d8df241

Merge branch 'develop' into feature/dev-network

216ea04

add convenient scripts for dev-network

95cb547

update dev-network scripts

c9663a4

add arlocal to the dev-network

87ec94a

Merge remote-tracking branch 'origin/develop' into feature/dev-network

e48f061

kyve and arweave setup in dev-network

f75a8f0

add 3 validators to the dev-entwork

7965014

Merge branch 'develop' into feature/dev-network

9133569

add --assume-yes option to the cli

03743f3

call cli query stake before starting validators in dev-network

666d2b7

add missing use of CONFIG_TEST validator's Item

9a177ab

use BigNumber type for amount in validator's StoreManager

bce42ab

update validator to kyvejs v1.0.1

13ada65

make the validator work with the DevNetwork

9f8d233

fund validators and join the pool when DevNetwork starts

22a3c02

get report from the last item in the bundle when fetching ReportData …

b6efe6e

…in the broker

update broker config schema and config wizard to follow common approa…

c69ff69

…ch for the KYVE pool config

fix broker's data endpoint to response if there is no messages for th…

858a5f7

…e query

simplyify publishing ProofOfReport message in the broker

16847ab

fix ReportManager caontract generating a proper reportJson before has…

cc5e24e

…hing

move assets for validators running in the DevNetwork

2946f9c

adjust KYVE config with convenient values for debugging within DevNet…

2fe055e

…work

add scripts and docs to setup and manage the DevNetwork

545252c

fix dev-network deploy command

b956b27

vercel bot deployed to Preview June 11, 2023 06:09 View deployment

fix weird behavior when compile contracts in container on VPS

4fe460a

increase delay before minting to validators' arweave accounts

72debbb

vercel bot deployed to Preview June 11, 2023 10:02 View deployment

victorshevtsov added 3 commits June 12, 2023 10:57

Merge branch 'develop' into feature/dev-network

40e6637

whitelist the broker accounts for DevNetwork

64261f6

make the internal arweave.net host reachable when compiling the contr…

e1a7eda

…acts

vercel bot deployed to Preview June 12, 2023 12:05 View deployment

rsoury reviewed Jun 12, 2023

View reviewed changes

rsoury merged commit bb0a0f8 into develop Jun 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/dev network #31

Feature/dev network #31

victorshevtsov commented Jun 11, 2023

vercel bot commented Jun 11, 2023 •

edited

Loading

gitguardian bot commented Jun 11, 2023 •

edited

Loading

rsoury left a comment

rsoury Jun 12, 2023

victorshevtsov Jun 12, 2023

rsoury Jun 12, 2023

victorshevtsov Jun 12, 2023

rsoury Jun 13, 2023

victorshevtsov Jun 13, 2023

rsoury Jun 12, 2023

victorshevtsov Jun 12, 2023

rsoury Jun 12, 2023

victorshevtsov Jun 12, 2023

rsoury Jun 12, 2023

victorshevtsov Jun 12, 2023

Feature/dev network #31

Feature/dev network #31

Conversation

victorshevtsov commented Jun 11, 2023

vercel bot commented Jun 11, 2023 • edited Loading

gitguardian bot commented Jun 11, 2023 • edited Loading

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

rsoury left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Jun 11, 2023 •

edited

Loading

gitguardian bot commented Jun 11, 2023 •

edited

Loading