Deep Account Analytics: Milestone 1 #1038
Conversation
|
@sourabhniyogi sorry for the delay here and here, we currently got a bit of a backlog. I'll be back with more feedback soon. |
There was a problem hiding this comment.
@sourabhniyogi sorry for the delay here.
Regarding your delivery I want to make a couple of point:
-
In general, the testing instructions are not very clear at this stage - for example, in the Manual Testing Guide section you assume that everything has been set up and is running correctly already. Please keep in mind that an evaluation includes testing all aspects of the software deliverables, which includes the set up process of any software component involved. I understand that you've successfully made a deployment already which can be used for testing as well, but please note that as part of the evaluation process the setup procedure still has to be tested.
-
I've noticed that there is a problem with installing dependencies on the polcaholic repo:
% nvm use stable
Now using node v20.3.1 (npm v9.6.7)
% npm i
npm WARN ERESOLVE overriding peer dependency
npm WARN While resolving: [email protected]
npm WARN Found: [email protected]
npm WARN node_modules/@acala-network/eth-providers/node_modules/graphql
npm WARN graphql@"~16.0.1" from @acala-network/[email protected]
npm WARN node_modules/@acala-network/eth-providers
npm WARN @acala-network/eth-providers@"2.4.21" from @acala-network/[email protected]
npm WARN node_modules/@acala-network/bodhi
npm WARN
npm WARN Could not resolve dependency:
npm WARN peer graphql@"14.x || 15.x" from [email protected]
npm WARN node_modules/@acala-network/eth-providers/node_modules/graphql-request
npm WARN graphql-request@"~3.6.1" from @acala-network/[email protected]
npm WARN node_modules/@acala-network/eth-providers
npm WARN
npm WARN Conflicting peer dependency: [email protected]
npm WARN node_modules/graphql
npm WARN peer graphql@"14.x || 15.x" from [email protected]
npm WARN node_modules/@acala-network/eth-providers/node_modules/graphql-request
npm WARN graphql-request@"~3.6.1" from @acala-network/[email protected]
npm WARN node_modules/@acala-network/eth-providers
npm WARN deprecated [email protected]: This package is broken and no longer maintained. 'mkdirp' itself supports promises now, please switch to that.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm WARN deprecated [email protected]: This module has been superseded by the multiformats module
npm ERR! code 1
npm ERR! path /Users/xxx/repos/polkaholic/node_modules/canvas
npm ERR! command failed
npm ERR! command sh -c node-pre-gyp install --fallback-to-build --update-binary
npm ERR! Failed to execute '/Users/xxx/.nvm/versions/node/v20.3.1/bin/node /Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --update-binary --module=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release --napi_version=9 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v115' (1)
npm ERR! node-pre-gyp info it worked if it ends with ok
npm ERR! node-pre-gyp info using [email protected]
npm ERR! node-pre-gyp info using [email protected] | darwin | arm64
npm ERR! node-pre-gyp http GET https://github.com/Automattic/node-canvas/releases/download/v2.10.1/canvas-v2.10.1-node-v115-darwin-unknown-arm64.tar.gz
npm ERR! node-pre-gyp ERR! install response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.10.1/canvas-v2.10.1-node-v115-darwin-unknown-arm64.tar.gz
npm ERR! node-pre-gyp WARN Pre-built binaries not installable for [email protected] and [email protected] (node-v115 ABI, unknown) (falling back to source compile with node-gyp)
npm ERR! node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.10.1/canvas-v2.10.1-node-v115-darwin-unknown-arm64.tar.gz
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info ok
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | darwin | arm64
npm ERR! gyp info find Python using Python version 3.9.6 found at "/Library/Developer/CommandLineTools/usr/bin/python3"
npm ERR! gyp info spawn /Library/Developer/CommandLineTools/usr/bin/python3
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args '/Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'make',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/xxx/repos/polkaholic/node_modules/canvas/build/config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args '/Users/xxx/Library/Caches/node-gyp/20.3.1/include/node/common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args '-Dnode_root_dir=/Users/xxx/Library/Caches/node-gyp/20.3.1',
npm ERR! gyp info spawn args '-Dnode_gyp_dir=/Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp',
npm ERR! gyp info spawn args '-Dnode_lib_file=/Users/xxx/Library/Caches/node-gyp/20.3.1/<(target_arch)/node.lib',
npm ERR! gyp info spawn args '-Dmodule_root_dir=/Users/xxx/repos/polkaholic/node_modules/canvas',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',
npm ERR! gyp info spawn args '--no-parallel',
npm ERR! gyp info spawn args '--generator-output',
npm ERR! gyp info spawn args 'build',
npm ERR! gyp info spawn args '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! /bin/sh: pkg-config: command not found
npm ERR! gyp: Call to 'pkg-config pixman-1 --libs' returned exit status 127 while in binding.gyp. while trying to load binding.gyp
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack at ChildProcess.onCpExit (/Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:325:16)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:511:28)
npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:293:12)
npm ERR! gyp ERR! System Darwin 23.1.0
npm ERR! gyp ERR! command "/Users/xxx/.nvm/versions/node/v20.3.1/bin/node" "/Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--update-binary" "--module=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release/canvas.node" "--module_name=canvas" "--module_path=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release" "--napi_version=9" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v115"
npm ERR! gyp ERR! cwd /Users/xxx/repos/polkaholic/node_modules/canvas
npm ERR! gyp ERR! node -v v20.3.1
npm ERR! gyp ERR! node-gyp -v v9.3.1
npm ERR! gyp ERR! not ok
npm ERR! node-pre-gyp ERR! build error
npm ERR! node-pre-gyp ERR! stack Error: Failed to execute '/Users/xxx/.nvm/versions/node/v20.3.1/bin/node /Users/xxx/.nvm/versions/node/v20.3.1/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --update-binary --module=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/Users/xxx/repos/polkaholic/node_modules/canvas/build/Release --napi_version=9 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v115' (1)
npm ERR! node-pre-gyp ERR! stack at ChildProcess.<anonymous> (/Users/xxx/repos/polkaholic/node_modules/@mapbox/node-pre-gyp/lib/util/compile.js:89:23)
npm ERR! node-pre-gyp ERR! stack at ChildProcess.emit (node:events:511:28)
npm ERR! node-pre-gyp ERR! stack at maybeClose (node:internal/child_process:1098:16)
npm ERR! node-pre-gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:304:5)
npm ERR! node-pre-gyp ERR! System Darwin 23.1.0
npm ERR! node-pre-gyp ERR! command "/Users/xxx/.nvm/versions/node/v20.3.1/bin/node" "/Users/xxx/repos/polkaholic/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" "--update-binary"
npm ERR! node-pre-gyp ERR! cwd /Users/xxx/repos/polkaholic/node_modules/canvas
npm ERR! node-pre-gyp ERR! node -v v20.3.1
npm ERR! node-pre-gyp ERR! node-pre-gyp -v v1.0.10
npm ERR! node-pre-gyp ERR! not ok
npm ERR! A complete log of this run can be found in: /Users/xxx/.npm/_logs/2023-11-20T10_16_26_474Z-debug-0.log-
Is there any software component to test in the substrate-etl repo or is its scope limited to static files?
-
Due to the project complexity it might be useful to have a call to clear things up. That's completely optional, but if you're interested feel free to book a slot here. Please note that you'll still have to provide a written testing guide so other people in the community can easily reuse your software components, if they're interested.
LMK if you have any questions!
|
@takahser -- This project is not suitable for Docker-based review. This is because Polkaholic.io, while open source, depends on a Google Cloud backend (BigTable, MySql, GS, BigQuery) and in this case Apache Superset. We have worked with @michaelis @ w3f who has audited both Milestone 1 and 2 to his satisfaction, which I hope should satisfy the majority of your audit concerns. Here is a log of @mkchungs 's conversation with him for the milestone 1/2: Chat_Export_1_PDA_deep_analytics.pdf I booked a time slot for Dec 11, but I recommend you check in with him before the call. |
There was a problem hiding this comment.
@sourabhniyogi thanks for your time yesterday. I've talked to Michalis and have also taken a look. What currently confuses me is where the pollid comes from and how to use it. I was expecting it to match the referendum id, but it seems like that's not the case. Example:
- Here are the votes on referenda#323:
- The votes on the polkaholic dashboard for pollid 323 have completely different voting numbers:
Could you elaborate?
This project is not suitable for Docker-based review.
I'm fine with not using docker. However, after refreshing my memory on my previous comment, I came to the conclusion that you'd still have to provide a guide/tutorial for how to set up the infrastructure. This is because we support open-source software that should be reusable by other community players. Also, the NPM issues should be resolved.
|
@sourabhniyogi FYI - I'll be out of office for the next 2 weeks. In case you deliver an update to the code and somebody should take a look before I'm back, feel free to ping @keeganquigley or @semuelle on this PR. |
|
@sourabhniyogi just checking - do you have any updates here? |
|
Happy New Year -- we're taking your request very seriously by doing some
new years clean up this week!
…On Tue, 2 Jan 2024 at 18:37, S E R A Y A ***@***.***> wrote:
@sourabhniyogi <https://github.com/sourabhniyogi> just checking - do you
have any updates here?
—
Reply to this email directly, view it on GitHub
<#1038 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACUXO65SJG5CGS47VULH53YMS76PAVCNFSM6AAAAAA6QDVMAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZUG44DCNZXGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@sourabhniyogi great to hear! Looking forward to your next update then. |
|
@sourabhniyogi will this PR be closed as well or are you still working on it? |
|
pinging @sourabhniyogi |
|
We would like to keep it open for the rest of 2024 in expectation that the
"account research on traces" problem will be asked by someone else and we
can mechanize it within Dune and that we solve both Milestones in Dune this
year. We have to execute on a bunch of other things first.
…On Thu, 11 Apr 2024 at 06:54, Keegan | W3F ***@***.***> wrote:
pinging @sourabhniyogi <https://github.com/sourabhniyogi>
—
Reply to this email directly, view it on GitHub
<#1038 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACUXO5M7AO4R476G7MHGO3Y42IXPAVCNFSM6AAAAAA6QDVMAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBZG42DONZUGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@sourabhniyogi closing this for now, since it's been open for more than half a year. Feel free to re-open it, once it's ready to be reviewed again. |
Milestone Delivery Checklist
Link to the application pull request: Deep Account Analytics in Three Tiers for the Polkadot Data Alliance