No RuntimeApis available on a parachain

[th7nder]

Hey,

The runtime APIs are not available in the exported metadata.scale via subxt tool nor via explore command.
It doesn't work on our parachain, but on the template as well.

$ git clone https://github.com/paritytech/polkadot-sdk-parachain-template && cd polkadot-sdk-parachain-template
$ cargo build --release
$ zombienet -p native spawn zombienet.toml
[...]

$ subxt explore -a --url ws://localhost:9988
Usage:
    subxt explore pallet <PALLET>
        explore a specific pallet
    subxt explore api <RUNTIME_API>
        explore a specific runtime api

Available <PALLET>'s are:
    Aura
    AuraExt
[...]
There are no <RUNTIME_API>'s available.

There should be at least the default ones, like AccountNonceApi etc.

Are there any additional steps that need to be performed to export Runtime API in the metadata?

[jsdw]

In order to see Runtime APIs, the parachain needs to be using uptodate enough Substrate libraries such that it exposes V15 metadata or above. To check this, you can look for the runtime API Metadata_metadata_versions. If you call this API (eg via Polkadot.js UI), it will return the metadata versions that the runtime is capable of handing back. If it doesn't exist at all then V14 metadata will be the only one that Subxt etc can ask for, and this does not contain any information abotu Runtime APIs.

[th7nder]

It supports both v14 and v15.
Not sure how can I dive deeper to debug what's the issue is.

[jsdw]

Not sure how can I dive deeper to debug what's the issue is.

Is the code for your parachain public anywhere? With some steps to reproduce, we could look more into it! That'd be the best way we can help you.

If not, the next step I'd take is checking what metadata you're getting back from a node, which I think you could do by outputting it as --format json using the subxt CLI tool and seeing what version number is mentioned near the top of the outputted JSON. If it's V14, then we'd have to try the Subxt CLI tool on your parachain ourselves to see why.

One thought here is to make sure you're using the very latest version of the subxt CLI tool (subxt --version should be 0.38): older versions did not support V15 metadata; perhaps you have such a version lying about on your system?

If the metadata is V15, then it would be a case of looking into your parachain runtime code to see what the RuntimeApiMetadataIR that is generated looks like. You can do this using cargo expand on your runtime and IIRC enabling the sp-api/frame-metadata feature (here: https://github.com/paritytech/polkadot-sdk/blob/master/substrate/primitives/api/Cargo.toml#L70). Does the generated metadata IR contain any Runtime APIs?

[th7nder]

@jsdw Thanks a lot for the pointers, your help is really appreciated.

1. The parachain we're working on is here: https://github.com/eigerco/polka-storage, if you have Nix - then downloading the repo, and running just testnet will spawn the entire chain available on "ws://localhost:42069".

2. I queried metadata in JSON, it's returning V15, so it should be compatible.

galeheim ➜  subxt git:(master) ./target/release/subxt metadata -a --url http://127.0.0.1:42069 --format json > metadata.json
galeheim ➜  subxt git:(master) ✗ head -5 metadata.json 
[
  1635018093,
  {
    "V15": {
      "types": {

3. Subxt version I have is freshly compiled from source and 0.38.0, however in our repo we're using Nix and the subxt on nix is still 0.37.1:

galeheim ➜  subxt git:(master) ✗ ./target/release/subxt version
subxt-cli 0.38.0-0dbcdbdd9c6

4. Then I went into expanding the runtime and it seems there are APIs in there:

galeheim ➜  runtime git:(feat/449/verify-post) ✗ cargo expand > expanded.rs                                     
   Compiling polka-storage-runtime v0.0.0 (/Users/konrad.stepniak/workspace/polka-storage/runtime)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 33.54s

[...]
    #[doc(hidden)]
    trait InternalImplRuntimeApis {
        #[inline(always)]
        fn runtime_metadata(&self)
            ->
                sp_api::__private::vec::Vec<sp_api::__private::metadata_ir::RuntimeApiMetadataIR> {
            <[_]>::into_vec(#[rustc_box] ::alloc::boxed::Box::new([primitives_proofs::runtime_decl_for_storage_provider_api::runtime_metadata::<Block,
                                    AccountId>(),
                            sp_consensus_aura::runtime_decl_for_aura_api::runtime_metadata::<Block,
                                    AuraId>(),
                            cumulus_primitives_aura::runtime_decl_for_aura_unincluded_segment_api::runtime_metadata::<Block>(),
                            sp_api::runtime_decl_for_core::runtime_metadata::<Block>(),
                            sp_api::runtime_decl_for_metadata::runtime_metadata::<Block>(),
                            sp_block_builder::runtime_decl_for_block_builder::runtime_metadata::<Block>(),
[...]

aaaand it looks like they should be there, in the metadata.
but they're not :(
(attaching metadata)

./target/release/subxt metadata -a --url http://127.0.0.1:42069 --format json > metadata.json

metadata.json

[th7nder]

Ok, I found the issue. It was a wild ride though.

Root cause

paritytech/polkadot-sdk#3817 -> introduced a refactoring of the parachain template.
It moves the impl_runtime_apis decl into another module called apis.
Effectivly, trait InternalImplRuntimeApis ends up being under crate::apis::InternalImplRuntimeApis, not in the crate::InternalImplRuntimeApis.

When construct_runtime macro is expanded and uses (&rt).runtime_metadata, it's expanded under crate::* and has no visibility into crate::apis::InternalImplRuntimeApis, because:
a) trait is private
b) construct macro is not importing api::*

So in the end, the 'default, backward compatible' implementation is used.

Solution

1. Remove api.rs from the runtime move it under the lib.rs, so it gets expanded properly.
2. Make the construct_runtime macro module-aware (InternalImplRuntimeApis trait public at least), so it the correct metadata can be used.

I'm going to make a PR for our chain with the 1st solution, as it solves the issue and allows to progress further, however, not sure what will be the best for the overall ecosystem ( https://github.com/paritytech/polkadot-sdk-parachain-template ). Do you have any thoughts on that?

// EDIT:

• funny thing n1. is that enabling feature "frame-metadata" on sp-api crate does not seem to have any affect on the exported APIs.

[ntn-x2]

Hello. Sorry, I came across this issue because we had the same idea of refactoring the runtime code into multiple smaller modules. @th7nder your write-up was very informative to help me find a workaround. I did find that "aliasing" the private trait with a public one, and providing a blanket implementation for the new trait seems to work. Here's a commit that does that, perhaps it's helpful to someone that will stumble upon the same issue: KILTprotocol/kilt-node@7132eeb.

[jsdw]

@th7nder great job digging into this; your explanation is really helpful!

From what I understand, impl_runtime_apis! does indeed need to be called exactly once and in the main lib.rs as it stands, so it sounds like an issue with the parachain template to me; I'd open an issue there to bring up what you said!

A perhaps related issue in polkadot-sdk is paritytech/polkadot-sdk#3067 re multiple impl_runtime_api blocks not being supported. It's currently being explored whether this can be fixed in a sane way or not!

[th7nder]

Thanks for all of the support guys!
I created an issue in polkadot-sdk repo, so closing this one.
paritytech/polkadot-sdk#6659

TL;DR for anyone looking at it some time later:
a) Move impl_runtime_apis! out of the api.rs module like here (eigerco/polka-storage#605)
b) Do some blanket implementation trickery like here (KILTprotocol/kilt-node@7132eeb)
c) ... wait for polkadot-sdk decision