Support decompiling transaction messages fetched using getTransaction that involve lookup tables in a single pass.

[lithdew]

Motivation

Suppose we use getTransaction to fetch the base64/base58-encoded bytes of a v0 transaction.

getTransaction additionally provides the list of addresses which the transaction loads from lookup tables as meta.loadedAddresses.

It would be great to be able to provide meta.loadedAddresses to decompileTransactionMessage in order to fully decode a transaction message.

Doing this manually at the moment is quite a bit of boilerplate and requires in-depth understanding as to how addresses are organized within a v0 transaction.

A workaround at the moment is to use decodeTransactionMessage, though decodeTransactionMessage in this case would inefficiently fetch the lookup table's addresses from the RPC even though we already have the data at hand from getTransaction.

Example use case

decompileTransactionMessage(compiledTransactionMessage, {
    loadedAddresses: result.meta?.loadedAddresses,
});

Details

N/A

[mcintyre94]

This sounds like a good idea!

I think we could implement it like this, contingent on the RPC ordering things sensibly (which I haven't checked):

• Take loadedAddresses as input, eg

"loadedAddresses": {
    "readonly": [
        "So11111111111111111111111111111111111111112",
        "5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1",
        "675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8"
    ],
    "writable": [
        "728XPhZKjAYWp6dys98pvQg1zukjRq5Cckt2tqpEDgrS",
        "9DZiJL5dwHVje2MeDNAWNZkxYvF6y7jzqZUxS5BPFFdn",
        "GJmxsfhhho2nej3Bc2kSpc7DGJBPXPAinAYTkTHsKRob"
    ]
},

• When we decompile the transaction message we will have an addressLookupTables field like this (on the CompiledTransactionMessage)

"addressTableLookups": [
    {
        "accountKey": "GtXcpBiwyhpd8sJrUDcBaRKWt3oUnsZijwBWP4hwJh3y",
        "readonlyIndexes": [
            23,
            3,
            0
        ],
        "writableIndexes": [
            155,
            158,
            153
        ]
    }
],

This should be sufficient to generate IAccountLookupMeta for the loaded addresses.

Eg So11111111111111111111111111111111111111112 becomes:

{
  address: 'So11111111111111111111111111111111111111112',
  addressIndex: 23,
  lookupTableAddress: 'GtXcpBiwyhpd8sJrUDcBaRKWt3oUnsZijwBWP4hwJh3y',
  role: AccountRole.READONLY

The assumption this is contingent on is how the RPC orders loadedAddresses if there are multiple lookup tables though.

Suppose we add another address table lookup:

"addressTableLookups": [
    {
        "accountKey": "GtXcpBiwyhpd8sJrUDcBaRKWt3oUnsZijwBWP4hwJh3y",
        "readonlyIndexes": [
            23,
            3,
            0
        ],
        "writableIndexes": [
            155,
            158,
            153
        ]
    },
    {
        "accountKey": "test",
        "readonlyIndexes": [0],
        "writableIndexes": [1]
    }
],

Then loadedAddresses needs to be:

"loadedAddresses": {
    "readonly": [
        "So11111111111111111111111111111111111111112",
        "5Q544fKrFoe6tsEbD7S8EmxGTJYAKtTVhAW5Q5pge4j1",
        "675kPX9MHTjS2zt1qfr1NYHuzeLXfQM9H24wFSUt1Mp8",
       "index 0 of test"
    ],
    "writable": [
        "728XPhZKjAYWp6dys98pvQg1zukjRq5Cckt2tqpEDgrS",
        "9DZiJL5dwHVje2MeDNAWNZkxYvF6y7jzqZUxS5BPFFdn",
        "GJmxsfhhho2nej3Bc2kSpc7DGJBPXPAinAYTkTHsKRob",
        "index 1 of test"
    ]
},

We will then have enough information to correctly map these addresses to their lookup table and index.

As an aside the function names I used here are pretty bad, probably need to think of something better and more descriptive.

[steveluscher]

It looks like it just concatenates the readable and writable addresses together.

https://github.com/anza-xyz/agave/blob/f9f8b60ca15fa721c6cdd816c99dfd4e9123fd77/sdk/program/src/message/versions/v0/loaded.rs#L39-L40

As for how it orders the addresses of the lookup table accounts themselves, it looks like it's just in the order found in the message.

https://github.com/anza-xyz/agave/blob/f9f8b60ca15fa721c6cdd816c99dfd4e9123fd77/sdk/program/src/message/versions/v0/mod.rs#L268-L275

Whatever we do, we shouldn't bother decompileTransactionMessage with this responsibility. We should instead add a utility that will produce a AddressesByLookupTableAddress for use with decompileTransactionMessage.

// NEW
const addressesByLookupTableAddress = createAddressesByLookupTableAddressFromLoadedAddresses({
    addressTableLookups, // compiledTransactionMessage.addressTableLookups
    loadedAddresses, // result.meta.loadedAddresses
});
// EXISTING
const message = decompileTransactionMessage(compiledTransactionMessage, {
    addressesByLookupTableAddress,
});

[lithdew]

@steveluscher thanks for the links with regards to ordering - I implemented createAddressesByLookupTableAddressFromLoadedAddresses for my codebase just now.

I confirmed that addresses are fetched and ordered correctly. Test code below references a random transaction I picked off of a recent slot.

it.only("works", async () => {
  const rpc = createRpc({ ... });

  const raw = await rpc
    .getTransaction(
      signature(
        "23ziNjdwh6bMYBkGnbTNHuYYmWa6tBmNrmvDBpp2U7DgSUFtzhXPkriDdyrCmVRWxs97yK7HKMmtP6msH655yTtM"
      ),
      {
        maxSupportedTransactionVersion: 0,
        encoding: "base64",
      }
    )
    .send();

  const tx = getTransactionDecoder().decode(
    Buffer.from(raw!.transaction[0], "base64")
  );
  const compiledTransactionMessage =
    getCompiledTransactionMessageDecoder().decode(tx.messageBytes);

  function createAddressesByLookupTableAddressFromLoadedAddresses({
    loadedAddresses,
    addressTableLookups,
  }: {
    loadedAddresses: {
      writable: readonly Address[];
      readonly: readonly Address[];
    };
    addressTableLookups: {
      lookupTableAddress: Address;
      writableIndices: readonly number[];
      readableIndices: readonly number[];
    }[];
  }) {
    const addressesByLookupTableAddress: AddressesByLookupTableAddress = {};

    const loadedWritableAddresses = [...loadedAddresses.writable];
    const loadedReadonlyAddresses = [...loadedAddresses.readonly];
    for (const lookup of addressTableLookups) {
      const lookupTableAddresses = new Array<Address>();
      for (const writableIndex of lookup.writableIndices) {
        lookupTableAddresses[writableIndex] = loadedWritableAddresses.shift()!;
      }
      for (const readableIndex of lookup.readableIndices) {
        lookupTableAddresses[readableIndex] = loadedReadonlyAddresses.shift()!;
      }
      addressesByLookupTableAddress[lookup.lookupTableAddress] =
        lookupTableAddresses;
    }

    return addressesByLookupTableAddress;
  }

  let addressesByLookupTableAddress: AddressesByLookupTableAddress | undefined =
    undefined;
  if (
    "addressTableLookups" in compiledTransactionMessage &&
    compiledTransactionMessage.addressTableLookups !== undefined &&
    raw?.meta?.loadedAddresses !== undefined
  ) {
    addressesByLookupTableAddress =
      createAddressesByLookupTableAddressFromLoadedAddresses({
        loadedAddresses: raw.meta.loadedAddresses,
        addressTableLookups: compiledTransactionMessage.addressTableLookups,
      });
  }

  const transactionMessage = decompileTransactionMessage(
    compiledTransactionMessage,
    {
      addressesByLookupTableAddress,
    }
  );

  console.dir(transactionMessage, { depth: Infinity });
});

[steveluscher]

Neat. If you want to write comprehensive tests for that, write a README entry, and PR it up for @solana/transaction-messages (I think that's where it belongs?) I'd be happy to accept that.

[steveluscher]

Feel free to knock out solana-labs/web3.js-issue-conveyer#12 while you're in this area.