Trying to deserialize Signed Transaction with Program

[giekaton]

Hi and thanks for the library.

I'm trying to deserialize a signed transaction with the program, but getting the following error, maybe you see what's wrong and can help me.

Error:

...Python\Python37\site-packages\canoser\rust_enum.py", line 69, in decode
    _name, datatype = cls._enums[index]
IndexError: list index out of range

My signed txn is this:
signed_txn = "200000003A24A61E05D129CACE9E0EFC8BC9E33831FEC9A9BE66F50FD352A2638A49B9EE200000000000000000000000040000006D6F766502000000020000000900000043414645204430304402000000090000006361666520643030640300000001000000CA02000000FED0010000000D1027000000000000204E0000000000008051010000000000"

I have defined structs like this:

class Addr(Struct):
    _fields = [
      ('addr', [Uint8, 32])
    ]

class TransactionArgument(RustEnum):
    _enums = [
        ('U64', Uint64),
        ('Address', Addr),
        ('ByteArray', [Uint8]),
        ('String', str),
    ]

class TransactionProgram(Struct):
    _fields = [
        ( 'code', [Uint8]),
        ( 'args', TransactionArgument),
        ( 'modules', [Uint8] ),
    ]

class Raw(Struct):
    _fields = [
        ('sender', Addr),
        ('sequence_number', Uint64),
        ('payload', TransactionProgram),
        ('max_gas_amount', Uint64),
        ('gas_unit_price', Uint64),
        ('expiration_time', Uint64),
    ]

And the rest of the code:

def test():
    txn_bytes = bytes.fromhex(signed_txn)
    result = Raw.deserialize(txn_bytes)
    print(result)
test()

[yuan-xy]

You can refer this: https://github.com/yuan-xy/libra-client/blob/master/libra/transaction.py and the test code: https://github.com/yuan-xy/libra-client/blob/master/test/test_canoser.py

…

________________________________ 发件人: Gie Katon <[email protected]> 发送时间: Thursday, September 26, 2019 5:13:34 PM 收件人: yuan-xy/canoser-python <[email protected]> 抄送: Subscribed <[email protected]> 主题: [yuan-xy/canoser-python] Trying to deserialize Signed Transaction with Proof (#1) Hi and thanks for the library. I'm trying to deserialize a signed transaction with the program, but getting the following error, maybe you see what's wrong and can help me. Error: ...Python\Python37\site-packages\canoser\rust_enum.py", line 69, in decode _name, datatype = cls._enums[index] IndexError: list index out of range My signed txn is this: signed_txn = "200000003A24A61E05D129CACE9E0EFC8BC9E33831FEC9A9BE66F50FD352A2638A49B9EE200000000000000000000000040000006D6F766502000000020000000900000043414645204430304402000000090000006361666520643030640300000001000000CA02000000FED0010000000D1027000000000000204E0000000000008051010000000000" I have defined structs like this: class Addr(Struct): _fields = [ ('addr', [Uint8, 32]) ] class TransactionArgument(RustEnum): _enums = [ ('U64', Uint64), ('Address', Addr), ('ByteArray', [Uint8]), ('String', str), ] class TransactionProgram(Struct): _fields = [ ( 'code', [Uint8]), ( 'args', TransactionArgument), ( 'modules', [Uint8] ), ] class Raw(Struct): _fields = [ ('sender', Addr), ('sequence_number', Uint64), ('payload', TransactionProgram), ('max_gas_amount', Uint64), ('gas_unit_price', Uint64), ('expiration_time', Uint64), ] And the rest of the code: def test(): txn_bytes = bytes.fromhex(signed_txn) result = Raw.deserialize(txn_bytes) print(result) test() ― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAC55MWO5NXTJCDHYPPIVPTQLR4L5A5CNFSM4I2XI272YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HN2PIQQ>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAC55MVAAPDCJUPAWO4F22TQLR4L5ANCNFSM4I2XI27Q>.

[yuan-xy]

Added detailed doc here: https://github.com/yuan-xy/canoser-python#must-define-canoser-struct-by-serialized-fields-and-sequence-not-the-definition-in-the-rust-struct
Hope it can help you.

[giekaton]

Hi, thank you for the references and documentation updates. I'm learning and trying to run the LCS, but not quite there yet.

I see that in your transaction.py script, there are all details defined for the signed transaction deserialization.

So I'm using your transaction.py file, at the bottom I'm adding my custom code:

signed_txn = "200000003A24A61E05D129CACE9E0EFC8BC9E33831FEC9A9BE66F50FD352A2638A49B9EE200000000000000000000000040000006D6F766502000000020000000900000043414645204430304402000000090000006361666520643030640300000001000000CA02000000FED0010000000D1027000000000000204E0000000000008051010000000000"

txn_bytes = bytes.fromhex(signed_txn)
result = SignedTransaction.deserialize(txn_bytes)

print(result)

When I run it, I get the error:

...Python\Python37\site-packages\canoser\cursor.py", line 9, in read_bytes
    raise IOError("{} exceed buffer size: {}".format(end, total))
OSError: 142 exceed buffer size: 138

Maybe it's because my txn is a RawTransaction with a Program (as in Libra LCS specification), and not with a WriteSet. Do I need to specify this in structs? If not, any ideas what I'm doing wrong?

[giekaton]

The above signed_txn is a RawTransaction with a Program example from https://github.com/libra/libra/tree/master/common/canonical_serialization

[yuan-xy]

Maybe you can tell me the transaction id, so I can figure out what's the problem there.

[giekaton]

Please check my previous comment. It's an example transaction from the readme. At the very bottom of that page it's deserialized.

[yuan-xy]

You are right. The above signed_txn is a RawTransaction.

txn_bytes = bytes.fromhex(signed_txn)
result = libra.RawTransaction.deserialize(txn_bytes)
print(result)

It works fine.

[giekaton]

It works with RawTransaction.deserial... (without "libra." at the beginning)

When I print the deserialized result, I get the below message. How can I turn it into an object like representation, as it is in the LCS readme? I'm a Python newbie ツ

I will need to return it to my Node.js client from a spawned Python module.

<__main__.RawTransaction object at 0x000001522541BF08>

[yuan-xy]

I'm writing a str method to pretty print this object, comming soon

[giekaton]

Thank you very much. Looking forward to it!

Also, maybe this is something you know, and it's an easy answer.

To query a custom txn by range, I use a grpc client and the returned signed_txn looks like this:

signed_txn: "IAAAAB/HpLI25vah9LY90dFahBXFrtJYvHxrTT2+jxUQSw6SAQAAAAAAAAACAAAAuAAAAExJQlJBVk0KAQAHAUoAAAAEAAAAA04AAAAGAAAADVQAAAAGAAAADloAAAAGAAAABWAAAAApAAAABIkAAAAgAAAACKkAAAAPAAAAAAAAAQACAAEDAAIAAgQCAAMCBAIDAAY8U0VMRj4MTGlicmFBY2NvdW50BG1haW4PcGF5X2Zyb21fc2VuZGVyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQACAAQADAAMARMBAQICAAAAAQAAACAAAACbF/h+UR+X4dxOXN5KqU17lMJVNBEMJsl85hiXNVgvwAAAAABAQg8AAAAAAOAiAgAAAAAAAAAAAAAAAAB5KotdAAAAACAAAADGMHrvNmuIrxuW5cmzptu7rmo/bCOohwQSSpRMv/zmdkAAAADKZ+SsmtKC07wa7vK3nYVdkkeoLEmrpxCr/zw6T50jmB6TNp0XNpCT8D1p6JSQ5TlOYEyPvjZjl3yS0lHmps0D"

This doesn't look like the LCS format and it doesn't work with bytes.fromhex(signed_txn).

Does it need to be first decoded into an LCS representation, or is it more likely that my grpc client is returning it in the wrong format?

When querying directly from Rust CLI, the same txn looks like below:

raw_txn: RawTransaction {
        sender: 1fc7a4b..4b0e92,
        sequence_number: 1,
        payload: {,
                transaction: peer_to_peer_transaction,
                args: [
                        {ADDRESS: 9b17...82fc0},
                        {U64: 1000000},
                ]
        },
        max_gas_amount: 140000,
        gas_unit_price: 0,
        expiration_time: 1569401465s,
},
 public_key: Ed25519PublicKey(
    PublicKey(CompressedEdwardsY: [198, 48, 122, 239, 54, ...
 )
 signature: Ed25519Signature(
    Signature( R: CompressedEdwardsY: [202, 103, 22 ...
 )

[yuan-xy]

Update canoser to 0.3.4, add simple pretty print support.

[yuan-xy]

after get a proto class txn, you can get the SignedTransaction object with:

stx = SignedTransaction.deserialize(txn.signed_txn)

you can refer to this code:
test_get_transaction of libra-client

[giekaton]

Pretty printing works great. 非常感谢你.

And thanks for the reference regarding the deserialization of the signed_txn. I think I will finish everything tomorrow, now time sleep.

[giekaton]

Hi, regarding my second question, in your file test_client.py, I see that your client gets the signed_txn and decodes it here:

def test_get_transaction():
    c = libra.Client("testnet")
    txn = c.get_transaction(1)
    assert len(txn.signed_txn) > 0
    stx = SignedTransaction.deserialize(txn.signed_txn)

The signed_txn returned from my Node.js grpc client is in the format as I mentioned above and if I try to deserialize it with SignedTransaction.deserialize(signed_txn), I get this error: TypeError: a bytes-like object is required, not 'str'.

Any ideas?

[yuan-xy]

The input of deserialize method should be bytes type, not str type. 在 2019年9月27日，16:43，Gie Katon <[email protected]<mailto:[email protected]>> 写道： Hi, regarding my second question, in your file test_client.py, I see that your client gets the signed_txn and decodes it here: def test_get_transaction(): c = libra.Client("testnet") txn = c.get_transaction(1) assert len(txn.signed_txn) > 0 stx = SignedTransaction.deserialize(txn.signed_txn) The signed_txn returned from my Node.js grpc client is in the format as I mentioned above and if I try to deserialize it with SignedTransaction.deserialize(signed_txn), I get this error: TypeError: a bytes-like object is required, not 'str'. Any ideas? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAC55MTKUSUXO36WBJ3363LQLXBSZA5CNFSM4I2XI272YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7YG6JY#issuecomment-535850791>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAC55MRDUGL7IBYO5JCMRXDQLXBSZANCNFSM4I2XI27Q>.

[giekaton]

Can I convert it to bytes with Python, or should the grpc client return it in the right format?

Below in the comments you can see the errors I get when trying to decode it from string.

signed_txn = "IAAAAB/HpLI25vah9LY90dFahBXFrtJYvHxrTT2+jxUQSw6SAQAAAAAAAAACAAAAuAAAAExJQlJBVk0KAQAHAUoAAAAEAAAAA04AAAAGAAAADVQAAAAGAAAADloAAAAGAAAABWAAAAApAAAABIkAAAAgAAAACKkAAAAPAAAAAAAAAQACAAEDAAIAAgQCAAMCBAIDAAY8U0VMRj4MTGlicmFBY2NvdW50BG1haW4PcGF5X2Zyb21fc2VuZGVyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQACAAQADAAMARMBAQICAAAAAQAAACAAAACbF/h+UR+X4dxOXN5KqU17lMJVNBEMJsl85hiXNVgvwAAAAABAQg8AAAAAAOAiAgAAAAAAAAAAAAAAAAB5KotdAAAAACAAAADGMHrvNmuIrxuW5cmzptu7rmo/bCOohwQSSpRMv/zmdkAAAADKZ+SsmtKC07wa7vK3nYVdkkeoLEmrpxCr/zw6T50jmB6TNp0XNpCT8D1p6JSQ5TlOYEyPvjZjl3yS0lHmps0D"

signed_txn_bytes = bytes.fromhex(signed_txn)
# ERROR: ValueError: non-hexadecimal number found in fromhex() arg at position 0

# signed_txn_bytes = signed_txn.encode()
# The deserializer accepts the encoded string, but it is not in the LCS format, so returns the ERROR: TypeError: 1094795593 is not equal to predefined value: 32

des = SignedTransaction.deserialize(signed_txn_bytes)
print(des)

[yuan-xy]

It’s not a hex str, so you can’t call bytes.fromhex. Hex str only contain 0-9a-f. Where did you get this signed_txn? You can use struct.unpack to parse it with correct format. 在 2019年9月27日，17:42，Gie Katon <[email protected]<mailto:[email protected]>> 写道： Can I convert it to bytes with Python, or should the grpc client return it in the right format? Below in the comments you can see the errors I get when trying to decode it from string. signed_txn = "IAAAAB/HpLI25vah9LY90dFahBXFrtJYvHxrTT2+jxUQSw6SAQAAAAAAAAACAAAAuAAAAExJQlJBVk0KAQAHAUoAAAAEAAAAA04AAAAGAAAADVQAAAAGAAAADloAAAAGAAAABWAAAAApAAAABIkAAAAgAAAACKkAAAAPAAAAAAAAAQACAAEDAAIAAgQCAAMCBAIDAAY8U0VMRj4MTGlicmFBY2NvdW50BG1haW4PcGF5X2Zyb21fc2VuZGVyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQACAAQADAAMARMBAQICAAAAAQAAACAAAACbF/h+UR+X4dxOXN5KqU17lMJVNBEMJsl85hiXNVgvwAAAAABAQg8AAAAAAOAiAgAAAAAAAAAAAAAAAAB5KotdAAAAACAAAADGMHrvNmuIrxuW5cmzptu7rmo/bCOohwQSSpRMv/zmdkAAAADKZ+SsmtKC07wa7vK3nYVdkkeoLEmrpxCr/zw6T50jmB6TNp0XNpCT8D1p6JSQ5TlOYEyPvjZjl3yS0lHmps0D" signed_txn_bytes = bytes.fromhex(signed_txn) # ERROR: ValueError: non-hexadecimal number found in fromhex() arg at position 0 # signed_txn_bytes = signed_txn.encode() # The deserializer accepts the encoded string, but it is not in the LCS format, so returns the ERROR: TypeError: 1094795593 is not equal to predefined value: 32 des = SignedTransaction.deserialize(signed_txn_bytes) print(des) — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAC55MQLPLQK3NPOCK3GL2DQLXIRNA5CNFSM4I2XI272YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7YL2JA#issuecomment-535870756>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAC55MWDU4COM23AGKTFLZDQLXIRNANCNFSM4I2XI27Q>.

[giekaton]

That's how my Node.js libra-grpc client returns it. I updated my .proto files, but maybe something is still missing, I need to double-check then, or maybe switch to your client.

I will also double-check the struct.unpack, I guess I need to refer to the original Libra Rust source code, or are there Python unpack examples in your client?

[giekaton]

I rebuilt the .proto files from another Libra branch, but the signed_txn is returned in the same format as before. I guess grpc client works fine, but the unpacking is what's necessary.

In your code, regarding struct.unpack there is only bytes to int_list unpacking:

def bytes_to_int_list(bytes_str):
    tp = struct.unpack("<{}B".format(len(bytes_str)), bytes_str)
    return list(tp)

So for my signed_txn unpacking, any ideas?

def unpack_signed_txn(???):
    txn = struct.unpack( ???, ??? )
    return print(txn)

[yuan-xy]

It seems to be a base64 encoded str. base64.b64decode("IAAAAB/HpLI2") b' \x00\x00\x00\x1f\xc7\xa4\xb26' 在 2019年9月27日，23:00，Gie Katon <[email protected]<mailto:[email protected]>> 写道： I rebuilt the .proto files from another Libra branch, but the signed_txn is returned in the same format as before. I guess grpc client works fine, but the unpacking is what's necessary. In your code, regarding struct.unpack there is only bytes to int_list unpacking: def bytes_to_int_list(bytes_str): tp = struct.unpack("<{}B".format(len(bytes_str)), bytes_str) return list(tp) So for my signed_txn unpacking, any ideas? def unpack_signed_txn(???): txn = struct.unpack( ???, ??? ) return print(txn) — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAC55MRP6JEVWDWIKVUGU73QLYNZJA5CNFSM4I2XI272YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ZFVUQ#issuecomment-535976658>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAC55MS3P6E45DASGV25NELQLYNZJANCNFSM4I2XI27Q>.

[giekaton]

It works!!! Thanks a lot. I just need to finish with the decoding.

{
  raw_txn: {
    sender: [31, 199, 164, 178, 54, 230, 246, 161, 244, 182, 61, 209, 209, 90, 132, 21, 197, 174, 210, 88, 188, 124, 107, 77, 61, 190, 143, 21, 16, 75, 14, 146],
    sequence_number: 1,
    payload: Script{
      code: [76, 73, 66, 82, 65, 86, 77, 10, 1, 0, 7, 1, 74, 0, 0, 0, 4, 0, 0, 0, 3, 78, 0, 0, 0, 6, 0, 0, 0, 13, 84, 0, 0, 0, 6, 0, 0, 0, 14, 90, 0, 0, 0, 6, 0, 0, 0, 5, 96, 0, 0, 0, 41, 0, 
0, 0, 4, 137, 0, 0, 0, 32, 0, 0, 0, 8, 169, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 1, 0, 2, 0, 1, 3, 0, 2, 0, 2, 4, 2, 0, 3, 2, 4, 2, 3, 0, 6, 60, 83, 69, 76, 70, 62, 12, 76, 105, 98, 114, 97, 65, 99, 99, 111, 117, 110, 116, 4, 109, 97, 105, 110, 15, 112, 97, 121, 95, 102, 114, 111, 109, 95, 115, 101, 110, 100, 101, 114, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 2, 0, 4, 0, 12, 0, 12, 1, 19, 1, 1, 2],
      args: [<__main__.TransactionArgument object at 0x0000021F6AAEECC8>, <__main__.TransactionArgument object at 0x0000021F6AAEEEC8>],
    },
    max_gas_amount: 140000,
    gas_unit_price: 0,
    expiration_time: 1569401465,
  },
  public_key: [198, 48, 122, 239, 54, 107, 136, 175, 27, 150, 229, 201, 179, 166, 219, 187, 174, 106, 63, 108, 35, 168, 135, 4, 18, 74, 148, 76, 191, 252, 230, 118],
  signature: [202, 103, 228, 172, 154, 210, 130, 211, 188, 26, 238, 242, 183, 157, 133, 93, 146, 71, 168, 44, 73, 171, 167, 16, 171, 255, 60, 58, 79, 157, 35, 152, 30, 147, 54, 157, 23, 54, 144, 147, 240, 61, 105, 232, 148, 144, 229, 57, 78, 96, 76, 143, 190, 54, 99, 151, 124, 146, 210, 81, 230, 166, 205, 3],
}

[giekaton]

To decode sender, it works like this: print(int_list_to_hex(txn.raw_txn.sender))

But the args also need to be pretty-printed. If you can update your code, so that it pretty prints also the args, that would be great.

[yuan-xy]

I will do it, maybe need a few days. 在 2019年9月28日，17:49，Gie Katon <[email protected]<mailto:[email protected]>> 写道： To decode sender, it works like this: print(int_list_to_hex(txn.raw_txn.sender)) But the args also need to be pretty-printed. If you can update your code, so that it pretty prints also the args, that would be great. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1?email_source=notifications&email_token=AAC55MSK2G3WNAGL4BZCOSTQL4SAVA5CNFSM4I2XI272YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD72VFRY#issuecomment-536171207>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAC55MULXNWJOMJMUZEYNO3QL4SAVANCNFSM4I2XI27Q>.

[giekaton]

Thanks, looking forward to it, that's the only part I'm missing. Meanwhile, I will finish integrating this LCS as a spawned child process in my Node.js grpc client. It already works, just need to run it on the real testnet. I will later share the code on GitHub.

[yuan-xy]

The code of address formatting is updated.

[giekaton]

Hi, thanks a lot, works great. I have already integrated your LCS into Libra Checker. I still need to finish some things, then will update the GitHub.

As far as I know, there is yet no LCS written in JavaScript, so your Python LCS running in Node.js as a child process is a cool workaround.

[yuan-xy]

Maybe I can port Canoser from python to JavaScript.

[giekaton]

There are no solutions yet:
perfectmak/libra-core#48
https://community.libra.org/t/rawtransaction-object-removed-from-transaction-proto/1771

So I guess LCS in JS would be very useful for the community.

[giekaton]

Hi Yuan,

I'm still using Canoser and following your code on libra-client. Great work, much appreciated.

Can you help me to decode the deserialized txn payload's "code" field?

My current python file looks like this:
https://gist.github.com/giekaton/ff139bd81c1d041aa300b572b67ca8a9

Example query: Transaction nr 44 (the result from Canoser is below)

I need to extract the transaction_type from the code field.

UserTransaction: {
  raw_txn: {
    sender: 000000000000000000000000000000000000000000000000000000000a550c18,
    sequence_number: 43,
    payload: Script: {
      code: 4c49425241564d0a010007014a000000060000000350000000060000000d56000000060000000e5c0000000600000005620000003300000004950000002000000008b50000000f000000000000010002000300010400020002040200030204020300063c53454c463e0c4c696272614163636f756e74094c69627261436f696e046d61696e0f6d696e745f746f5f616464726573730000000000000000000000000000000000000000000000000000000000000000000100020004000c000c0113010102,
      args: [
        Address: dfef62fa6d938c838d4bbbcf394fe4e5d96392578efaa338d0510bb66bff7167,
        U64: 100000000,
      ],
    },
    max_gas_amount: 140000,
    gas_unit_price: 0,
    expiration_time: 1572491151,
  },
  public_key: 664f6e8f36eacb1770fa879d86c2c1d0fafea145e84fa7d671ab7a011a54d509,
  signature: 74366a7d290f8dfa0de3e500b7683b74db33d17ceade19c4cc553b2b1bb8d2e148e0409a7c79ed2e089c2ff4b74189f2f9ef5840cab8af81a64b30a4b7918703,
}

[yuan-xy]

I'm currently writing an openAPI for Libra, you may use the API to solve the problem later. But now, you can do like this:

from libra.bytecode import bytecodes

#tx = tx.to_json_serializable()
#payload = tx['raw_txn']['payload']

def get_tx_abbreviation_name(payload, version):
    if version == 0:
        return "Genesis"
    try:
        payload['Script'] = payload.pop('Program')
    except KeyError:
        pass
    if list(payload)[0] != "Script":
        return list(payload)[0]
    code = hex_to_int_list(payload['Script']['code'])
    if code == bytecodes["mint"]:
        return "mint"
    if code == bytecodes["peer_to_peer_transfer"]:
        return "p2p"
    if code == bytecodes["create_account"]:
        return "new account"
    if code == bytecodes["rotate_authentication_key"]:
        return "rotate key"
    return "script"

[giekaton]

It works and I learned a lot from your code snippet. Thanks. Looking forward to the openAPI.