Testing framework

[janaakhterov]

Objective: Create a testing framework where all integration tests are defined in one place, and can be run on any SDK and on any network.

Current workflow:

• Create integration tests which set all fields to some value, and confirm the request goes through. Usually requires querying the info from Hedera Services using *InfoQuery .
• Review new response codes within ResponseCode.proto.
• Create integration tests which set fields to certain values to correctly produce an expected status.
• Copy these integration tests into each SDK to confirm each SDK functions correctly and identically.
• Have an outside person test the SDKs to make sure we haven't missed any.
• Add missing integration tests if any.

Proposed Behavior:

• Define integration tests in a central location using configuration files which specify:
  • Which requests to create
  • Which fields to set on these requests
  • The order to execute these requests
  • The expected response of each request
  • The networks this test is applicable to
• Run these integration test configurations against any network on any SDK even if an SDK doesn't currently have certain requests implemented

Benefit:

• Having all the integration test defined in a central location:
  • Makes it easy to read and write all the expected behavior
• Having the integration tests separated from the SDKs themselves:
  • Allows the integration test to be defined in parallel as the SDK is being developed
  • Allows us to prove all the SDKs function identically

Option 1: Build a single testing framework that uses a REST API to execute request on each SDK

Pros:

• Only need to write the testing framework once meaning there is less chance for mistakes with the testing framework itself
• Lower chance for mistakes to happen within the integration tests since they're only written once
• Makes it easy to run integration tests against any SDK on any network
• Does not need a CLI tool

Cons:

• Requires time to construct these testing configuration files
• Harder to reason about each SDK's tests since they're not located in the SDK nor the same language.
• Each SDK needs to implement a REST API

e.g.

• Body:

{
    "request": {
        "type": "ContractCreateTransaction":,
        "data": {
            "adminKey": "clientProperty(operatorKey)",
            "gas": "2000",
            "bytecodeFileId": "stepProperty(fileCreateTransaction.fileId)",
            "contractMemo": "[e2e::ContractCreateTransaction]"
        }
    },
    "operatorId": "0.0.47439",
    "signers": [
        "302e020100300506032b6570042204208f4014a3f7f7a6c7147070da98d88f9cea074c13ed0554783471825d801888cf"
    ]
}

• Response:

{
    "response": {
        "receipt": {
            "status": "SUCCESS",
            "contractId": "0.0.154154"
            // ... 
        }
    }
}

Option 2: Build a single testing framework that uses a CLI tool to execute request on each SDK

Pros:

• Only need to write the testing framework once meaning there is less chance for mistakes with the testing framework itself
• Lower chance for mistakes to happen within the integration tests since they're only written once
• Makes it easy to run integration tests against any SDK on any network
• Does not need a REST API

Cons:

• Requires time to construct these testing configuration files
• Harder to reason about each SDK's tests since they're not located in the SDK nor the same language.
• Each SDK needs to create a CLI to execute requests

Notes:

• The input/output of this CLI tool would be identical to the REST API

Option 3: Build the testing framework in each SDK language

Pros:

• Does not need a REST API
• Does not need a CLI tool
• Lower chance for mistakes to happen within the integration tests since they're only written once

Cons:

• Requires time to construct these testing configuration files
• Harder to reason about each SDK's tests since they're not located in the SDK nor the same language.
• Each SDK will need to implement the same behavior for testing which includes:
  • Parsing entire configuration file
  • Parsing keywords in values
  • Printing the same logs
  • Implementing behavior of asserting responses are identical
  • Implement same state which holds onto current constants, accounts, steps performed, etc.

Option 4: Centralized location for test, but no testing framework

Pros:

• Do not need to allocate any time into building a testing framework
• Does not need a REST API
• Does not need a CLI tool
• Easier to reason about each SDK since all the tests are defined in the same language

Cons:

• Requires time to construct these testing configuration files
• Requires developers to make sure the same tests exist in each SDK which takes developer time.
• Higher chance of mistakes happening.

Option 5: Do nothing

Pros:

• Do not need to allocate any time into building a testing framework
• Does not need a REST API
• Does not need a CLI tool
• Easier to reason about each SDK since all the tests are defined in the same language

Cons:

• No centralized location to define expected behavior of SDKs and Hedera Services
• Requires developers to review all tests to make sure identical tests are created which wastes a lot of developer time
• Higher probability of mistakes happening

---

Example test configuration file which defines the current a test on the expected behavior of a ContractBytecodeQuery

{
    // Define all the constants for this test
    "constants": {
        // For smart contracts we just save the bytecode in a constant field
        "smartContractBytecode": "608060405234801561001057600080fd5b506040516104d73803806104d78339818101604052602081101561003357600080fd5b810190808051604051939291908464010000000082111561005357600080fd5b90830190602082018581111561006857600080fd5b825164010000000081118282018810171561008257600080fd5b82525081516020918201929091019080838360005b838110156100af578181015183820152602001610097565b50505050905090810190601f1680156100dc5780820380516001836020036101000a031916815260200191505b506040525050600080546001600160a01b0319163317905550805161010890600190602084019061010f565b50506101aa565b828054600181600116156101000203166002900490600052602060002090601f016020900481019282601f1061015057805160ff191683800117855561017d565b8280016001018555821561017d579182015b8281111561017d578251825591602001919060010190610162565b5061018992915061018d565b5090565b6101a791905b808211156101895760008155600101610193565b90565b61031e806101b96000396000f3fe608060405234801561001057600080fd5b50600436106100415760003560e01c8063368b87721461004657806341c0e1b5146100ee578063ce6d41de146100f6575b600080fd5b6100ec6004803603602081101561005c57600080fd5b81019060208101813564010000000081111561007757600080fd5b82018360208201111561008957600080fd5b803590602001918460018302840111640100000000831117156100ab57600080fd5b91908080601f016020809104026020016040519081016040528093929190818152602001838380828437600092019190915250929550610173945050505050565b005b6100ec6101a2565b6100fe6101ba565b6040805160208082528351818301528351919283929083019185019080838360005b83811015610138578181015183820152602001610120565b50505050905090810190601f1680156101655780820380516001836020036101000a031916815260200191505b509250505060405180910390f35b6000546001600160a01b0316331461018a5761019f565b805161019d906001906020840190610250565b505b50565b6000546001600160a01b03163314156101b85733ff5b565b60018054604080516020601f600260001961010087891615020190951694909404938401819004810282018101909252828152606093909290918301828280156102455780601f1061021a57610100808354040283529160200191610245565b820191906000526020600020905b81548152906001019060200180831161022857829003601f168201915b505050505090505b90565b828054600181600116156101000203166002900490600052602060002090601f016020900481019282601f1061029157805160ff19168380011785556102be565b828001600101855582156102be579182015b828111156102be5782518255916020019190600101906102a3565b506102ca9291506102ce565b5090565b61024d91905b808211156102ca57600081556001016102d456fea264697066735822122084964d4c3f6bc912a9d20e14e449721012d625aa3c8a12de41ae5519752fc89064736f6c63430006000033",
    },

    // Define the types of accounts which can execute this test
    // Some requests require system keys such as the genesis key
    //
    // If this list contains only "genesis" and the operator is not `0.0.2` this test
    // will be skipped
    "accountTypes": [ "genesis", "regular" ],

    // Define the newtorks this test is applicable to
    // Perhaps we can also support `integrationnet` to allow testing
    // requests that are a WIP
    //
    // If the client network is not in this list, the test will be skipped
    "networks": [ "testnet", "previewnet", "mainnet" ],

    // Define a list of account that need to be created to run this test
    //
    // If this list is of length 1, the account created will become the client operator,
    // otherwise you can use `accounts(<index>)` e.g. `accounts(0)` to get the first account
    "accounts": [{
        // Define the initial balance of this account
        "initialBalance": "100 ℏ",

        // Define if the account should deleted after all the test steps
        // and all the test step deletions
        "delete": true
    }],

    // Define the steps for this test
    // The steps defined here will be executed in order
    "steps": [
        // Create a file containing the contracts bytecode
        {
            // Name this test step to use the result in a following test step
            "name": "fileCreateTransaction",

            // Add a description of what this test step is trying to acheive
            // Can be used to just write comments
            "description": "Create the file containing the bytecode of the contract",

            // Define if the receipt should be fetched for this request automatically
            //
            //  - If this value is `true` the response of this step will be `TransactionReceipt`
            //  - If this value is `false` the response of this step will be `TransactionResponse`
            //
            //  Notes:
            //  - This field is ignored if the request is not a transaction
            //
            // The default value is `true`
            //
            "receipt": true,

            // Define if we should delete the entity that is created by this transaction
            // 
            //  - If this value is `true` the entity created by this request will be deleted after
            //    all the test steps have completed
            //  - If this value is `false` no further computation is done
            //
            //
            //  Notes:
            //  - This field is ignored for any request that isn't `*CreateTransaction`
            //  - Entities are deleted in the reverse order they were created in
            //    e.g. if we create a file here, and in the next step we create a contract
            //    the contract will be deleted first, then the file
            //  - If you're creating an immutable entity be sure to set this value to `false`
            //    otherwise a delete transaction will be executed resulting in an error
            //
            //  the defaule value is `true`
            //
            "delete": true,

            // Define the request to submit
            "request": {
                "type": "FileCreateTransaction",
                "data": {
                    // Any string value that looks like `keyword(.*)` is going to be parsed
                    // by the testing framework and filled in with the relevant information.
                    // In this case the `Client.getOperatorKey()` will be used here.
                    "keys": [ "clientProperty(operatorKey)" ],

                    // This also follows the pattern `keyword(.*)` so the testing framework
                    // will parse this; the keyword `constants` means use the top level field
                    // `constants` and retreive the value for `smartContractBytecode`
                    "contents": "constant(smartContractBytecode)"
                }

                // Define a list of private keys that need to sign this request
                //
                // Notes:
                //  - The operator of the client by default signs all requests
                //  - These keys could be defined as constants and used here as `constant(key1)`
                //
                // "signers": [
                //     "<key1>",
                //     "<key2>"
                // ]
            },

            // Define what the response is expected
            "response": {
                // Define the expected precheck for this request
                // Since `request.receipt` is `true` *and* the request is a transaction this is ignored
                // "precheck": {
                //     "status": "OK"
                // },

                // Since `request.receipt` is `true` the response is a `TransactionReceipt`
                "receipt": {
                    // Assert this field is equal to `SUCCESS`
                    "status": "SUCCESS",

                    // Assert this field is not null
                    "fileId": "assertNotNull()"
                }
            }
        },

        // Create the contract
        {
            "name": "createContractTransaction",
            "description": "Create a contract with the file aforementioned file",
            "request": {
                "type": "ContractCreateTransaction":,
                "data": {
                    "adminKey": "clientProperty(operatorKey)",
                    "gas": "2000",
                    "bytecodeFileId": "stepProperty(fileCreateTransaction.fileId)",
                    "contractMemo": "[e2e::ContractCreateTransaction]"
                }
            },
            "response": {
                "receipt": {
                    "status": "SUCCESS",
                    "contractId": "assertNotNull()"
                }
            }
        },

        // Query the bytecode for the contract
        {
            "name": "contractBytecodeQuery",
            "description": "Query the bytecode for the contract and make sure it is what we expecte",
            "request": {
                "type": "ContractBytecodeQuery",
                "data": {
                    "contractId": "stepProperty(contractCreateTransaction.contractId)"
                }
            },
            "response": {
                // Since `ContractBytecodeQuery` returns bytes directly instead of some wrapper type
                // we compare the entire response instead of a field of the response
                "data": "constant(smartContractBytecode)"
            }
        }
    ]
}

The equivalent test in Java would look like:

assertDoesNotThrow(() -> {
    var testEnv = new IntegrationTestEnv(1);

    @Var var response = new FileCreateTransaction()
        .setKeys(testEnv.operatorKey)
        .setContents(SMART_CONTRACT_BYTECODE)
        .execute(testEnv.client);

    var fileId = Objects.requireNonNull(response.getReceipt(testEnv.client).fileId);

    response = new ContractCreateTransaction()
        .setAdminKey(testEnv.operatorKey)
        .setGas(2000)
        .setConstructorParameters(new ContractFunctionParameters().addString("Hello from Hedera."))
        .setBytecodeFileId(fileId)
        .setContractMemo("[e2e::ContractCreateTransaction]")
        .execute(testEnv.client);

    var contractId = Objects.requireNonNull(response.getReceipt(testEnv.client).contractId);

    var bytecode = new ContractByteCodeQuery()
        .setContractId(contractId)
        .execute(testEnv.client);

    assertEquals(Hex.toHexString(bytecode), SMART_CONTRACT_BYTECODE);

    new ContractDeleteTransaction()
        .setContractId(contractId)
        .execute(testEnv.client)
        .getReceipt(testEnv.client);

    new FileDeleteTransaction()
        .setFileId(fileId)
        .execute(testEnv.client)
        .getReceipt(testEnv.client);

    testEnv.close();
});

Configuration file keywords

• constant
  • retrieve the value for the constant with the name passed in
• accounts
  • retrieve one of the accounts that were created
  • The value returned has two fields, key and id
• stepProperty
  • retrieve the value for a previous test step
• clientProperty
  • retrieve the value for a certain property that exist on Client
• assertNotNull
  • assert the value is not null
• assertNull
  • assert the value is null
• assertEquals
  • assert the value is equal to value passed in
  • This is implied if no other assert* is used

[SimiHunjan]

@rbair23 please see above

[rbair23]

Let's also consider the quantity of tests involved. Our testing should be a pyramid, where the bulk (base) of the pyramid are unit tests and integration tests are closer to the top. If we're thinking we need a large quantity of integration tests, we may consider doing things differently than if we're thinking of a smaller number of integration tests.

[janaakhterov]

It does make sense that we should have more unit tests than integration tests, and I don't think this framework would change that since each SDK and Hedera Services already maintain their own unit tests. To clarify, the tests defined in the testing framework would only be written once and will be used to confirm that both the SDKs and Hedera Services run correctly, unlike right now which require each SDK and Hedera Services to create and maintain their own integration tests.

[gregscullard]

while looking at unit tests on a different project, I came across gRPC mocking frameworks which could be used to mock the hedera services (at least in Java). I have a small sample project that appears to work as a test (see attached ZIP)

https://github.com/Fadelis/grpcmock

grpcMock.zip

There is also a more complex "over the wire" solution: https://trafficparrot.com/tutorials/mocking-and-simulating-grpc.html which could support all the SDKs.

[janaakhterov]

Mocking is not a bad idea for unit tests, but that's not what I was trying to achieve with this testing framework. What I was looking for is testing both Hedera SDK's and Hedera Services e2e tests to make sure both pieces function correctly.

[rbair23]

We use the Java SDK + Services + Mirror Node in our e2e tests. Do we need the same for all the other SDKs? Or would a mock server be better?

Essentially what you're trying to define here is a TCK that we can throw any SDK at and verify that it is a valid SDK. For that we don't necessarily need a real Hedera server (although having that as a possibility is really nice and should be a design goal).

I was thinking about the TCK angle some more. The point to having a CLI or a web server for each SDK is to work as an integration for a TCK. Is it possible to actually just use a single TCK application, maybe using Graal, and have it call VM-based (Java, JS) or native (Ruby, Go, etc)? Are the interfaces for the different SDKs similar enough to allow for that?

[janaakhterov]

Firstly I need to mention that I don't have that much experience with mocking servers. However, from what I've seen mocking a full blown Hedera Services would be quite challenging and could lead to mistakes written specifically in the mock implementation of Hedera Services which is why I advocate for testing a live instance of Hedera Services instead of mocking it. I'd like the e2e tests to test absolutely everything related to the SDK and Hedera Services which I think would require a very complex mock server. As for testing each SDK I can't argue enough for this. At the moment the Java SDK is treated as a first class citizen while the other SDKs are trying their best to imitate the Java SDK which I'd argue is not a good development strategy.

As for your last point I can't say much since I don't know what TCK or Graal are. However, I can say that the interface for the different SDKs aren't that similar. Yes, they do use the builder pattern for each request, but that's really as far as the similarities go. The execution of a request and handling of errors is different enough that I'd say it's better to implement some tool in each SDK to actually execute a request. Although, maybe I misunderstood what you mean by similarities.