Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

02 - Script to parser SDK python tests #144

Open
wants to merge 1 commit into
base: 01_evaluation_test_info_pull_sdk_docker_image
Choose a base branch
from

Conversation

hiltonlima
Copy link
Contributor

@hiltonlima hiltonlima commented Sep 12, 2024

The purpose of this PR is to resolve the display of the names of the python test steps.

Now we will leave it up to the SDK to deliver the information for each test. The backend will only pass the name of the test to the frontend.

The information is obtained by calling the get_test_info() method. Here is an example of the information for the TC_ACE_1_2 test:

{
        "class_name": "TC_ACE_1_2",
        "desc": "test_TC_ACE_1_2",
        "function": "test_TC_ACE_1_2",
        "path": "sdk/TC_ACE_1_2",
        "pics": [],
        "steps": [
            {
                "description": "Run entire test",
                "expectation": "",
                "is_commissioning": false,
                "test_plan_number": 1
            }
        ]
    },

Issue

project-chip/certification-tool#306

With this change no changes to the frontend will be necessary. And we can close this issue.

Code changes

  • Added parsed file for python sdk tests
  • Added script to generate to parse python files
  • Added '--get_test_info' parameter into 'test_harness_client.py' @rquidute

Output report

###########################################################################
###############################   REPORT   ################################
###########################################################################
>>>>>>>> Output JSON file: /app/backend/test_collections/matter/sdk_tests/python_tests_17b1a38e909e7874593bcb87c31be03a5866f1d4.json
>>>>>>>> Total of test functions: 185
>>>>>>>> Total of test functions (doesn't start with 'test_TC_'): 12
>>>>>>>> Total of scripts with error: 1
Error 0: Failed running command: ['sdk/TC_CCTRL_2_3', 'TC_CCTRL_2_3', '--get_test_info']
###########################################################################

@hiltonlima hiltonlima changed the title 02 Script to parser SDK python tests 02 - Script to parser SDK python tests Sep 16, 2024
@hiltonlima hiltonlima marked this pull request as draft September 16, 2024 13:35
@hiltonlima hiltonlima force-pushed the 01_evaluation_test_info_pull_sdk_docker_image branch from c5f9de9 to c52f447 Compare September 19, 2024 19:22
@hiltonlima hiltonlima force-pushed the 02_script_to_parsing_sdk_python_tests branch from 0c6f9b1 to c09cbc6 Compare September 19, 2024 19:23
Comment on lines +173 to +191
# collection = PythonCollectionDeclaration(
# name="Custom SDK Python Tests", folder=python_test_folder
# )

# suites = __parse_python_tests(
# python_test_version="custom",
# mandatory=False,
# )

# for suite in suites:
# if not suite.test_cases:
# continue
# suite.sort_test_cases()
# collection.add_test_suite(suite)

# if not collection.test_suites:
# return None

# return collection
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: After PoC Validation

@hiltonlima hiltonlima marked this pull request as ready for review September 19, 2024 19:25

with open(JSON_OUTPUT_FILE_PATH, "r") as json_file:
json_data = json.load(json_file)
json_file.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
json_file.close()

I think when using with statement, it not necessary to close the file. Please check this information.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're correct Romulo, it's not necessary to close.
It's the purpose of the with statement indeed.

Copy link

@fabiowmm fabiowmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's our end goal here? Let's clearly state our goals for the PRs we open. It may be obvious for the author but not so for the reviewers.

Copy link
Contributor

@ccruzagralopes ccruzagralopes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remember to fix the CI issues

print(f">>>>>>>> Total of test functions: {test_function_count}")
print(
(
">>>>>>>> Total of test functions (doesn't start with 'test_TC_'): "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
">>>>>>>> Total of test functions (doesn't start with 'test_TC_'): "
">>>>>>>> Total of test functions that don't start with 'test_TC_': "

or maybe

Suggested change
">>>>>>>> Total of test functions (doesn't start with 'test_TC_'): "
">>>>>>>> Total of invalid test functions (don't start with 'test_TC_'): "

?

@@ -119,24 +171,20 @@ def __test_methods(class_def: ast.ClassDef) -> list[FunctionDefType]:
return all_methods


def __test_case_names(methods: list[FunctionDefType]) -> list[str]:
def __test_case_name(function_name: str) -> Optional[str]:
"""Extract test case names from methods that match the pattern "test_TC_[\\S]+".
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Extract test case names from methods that match the pattern "test_TC_[\\S]+".
"""Extract test case name from methods that match the pattern "test_TC_[\\S]+".

Copy link

@fabiowmm fabiowmm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job.

@hiltonlima hiltonlima added v1.5 enhancement New feature or request labels Oct 23, 2024
@hiltonlima hiltonlima force-pushed the 01_evaluation_test_info_pull_sdk_docker_image branch from d3e5599 to 48f5e00 Compare October 24, 2024 12:59
@hiltonlima hiltonlima force-pushed the 02_script_to_parsing_sdk_python_tests branch from 1a3fccc to a79a5fd Compare October 24, 2024 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request v1.5
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants