Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Higher test coverage for tests/client-sdk #651

Open
yanxi0830 opened this issue Dec 18, 2024 · 0 comments
Open

Higher test coverage for tests/client-sdk #651

yanxi0830 opened this issue Dec 18, 2024 · 0 comments
Assignees
@vladimirivic

Comments

@yanxi0830
Copy link
Contributor

yanxi0830 commented Dec 18, 2024 •
edited
Loading

🚀 Describe the new functionality needed

Why

  • We want a comprehensive & consolidated test suite covering all functionalities

What

  • Audit existing tests on functionalities in providers/tests
  • Audit examples in llama-stack-apps & showcase notebooks
  • Improve test coverage for client-sdk

Requirements

  • add more integration tests in tests/client-sdk covering functionalities in llama-stack-apps

Code Pointer

💡 Why is this needed? What if we don't build it?

reliability

Other thoughts

No response
@yanxi0830 yanxi0830 mentioned this issue Dec 18, 2024
Open
@vladimirivic vladimirivic mentioned this issue Dec 31, 2024
Open
vladimirivic added a commit that referenced this issue Jan 1, 2025
@vladimirivic
Consolidating Memory tests under client-sdk
ff09600 
Summary:
Part of #651

Requirements
* add more integration tests in tests/client-sdk covering functionalities in llama-stack-apps

Porting tests from
* llama_stack/providers/tests/memory/test_memory.py

Ensuring we cover some basic functions
* MemoryResource src/llama_stack_client/resources/memory.py
* MemoryBanksResource src/llama_stack_client/resources/memory_banks.py

Test Plan:
Run against the stack as lib
```
LLAMA_STACK_CONFIG=tests/client-sdk/memory/resources/run.yaml pytest tests/client-sdk/memory -v

tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 20%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 40%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 60%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 80%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```


Run agianst the local server
```
LLAMA_STACK_BASE_URL=http://localhost:5000 pytest tests/client-sdk/memory -v


tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 20%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 40%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 60%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 80%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```
@vladimirivic vladimirivic mentioned this issue Jan 1, 2025
Open
vladimirivic added a commit that referenced this issue Jan 1, 2025
@vladimirivic
Consolidating Memory tests under client-sdk
8c528c4 
Summary:
Part of #651

Requirements
* add more integration tests in tests/client-sdk covering functionalities in llama-stack-apps

Porting tests from
* llama_stack/providers/tests/memory/test_memory.py

Ensuring we cover some basic functions
* MemoryResource src/llama_stack_client/resources/memory.py
* MemoryBanksResource src/llama_stack_client/resources/memory_banks.py

Test Plan:
Run against the stack as lib
```
LLAMA_STACK_CONFIG=tests/client-sdk/memory/resources/run.yaml pytest tests/client-sdk/memory -v

tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 20%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 40%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 60%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 80%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```


Run agianst the local server
```
LLAMA_STACK_BASE_URL=http://localhost:5000 pytest tests/client-sdk/memory -v


tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 20%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 40%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 60%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 80%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```
vladimirivic added a commit that referenced this issue Jan 3, 2025
@vladimirivic
Consolidating Memory tests under client-sdk
75e8a3a 
Summary:
Part of #651

Requirements
* add more integration tests in tests/client-sdk covering functionalities in llama-stack-apps

Porting tests from
* llama_stack/providers/tests/memory/test_memory.py

Ensuring we cover some basic functions
* MemoryResource src/llama_stack_client/resources/memory.py
* MemoryBanksResource src/llama_stack_client/resources/memory_banks.py

Test Plan:
Run against the stack as lib
```
LLAMA_STACK_CONFIG=tests/client-sdk/memory/resources/run.yaml pytest tests/client-sdk/memory -v

tests/client-sdk/memory/test_memory.py::test_memory_bank_retrieve PASSED                                                                                                                 [ 16%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 33%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 50%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 66%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 83%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```


Run agianst the local server
```
LLAMA_STACK_BASE_URL=http://localhost:5000 pytest tests/client-sdk/memory -v


tests/client-sdk/memory/test_memory.py::test_memory_bank_list PASSED                                                                                                                     [ 20%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_register PASSED                                                                                                                 [ 40%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_unregister PASSED                                                                                                               [ 60%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_inline_and_query PASSED                                                                                                  [ 80%]
tests/client-sdk/memory/test_memory.py::test_memory_bank_insert_from_url_and_query PASSED                                                                                                [100%]
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vladimirivic vladimirivic

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vladimirivic @yanxi0830