collection of Jupyter Notebook in Python that can be used to test your Watson Assistant workspace
This notebook can be used for Regression testing and Blind testing.
- Regression Test Scope: This can be used to test a set of testing phrases that are not part of the training set in Watson Assistant (WA). The idea is that you would re-run this test over time after the improvement phase, to check the health status of your workspace, and make sure that the workspace is still behaving in a consistent manner.
- Blind Test Scope: Simply analyse a set of testing phrases that are not part of the training set in Watson Assistant. This can be a one-off task requested by stakeholders.
This notebook can be used for Regression testing and Blind testing in projects where multiple-workspaces (multiple-skills) are used. Specifically, it evaluates the SPRAY method: each testing phrase is sent to each sub-workspace/skill, individually.
This notebook will allow the user to perform a K-fold cross validation test on your Watson Assistant workspace
This notebook can be used to search for the optimal value of the confidence level threshold to use in your Waston Assistant workspace.
This notebook will be used for diagnosis of incorrect intent detection problem. We want to use unigrams, bigrams and trigrams to do so.
The idea is that each intent is sensitive to particular words/combination of words. The scope is to find a fast way to detect why, for instance, a testing phrase didn't trigger the right intent.This notebook will help the user understanding why which words and in which positions are the most important in the intent detection phase.
The notebooks are using Python 3.6. It is recommended to install Anaconda for Python 3.x since it is compatible with most of the packages. Please download and install Anaconda from:
https://www.anaconda.com/download
If you have an instance of Watson Studio, you can upload the notebooks directly into your project.
Go to the "asset" tab of your project, and in the notebook section click on "add new notebook" and follow the instruction.
When choosing the run time, choose a default Python runtime.
Make sure to use the notebooks optimised for Watson Studio in the folder "For Watson Studio", because they will explain step by step how to set up the access token, how to store and save your datasets using Watson Studio.