This section provides detailed instructions for reproducing the example usage scenarios of the BioImage.IO Chatbot illustrated in Figure 2 of the main text:
These steps will guide users through querying documents, utilizing online services, executing AI models, and developing extensions.
Launch the chatbot through the BioImage.IO website here or use the dedicated user interface.
- A video showcasing information retrieval (as described in senario a-c)
- A video showcasing AI model execution (as described in senario d)
- Initiate a Query: Type a question related to bioimage analysis, e.g., "What are the best practices for optimizing model performance on bioimage.io?"
- Review the Chatbot's Response: The chatbot will provide an answer that includes information extracted from the BioImage Model Zoo documentation.
- Initiate a Query: Ask the chatbot to find protein information in the Human Protein Atlas by typing "Tell me about PML protein and show me the cell images"
- Interpret the Results: The chatbot will respond by constructing an API call to the Protein Atlas database and displaying the relevant information about the PML protein, including cell images.
- Initiate a Query: Ask the chatbot to find cell images at the G1 phase by typing "Please, find datasets of cell images at G1 phase."
- Interpret the Results: The chatbot will initiate an API call to the BioImage Archive server, and return results such as a study titled "DeepCycle: Deep learning reconstruction of the closed cell cycle trajectory from single-cell unsegmented microscopy images."
- Prereqsitues: Ensure you have Chrome or a Chromium-based browser installed on your computer.
- Download Image Data: Begin by creating a new folder on your computer named
test-images
. Download the image data file from this link and save it into thetest-images
folder. - Initiate Bioimage Analyst: Navigate to the BioImage.IO chatbot interface at https://bioimage.io/chat/. Note that only Chrome or Chromium-based browser is supported at the moment. Select "Bioimage Analyst(Bridget)" located in the upper right corner of the chatbot interface.
- Mount your Data Folder: Within the chat interface, click on the "Mount Files" button located below the dialog window. This action will allow you to mount the test-image folder that contains your downloaded image data. The chatbot will confirm the successful mounting of the folder, you can now ask it to list the files contained within, and ensuring that your data is ready for analysis.
- Perform segmentation using Cellpose model: Type "Segment the image
/mnt/nuclei.tif
using Cellpose" to run the Cellpose model on the image data. Upon successful execution of the model, the chatbot will notify you that the segmentation process is complete and will display the analyzed results. Optionally, you can ask it to "count the number of nuclei in the image" if successfully segmented, "plot the size distribution of nuclei", or you can tell it to "use the visual inspection tool to analyze the figure and create a report about the size distribution".
Follow the steps below to develop a new extension for microscope stage control and image capture. For a detailed tutorial, visit our GitHub repository or access the Jupyter Notebook directly through ImJoy here.
-
Pre-requisites: You will need a microscope and the squid control software
-
Create microscope extension: Following the example in the above chatbot extension example notebook, create a hypha service extension for controlling the microscope:
- Setup the Developer Environment: Open a Jupyter Notebook. Install and import the
imjoy_rpc
,hypha_rpc
andpydantic
packages. - Define Input Schemas: Create classes for
MoveStageInput
andSnapImageInput
to structure the user input. (Note: To help the chatbot understand the "center", you will need to tell the chatbot about the boundaries of the stage via the docstring of theMoveStageInput
class) - Implement Control Functions: Write asynchronous functions
move_stage
andsnap_image
. - Setup Extension Interface: Develop the extension interface and define a schema getter function.
- Register the Extension: Register the extension as hypha server and connect to the the chatbot.
- Setup the Developer Environment: Open a Jupyter Notebook. Install and import the
-
Initiate a Query: Ask the chatbot to "Please move to the center and snap an image".
-
Interpret the Results: The chatbot will execute the
move_stage
function to move the microscope stage to the center and then capture an image using thesnap_image
function. The chatbot will confirm the successful completion of the tasks.