Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation for refining workflow #68

Merged
merged 4 commits into from
May 24, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ The easiest way to get started is to drop a folder (typically a folder from with
- `E`, to enable edge coloring (by default, if using this in refinement GUI mode, points with a confidence lower than 0.6 are marked
in red)
- `F`, to toggle between animal and body part color scheme.
- `V`, to toggle visibility of the selected layer.
- `backspace` to delete a point.
- Check the box "display text" to show the label names on the canvas.
- To move to another folder, be sure to save (Ctrl+S), then delete the layers, and re-drag/drop the next folder.
Expand All @@ -61,6 +62,7 @@ Only when saving segmentation masks does a save file dialog pop up to name the d
keypoint annotations are otherwise automatically saved in the corresponding folder as `CollectedData_<ScorerName>.h5`.
- As a reminder, DLC will only use the H5 file; so be sure if you open already labeled images you save/overwrite the H5.
- Note, before saving a layer, make sure the points layer is selected. If the user clicked on the image(s) layer first, does `Save As`, then closes the window, any labeling work during that session will be lost!
- Modifying and then saving points in a `machinelabels...` layer will add to or overwrite the existing `CollectedData` layer and will **not** save to the `machinelabels` file.


### Video frame extraction and prediction refinement
Expand Down Expand Up @@ -106,13 +108,50 @@ Suggested workflows, depending on the image folder contents:
3. **Refining labels** – the image folder contains a `machinelabels-iter<#>.h5` file.

The process is analog to *2*.
Open *napari* and open an image folder.
If the video was originally labeled, *and* had outliers extracted it will contain a `CollectedData_<ScorerName>.h5` file and a `machinelabels-iter<#>.h5` file. In this case, select the `machinelabels` layer in the GUI, and type `e` to show edges. Red indicates likelihood < 0.6. As you navigate through frames, images with labels with edges will need to be refined (moved, deleted, etc). Images with labels without edges will be on the `CollectedData` (previous manual annotations) layer and shouldn't need refining. However, you can switch to that layer and fix errors. You can also right-click on the `CollectedData` layer and select `toggle visibility` to hide that layer. Select the `machinelabels` layer before saving which will append your refined annoations to `CollectedData`.

If the folder only had outliers extracted and wasn't originally labeled, it will not have a `CollectedData` layer. Work with the `machinelabels` layer selected to refine annoation positions, then save.

In this case, it is not necessary to open the DLC project's `config.yaml` file, as all necessary metadata is read from the `h5` data file.

Saving works as described in *1*.

4. **Drawing segmentation masks**

Drop an image folder as in *1*, manually add a *shapes layer*. Then select the *rectangle* in the layer controls (top left pane),
and start drawing rectangles over the images. Masks and rectangle vertices are saved as described in [Save Layers](#save-layers).
Note that masks can be reloaded and edited at a later stage by dropping the `vertices.csv` file onto the canvas.

### Workflow flowchart

```mermaid
%%{init: {"flowchart": {"htmlLabels": false}} }%%
graph TD
id1[What stage of labeling?]
id2[deeplabcut.label_frames]
id3[deeplabcut.refine_labels]
id4[Add labels to, or modify in, \n `CollectedData...` layer and save that layer]
id5[Modify labels in `machinelabels` layer and save \n which will create a `CollectedData...` file]
id6[Have you refined some labels from the most recent iteration and saved already?]
id7["All extracted frames are already saved in `CollectedData...`.
1. Hide or trash all `machinelabels` layers.
2. Then modify in and save `CollectedData`"]
id8["
1. hide or trash all `machinelabels` layers except for the most recent.
2. Select most recent `machinelabels` and hit `e` to show edges.
3. Modify only in `machinelabels` and skip frames with labels without edges shown.
4. Save `machinelabels` layer, which will add data to `CollectedData`.
- If you need to revisit this video later, ignore `machinelabels` and work only in `CollectedData`"]

id1 -->|I need to manually label new frames \n or fix my labels|id2
id1 ---->|I need to refine outlier frames \nfrom analyzed videos|id3
id2 -->id4
id3 -->|I only have a `machinelabels...` file|id5
id3 ---->|I have both `machinelabels` and `CollectedData` files|id6
id6 -->|yes|id7
id6 ---->|no, I just extracted outliers|id8
```

### Labeling multiple image folders

Expand Down