Skip to content

Commit 26b31ef

Browse files
Add GradCAM for object detection (#75)
* Add example with GradCAM for YOLOv5 in object detection * Remove constraints for min and max bbox size * Extract base ObjectDetector class * Remove previous example with object detection task * Add original code source URL in docstring * Move fetching layer from YOLO model function, add target_layer parameter to GradCAM * Add dataclasses for complex object detection output data types * Refactor code, add docstrings, fix names, add workaround with np.abs for bbox coordinates * Add support for torchvision SSD model for object detection, refactoring * Rename directories * Refactor, change BaseObjectDetector class definition, make SSD model inherit from it * Refactor modules structure, add GradCAM to object detection module * Remove unused imports * Refactor, move OD models to separate module, fix YOLO prediction generation * Apply pre-commit hooks to all files * Replace excessive unnecessary dependency with simple parsing * Remove model warmup run * Remove unused path in forward pass algorithm * Replace custom implementations with torchvision imports * Add unit tests for object detection utils * Remove obsolete directory * Fix YOLOv5 bbox conversion * Add unit test for object detection visualization utils, refactor * Refactor GradCAM for OD * Remove redundant device argument to YOLOv5ObjectDetector class initializer * Fix unit tests for image preprocessing - use rectangle instead of square shapes * Simplify forward function in WrapperYOLOv5ObjectDetectionModule class * Add custom GradCAM base algorithm implementation for classification and object detection * Move object detection model examples to examples directory * Restructure library directory structure for explainer algorithms, move object detection example script to notebook * Add interpolation method parameter to resize_image function * Fix unit tests and imports * Remove adding epsilon in preprocess object detection image function * Fix example notebook - add assertions of YOLOv5 image shape * Fix basic usage notebook after refactoring class names and directory structure * Enable changing image ratio to match YOLOv5 requirements of image shape * Update README * Refactor object detection custom modules * Refactor GradCAM for object detection
1 parent 1f57591 commit 26b31ef

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

60 files changed

+3539
-2177
lines changed

.github/workflows/publish_docs.yaml

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ jobs:
1111

1212
- name: Checkout
1313
uses: actions/checkout@v3
14-
14+
1515
- name: Setup Python
1616
uses: actions/setup-python@v4
1717
with:
@@ -26,18 +26,18 @@ jobs:
2626
restore-keys: |
2727
venv-${{ runner.os }}-${{ env.PYTHON_VERSION }}-${{ hashFiles('poetry.lock') }}
2828
venv-${{ runner.os }}-${{ env.PYTHON_VERSION }}-
29-
29+
3030
- name: Setup poetry
3131
uses: abatilo/[email protected]
3232
with:
3333
poetry-version: ${{ env.POETRY_VERSION }}
34-
34+
3535
- name: Install dependencies
3636
shell: bash
3737
run: |
3838
poetry install --with docs
3939
working-directory: ""
40-
40+
4141
- name: Get Package Version
4242
id: get_version
4343
run: echo ::set-output name=VERSION::$(poetry version | cut -d " " -f 2)
@@ -77,7 +77,7 @@ jobs:
7777
echo ${{ github.ref }}
7878
echo "BRANCH_NAME=$(echo ${GITHUB_REF##*/} | tr / -)" >> $GITHUB_ENV
7979
cat $GITHUB_ENV
80-
80+
8181
- uses: actions/checkout@v3
8282
name: Check out gh-pages branch (full history)
8383
with:
@@ -109,13 +109,13 @@ jobs:
109109

110110
- name: Run docs-versions-menu
111111
run: docs-versions-menu
112-
112+
113113
- name: Set git configuration
114114
shell: bash
115115
run: |
116116
git config user.name github-actions
117117
git config user.email [email protected]
118-
118+
119119
- name: Commit changes
120120
shell: bash
121121
run: |
@@ -124,7 +124,7 @@ jobs:
124124
git add -A --verbose
125125
echo "# GIT STATUS"
126126
git status
127-
echo "# GIT COMMIT"
127+
echo "# GIT COMMIT"
128128
git commit --verbose -m "Auto-update from Github Actions Workflow" -m "Deployed from commit ${GITHUB_SHA} (${GITHUB_REF})"
129129
git log -n 1
130130

Dockerfile

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,12 @@ ARG BUILD_PYTHON_DEPS=" \
77
libffi-dev \
88
libgdbm-dev \
99
libncurses5-dev \
10-
libncursesw5-dev \
10+
libncursesw5-dev \
1111
libnss3-dev \
1212
libreadline-dev \
13-
libsqlite3-dev \
13+
libsqlite3-dev \
1414
libssl-dev \
15-
xz-utils \
15+
xz-utils \
1616
zlib1g-dev \
1717
liblzma-dev \
1818
"

FAQ.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
<b>While trying to use the library installed from source you encounter error: ``RuntimeError: CUDA error: no kernel image is available for execution on the device``</b>
77
</summary>
88

9-
This error indicates that you actually have pytorch version which does not have CUDA enabled. To solve that you should refer to [https://pytorch.org/get-started/previous-versions/](https://pytorch.org/get-started/previous-versions/).
9+
This error indicates that you actually have pytorch version which does not have CUDA enabled. To solve that you should refer to [https://pytorch.org/get-started/previous-versions/](https://pytorch.org/get-started/previous-versions/).
1010

1111
</details>
1212
<details>
@@ -26,7 +26,7 @@ Source: [StackOverflow](https://stackoverflow.com/questions/55313610/importerror
2626
<b>While trying to install poetry you get an error: ``ModuleNotFoundError: No module named 'distutils.cmd'``</b>
2727
</summary>
2828

29-
It helps to install the following:
29+
It helps to install the following:
3030
```bash
3131
apt-get install python3-distutils
3232
```

README.md

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,15 +26,15 @@ add support for text, tabular and multimodal data problems in the future.
2626
# Installation
2727

2828
Installation requirements:
29-
* `Python` >= 3.7 & < 4.0
29+
* `Python` >=3.7.2,<3.11
3030

3131
**Important**: For any problems regarding installation we advise to refer first to our [FAQ](FAQ.md).
3232

3333
## GPU acceleration
3434

3535
To use the torch library with GPU acceleration, you need to install
3636
a dedicated version of torch with support for the installed version of CUDA
37-
drivers in the version supported by the library, at the moment `torch==1.12.1`.
37+
drivers in the version supported by the library, at the moment `torch>=1.12.1,<2.0.0`.
3838
A list of `torch` wheels with CUDA support can be found at
3939
[https://download.pytorch.org/whl/torch/](https://download.pytorch.org/whl/torch/).
4040

@@ -72,7 +72,7 @@ from pytorch_lightning.loggers import WandbLogger
7272

7373
import wandb
7474
from foxai.callbacks.wandb_callback import WandBCallback
75-
from foxai.context_manager import Explainers, ExplainerWithParams
75+
from foxai.context_manager import CVClassificationExplainers, ExplainerWithParams
7676

7777
...
7878
wandb.login()
@@ -81,10 +81,10 @@ from foxai.context_manager import Explainers, ExplainerWithParams
8181
wandb_logger=wandb_logger,
8282
explainers=[
8383
ExplainerWithParams(
84-
explainer_name=Explainers.CV_INTEGRATED_GRADIENTS_EXPLAINER
84+
explainer_name=CVClassificationExplainers.CV_INTEGRATED_GRADIENTS_EXPLAINER
8585
),
8686
ExplainerWithParams(
87-
explainer_name=Explainers.CV_GRADIENT_SHAP_EXPLAINER
87+
explainer_name=CVClassificationExplainers.CV_GRADIENT_SHAP_EXPLAINER
8888
),
8989
],
9090
idx_to_label={index: index for index in range(0, 10)},
@@ -145,12 +145,12 @@ The recommended version of CUDA is `10.2` as it is supported since version
145145
with the current version of `torch`:
146146
https://pytorch.org/get-started/previous-versions/.
147147

148-
As our starting Docker image we were using the one provided by Nvidia: ``nvidia/cuda:10.2-devel-ubuntu18.04``.
148+
As our starting Docker image we were using the one provided by Nvidia: ``nvidia/cuda:10.2-devel-ubuntu18.04``.
149149

150-
If you wish an easy to use docker image we advise to use our ``Dockerfile``.
150+
If you wish an easy to use docker image we advise to use our ``Dockerfile``.
151151

152152
## pyenv
153-
Optional step, but probably one of the easiest way to actually get Python version with all the needed aditional tools (e.g. pip).
153+
Optional step, but probably one of the easiest way to actually get Python version with all the needed aditional tools (e.g. pip).
154154

155155
`pyenv` is a tool used to manage multiple versions of Python. To install
156156
this package follow the instructions on the project repository page:
@@ -215,7 +215,7 @@ the project directory.
215215
## Pre-commit hooks setup
216216

217217
To improve the development experience, please make sure to install
218-
our [pre-commit][https://pre-commit.com/] hooks as the very first step after
218+
our [pre-commit](https://pre-commit.com/) hooks as the very first step after
219219
cloning the repository:
220220

221221
```bash

example/gradcam_object_detection/custom_models/__init__.py

Whitespace-only changes.

example/gradcam_object_detection/custom_models/ssd/__init__.py

Whitespace-only changes.
Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
"""File contains SSD ObjectDetector class."""
2+
from collections import OrderedDict
3+
from typing import Dict, List, Optional, Tuple, Union
4+
5+
import torch
6+
import torch.nn.functional as F
7+
from torchvision.models.detection import _utils as det_utils
8+
from torchvision.models.detection.image_list import ImageList
9+
from torchvision.models.detection.ssd import SSD
10+
from torchvision.ops import boxes as box_ops
11+
12+
from foxai.explainer.computer_vision.object_detection.base_object_detector import (
13+
BaseObjectDetector,
14+
)
15+
from foxai.explainer.computer_vision.object_detection.types import PredictionOutput
16+
17+
18+
class SSDObjectDetector(BaseObjectDetector):
19+
"""Custom SSD ObjectDetector class which returns predictions with logits to explain.
20+
21+
Code based on https://github.com/pytorch/vision/blob/main/torchvision/models/detection/ssd.py.
22+
"""
23+
24+
def __init__(
25+
self,
26+
model: SSD,
27+
class_names: Optional[List[str]] = None,
28+
):
29+
super().__init__()
30+
self.model = model
31+
self.class_names = class_names
32+
33+
def forward(
34+
self,
35+
image: torch.Tensor,
36+
) -> Tuple[List[PredictionOutput], List[torch.Tensor]]:
37+
"""Forward pass of the network.
38+
39+
Args:
40+
image: Image to process.
41+
42+
Returns:
43+
Tuple of 2 values, first is tuple of predictions containing bounding-boxes,
44+
class number, class name and confidence; second value is tensor with logits
45+
per each detection.
46+
"""
47+
# get the original image sizes
48+
images = list(image)
49+
original_image_sizes: List[Tuple[int, int]] = []
50+
for img in images:
51+
img_shape_hw = img.shape[-2:]
52+
assert (
53+
len(img_shape_hw) == 2
54+
), f"expecting the last two dimensions of the Tensor to be H and W instead got {img.shape[-2:]}"
55+
original_image_sizes.append((img_shape_hw[0], img_shape_hw[1]))
56+
57+
# transform the input
58+
image_list: ImageList
59+
targets: Optional[List[Dict[str, torch.Tensor]]]
60+
image_list, targets = self.model.transform(images, None)
61+
62+
# Check for degenerate boxes
63+
if targets is not None:
64+
for target_idx, target in enumerate(targets):
65+
boxes = target["boxes"]
66+
degenerate_boxes = boxes[:, 2:] <= boxes[:, :2]
67+
if degenerate_boxes.any():
68+
bb_idx = torch.where(degenerate_boxes.any(dim=1))[0][0]
69+
degen_bb: List[float] = boxes[bb_idx].tolist()
70+
assert False, (
71+
"All bounding boxes should have positive height and width. "
72+
+ f"Found invalid box {degen_bb} for target at index {target_idx}."
73+
)
74+
75+
# get the features from the backbone
76+
features: Union[Dict[str, torch.Tensor], torch.Tensor] = self.model.backbone(
77+
image_list.tensors
78+
)
79+
if isinstance(features, torch.Tensor):
80+
features = OrderedDict([("0", features)])
81+
82+
features_list = list(features.values())
83+
84+
# compute the ssd heads outputs using the features
85+
head_outputs = self.model.head(features_list)
86+
87+
# create the set of anchors
88+
anchors = self.model.anchor_generator(image_list, features_list)
89+
90+
detections: List[Dict[str, torch.Tensor]] = []
91+
detections, logits = self.postprocess_detections(
92+
head_outputs=head_outputs,
93+
image_anchors=anchors,
94+
image_shapes=image_list.image_sizes,
95+
)
96+
detections = self.model.transform.postprocess(
97+
detections, image_list.image_sizes, original_image_sizes
98+
)
99+
100+
detection_class_names = [str(val.item()) for val in detections[0]["labels"]]
101+
if self.class_names:
102+
detection_class_names = [
103+
str(self.class_names[val.item()]) for val in detections[0]["labels"]
104+
]
105+
106+
# change order of bounding boxes
107+
# at the moment they are [x2, y2, x1, y1] and we need them in
108+
# [x1, y1, x2, y2]
109+
detections[0]["boxes"] = detections[0]["boxes"].detach().cpu()
110+
for detection in detections[0]["boxes"]:
111+
tmp1 = detection[0].item()
112+
tmp2 = detection[2].item()
113+
detection[0] = detection[1]
114+
detection[2] = detection[3]
115+
detection[1] = tmp1
116+
detection[3] = tmp2
117+
118+
predictions = [
119+
PredictionOutput(
120+
bbox=bbox.tolist(),
121+
class_number=class_no.item(),
122+
class_name=class_name,
123+
confidence=confidence.item(),
124+
)
125+
for bbox, class_no, class_name, confidence in zip(
126+
detections[0]["boxes"],
127+
detections[0]["labels"],
128+
detection_class_names,
129+
detections[0]["scores"],
130+
)
131+
]
132+
133+
return predictions, logits
134+
135+
def postprocess_detections(
136+
self,
137+
head_outputs: Dict[str, torch.Tensor],
138+
image_anchors: List[torch.Tensor],
139+
image_shapes: List[Tuple[int, int]],
140+
) -> Tuple[List[Dict[str, torch.Tensor]], List[torch.Tensor]]:
141+
bbox_regression = head_outputs["bbox_regression"]
142+
logits = head_outputs["cls_logits"]
143+
confidence_scores = F.softmax(head_outputs["cls_logits"], dim=-1)
144+
pred_class = torch.argmax(confidence_scores[0], dim=1)
145+
pred_class = pred_class[None, :, None]
146+
147+
num_classes = confidence_scores.size(-1)
148+
device = confidence_scores.device
149+
150+
detections: List[Dict[str, torch.Tensor]] = []
151+
152+
for boxes, scores, anchors, image_shape in zip(
153+
bbox_regression, confidence_scores, image_anchors, image_shapes
154+
):
155+
boxes = self.model.box_coder.decode_single(boxes, anchors)
156+
boxes = box_ops.clip_boxes_to_image(boxes, image_shape)
157+
158+
image_boxes: List[torch.Tensor] = []
159+
image_scores: List[torch.Tensor] = []
160+
image_labels: List[torch.Tensor] = []
161+
for label in range(1, num_classes):
162+
score = scores[:, label]
163+
164+
keep_idxs = score > self.model.score_thresh
165+
score = score[keep_idxs]
166+
box = boxes[keep_idxs]
167+
168+
# keep only topk scoring predictions
169+
num_topk = det_utils._topk_min( # pylint: disable = (protected-access)
170+
score, self.model.topk_candidates, 0
171+
)
172+
score, idxs = score.topk(num_topk)
173+
box = box[idxs]
174+
175+
image_boxes.append(box)
176+
image_scores.append(score)
177+
image_labels.append(
178+
torch.full_like(
179+
score, fill_value=label, dtype=torch.int64, device=device
180+
)
181+
)
182+
183+
image_box: torch.Tensor = torch.cat(image_boxes, dim=0)
184+
image_score: torch.Tensor = torch.cat(image_scores, dim=0)
185+
image_label: torch.Tensor = torch.cat(image_labels, dim=0)
186+
187+
# non-maximum suppression
188+
keep = box_ops.batched_nms(
189+
boxes=image_box,
190+
scores=image_score,
191+
idxs=image_label,
192+
iou_threshold=self.model.nms_thresh,
193+
)
194+
keep = keep[: self.model.detections_per_img]
195+
196+
detections.append(
197+
{
198+
"boxes": image_box[keep],
199+
"scores": image_score[keep],
200+
"labels": image_label[keep],
201+
}
202+
)
203+
# add batch dimension for further processing
204+
keep_logits = logits[0][keep][None, :]
205+
return detections, list(keep_logits)

example/gradcam_object_detection/custom_models/yolov5/__init__.py

Whitespace-only changes.

0 commit comments

Comments
 (0)