WIP:avoid unnecessary segmentation #783
Conversation
|
That's a great start! The next steps are:
|
| # difference_image = extract_difference_image(new_image, old_image, tolerance=0.05) | ||
| # difference_image.show() | ||
|
|
||
| new_image = Image.open('./winCalNew.png') |
There was a problem hiding this comment.
Please move to a main function and remove commented out code.
| import cv2 | ||
| from skimage.metrics import structural_similarity as ssim | ||
|
|
||
| def extract_difference_image( |
There was a problem hiding this comment.
Please add two newlines before/after a standalone function, as per Google PyDocStyle.
| @@ -0,0 +1,45 @@ | |||
| from PIL import Image | |||
There was a problem hiding this comment.
Please add brief module documentation and example usage.
There was a problem hiding this comment.
Maybe I should remove this file? already implemented same function in vision.py
openadapt/strategies/visual.py
Outdated
| @@ -47,9 +47,11 @@ | |||
| from dataclasses import dataclass | |||
| from pprint import pformat | |||
| import time | |||
| from xml.etree.ElementPath import find | |||
openadapt/strategies/visual.py
Outdated
|
|
||
| from loguru import logger | ||
| from PIL import Image, ImageDraw | ||
| from typing import List |
There was a problem hiding this comment.
Please use list instead of List
openadapt/strategies/visual.py
Outdated
| combined_centroids = previous_centroids + new_centroids | ||
|
|
||
| return Segmentation( | ||
| image=combined_image, |
There was a problem hiding this comment.
I believe this should be the new image, not the previous image.
openadapt/strategies/visual.py
Outdated
|
|
||
| combined_image = previous_segmentation.image | ||
| combined_masked_images = previous_segmentation.masked_images + new_masked_images | ||
| combined_descriptions = previous_segmentation.descriptions + new_descriptions |
There was a problem hiding this comment.
I believe we need to filter out the masked_images and the descriptions that are not relevant for the new image, i.e. those that overlap with difference areas between the old and the new image.
openadapt/strategies/visual.py
Outdated
| new_bounding_boxes, new_centroids = vision.calculate_bounding_boxes(new_masks) | ||
|
|
||
| combined_bounding_boxes = previous_bounding_boxes + new_bounding_boxes | ||
| combined_centroids = previous_centroids + new_centroids |
There was a problem hiding this comment.
As above, we need to filter out the irrelevant boxes/centroids.
openadapt/strategies/visual.py
Outdated
| @@ -382,6 +412,26 @@ def get_window_segmentation( | |||
| # TODO XXX: create copy of similar_segmentation, but overwrite with segments of | |||
| # regions of new image where segments of similar_segmentation overlap non-zero | |||
| # regions of similar_segmentation_diff | |||
| new_image = vision.extract_difference_image( | |||
There was a problem hiding this comment.
What do you think about naming this difference_image rather than new_image?
There was a problem hiding this comment.
makes sense, I'll change this
openadapt/vision.py
Outdated
|
|
||
| # Create a mask of the differences | ||
| mask = np.zeros_like(new_image_np) | ||
| cv2.drawContours(mask, contours, -1, (255), thickness=cv2.FILLED) |
There was a problem hiding this comment.
I believe contours should not be necessary. A more direct approach involves getting the indexes of the pixels whose difference is above a threshold, and setting only those pixels directly.
…or the new image, removed unnecessary contours in extract_difference_image
Animesh404
changed the title
Animesh404
changed the title
What kind of change does this PR introduce?
attempt for issue #614 to extract the portion of active window that is different and combine with similar image
Summary
Trying to extract the portion of the active window that is different (i.e. greater than some tolerance) from the window retrieved, this can be used later to segment + describe this portion rather than the full thing.
reference image segments
new image segments
non-matching image section
matching image section
non-matching image section segments
matching image section segments
combined matching image section segments with non-matching image section segments
Final result
Checklist
How can your code be run and tested?
try to run
experiments/visualizing_segments.py