feat: Add segmentation GVI, refactor scoring images #55

dragonejt · 2024-07-03T01:53:55Z

Notes

Closes use class segmentation for % vegetation/canopy estimation #10
Should output_file Path be a required argument or an option with a sensible default?

Changes

Add PyTorch and HuggingFace Transformers for the segmentation GVI method
Add output_file as argument to assign_images script
Rename assign_gvi_to_points to score_images
Add ScoringMethod interface and ScoringSelector enum
Add PixelCounting GVI scoring method which implements ScoringMethod, uses Treepedia pixel counting method
Add Segmentation GVI scoring method which implements ScoringMethod, uses mask2former model from huggingface

Testing

Running script with both segmentation and pixel counting methods produced scores in the geodataframe

danbjoseph · 2024-07-19T15:39:15Z

i was able to run a successful segmentation. thanks for this @dragonejt ! 🎉
there was comment on the issue about "do we want to divide the image into 4 images with 90 degree field of vision instead of 1 image with 360 field of vision before the segmentation step?" - was that considered in this PR? or are we just ingesting the source image as is?
i think we can deal with the question about a default output_file in a different PR? i logged it as default output_file for every step #58
can you do at least a partial update to the docs in the README as part of this PR? can you note which category(ies) from the output are being saved for the value? can you link to the mask2former page on huggingface?
- related to the category(ies) there's this comment from use class segmentation for % vegetation/canopy estimation #10 (i didn't copy over the image being referenced): "see the sample segmented image above. there's a more pastel green (grassy looking area) and a more neon green (appears to be mostly trees). confirm the different possible labels related to vegetation (there are at least 2)."
can we have an option flag to save out the segmentation results? same folder as the images, same filename with maybe "_segmentation_YYYYMMDD" appended? however, StreetView-NatureVisibility does the images side-by-side in one file and shows all the categories. ours would just the segmentation image by itself and it would be just 2 colors for vegetation and not-vegetation (or even 1 color, green for vegetation and then transparent for the null?). this request can also be logged as an issue and done after merging this PR.

dragonejt · 2024-07-20T00:23:55Z

Hey Dan!

We are currently ingesting the image as-is from the JPEG format. No modification has been done, I believe.
Sure output_file can be in a different PR. Do you want me to revert the current output_file changes then (that make it more similar to create_points.py)?
I'll add README pages shortly.
I can also take a look at saving the segmentation images, though I agree that it would probably be best put in a separate PR.

danbjoseph · 2024-07-20T15:18:10Z

For...

Do you want me to revert the current output_file changes then (that make it more similar to create_points.py)?

...I don't fully understand the 2 options being offered.

I've logged save out visualizations of image segmentation #59 for the future.

dragonejt · 2024-07-24T23:39:33Z

Previously the script determined the output file path on its own instead of accepting a CLI argument, do we want to revert back to determining its own output file path for now?

danbjoseph · 2024-07-25T00:25:09Z

Previously the script determined the output file path on its own instead of accepting a CLI argument, do we want to revert back to determining its own output file path for now?

ah, no thanks. i like inputting the file path.

jayqi · 2024-07-25T00:11:33Z

pyproject.toml

+  "torch",
+  "transformers",
  "tqdm",


Suggested change

"torch",

"transformers",

"tqdm",

"torch",

"tqdm",

"transformers",

jayqi · 2024-07-25T00:29:15Z

src/score_images.py

+    thread_map(
+        score_image,
+        gdf.index,
+        total=len(gdf.index),
+        desc="Calculating GVI of Images/Points",
+        unit="images",
+    )


I'm not sure if thread_map makes sense for this. My guess is that scoring images is probably going to be compute bound rather than I/O bound—though I'm not sure about this. I guess maybe pixel counting would be I/O bound but the segmentation is probably compute-bound.

It's possible process_map would speed things up more. Most likely it would for the pixel counting method. It might not for the segmentation model, if the segmentation model is internally parallelized already.

I wonder then if we want a score_images that takes a collection of images to be the primary API (rather than a score_image for a single image, though it can still be useful to keep that around for utility) and then the individual scoring methods can determine the most effective parallelization approach for themselves.

jayqi · 2024-07-25T00:30:37Z

src/scoring/segmentation.py

+        w4 = int(width / 4)
+        h4 = int(height / 4)
+        hFor43 = int(w4 * 3 / 4)
+        images = []
+        pickles = []
+        # Crop the panoramic image based on road centers
+        for w in range(4):
+            x_begin = w * w4
+            x_end = (w + 1) * w4
+            cropped_image = image.crop((x_begin, h4, x_end, h4 + hFor43))
+            cropped_segmentation = segmentation[h4 : h4 + hFor43, x_begin:x_end]
+            images.append(cropped_image)
+            pickles.append(cropped_segmentation)
+        return images, pickles


I'm a little suspicious about this being hardcoded to these particular widths and heights. Do we even need them? If we're segmenting, wouldn't the model identify these pixels as not relevant if they're not relevant?

Also, why do we split up the image into 4 pieces? It seems like we just sum them back up again later in _get_GVI.

Anyways, I'm also okay with investigating and fixing this later though.

if it's 1 piece with a 360 field of view it's distorted? and if it's re-projected into 4 images with 90 degree field of view then the shapes of things aren't distorted?

i missed that this code is doing something based on road centers the NatureVisibility project describes 3 scenarios for 3. Clean and process data

Panoramic Image Cropping using Road Centers

Panoramic Image without Road Center Cropping

Non-Panoramic Image

I thought we should do # 2 ?

I think this might be #2 based on the feature flag? The comments say that it cuts by road center, but crop_panoramic_images is only called when cut_by_road_centers is false. Also, do we need to do the cutting of images into 4 pieces? We don't do this for the pixel counting method, so I'm wondering if we want to keep the image modification process as simple as possible or continue to do these modifications. Pixel counting doesn't crop off the bottom 20% of the image for the car either I think, so I don't think the segmentation GVI and pixel counting GVI are very comparable or of the same scale currently

Here's the feature flag for reference: https://github.com/Spatial-Data-Science-and-GEO-AI-Lab/StreetView-NatureVisibility/blob/f4e6b5f53890db13bc32154682591937ba2271d0/modules/process_data.py#L276

can we have an optional argument that is --crop-vehicle 20.0 with 20 being the default (20% of image bottom cropped) if the argument isn't provided - adding this to the config file created in Initial working implementation for config files for create_points #48. this option could apply to all score_images workflows?

the Indonesian Red Cross is going to be doing an imagery collection test project in October, and I think they will be using cameras mounted on helmets of people riding motorbikes, so cropping may not be as beneficial compared to if the camera is in the middle of a car roof.

regarding the cutting into 4. if it's 1 piece with a 360 field of view it's distorted? see the example below. and if it's re-projected into 4 images with 90 degree field of view then the shapes of things aren't distorted?

this paper mentions: "distortion-aware modules to address extreme object deformations and panorama distortions that result from equirectangular representation" and "While extensive research has been conducted on pinhole based learning methods, approaches tailored for processing ultra-wide panoramic images and inherently accounting for spherical deformations remain ongoing research."

is there anything from the NatureVisibility project that we should strip out?

i've updated issue reproject 360 image for segmentation (and other analysis options) #13 with a link to a good Stack Overflow post that has an interesting deep dive into re-projecting equirectangular panoramas. it would be great to include the reprojection as part of this PR, however we can also leave it for a future PR

jayqi · 2024-07-25T00:31:44Z

src/scoring/segmentation.py

+            # Cut panoramic image in 4 equal parts
+            # Crop the image and its segmentation based on
+            # the previously found road centers
+            images, pickles = self._crop_panoramic_images(image, segmentation)


Why are the segmentations called "pickles"?

https://docs.python.org/3/library/pickle.html I think

jayqi · 2024-07-25T00:33:25Z

src/scoring/segmentation.py

+            pickles.append(cropped_segmentation)
+        return images, pickles
+
+    def _get_GVI(self, segmentations: list[torch.Tensor]):


Suggested change

def _get_GVI(self, segmentations: list[torch.Tensor]):

def _get_gvi(self, segmentations: list[torch.Tensor]):

Code style preference.

Will do, but also, code style issues should be caught by linter or autoformatter rules, not by manual review. For the future, let's review the Ruff rules and see which extra ones we want to add, such as variable capitalization.

jayqi · 2024-07-25T00:34:11Z

src/scoring/segmentation.py

+            GVI, segment_scores = self._get_GVI(pickles)
+            return GVI


Suggested change

GVI, segment_scores = self._get_GVI(pickles)

return GVI

gvi, segment_scores = self._get_gvi(pickles)

return gvi

Code style

ioalexei · 2024-10-02T14:38:02Z

I'm taking a look at this to work on #59 - I can get it to work with the PIXELS argument, but running with SEGMENTATION causes my session to crash. Any suggestions for how I can identify the cause?

…py-estimation

feat: Add segmentation GVI, refactor scoring images

de73342

dragonejt added the enhancement New feature or request label Jul 3, 2024

dragonejt requested review from jayqi and danbjoseph July 3, 2024 01:53

dragonejt self-assigned this Jul 3, 2024

dragonejt linked an issue Jul 3, 2024 that may be closed by this pull request

use class segmentation for % vegetation/canopy estimation #10

Open

jayqi mentioned this pull request Jul 10, 2024

Evaluate accuracy of mask2former segmentation model #57

Open

danbjoseph mentioned this pull request Jul 20, 2024

save out visualizations of image segmentation #59

Open

jayqi requested changes Jul 25, 2024

View reviewed changes

dragonejt mentioned this pull request Aug 3, 2024

Review Linter and Formatter Rules #70

Open

danbjoseph mentioned this pull request Aug 26, 2024

implement segmentation method for green view score #41

Closed

dragonejt added this to the Indonesia SLI Capture milestone Sep 26, 2024

danbjoseph added 2 commits October 25, 2024 08:36

docs: update references

ef0c31f

Merge branch 'main' into 10-use-class-segmentation-for-vegetationcano…

0e08fcb

…py-estimation

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add segmentation GVI, refactor scoring images #55

feat: Add segmentation GVI, refactor scoring images #55

dragonejt commented Jul 3, 2024 •

edited

Loading

danbjoseph commented Jul 19, 2024

dragonejt commented Jul 20, 2024

danbjoseph commented Jul 20, 2024

dragonejt commented Jul 24, 2024

danbjoseph commented Jul 25, 2024

jayqi Jul 25, 2024

jayqi Jul 25, 2024

jayqi Jul 25, 2024

danbjoseph Jul 25, 2024 •

edited

Loading

dragonejt Aug 4, 2024

danbjoseph Aug 4, 2024 •

edited

Loading

danbjoseph Sep 3, 2024 •

edited

Loading

jayqi Jul 25, 2024

dragonejt Jul 25, 2024 •

edited

Loading

jayqi Jul 25, 2024

dragonejt Jul 25, 2024

jayqi Jul 25, 2024

ioalexei commented Oct 2, 2024

	def _get_GVI(self, segmentations: list[torch.Tensor]):
	def _get_gvi(self, segmentations: list[torch.Tensor]):

feat: Add segmentation GVI, refactor scoring images #55

Are you sure you want to change the base?

feat: Add segmentation GVI, refactor scoring images #55

Conversation

dragonejt commented Jul 3, 2024 • edited Loading

Notes

Changes

Testing

danbjoseph commented Jul 19, 2024

dragonejt commented Jul 20, 2024

danbjoseph commented Jul 20, 2024

dragonejt commented Jul 24, 2024

danbjoseph commented Jul 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danbjoseph Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danbjoseph Aug 4, 2024 • edited Loading

Choose a reason for hiding this comment

danbjoseph Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dragonejt Jul 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ioalexei commented Oct 2, 2024

dragonejt commented Jul 3, 2024 •

edited

Loading

danbjoseph Jul 25, 2024 •

edited

Loading

danbjoseph Aug 4, 2024 •

edited

Loading

danbjoseph Sep 3, 2024 •

edited

Loading

dragonejt Jul 25, 2024 •

edited

Loading