Photoanalysis scores with examples - Sonoma

[cfc62]

This is building off of the statistical view of photoanalysis scores I wrote up earlier this year, if you haven't looked at that, it's here: #1173

For this write up, I selected the extreme scores to see if I could discern what the scores represented. For many it's pretty clear, but there are some that are still head scratchers.

Well-timed shot - scores higher than 0.65

These are mainly sunset/sunrise with some action photos (surfing, motorcycle stunts, airplanes in flight) mixed in:

Well-timed shot - scores lower than -0.125

Now we look at the lowest scores for this for contrast, and here I'd describe these shots as hurried and generally pretty poor - but not completely so. If I had to guess, these look like they are detecting overall blur or subject blur, as if someone stepped into the shot, and/or that the background is in focus but the (potential) human subjects in the foreground are not:

Well-framed subject - scores higher than 0.75

Pretty much what it sounds like, but I suspect it's also checking for scene complexity and possibly rule of thirds as well.

Well-framed subject - scores less than -0.64

Busier scenes visually overall, with no clear subject. Not necessarily technically bad, and may be memorable shots, but certainly not photographic art:

Well-chosen subject - scores higher than 0.34

I struggle with this one, but overall I believe this to be scene complexity, in that the higher scores generally are more simple:

Well-chosen subject - scores less than -0.3

Not all of the lower scoring photos have scene complexity, at least to my eyes. But I'm a human and perceptually we all filter things out (and in) that are/are not there.

Some of these seem to be when people are looking away from the camera, head cut off in the frame and so on, but hard to say. When doing street photography, I don't intrude, nor am I going to post any pictures of people's faces here.

Tastefully Blurred - scores higher than 0.5

Bokeh and tilt-shift.

Tastefully Blurred - scores lower than -0.6

Typically motion blur, camera blur or mis-focus events, but it can also tag photos where you're intending to show motion with blur. Overall though, a negative score here is pretty indicative of a bad shot.

More to come...

[RhetTbull]

Thanks for the examples! (Great photos by the way!) It is helpful to see what these scores correlate to. On my "projects to work on someday" list is a tool to help surface the "best" shot given a natural language description. Understanding how these scores work would be useful for that.

[cfc62]

Focus score - higher than 0.25

I find a lot of overlap here with "Tastefully Blurred" - so much so I really can't sense much difference, except that perhaps the difference isn't for the high scores, but for the low scores. More on that in a bit.

As with Tastefully Blurred, the high scorers usually have some bokeh, but here I think there's more leeway in that it appears that if any part of the picture is sharp, the score is OK.

Focus score - equal to 0 (zero)

Unlike the low scores for "Tastefully Blurred" here the entire image is out of focus or otherwise impaired:

With the score distribution being what it is, I could only run a query for focus = 0. But clearly Focus being very low means the entire image is poor.

Symmetry score - higher than 0.14

Pretty self-explanatory, these show some level of symmetry:

Symmetry score - less than 0.0011

I don't think this score being low really implies anything other than a lack of symmetry in the photo. But strangely, at the same time, these images are less appealing overall:

Reflection score - greater than 0.1

This one was fun to figure out! For the high scores, it's a natural reflection that enhances the image, typically but not always water:

Reflection score - lower than -0.12

Unlike other scores where lower scores just indicate a lack of whatever, here the lower scores clearly indicate where reflections - mainly in glass - are hurting the image (don't forget your circular polarizers!):

[cfc62]

Next comes "Post Processing" and I have no earthly idea what's going on here. It's not scoring of adjusted/edited/tweaked photos as most are unedited in any way.

Let me sleep on it and get back to it later.

Pleasant Perspective - scores above 0.58

These are typically looking up or down at less severe angles, or if at more severe angles, the photo still "works:"

Pleasant Perspective - scores below -0.2

Typically images shot at a more extreme upwards or downwards angle - it does a good job in finding images that just... don't... look... right:

[cfc62]

Pleasant Pattern - score above 0.5

Just what its name says, there's a pattern in the photo; usually it's architectural but not always.

Pleasant Pattern - scores less than 0.008

Mainly nature shots with some people sprinkled in. Lots of surfing photos, which is a combo of a nature shot with person in it. As with some of the other scores, I find it interesting that when a score is lower, the overall selection of images is also subjectively less impressive - but with quite a few exceptions, to be clear.

Pleasant Lighting - score above 0.475

Vast majority of these have clear blue sky in them, with some sunsets and fall colors sprinkled in, as well as green grass:

Pleasant Lighting - score below -0.75

A solid indication that something has gone wrong with the light in the photo; the majority are indeed poor shots but perhaps 10-20% are still good and accurately captured what I was looking for. If I had to say, I'd suspect it's looking for large areas that are blown out or zeroed out, and poor contrast. If the images are RAW, perhaps they can be massaged.

This last one is fine & a keeper:

Composition - score above 0.55

I can't really discern how it's picking these out, but I think it's doing a pretty good job:

Composition - score below -0.9

Much like how I can't really figure out how it scores photos high for composition, I haven't figured out why they score low, but in general I think it's pretty accurate, maybe 10% keepers.

Again just to note, I do take snapshots as memory joggers, times when things are just alien, novel or just of note in some way - chaotic, busy or overwhelming, or serene and relaxing. It's not always art, it's documentation. I want to be reminded of these times. Would I cull these from my collection? Case-by-case basis...

Tilt - scores above 0.025

I had high hopes for Tilt scoring, I do take a lot of pictures with a visible horizon/sealine and I was hoping this score would point me to photos that I could correct.

Unfortunately, while most photos with a tilt are given a lower score, quite a few aren't, and vice-versa. And for quite a few of the high scoring photos, I have no idea of how the score was arrived at. Such is AI/ML, of course.

Overall, this is still a decent score in that higher scores are generally better photos, but not always.

Tilt - scores below -0.18

Yes, most of these are bad but frustratingly, 20% or more are fine:

Here, the road is level, but the bridge is ascending to the left, throwing off the calculation:

[cfc62]

Continuing...

Noise - score above -0.0015

Basically, images without noise. Most are taken in bright environments, allowing for lower ISOs amongst other noise-fighting settings.

Noise - scores below -0.3

Noisy images. The examples may not look noisy in thumbnail view, but they are. Items here may - especially if in RAW format - be candidates for de-noising AI usage.

Important note: some scores are simply incorrect here - failures included a blacktop road and a picture of a crowd and a floor with speckles in it.

Many of the photos with high noise are also from older cameras, which in part explains why the overall aesthetic score rises with newer camera models.

Low Light - score above 0.998

As a reminder, this is a score where most of the photos are scored at 0.0 - there's plenty of light in the scene, so no examples needed for that.

But looking at the other side, when the score is high, it's definitely a low-light scene. Many are indeed outright failures, but there are keepers which accurately reflect what I was intending to capture:

[cfc62]

Lively Color - scores above 0.3

Colorful images.

Lively Color - score below -0.3

Either lacking in color "punch," generally monochromatic, muddy or otherwise less than pleasing color:

Dusty/cloudy days certainly don't help:

But sometimes the color is the color, but it still gets scored harshly:

Harmonious Color - scores above 0.3

I am not sure I believe in 100% of "harmonious color" theory, but at the same time quite a bit of it rings true.

In very broad brushes, I would tend to think if you're intentionally looking to set a theme or feeling for one photo (as you might for model or product photography), this score might be more important to you. For me, as I capture what interests me, I care less about it.

Harmonious Color - scores below -0.05

Lots of shots on cloudy, dusty, hazy days:

[cfc62]

Intrusive Object Presence - scores above -0.012

To be completely honest, I had high hopes for this score. I think that when starting out, many photographers will not realize that there's something in the scene that will be distracting once it's viewed later.

While this score does... okay, it fails a lot too. I'll show those, but first, some example non-failures:

All fine and good, but then...

(Amazon box)

(Lamppost)

(Lens flare)

(Another lamppost)

...and so on. Sure, sometimes you can move to another location, but sometimes you can't, and it would have been great to ID photos with object issues, editing software is getting better day after day and some of these elements can easily be removed now.

Intrusive Object Presence - scores below -0.8

Unfortunately, the vast vast majority of photos that get these low scores are ones where something happened at the edge of a photo. I do a lot of shooting through windshields and other windows, and it will detect if I get a corner in the frame, but that's about it. Well, that and getting my hand/fingers in the frame.

Now - if you're more intentional about shooting, you will likely have less of these.

[RhetTbull]

Thanks for posting all this analysis--really interesting! I've been thinking about what an app would look like that makes use of this data to help curate good photos, find photos to delete, etc. For example a photo browser that let you quickly sort on different scores, mark favorites, mark for deletion, etc.

[cfc62]

Well, exactly! Apple is being their usual opaque selves and generating these scores, but a) never mentioning that they are doing so b) not explaining what the scores are or represent, and c) hardly leveraging them at all.

Now, I suspect (see my other Sonoma post) that they're amping up the AI/ML scoring for a lot of things (not just photos) and the next releases os iOS and MacOS will start leveraging that data, but we'll see.

C'mon Apple, help me out here! Show me the poor photos (out of focus; blurry; poor lighting) and let me either discard them, fix them, or tell the algorithm that they're actually good photos. And let me curate photos too. (I'm sticking to my guns that curation of 0.5 means it's not curated at all, I'll show that in another post)

[RobBW]

I found this page because I was searching for an app such as you both are talking about here. OSXPhotos is a brilliant foundation for such work but I note your conversation appears to have stopped 9 months ago.
Before I launch into reinventing the wheel it would be helpful to know if you have made further unreported progress?

[cfc62]

I note that Sonoma 14.2 brought some changes to Photos so I'll need to see if anything changed materially.

Oddly enough, while I haven't spotted a lot of photoanalysisd usage, photolibraryd was very high.

[cfc62]

I haven't forgotten this thread, just a bit behind with the holidays...

[cfc62]

Continuing... yes, I know some of the photos are dupes from above.

Interesting Subject score above 0.5

Not sure what it's keying on here, but I think whatever it's looking at seems to be working... it also loves surfing pictures, even ones that aren't all that great.

Interesting Subject scores less than -0.875

A lot of through car window shots, screenshots, pictures of other displays or pictures, direct flash...

That last one is likely completely meaningless to anyone other than me. Yet another memory marker.

Interaction score above 0.06

As noted previously, this is where a lot of movies get scored higher, so I'll ignore that and just look at photos... and I still have not a great idea of what this score represents. There are a bunch of photos with people in them, but also a bunch of photos with no one in them at all.

Note, this image is HDR so the screen cap doesn't do it justice.

Some of you may know who this is...

[cfc62]

Immersiveness score above 0.7

Easy: You're going somewhere. Driving down a road, walking down a path, etc.

When looking at Immersiveness scores below 0.001, there's no pattern I can discern.

[cfc62]

Failure score below -0.2

Yep, these are bad. Focus problems, camera blur, subject blur, exposure issues...

Now, while 90% plus of my photos and videos with this low score are bad, it did mark a low-light time lapse of stars as a failure, so automatically chucking assets based in this score would be a mistake.

Behavioral score above 0.91

I thought this would be easy, these are going to be pictures with people... behaving in them. Nope.

While the majority of assets do have people in them, a large number have no people in them at all.

So, I don't understand this score. It could be that the AI is not very good at this score?

However...

Behavioral score below 0.1

With the big exception of surfers, there are almost no people in these photos.

In general, I'd say Behavioral is a general idea if people are in the asset. Perhaps Photos uses this as a way to only do face detection on photos where people are depicted.

[cfc62]

Just two scores left!

Overall scores above 0.8

Photos is very stingy with this score, and only a tiny tiny sliver of my assets get this high grade:

(another HDR photo that may not look great)

Do these represent my best photos? Not by a long shot. Yes, some of my good ones are in here, but a few sub-par ones got in as well.

Overall score less than 0.03

Again, Photos hates shots of paper, screens and so on. I won't be tedious and show you these all over again.

Curation score above 0.7

I'll remind the reader that not all photos or videos are curated, in fact only a small number are. From what I can tell, each asset begins at 0.5 and if Photos (somehow) decides to give it a Curation score, then the score will move from that point - sometimes up, sometimes down.

Here, I'd say that yes, these are good photos. I can't find a bad one in the bunch. Are there other photos I like better in my collection? Absolutely.

Once again, a lot of these would be repeats from above so I'm not including more examples.

Curation score below 0.3

I've got quibbles with this. Yes, Photos hates pictures of pictures and signs but good photos get swept up in the process. Same for scene complexity and colors. Just because a composition has complexity in the scene or it was a cloudy/hazy/sandstorm day doesn't necessarily mean it should automatically be curated downwards.

That's a long way of saying I see plenty of good and even great pics with these lower scores. Yes, they're in the minority but it's not a tiny minority.

The above is a screenshot of a time lapse of a Japanese taxi rank filling and emptying in a very orderly fashion. It's great... not sure why it got such a low curation score.

[cfc62]

Still no idea what "Post Processing" represents.

Overall, I still think we're looking at a work in progress, warts and all. The scores that our CPUs and GPUs and NPUs strain to compute are also nearly un-leveraged by Apple, but I suspect this may change in the next iOS and MacOS releases.

[RobBW]

Have you explored the effect of treating some of these scores as pass/fail filters arranged in serial and parallel combinations. My instinct is that the 'overallscore' is created in such a way. It might be called "Post Processing"
By the way your comprehensive statistical analysis of the Photos scores is impressive. You have saved me many hours of playing around and working out how to do it.

[cfc62]

Hi!

The problem with having cool statistical tools and a large and rich data set is that there's a nearly unlimited number of analyses that can be done. I'm sure I missed plenty!

You might have missed it but I posted more of a data description before this thread, scroll all the way up to the top to see the link to that. I didn't see much in the way of data distributions that would lead me to think that there were many bimodal or pass/fail scores, but Apple is opaque here and likely always will be. The score scales vary across scores, and the naming is... unclear. Without an official decoder ring I suspect there's not much more we can glean from the scores.

...and I'm sure the scores, definitions and models changed very much with Sequoia. If I still have access to those stats tools and I have some time, sometime in the next few months I'll have another whack at it.