Script to generate image diff grid

[forsyth2]

Summary

Objectives:

• Create a PDF of diffs from the image_check_failures directory.
• More specifically, each row corresponds to a plot, and each column to actual/expected/diff

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[forsyth2]

Could probably use some logic with os.walk to show diffs from multiple subdirs of the image_check_failures directory.

[forsyth2]

Ok, I think I have a script that can actually display all the image diffs in a single PDF. I'll test it out more on my latest round of Unified testing. Example output, using my last test cycle's diffs: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_rc3_unified_rc8/v3.LR.historical_0051/image_check_failures_comprehensive_v3//image_diff_grid_v5.pdf

Updated stand-alone script

# Stand-alone python script

import os
from math import ceil

import matplotlib.backends.backend_pdf
import matplotlib.image as mpimg
import matplotlib.pyplot as plt

from mache import MachineInfo


def make_image_diff_grid(directory, pdf_name="image_diff_grid.pdf"):
    machine_info = MachineInfo(machine=os.environ["E3SMU_MACHINE"])
    web_portal_base_path = machine_info.config.get("web_portal", "base_path")
    web_portal_base_url = machine_info.config.get("web_portal", "base_url")
    print(f"web_portal_base_path: {web_portal_base_path}")
    print(f"web_portal_base_url: {web_portal_base_url}")

    if not directory.startswith(web_portal_base_path):
        print(f"Directory {directory} is not a subdir of web portal base path: {web_portal_base_path}")
        return
    pdf_path = f"{directory}/{pdf_name}"
    pdf = matplotlib.backends.backend_pdf.PdfPages(pdf_path)
    print(f"Saving to:\n{pdf_path}")
    subdir = directory.removeprefix(web_portal_base_path)
    print(f"Go to:\n{web_portal_base_url}/{subdir}/{pdf_name}")

    prefixes = []
    #print(f"Walking directory: {directory}")
    for root, _, files in os.walk(directory):
        #print(f"root: {root}")
        for file_name in files:
            #print(f"file_name: {file_name}")
            if file_name.endswith("_diff.png"):
                prefixes.append(f"{root}/{file_name.split('_diff.png')[0]}")
    rows = len(prefixes)
    cols = 3  # actual, expected, diff
    print(f"Constructing a {rows}x{cols} grid of image diffs")

    rows_per_page = 4
    num_pages = ceil(rows / rows_per_page)
    for page in range(num_pages):
        fig, axes = plt.subplots(rows_per_page, cols)
        print(f"Page {page}")
        for i, ax_row in enumerate(axes):
            count = page * 3 + i
            if count > len(prefixes) - 1:
                break
            # We already know all the files are in `directory`; no need to repeat it.
            short_title = prefixes[count].removeprefix(directory)
            print(f"short_title {i}: {short_title}")
            ax_row[1].set_title(short_title, fontsize=6)
            img = mpimg.imread(f"{prefixes[count]}_actual.png")
            ax_row[0].imshow(img)
            ax_row[0].set_xticks([])
            ax_row[0].set_yticks([])
            img = mpimg.imread(f"{prefixes[count]}_expected.png")
            ax_row[1].imshow(img)
            ax_row[1].set_xticks([])
            ax_row[1].set_yticks([])
            img = mpimg.imread(f"{prefixes[count]}_diff.png")
            ax_row[2].imshow(img)
            ax_row[2].set_xticks([])
            ax_row[2].set_yticks([])
        fig.tight_layout()
        pdf.savefig(1)
        plt.close(fig)
    plt.clf()
    pdf.close()
    print(f"Reminder:\n{web_portal_base_url}/{subdir}/{pdf_name}")


if __name__ == "__main__":
    #path = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_rc3_unified_rc8/v3.LR.historical_0051/image_check_failures_comprehensive_v3/mpas_analysis/ts_1985-1995_climo_1990-1995/sea_ice/"
    path = "/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_comprehensive_v3_www/test_zppy_rc3_unified_rc8/v3.LR.historical_0051/image_check_failures_comprehensive_v3/"
    make_image_diff_grid(path, "image_diff_grid_v5.pdf")

[xylar]

@forsyth2, I think that looks really promising! None of the MPAS-Analysis differences you're showing in the example PDF are concerning to me. The changes in the MOC plots are very small and those in the sea-ice plots are due to intentional changes made by the MPAS-Seaice developers.

[forsyth2]

@chengzhuzhang Take a look at the image diff grids I linked in #634 (reply in thread). If you like how those look (the format, not the content), I'll go ahead and merge this PR. Thanks.

[forsyth2]

Can customize how many rows per page, though more than 5 isn't very readable

[chengzhuzhang]

the images are too small and lose all the details, and diffs on diff images are hard to see, for the e3sm_diags figures. Maybe we can try two rows per page?

[forsyth2]

Yes, rows-per-page is customizable from the script call, so doesn't really affect the merging of this PR. (This suggestion would be more for me generating plots in my RC testing)

I should point out though that the e3sm_diags diffs in #634 (reply in thread) were numbering 200 pages of 5 rows per page. So, we'd be looking at 400+ pages of plots using fewer rows per page. My intention was to get as quick as possible an overview of what is going on. Then the single-image diffs still exist to investigate further.

[chengzhuzhang]

yes, there are so many diff images and it took lots of clicks to go through each. Have a centralized file to overview those would be helpful. We still need the necessary resolution/definition to actually spot the diffs in the overview file.

[forsyth2]

Agreed, we can certainly tailor the rows per page as needed. I'll make updated overviews for the rc10 diffs using 2 rows per page.

[forsyth2]

Hmm, using 2 rows per page, almost of all the links just go to empty PDFs. I can't tell if it's because I automated the generation of all the grids (instead of doing them one by one) or because the number of pages just became too much to handle. Will look into more.

Test name	missing	mismatched	correct
test_comprehensive_v2_e3sm_diags_images	81	1023	1802
test_comprehensive_v2_mpas_analysis_images	708	45	229
test_comprehensive_v2_ilamb_images	0	20	462
test_comprehensive_v3_e3sm_diags_images	102	1116	2512
test_comprehensive_v3_mpas_analysis_images	726	45	229
test_comprehensive_v3_ilamb_images	0	24	447
test_bundles_e3sm_diags_images	0	550	922
test_bundles_ilamb_images	0	16	225

Some issues popping up:

• v2 e3sm_diags: Something seems to happen around page 340, where all plots go blank.
• v2 mpas_analysis, v2 ilamb, v3 e3sm_diags, v3 mpas_analysis, v3 ilamb, bundles e3sm_diags, bundles ilamb: All pages are empty

[forsyth2]

Using the make_all_grids function but with 5 rows per page: v2 e3sm_diag works, but nothing else does. Maybe a figure isn't getting closed properly?

Test name	missing	mismatched	correct
test_comprehensive_v2_e3sm_diags_images	81	1023	1802
test_comprehensive_v2_mpas_analysis_images	708	45	229
test_comprehensive_v2_ilamb_images	0	20	462
test_comprehensive_v3_e3sm_diags_images	102	1116	2512
test_comprehensive_v3_mpas_analysis_images	726	45	229
test_comprehensive_v3_ilamb_images	0	24	447
test_bundles_e3sm_diags_images	0	550	922
test_bundles_ilamb_images	0	16	225

[forsyth2]

Hmm it actually appears to be both problems: 1) too many large PDFs being created by a single script and 2) with fewer rows per page, the e3sm_diags PDFs alone are too large.

Running just v2 e3sm_diags with 2 rows per page: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output//ac.forsyth2/zppy_weekly_comprehensive_v2_www/test_unified_rc10_20250220/v2.LR.historical_0201/image_check_failures_comprehensive_v2/e3sm_diags/image_diff_grid_try4.pdf again stops plotting anything useful after page ~340.

[xylar]

@forsyth2, I think the usefulness of this approach also is lost after page 340, so maybe the point is that this many diffs needs to be divided up somehow.

[forsyth2]

Ok I confirmed the plots still work if generated by separate calls to the script: v2 e3sm_diags, v2 mpas_analysis. So it does seem to be that a single call of the script can simply not handle having too many pages of output.

I think the usefulness of this approach also is lost after page 340, so maybe the point is that this many diffs needs to be divided up somehow.

@xylar I agree. The initial implementation of this script didn't even walk subdirectories, so I was only running it on the lowest-level subdirectories. But of course that got cumbersome, so now I just run it for each task. Here though, e3sm_diags has over 1000 diffs to display.

That is why I aimed to just put a high density of plots per page:

My intention was to get as quick as possible an overview of what is going on. Then the single-image diffs still exist to investigate further.

To @chengzhuzhang's point though, the diffs have to be visible enough to get some information from them.

There are several possible options, none of which are particularly great:

1. Group images with similar diffs together. E.g., "oh all these plots are different because the axis labels changed". This is the ideal grouping, but I think we'd need to use AI or something like that to implement it automatically.
2. Choose an arbitrary task subdir to walk (maybe a certain job's results, instead of all diags results?). Pro: shorter PDFs, Con: more PDFs to find/link to/click on.
3. My approach above of packing as many plots as possible per page. Pro: high density of information, Con: can't decipher small diffs.

[forsyth2]

Did my best to provide a good label to each row. e3sm_diags has long subdirectory paths so the row titles grow quite long, unfortunately.

[chengzhuzhang]

The labels are good!