Add Conditional Value at Risk (CVAR) metric by abdelrahman-ayad · Pull Request #100 · NatLabRockies/PRAS

abdelrahman-ayad · 2026-02-02T19:18:23Z

This PR adds the conditional value at risk (CVAR) metric to assess tail risk shortfalls. CVAR measures the expected value observations above a pre-determined threshold $\alpha$ (Nth percentile).
The current CVAR implementation measures two risk metrics: total unserved energy and capacity shortfalls, as follow:

CVAR of unserved energy: Mean of the worst ($1-\alpha$) unserved energy across all samples, calculated on the Shortfall and ShortfallSamples. To calculate the values from the Shortfall, an additional vector of total unserved energy for each sample is stored before being removed from memory.
CVAR of capacity shortfall samples: Mean of the worst ($1-\alpha$) maximum capacity shortfall in each sample, calculated on the `ShortfallSamples.
CVAR of capacity shortfall: Mean of the worst ($1-\alpha$) capacity shortfall periods, calculated on the `Shortfall.

Note: The CVAR metric reports the unserved energy by default (1), and it has capacity cvar fields for (2) if calculated from the ShortfallSamples or (3) if calculated from the Shortfall

Attached are @benchmark testing for Shortfall and ShortfallSamples comparison after implementing the CVAR metrics.
Main branch:

CVAR branch:

codecov-commenter · 2026-02-02T19:21:19Z

Codecov Report

❌ Patch coverage is 90.84507% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.57%. Comparing base (fbde94e) to head (7a610f6).

Files with missing lines	Patch %	Lines
PRASCore.jl/src/Results/Results.jl	0.00%	5 Missing ⚠️
PRASCore.jl/src/Results/Shortfall.jl	92.45%	4 Missing ⚠️
PRASCore.jl/src/Results/ShortfallSamples.jl	93.84%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #100      +/-   ##
==========================================
+ Coverage   83.13%   83.57%   +0.43%     
==========================================
  Files          45       45              
  Lines        2325     2466     +141     
==========================================
+ Hits         1933     2061     +128     
- Misses        392      405      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…le CVAR

…cumulator

…capacity calculations

Copilot

Pull request overview

Adds a Conditional Value at Risk (CVAR) (and normalized NCVAR) reliability metric for tail-risk assessment of shortfalls, including capacity-shortfall variants, and updates simulation recording + tests accordingly.

Changes:

Introduces CVAR / NCVAR metric types (plus display/validation) and computes them for ShortfallResult and ShortfallSamplesResult.
Extends result recording/finalization to retain per-sample unserved energy and max capacity shortfall needed for CVAR.
Adds/updates unit tests and reference values (including system test data) to cover CVAR/NCVAR behavior.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
PRASCore.jl/src/Results/metrics.jl	Adds `CVAR`/`NCVAR` metric structs and `show` methods; adjusts `MeanEstimate` for singleton samples.
PRASCore.jl/src/Results/Shortfall.jl	Stores per-sample unserved energy and adds CVAR/NCVAR computations for `ShortfallResult`.
PRASCore.jl/src/Results/ShortfallSamples.jl	Adds max capacity shortfall storage and CVAR/NCVAR computations for `ShortfallSamplesResult`.
PRASCore.jl/src/Results/Results.jl	Exports CVAR/NCVAR and adds broadcast convenience methods.
PRASCore.jl/src/Simulations/recording.jl	Records per-sample UE totals into the shortfall accumulator.
PRASCore.jl/src/Systems/TestData.jl	Adds expected CVAR reference values for simulation tests.
PRASCore.jl/test/Results/metrics.jl	Adds tests for `CVAR`/`NCVAR` formatting and domain checks.
PRASCore.jl/test/Results/shortfall.jl	Adds correctness tests for CVAR/NCVAR on results (overall/region/period).
PRASCore.jl/test/Simulations/runtests.jl	Adds simulation-level assertions for CVAR values.
PRASCore.jl/test/dummydata.jl	Adds dummy alpha/sample vectors for new result fields.
PRAS.jl/test/runtests.jl	Expands integration tests to validate CVAR/NCVAR return types for both result kinds.

Comments suppressed due to low confidence (1)

PRASCore.jl/test/Results/shortfall.jl:43

Typo in variable name: cap_shortfal is missing an l (should be cap_shortfall) to match intent and improve readability.

    cap_shortfal = vec(reshape(result.capacity_shortfall_mean, 1, :))
    var = quantile(cap_shortfal, alpha)
    tail_losses = cap_shortfal[cap_shortfal .>= var]

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-23T23:16:20Z

PRASCore.jl/src/Results/Shortfall.jl

    nsamples = first(acc.unservedload_total.stats).n

    p2e = conversionfactor(L,T,P,E)
+    capacity_shortfall_mean = ue_regionperiod_mean


capacity_shortfall_mean = ue_regionperiod_mean aliases the same matrix; the subsequent in-place scaling (.*=) will also scale capacity_shortfall_mean, so it will no longer be in capacity units. Make capacity_shortfall_mean a copy (or avoid in-place scaling) before converting ue_regionperiod_mean.

Suggested change

capacity_shortfall_mean = ue_regionperiod_mean

capacity_shortfall_mean = copy(ue_regionperiod_mean)

Copilot · 2026-02-23T23:16:20Z

PRASCore.jl/src/Results/ShortfallSamples.jl

+
+    estimate = x[]
+    var = quantile(estimate, alpha)
+    tail_losses = estimate[estimate .> var]


This overload uses a strict > comparison for tail selection, while the other CVAR overloads (and tests) use .>=. This can drop observations equal to the quantile threshold and makes behavior inconsistent across overloads. Use the same inclusion rule consistently (typically .>= var).

Suggested change

tail_losses = estimate[estimate .> var]

tail_losses = estimate[estimate .>= var]

Copilot · 2026-02-23T23:16:20Z

PRASCore.jl/src/Results/ShortfallSamples.jl

+function NCVAR(x::ShortfallSamplesResult{N,L,T,P}, cvar::CVAR) where {N,L,T,P}
+    demand = sum(x.regions.load)
+
+    if demand > 0
+        ncvar = div(cvar.cvar, demand/1e6)
+        var = div(cvar.var, demand/1e6)
+    else
+        ncvar = MeanEstimate(0.)
+        var = MeanEstimate(0.)
+    end
+
+    return NCVAR(ncvar, cvar.alpha, var)


Two issues: (1) NCVAR stores var::Float64, but the else branch sets var = MeanEstimate(0.), which will fail construction/type expectations. (2) div on floating-point inputs truncates (and may not be defined for MeanEstimate), which is not appropriate for normalization. Use true division for scaling, and ensure the var argument passed to NCVAR(...) is always a Float64.

Copilot · 2026-02-23T23:16:21Z

PRASCore.jl/src/Results/Shortfall.jl

+function NCVAR(x::ShortfallResult{N,L,T,E}, cvar::CVAR) where {N,L,T,E}
+    demand = sum(x.regions.load)
+
+    if demand > 0
+        ncvar = div(cvar.cvar, demand/1e6)
+        var = div(cvar.var, demand/1e6)
+    else
+        ncvar = MeanEstimate(0.)
+        var = MeanEstimate(0.)
+    end
+
+    return NCVAR(ncvar, cvar.alpha, var)


Same normalization problems as in ShortfallSamples.jl: div will truncate for floats (and likely won’t work for MeanEstimate), and the else branch passes a MeanEstimate where NCVAR expects var::Float64. Switch to non-truncating division and keep var as Float64 on all paths.

Copilot · 2026-02-23T23:16:21Z

PRASCore.jl/src/Results/Results.jl

+CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, t::ZonedDateTime) =
+    CVAR.(x, alpha, x.regions.names, t)
+
+CVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) =
+    CVAR.(x, alpha, r, x.timestamps)
+
+CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) =
+    CVAR.(x, alpha, x.regions.names, permutedims(x.timestamps))    
+
+NCVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) =
+    NCVAR.(x, alpha, r, x.timestamps)
+
+NCVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) =
+    NCVAR.(x, alpha, x.regions.names, permutedims(x.timestamps))


These broadcast convenience methods call NCVAR.(x, alpha, ...), but in this PR the implemented NCVAR overloads take a cvar::CVAR metric (e.g., NCVAR(x, cvar) / NCVAR(x, cvar, r)), not (x, alpha, ...). As written, these definitions will raise MethodError unless additional (x, alpha, ...) overloads exist. Prefer updating these convenience methods to accept cvar::CVAR (or implement the corresponding NCVAR(x, alpha, ...) methods consistently).

Suggested change

CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, t::ZonedDateTime) =

CVAR.(x, alpha, x.regions.names, t)

CVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) =

CVAR.(x, alpha, r, x.timestamps)

CVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) =

CVAR.(x, alpha, x.regions.names, permutedims(x.timestamps))

NCVAR(x::AbstractShortfallResult, alpha::Float64, r::AbstractString, ::Colon) =

NCVAR.(x, alpha, r, x.timestamps)

NCVAR(x::AbstractShortfallResult, alpha::Float64, ::Colon, ::Colon) =

NCVAR.(x, alpha, x.regions.names, permutedims(x.timestamps))

CVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, t::ZonedDateTime) =

CVAR.(x, cvar, x.regions.names, t)

CVAR(x::AbstractShortfallResult, cvar::CVAR, r::AbstractString, ::Colon) =

CVAR.(x, cvar, r, x.timestamps)

CVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, ::Colon) =

CVAR.(x, cvar, x.regions.names, permutedims(x.timestamps))

NCVAR(x::AbstractShortfallResult, cvar::CVAR, r::AbstractString, ::Colon) =

NCVAR.(x, cvar, r, x.timestamps)

NCVAR(x::AbstractShortfallResult, cvar::CVAR, ::Colon, ::Colon) =

NCVAR.(x, cvar, x.regions.names, permutedims(x.timestamps))

Copilot · 2026-02-23T23:16:21Z

PRASCore.jl/src/Results/metrics.jl

+        val(cvar) >= 0 || throw(DomainError(
+            "$val is not a valid CVAR"))


The error message interpolates $val, which is not the CVAR value and will render as the val function rather than the offending number. Interpolate the actual invalid value (e.g., val(cvar)), and consider including which field failed (e.g., cvar vs capacity_cvar).

Copilot · 2026-02-23T23:16:21Z

PRASCore.jl/src/Results/metrics.jl

+
+    function NCVAR(ncvar::MeanEstimate, alpha::Float64, var::Float64)
+        val(ncvar) >= 0 || throw(DomainError(
+            "$val is not a valid NCVAR"))


Same as CVAR: this interpolates $val (the function), not the numeric value that violated the domain constraint. Interpolate the invalid estimate (e.g., val(ncvar)) so the message is actionable.

Suggested change

"$val is not a valid NCVAR"))

"$(val(ncvar)) is not a valid NCVAR"))

abdelrahman-ayad added 8 commits January 30, 2026 16:58

initial CVAR functions commit

bd3c07a

added normalized CVAR (NCVAR)

b1f105a

formulate normalized CVAR based on CVAR struct

e633074

broadcast CVAR and NCVAR - to check

fac883c

broadcast CVAR and NCVAR - to check

37ab22c

add tests for CVAR and NCVAR

2091f49

CVAR and NCVAR shortfall and metrics test

04e6f25

broadcast CVAR and NCVAR functions

3547a87

abdelrahman-ayad requested review from scdhulipala and sriharisundar February 2, 2026 19:18

abdelrahman-ayad self-assigned this Feb 2, 2026

abdelrahman-ayad added the enhancement New feature or request label Feb 2, 2026

abdelrahman-ayad added 15 commits February 4, 2026 17:51

report var value and add field to distinguish between period and samp…

1140338

…le CVAR

Shortfall based CVAR - worst periods

dac2262

add type field for CVAR and store var

a8dd8dd

add ShortfallResult tests

a8c3e7f

metrics.jl update

0439599

add unservedload_sample and unservedload_region_sample to ShortfallAc…

6dded9d

…cumulator

update CVAR implementation in ShortfallSamples to include energy and …

5613f0b

…capacity calculations

update to the recording and reset! functions

5befde4

udpate TestData with the CVAR values

914e541

update DummyData with CVAR values

5bf96cf

include CVAR tests

a979801

rename capacity_cvar

35579e2

add capacity_cvar to shortfall and shortfallsamples

0959c23

update Dummydata

59e7983

CVAR capacity tests

7a610f6

abdelrahman-ayad marked this pull request as draft February 23, 2026 19:59

abdelrahman-ayad requested a review from Copilot February 23, 2026 23:10

Copilot AI reviewed Feb 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Add Conditional Value at Risk (CVAR) metric#100

Add Conditional Value at Risk (CVAR) metric#100
abdelrahman-ayad wants to merge 23 commits intomainfrom
aa/CVAR

abdelrahman-ayad commented Feb 2, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Feb 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Copilot AI Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	capacity_shortfall_mean = ue_regionperiod_mean
	capacity_shortfall_mean = copy(ue_regionperiod_mean)

	tail_losses = estimate[estimate .> var]
	tail_losses = estimate[estimate .>= var]

		val(cvar) >= 0 \|\| throw(DomainError(
		"$val is not a valid CVAR"))

	"$val is not a valid NCVAR"))
	"$(val(ncvar)) is not a valid NCVAR"))

Comments

Conversation

abdelrahman-ayad commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abdelrahman-ayad commented Feb 2, 2026 •

edited

Loading

codecov-commenter commented Feb 2, 2026 •

edited

Loading