Task - RBAC AuthZ - Multi scope roles: benchmark to identify any performance concerns with glob support

[rodmgwgu]

Run benchmarks on permission queries before and after the implementation for the glob matching support.

Define a set of 5 scenarios and test by repeatedly calling the is_user_allowed function and counting the time.

Document results in the wiki and add a link to it in the task comments.

See https://openedx.atlassian.net/wiki/spaces/OEPM/pages/5404590081/Technical+Discovery+Notes+-+AuthZ+for+Course+Authoring

[mariajgrimaldi]

Performance Report: Exact vs Glob Course Assignments

Load test comparison to evaluate whether glob-based role assignments (course-v1:WGU+*) introduce a performance regression relative to exact assignments (course-v1:WGU+CS+2026).

Related PR: #225

Test Environment

Component	Detail
Load generator	AWS EC2 micro running k6
Target instance	Open edX (muscat.eu1.edunext.cloud)
Concurrent VUs	100
Permissions per request	5
Assignment scopes (exact/baseline)	course-v1:WGU+CS+2026, course-v1:edunext+PY+2026, course-v1:OpenedX+DMS+2026
Assignment scopes (glob)	course-v1:WGU+*, course-v1:edunext+*, course-v1:OpenedX+*
Validation scopes (all runs)	course-v1:WGU+CS+2026, course-v1:edunext+PY+2026, course-v1:OpenedX+DMS+2026
Roles	course_admin, course_staff, course_editor (~34 users each)
Users	100
Profile	authz_large

Three tests ran sequentially on the same instance:

1. Exact (with PR changes) — started at 13:19 UTC
2. Glob (with PR changes) — started at 14:19 UTC
3. Baseline (no changes) — started at 21:06 UTC, same exact assignments without the PR changes

k6 Results Summary

End-to-end times measured from the k6 load generator (EC2 micro) to the Open edX server. These are significantly higher than the server-side times reported by Scout — likely due to network overhead (see Scout section below). The relevant comparison is across the runs, not the absolute values.

Response Times (http_req_duration)

Percentile	Baseline	Exact	Glob
min	135 ms	157 ms	142 ms
avg	9,487 ms	9,774 ms	8,705 ms
median	9,090 ms	8,984 ms	7,852 ms
p90	19,420 ms	19,308 ms	17,058 ms
p95	19,586 ms	21,700 ms	18,544 ms
p99	20,912 ms	30,601 ms	21,225 ms
max	22,160 ms	35,687 ms	23,216 ms

Throughput & Reliability

All three runs completed with zero validation errors and 100% check pass rate. The ~1% HTTP error rate corresponds to the initial auth token requests (100 users authenticating), not to permission validation failures.

Metric	Baseline	Exact	Glob
Iterations completed	8,053	7,883	8,754
Throughput	3.30 iter/s	3.06 iter/s	3.32 iter/s
HTTP requests	8,386	8,220	9,087
Error rate	1.34%	1.22%	1.10%
Test duration	~41 min	~43 min	~44 min

Thresholds

Defined in the k6 test configuration. All three runs show the same threshold results: http_req_failed and permission_validation_errors pass; all duration thresholds fail. These thresholds are calibrated for ideal conditions and are expected to fail under sustained 100-VU load on infrastructure not optimized for load testing.

Scout APM Analysis (PermissionValidationMeView)

PermissionValidationMeView is the endpoint under test — it's where Casbin enforcement happens. Scout instruments it server-side, so these numbers reflect actual server processing time.

k6 measures ~8.7-9.8s avg while Scout measures ~351-392ms avg. This gap might be network overhead (round-trip, TLS, request queuing under 100 VUs) and is also affected by the infrastructure setup — none of this is optimized for load testing. The gap is consistent across all three runs, so it does not affect the comparison.

Full Scout report: https://scoutapm.com/shares/7aa92927-8713-40fb-b34c-7c05b944132d

Metric	Baseline	Exact	Glob
Response time (avg)	391.8 ms	387.0 ms	351.1 ms
Response time (median)	441.6 ms	614.7 ms	548.9 ms
Throughput	~138K requests	~113K requests	~113K requests
Object allocations	76.4	87.7	109.4
Errors	0.0	0.0	0.0

Query Breakdown

View processing dominates across all three runs (~96-97% of total time). Database queries account for less than 4%. The query profile is the same across all runs — glob matching does not introduce additional queries.

Layer	Baseline	Exact	Glob
View	~96%	~96%	~97%
DB queries	~4%	~4%	~3%

The object allocations increase from baseline (76.4) to exact (87.7) to glob (109.4) is consistent with in-memory pattern matching via keyMatch (string prefix matching at the * position — see pycasbin source). This is the matcher configured in the openedx-authz model. The increase is negligible.

Interpretation

1. No performance regression. All three runs land in a comparable range (~8.7-9.8s avg from k6, ~351-392ms avg from Scout). The baseline without PR changes performs the same as exact and glob with changes.

2. No additional database queries. Same query profile across all runs. The cost is in runtime (Casbin's in-memory keyMatch evaluation), not I/O.

Raw Data

• Baseline run (no changes)
• Exact run
• Glob run

Conclusion

No evidence that glob-based course assignments introduce a performance penalty. The cost is confined to Casbin's in-memory evaluation (keyMatch at enforcement time) — no additional queries, negligible runtime overhead at this scale.

Note on Scaling

Casbin evaluates each policy rule against the matcher per request. More policies means more evaluations, so the runtime cost grows with the policy set. This is a tradeoff we already committed to when choosing Casbin, but we should continue monitoring it as we did here.

[rodmgwgu]

@mariajgrimaldi Thanks for the detailed report! Based on the results, I think we are good to continue with this approach.