Commit e561842
fix(operator): increase resource limits to prevent OOMKill at scale (#1299)
## Summary
The agentic-operator is OOMKilled on vteam-uat (128 restarts,
CrashLoopBackOff for 2+ days on node `ip-10-0-15-94.ec2.internal`). The
cluster has grown beyond what the original resource limits can handle:
| Resource | Count |
|----------|-------|
| AgenticSessions | 4,319 |
| ProjectSettings | 674 |
| Namespaces | 763 |
| Pods (total) | 477 |
The controller-runtime in-memory cache stores all watched resources.
With the old 512Mi limit, the operator OOMs within ~60 seconds of
startup every time.
**Changes:**
- Memory: 128Mi/512Mi → 512Mi/4Gi (request/limit)
- CPU: 50m/200m → 100m/2 cores (request/limit)
- Add `GOMEMLIMIT=3500MiB` so Go's GC gets aggressive before hitting the
container ceiling
A follow-up issue has been filed for code-level scalability improvements
(cache scoping, unbounded maps, connection pooling, etc.).
## Test plan
- [ ] Deploy to vteam-uat, confirm operator pod starts and stays Running
- [ ] `oc top pod -n ambient-code -l app=agentic-operator` — check
steady-state memory usage
- [ ] Wait 10+ minutes, confirm zero restarts
- [ ] Verify sessions can be created/reconciled normally
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Optimized operator resource allocation, including increased CPU and
memory configurations, and added memory management settings for improved
operational performance and stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Ambient Code Bot <bot@ambient-code.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 8a2310a commit e561842
1 file changed
Lines changed: 8 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
145 | 149 | | |
146 | 150 | | |
147 | 151 | | |
| |||
153 | 157 | | |
154 | 158 | | |
155 | 159 | | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
| 160 | + | |
160 | 161 | | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
161 | 165 | | |
162 | 166 | | |
163 | 167 | | |
| |||
0 commit comments