Avni server performance degrades after awhile due to memory leak/restrictions

[himeshr]

Motivation

Avni server performance degrades after awhile due to memory leak/restrictions

Acceptance Criteria

Avni-server should release unused entities for Garbage collection and not cause Memory footprint to bloat for more than specified limit and end up in degrading server performance.

[himeshr]

Based on our analysis today, we observed the following:

• Heap Memory was being freed up as part of "Full GC" periodically.
• GC cpu utilization peaked to more than 50% and lasted for several minutes when we observe inoperability on our app
• newRelic Real-time profiling which was enabled did not provide vital cues regarding HeapMemory foot-print and hence was disabled
• After the recent restart of prod app server, we fetched the GC log file and we analysed the info using gceasy app. Link here
• We did not find any event specific triggers that resulted in Heap Memory usage shooting up and its rather a gradual ascent

Current Todos

• We wanna fetch the gc logs after a few more days of server uptime and then do a gc log analysis
• configure gc to run much much before the app gets into a non responsive state

[himeshr]

All GC log files are stored here for future reference:
https://drive.google.com/drive/folders/10FL8Zgu7SJz3vB_PE3F0KF2zUIkuDDw1?usp=sharing

Latest gc log file analysis is available here