I am writing a program in Go that depends on regexp. The memory usage as shown in htop is steadily increasing onto the point where the program is crashing. (> 5-7GB)
Yet when I analyze the memory usage with pprof it shows me that only a tiny fraction is actually in use. (~70MB)
I tried manually triggering GC:
go func() {
for range time.Tick(30*time.Second){
runtime.GC()
debug.FreeOSMemory()
runtime.ReadMemStats(&m)
fmt.Printf("HeapSys: %d, HeapAlloc: %d, HeapIdle: %d, HeapReleased: %d\n", m.HeapSys, m.HeapAlloc,
m.HeapIdle, m.HeapReleased)
}}()
// Sample output(not from the same run as the other numbers):
// HeapSys: 347308032, HeapAlloc: 123637792, HeapIdle: 194322432, HeapReleased: 0
This shows no effect. My understanding was that golang only requests memory if it needs more and cleans up every once in a while (Running with GCTRACE shows that gc is actually running often) I do not understand why golang continues to request memory when the actually used memory is comparatively small.
Does someone know the underlying cause of the gap and how I can prevent my program from eating memory?
As the graphs are SVGs I cannot embed them:
- pprof inuse_space https://files.niels-ole.com/memory9.svg (acceptable memory usage)
- pprof alloc_space https://files.niels-ole.com/memory9-alloc.svg (frequently called regular expression allocating vast amounts of memory)
Possibly related: how to analyse golang memory Does not explain why the memory continues to increase(and is not reused).