Go inuse_memory low while residential memory is steadily increasing

851 views Asked by At

I am writing a program in Go that depends on regexp. The memory usage as shown in htop is steadily increasing onto the point where the program is crashing. (> 5-7GB)

Yet when I analyze the memory usage with pprof it shows me that only a tiny fraction is actually in use. (~70MB)

I tried manually triggering GC:

go func() {
for range time.Tick(30*time.Second){
    runtime.GC()
    debug.FreeOSMemory()
    runtime.ReadMemStats(&m)
    fmt.Printf("HeapSys: %d, HeapAlloc: %d, HeapIdle: %d, HeapReleased: %d\n", m.HeapSys, m.HeapAlloc,
        m.HeapIdle, m.HeapReleased)
}}()
// Sample output(not from the same run as the other numbers):
// HeapSys: 347308032, HeapAlloc: 123637792, HeapIdle: 194322432, HeapReleased: 0

This shows no effect. My understanding was that golang only requests memory if it needs more and cleans up every once in a while (Running with GCTRACE shows that gc is actually running often) I do not understand why golang continues to request memory when the actually used memory is comparatively small.

Does someone know the underlying cause of the gap and how I can prevent my program from eating memory?

As the graphs are SVGs I cannot embed them:

Possibly related: how to analyse golang memory Does not explain why the memory continues to increase(and is not reused).

0

There are 0 answers