We are using MongoDB version 4.2.11. We have one primary and two secondary nodes of MongoDB on Kubernetes. During load testing phase, despite having ample disk space we got a WiredTiger exception saying disk space is full.
2022-05-01T19:57:03.058+0000 W FTDC [ftdc] Uncaught exception in 'FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data capture: /var/data/mon godb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem.Shutting down the full-time diagnostic data capture subsystem.
10622 2022-05-01T19:57:04.534+0000 I NETWORK [conn12883] received client metadata from 127.0.0 .1:33116 conn12883: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux",name: "linux", architecture: "x64", version: "5.4.0-87-generic" }, platform: "Node.js v12.16.1, LE, mongodb-core: 2.1.20" }
10623 2022-05-01T19:57:04.607+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [1651435024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_ write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/ WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device
2022-05-01T19:57:04.608+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [16514 35024:608377][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_ write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:608377][80:0x7f0b7fe7e70 0], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/ WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No sp ace left on device
10625 2022-05-01T19:57:04.608+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [16514 35024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_u pdate, 391: WiredTiger.turtle: fatal turtle file update error: No space left on device Raw : [1651435024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_ turtle_update, 391: WiredTiger.turtle: fatal turtle file update error: No space left on de vice
2022-05-01T19:57:04.608+0000 E STORAGE [WTCheckpointThread] WiredTiger error (-31804) [1 651435024:608496][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_pani c, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1`enter code here`5024:608496][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_panic, 489: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-05-01T19:57:04.608+0000 F [WTCheckpointThread] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 486
2022-05-01T19:57:04.608+0000 F[WTCheckpointThread] \n\n***aborting after fassert() failure\n\n