Your Java application is running smoothly, blissfully unaware of the danger lurking just around the corner. Then—without warning—it vanishes without a trace. Upon investigation, you discover the virtual remains of countless slain JVMs, each marked with the same cryptic number: 137. A serial killer is on the loose in your cluster.

This true crime investigation will walk you through the differences between JVM memory areas, the mechanics of container OOMKills, analysing Grafana dashboards for memory and thread anomalies, interpreting heap dumps, and utilising JVM Native Memory Tracking (NMT) to uncover hidden native memory consumption.

You’ll master practical forensic techniques and proven prevention strategies—from memory tuning tactics and balancing heap vs. non-heap allocations to leveraging large language models (LLMs) for advanced forensic analysis. These skills will help ensure your applications stay alive—even when something’s out to get them.

Just like in a real murder mystery, we may not know whodunnit... but with Martijn Dashorst leading the investigation, you're sure to get closer to the truth.

Devoxx

A Java developer investigates why a Quarkus monitoring application running in Kubernetes keeps getting killed with exit code 137 (OOM kill). The talk walks through the forensic process: examining Kubernetes logs, Grafana metrics, heap dumps, and native memory tracking reports. Key findings include that 150 threads with 2MB default stack size consume ~300MB of native memory, HTTPS buffers add overhead, and native memory fragmentation may be a culprit. The investigation covers Java's three memory areas (heap, non-heap, native), JVM tuning parameters, and practical tips like keeping the container alive with extra memory to gather evidence, exacerbating the issue to reproduce it faster, and using LLMs cautiously for analysis. The root cause remains unresolved, with a Netty committer suggesting that Quarkus/Netty uses unsafe memory allocation not trackable by JVM tools, recommending pmap for further investigation.

Catching the 137-Killer: A Java Memory Forensics Investigation by Martijn Dashorst