A Java service's p99 latency spikes every few minutes. How do you find the cause?
BackendSenior
In shortMeasure top-down instead of guessing. Check traces to see whether the time is in this JVM, correlate spikes with GC pauses, CPU and pool waits, then profile with JFR or async-profiler. Fix the workload first; tune the GC only when pauses are the proven cause.
Model answer
1. Is the time in this JVM? Open traces of slow requests. If the time is in a database query or a downstream call, the JVM isn’t the problem.
2. What happens at the same moment? Put p99 latency next to:
- GC pause times (
jvm.gc.pause, GC logs). A periodic multi-secondPause Fullon G1 means concurrent marking fell behind, often because of humongous allocations. - CPU usage and CPU throttling in the container.
- Connection pool and thread pool waits (HikariCP pending threads, Tomcat busy threads).
3. Profile. Record with Java Flight Recorder (around 1% overhead, can run continuously) or async-profiler during a spike. Read the CPU and allocation flame graphs by width; check lock contention and thread states for time spent waiting.
4. Fix in order of payoff:
- Fix the workload: hot allocations, needless work, N+1 queries, oversized responses.
- Size the heap and container correctly.
- Choose the collector: G1 by default, ZGC for strict latency or very large heaps.
- Change single flags, measuring before and after each one.
What a strong answer adds: allocation rate drives GC frequency, so reducing allocation often helps more than GC tuning. JIT warm-up explains slow first requests after a deploy, and readiness gating hides it from users.
What they're looking for
A methodical order (traces, then saturation signals, then profiling), knowing how to read GC logs and what a G1 full GC means, JFR and flame graphs, allocation rate as the driver of GC, and changing one thing at a time with before and after numbers.
Common mistakes
- Raising the heap or copying GC flags before looking at data.
- Looking only at average latency.
- Ignoring time spent waiting for connection or thread pools, which looks like slowness but uses no CPU.
- Benchmarking a suspect with a hand-written timing loop.
More backend questions
- What is the N+1 query problem, and how do you find and fix it?Mid-level
- What does Spring Boot's auto-configuration do, and how do you change what it set up?Mid-level
- How does a Spring backend authenticate requests from a mobile app, and how do you make sure users only see their own data?Mid-level
- Your Spring Boot service in Kubernetes keeps restarting with exit code 137. How do you investigate and fix it?Senior