Skip to content
Commitwise
Menu
Java Backend

Observability: logs, metrics and tracing

Know what your backend is doing in production: structured logs with trace ids, Micrometer metrics for rate, errors and latency percentiles, and distributed traces across services, in Spring Boot with Java and Kotlin, plus the mistakes that make each one useless.

intermediate

The core idea with one simple example.Step by step, with more examples and exercises.A cheat sheet: key APIs, gotchas and trade-offs.Internals, trade-offs and production pitfalls.

“Checkout is slow for some users.” Is it the database, the payment provider, one pod, one country, one app version? Without observability, the answer is guesswork and SSH sessions. With it, you open a dashboard, see that p99 latency for POST /orders jumped at 14:02, click into a slow trace, see 2.8 seconds spent in the fraud-check call, and find its error log with the same trace id. That chain, from metrics to traces to logs, is what this lesson builds.

New to some of these? Open what you need, or skip ahead.

Basics:What is Spring Boot?

Spring Boot is the most popular framework for Java and Kotlin backends. You write classes and mark them with annotations; Spring creates the objects (called beans), connects them to each other (dependency injection: a controller gets the service it needs through its constructor) and runs a web server that sends HTTP requests to your methods.

Starters (spring-boot-starter-web, spring-boot-starter-data-jpa…) add features by adding a dependency, with sensible defaults you can change in application.yml.

Basics:What are errors and exceptions?

When something goes wrong, code can throw an exception: an object describing the problem, which stops the current function and travels up to whoever catches it. An exception carries a message and a stack trace: the list of method calls that led to the error, which is often the fastest way to find the cause.

Part Not started

A ship’s logbook records what happened and when: “14:02, engine 2 temperature high”. When something goes wrong, the crew reads back through it. Application logs are the same: lines your code writes as it runs, kept so you can find out later what happened.

In production, there are millions of lines from dozens of servers, so the important thing is being able to search them: “all errors for order 1042” or “everything that happened during this one request”.

Logs are the highest-cardinality, highest-cost signal: great for the details of a specific event, poor for trends. Make them structured (JSON with stable field names, so the log platform indexes them), correlated (trace and span ids in every line via MDC, which Spring Boot adds automatically when tracing is on), leveled (INFO for business events, WARN for degraded behavior, ERROR for failures that need attention, DEBUG off in production) and safe (no tokens, passwords, card data or unnecessary personal data).

@Service
class CheckoutService(private val payments: PaymentClient) {
    private val log = LoggerFactory.getLogger(CheckoutService::class.java)

    fun checkout(customerId: String, cart: Cart): OrderConfirmation {
        log.atInfo()
            .setMessage("Checkout started")
            .addKeyValue("customerId", customerId)
            .addKeyValue("items", cart.items.size)
            .log()
        try {
            val confirmation = payments.charge(customerId, cart.totalCents)
            log.atInfo().setMessage("Order placed").addKeyValue("orderId", confirmation.orderId).log()
            return confirmation
        } catch (e: PaymentDeclinedException) {
            log.warn("Payment declined for customer {}", customerId, e)   // exception last: stack trace kept
            throw e
        }
    }
}
@Service
class CheckoutService {
    private static final Logger log = LoggerFactory.getLogger(CheckoutService.class);

    OrderConfirmation checkout(String customerId, Cart cart) {
        log.atInfo()
           .setMessage("Checkout started")
           .addKeyValue("customerId", customerId)
           .addKeyValue("items", cart.items().size())
           .log();
        try {
            var confirmation = payments.charge(customerId, cart.totalCents());
            log.atInfo().setMessage("Order placed")
               .addKeyValue("orderId", confirmation.orderId()).log();
            return confirmation;
        } catch (PaymentDeclinedException e) {
            log.warn("Payment declined for customer {}", customerId, e);   // exception last: stack trace kept
            throw e;
        }
    }
}

// application.yml (Spring Boot 3.4+): JSON logs on the console
// logging.structured.format.console: ecs
{"@timestamp":"2026-11-09T14:02:11.417Z","log.level":"INFO","message":"Order placed",
 "orderId":"o-1042","trace.id":"4bf92f3577b34da6a3ce929d0e0e4736","span.id":"00f067aa0ba902b7",
 "service.name":"checkout"}
  • Fields, not sentences: orderId is a field the log system can filter on, instead of text you have to search inside.
  • The trace id in every line connects all logs from one request, even across services.
  • Levels: INFO for normal business events, WARN when something went wrong but was handled, ERROR when someone needs to look, DEBUG for details that are off in production.
  • Never log passwords, tokens, full card numbers, or more personal data than you need.
Spot the bugErrors with no way to find the cause

Tap the line with the bug.

try {
    inventory.reserve(order)
} catch (e: Exception) {
    log.error("Reservation failed: " + e.message)
    throw CheckoutException("Could not reserve stock")
}
try {
    inventory.reserve(order);
} catch (Exception e) {
    log.error("Reservation failed: " + e.getMessage());
    throw new CheckoutException("Could not reserve stock");
}

Predict the output

The production log level is INFO. During one request, the code calls log.debug('Cart loaded'), log.info('Checkout started'), log.warn('Coupon expired') and log.error('Payment failed', e). Which lines appear in the logs?

Logging at scale

  • Cost: log volume is usually the largest observability bill. Sample noisy INFO, drop health-check access logs, and keep DEBUG off; use metrics for counts instead of logging every event.
  • Context propagation: MDC is thread-local; with coroutines use MDCContext(), with reactive code Micrometer’s context propagation, with virtual threads it works as usual. Spring Boot fills traceId/spanId automatically when Micrometer Tracing is present.
  • Audit vs diagnostic logs: “who changed the payout account” is an audit record with retention and integrity requirements; store it in a database or an append-only store, not just in application logs that rotate.
  • Structured JSON (logging.structured.format.console: ecs|logstash|gelf in Boot 3.4+), stable field names, SLF4J key-value API; trace/span ids in MDC.
  • Log exceptions as the last argument (stack trace), chain causes, log once where handled; parameterized messages, not concatenation.
  • Levels with intent; DEBUG off in production, per-logger overrides via Actuator loggers.
  • No secrets/PII; volume is cost: sample, drop health checks, prefer metrics for counts. MDC across coroutines (MDCContext) and reactive (context propagation).

Check this part

  1. How should an exception be logged?

  2. Why log orderId as a field instead of inside the message text?

  3. What connects log lines from one request across several services?

Logs are timestamped events. Write them as structured JSON with fields (order id, user id, trace id) instead of free text, log exceptions with their stack trace, choose levels deliberately, and never log secrets or personal data.

Part Not started

Metrics: the numbers behind the dashboards

A car’s dashboard doesn’t show a log of every engine revolution; it shows a few numbers that summarize the moment: speed, temperature, fuel. When the temperature needle climbs, you know something’s wrong before you know what.

Metrics are your backend’s dashboard: requests per second, how many fail, how long they take, how full the connection pool is. They’re cheap to store, fast to graph, and the basis for alerts that wake someone up before users complain.

Metrics are low-cardinality time series: cheap, aggregatable, alertable. For request-driven services, start with RED (rate, errors, duration) per endpoint; for resources, USE (utilization, saturation, errors). Micrometer is the facade in Spring Boot (Prometheus, OTLP, Datadog… via registries), and Boot already instruments HTTP server/client requests, JVM, pools, caches and data sources. Your job is business metrics, sane tags and percentile histograms where latency matters.

@Service
class CheckoutService(registry: MeterRegistry, private val payments: PaymentClient) {
    private val ordersPlaced = Counter.builder("shop.orders.placed")
        .description("Orders placed successfully")
        .register(registry)
    private val paymentTimer = Timer.builder("shop.payment.duration")
        .publishPercentileHistogram()                         // enables p95/p99 in Prometheus
        .register(registry)

    fun checkout(customerId: String, cart: Cart): OrderConfirmation {
        val confirmation = paymentTimer.recordCallable { payments.charge(customerId, cart.totalCents) }!!
        ordersPlaced.increment()
        return confirmation
    }
}
@Service
class CheckoutService {
    private final Counter ordersPlaced;
    private final Timer paymentTimer;
    private final PaymentClient payments;

    CheckoutService(MeterRegistry registry, PaymentClient payments) {
        this.payments = payments;
        this.ordersPlaced = Counter.builder("shop.orders.placed")
                .description("Orders placed successfully")
                .register(registry);
        this.paymentTimer = Timer.builder("shop.payment.duration")
                .publishPercentileHistogram()                 // enables p95/p99 in Prometheus
                .register(registry);
    }

    OrderConfirmation checkout(String customerId, Cart cart) {
        var confirmation = paymentTimer.record(() -> payments.charge(customerId, cart.totalCents()));
        ordersPlaced.increment();
        return confirmation;
    }
}

// application.yml: expose /actuator/prometheus for scraping
// management.endpoints.web.exposure.include: health,prometheus
TypeMeasuresExample
CounterSomething that only goes upOrders placed, emails sent
TimerHow long something takes, and how oftenPayment call duration
GaugeA current value that goes up and downItems in a queue, open connections

Spring Boot already measures every HTTP request (http.server.requests: count, errors, duration per endpoint), the JVM and the database pool. You add the ones about your business: orders placed, payments declined, coupons used.

Averages lie

Predict the output

An endpoint served 1,000 requests: 980 took 20 ms and 20 took 3,000 ms. What's the average latency, and what's the p99?

Spot the bugPrometheus runs out of memory

Tap the line with the bug.

fun recordCheckout(customerId: String, country: String, durationMs: Long) {
    Timer.builder("shop.checkout.duration")
        .tag("country", country)
        .tag("customerId", customerId)
        .register(registry)
        .record(Duration.ofMillis(durationMs))
}
void recordCheckout(String customerId, String country, long durationMs) {
    Timer.builder("shop.checkout.duration")
        .tag("country", country)
        .tag("customerId", customerId)
        .register(registry)
        .record(Duration.ofMillis(durationMs));
}

  • RED per endpoint (rate, errors, duration) + USE for resources; Boot auto-instruments HTTP, JVM, pools, caches. Add business metrics with Micrometer Counter/Timer/Gauge or @Observed.
  • Percentiles from histograms (publishPercentileHistogram), never averages; percentiles aren’t averageable across instances.
  • Tags from small fixed sets (route templates, status class, region, version); ids belong in logs/traces. Cardinality = product of tag value counts × histogram buckets.
  • Alert on symptoms (SLO burn rates on error/latency), not on every cause.

Check this part

  1. Which Micrometer meter fits "number of orders placed"?

  2. Why look at p99 latency instead of the average?

  3. Why not tag a metric with customerId?

Metrics are numbers aggregated over time: request rate, error rate and latency (RED), plus resource use. Record them with Micrometer counters, timers and gauges, look at latency percentiles (p95, p99) rather than averages, and keep tag values to a small fixed set.

Part Not started

Distributed tracing

A parcel’s tracking page shows every step: picked up, sorting center, truck, local depot, delivered, each with a time. If it’s late, you see exactly which step took two days.

A trace is a tracking page for one request. It’s made of spans: one for the incoming request, one for each database query, one for each call to another service, each with a start and an end. When checkout is slow, the trace shows that 2.8 of its 3 seconds were spent waiting for the fraud-check service.

Distributed tracing records causally linked spans across process boundaries. The W3C traceparent header carries trace id, parent span id and sampling decision; each service continues the trace and exports spans (OTLP) to a backend (Tempo, Jaeger, Zipkin, vendor APMs). Spring Boot’s Micrometer Tracing (with the OpenTelemetry bridge) instruments inbound/outbound HTTP, RestClient/WebClient, JDBC (with datasource-micrometer), Kafka and @Observed methods, and puts ids into the logging MDC. Sampling controls cost: head sampling (decide at the start, e.g. 10%) is cheap; tail sampling (decide after seeing errors/latency) keeps the interesting traces.

One checkout request, traced

traceparentMobile appAPI gateway15 msCheckout3,050 ms totalPostgreSQL3 queries, 12 msFraud check2,810 msKafkaOrderPlaced

Tap a box to see what it does.

// build: io.micrometer:micrometer-tracing-bridge-otel + io.opentelemetry:opentelemetry-exporter-otlp
// application.yml:
//   management.tracing.sampling.probability: 0.1          # keep 10% of traces
//   management.otlp.tracing.endpoint: http://otel-collector:4318/v1/traces

@Service
class FraudCheck(
    private val observations: ObservationRegistry,
    private val scoring: ScoringClient,
) {
    fun check(order: Order): Score =
        Observation.createNotStarted("fraud.check", observations)          // a span + a timer metric
            .lowCardinalityKeyValue("paymentMethod", order.paymentMethod.name)
            .observe(Supplier { scoring.score(order) })!!
}
// build: io.micrometer:micrometer-tracing-bridge-otel + io.opentelemetry:opentelemetry-exporter-otlp
// application.yml:
//   management.tracing.sampling.probability: 0.1          # keep 10% of traces
//   management.otlp.tracing.endpoint: http://otel-collector:4318/v1/traces

@Service
class FraudCheck {
    private final ObservationRegistry observations;
    private final ScoringClient scoring;

    FraudCheck(ObservationRegistry observations, ScoringClient scoring) {
        this.observations = observations;
        this.scoring = scoring;
    }

    Score check(Order order) {
        return Observation.createNotStarted("fraud.check", observations)   // a span + a timer metric
                .lowCardinalityKeyValue("paymentMethod", order.paymentMethod().name())
                .observe(() -> scoring.score(order));
    }
}

Most spans appear without any code: Spring Boot traces incoming requests, outgoing HTTP calls made with its RestClient, and more. You add an Observation around important work of your own, and get both a span in the trace and a timer metric. The trace id also shows up in every log line, so from a slow trace you jump straight to its logs.

Exerciseeasy

Make a slow endpoint observable

GET /api/recommendations is sometimes slow. It calls the catalog service over HTTP, runs a ranking function, and reads from Redis. Add what you’d need to find out which part is slow: what’s measured automatically, and what you add.

›

Some calls are traced automatically by Spring Boot; the ranking function is your own code. One Observation gives you a span and a metric at once.

SolutionTry it yourself first
  • Automatic (with Micrometer Tracing and the OTLP exporter configured): the incoming request span, the outgoing HTTP call to the catalog (if made with Boot’s auto-configured RestClient.Builder), Redis commands (Lettuce instrumentation), and the http.server.requests timer with p95/p99 per endpoint.
  • Add: an Observation around ranking, so it appears as its own span and metric:
fun rank(items: List<Item>, userSegment: String): List<Item> =
    Observation.createNotStarted("recommendations.rank", observations)
        .lowCardinalityKeyValue("segment", userSegment)
        .highCardinalityKeyValue("candidates", items.size.toString())
        .observe(Supplier { ranker.rank(items, userSegment) })!!
List<Item> rank(List<Item> items, String userSegment) {
    return Observation.createNotStarted("recommendations.rank", observations)
            .lowCardinalityKeyValue("segment", userSegment)           // a few segments: fine as a tag
            .highCardinalityKeyValue("candidates", String.valueOf(items.size()))   // span only, not a metric tag
            .observe(() -> ranker.rank(items, userSegment));
}
  • Then: on the dashboard, compare p99 of the request, the catalog call and the ranking span; open a slow trace to see which child span dominates; use its trace id to read the logs. Low-cardinality key-values become metric tags; high-cardinality ones only go on the span.
Exercisemedium

Investigate a latency incident

At 14:02, an alert fires: the error-budget burn rate for checkout latency is 10× normal. You have Prometheus metrics, Tempo traces (10% head sampling) and logs in Loki, all with trace ids. Write the investigation as a sequence of questions and the signal that answers each, and what you’d change in the setup afterwards.

›

Narrow from “what” to “where” to “why”: scope with metrics, locate with traces, explain with logs. Then ask whether sampling hid the evidence.

SolutionTry it yourself first
  1. Is it real, and how big? Metrics: p99 and error rate of http.server.requests{uri="/api/orders",method="POST"} vs baseline; requests per second (a traffic spike?).
  2. Who’s affected? Metrics split by low-cardinality tags: per pod (one bad instance?), per region, per app version, per payment method. A deploy annotation at 14:00 would be the first suspect.
  3. Which dependency? Client metrics (http.client.requests by target), DB pool metrics (hikaricp.connections.pending, acquire time), Kafka producer latency. A saturated pool shows as pending connections, not as slow queries.
  4. Where in the request? Traces: filter by service and duration > 2 s in the window; the span breakdown shows which child dominates (e.g., fraud.check → external scoring API with retries).
  5. Why? Logs for those trace ids: timeouts, retry warnings, error responses from the provider; correlate with the provider’s status page.
  6. Mitigate: tighten the fraud-check timeout and fall back to “manual review” instead of blocking checkout; scale or roll back if a deploy caused it.
  7. Improve the setup: tail-based sampling in the OpenTelemetry Collector (keep all traces with errors or > 1 s, sample the rest), an exemplar link from latency histograms to trace ids, a dashboard panel per critical dependency, and an alert on the dependency itself so next time the page names the cause.
  • Traces: spans linked by W3C traceparent (HTTP headers, message headers); Micrometer Tracing + OTel bridge exports via OTLP; Boot instruments HTTP server/client, Kafka, and @Observed/Observation API (span + timer in one).
  • Low-cardinality key-values → metric tags; high-cardinality → span attributes only.
  • Sampling: head (cheap, management.tracing.sampling.probability) vs tail in the Collector (keep errors/slow traces); exemplars link histograms to traces.
  • Workflow: metrics scope → traces locate → logs explain; alert on SLO burn rates.

Check this part

  1. What does a trace show?

  2. How do logs and traces connect?

  3. With 10% head sampling, why might the slow request you need be missing, and the fix?

A trace follows one request through every service it touches, as a tree of timed spans. Context travels in the traceparent header, Spring Boot creates spans for HTTP, database and messaging calls with Micrometer Tracing and OpenTelemetry, and trace ids in logs link everything together.

Key takeaways

  • Logs record events: make them structured, include the trace id, log exceptions with their stack trace, and never log secrets.
  • Metrics summarize behavior over time: counters, timers and gauges, with latency percentiles instead of averages.
  • Traces follow one request across services as timed spans.
  • Together: metrics tell you something is wrong, traces show where, logs explain why.
  • Structured, correlated, leveled, safe logs; volume is cost.
  • RED/USE metrics with histograms; small fixed tag sets; alert on SLO burn.
  • Micrometer Observation gives spans and metrics together; propagate context through HTTP and messaging.
  • Tail sampling and exemplars make the interesting traces findable.

Finished this page?Nice work, this page is complete!

Mark it as done to track your progress. It's saved in this browser, no account needed.

Its cards are now in your Bites review ⚡
Esc

Type to search the whole site.