Step 1: The Challenge of Complexity
Imagine a bustling restaurant. Each chef works hard in their kitchen station, and the manager sees everyone cooking. But for some reason, customers wait too long for their food. Why? The kitchen's dashboard shows all chefs are busy, but there's a hold-up in one small step—the sauce!
Twitter faced a similar problem when they moved from a single, large program to microservices.
What is a Microservice?
Microservice is a small, independent piece of a larger application that does one thing well. Imagine one chef responsible just for sauces!
Here's where microservices help:
- Smaller services: Each service is manageable and easier to understand.
- Clear ownership: Each team owns a service, like a chef owning a dish.
- Better scaling: Services grow independently, just like a chef who can whip up more salad without affecting the steak station.
Step 2: The Problem with Microservices
With multiple microservices in play, something new happened: a slowdown. Users experienced delays, even when individual services reported no problems.
Let's think back to the restaurant. Every individual kitchen station says, "All good," but orders don't come out fast enough because the sauce-making step is slow.
To solve this, Twitter needed to understand the whole journey from the user's order to the final dish—enter tracing.
Step 3: Introducing Zipkin
Twitter introduced Zipkin for tracing requests across their services.
What is Tracing?
Tracing is like creating a detailed breadcrumb trail that shows how an order travels through each kitchen station. It connects all the steps together.
How Does Zipkin Work?
- Trace Context: As a request (a customer's order) moves through services (kitchen stations), Zipkin gives it a unique ID, like a receipt number.
- X-B3-TraceId: ID for the full journey.
- X-B3-SpanId: ID for each individual step.
- X-B3-Sampled: Marks if this journey is tracked.
X-B3-TraceId: 80f198ee56343ba8
X-B3-SpanId: e457b5a2e4d86bd1
X-B3-Sampled: 1- Spans: Each step in a request's journey is a span. It records timings and important notes, like "sauce took 5 minutes".
What is a Span?
A span is a single step in a multi-step journey. Think of it as noting when a sauce starts and finishes.
- Annotations: These are milestones like "chef started cooking" or "chef finished".
- cs: Client send (order placed)
- sr: Server receive (order arrives at station)
- ss: Server send (order leaves the station)
- cr: Client receive (order completed)
With these annotations, Zipkin maps out every order's journey.
Step 4: Operating Zipkin
Every tool comes with its own set of challenges.
- Storage Costs: Storing many trace logs can get pricey.
- Sampling: Not every trip is logged. Imagine only keeping some order receipts. Too many, and storage fills up. Too few, and you miss slowdowns.
- Instrumentation Drift: Services might change over time, causing mismatches.
Twitter decided that the expense of Zipkin was worth the insights.
"Observability makes complexity visible and accountable."
The Core Lessons
- Microservices break a big job into manageable pieces but can hide delays in combined journeys.
- Tracing (with tools like Zipkin) shows the complete journey of requests, pinpointing slow parts exactly.
- Annotated Traces (spans and notes) help visualize how each step in a request's journey contributes to the whole experience.
- Operating a Tracing System comes with its own challenges, like managing storage and choosing which journeys to log.
Microservices and tracing together give teams the clarity to tackle hidden issues and ensure users have a smooth, timely experience.