Metrics and logs were the easy part, and I wasn’t ready for traces.
It took me a while to get it, first getting a grasp on the vocabulary, then putting it all together and managing to see a tree. I won’t blame myself too much, because a trace tree has several ways to fail silently, and I hit them one after another on the same code.

This is the post I wish I’d read first. Not an OpenTelemetry tutorial: the handful of things that stood between me and a working job > phase > task tree, in the order I tripped over them.

Why traces, and not just more logs Link to heading

Logs and metrics answer questions you decided on in advance. You chose what to log, which counter to bump. They’re proactive: you predict what you’ll want to know, then instrument for it.
When something you didn’t predict goes wrong, you add a log line, redeploy and wait.

Traces flip that. You instrument the structure once (who called what, in what order, nested how) and then you can ask questions you never anticipated, after the fact, without touching the code. “Why was this specific task re-run, and by which worker?” isn’t a question you planned for. With a trace, the answer is already sitting in the data, waiting for you to visualize it.

That’s the reactive power, and it’s why traces are worth the extra pain: they turn “I need to reproduce this with more logging” into “let me go look.” The pain is just that all the ways to get it wrong are quiet.

Vocabulary, so the rest makes sense Link to heading

Five words, because the failures below are mostly about confusing them.

  • Trace: one tree. Everything sharing a trace_id.
  • Span: one node, in a tree: a start, an end, some attributes, and a parent.
  • Span event: a timestamped annotation inside a span. span.AddEvent("reaped"). It lives in the trace.
  • Wide event: a fat structured log line. Lives in the log system, not the trace. It can carry trace_id and span_id as attributes, which makes it correlatable with a span, but it is not attached to one.
  • Context: Go’s context.Context, carrying the current span. This, and only this, links a child to its parent.

That last point is the whole article. Parent-child comes from the context you pass to Start. Nothing else.
Not events, not naming, not attributes. If the tree is wrong, the context is wrong.

The four traps below aren’t random, they follow a span’s life cycle: it has to be born (real provider), it has to travel across a process boundary, it has to attach to the right parent, and it has to die so it gets exported. I hit them in exactly that order, and that order is also how you debug them.

Trap 1 - birth: the no-op tracer Link to heading

otel.Tracer("x") doesn’t fail if you call it before the provider is set up. It returns a no-op tracer: it produces spans that are never recorded, never exported, and useless as parents. No panic, no warning…just a silent dud. So if you grab your tracer and start a long-lived span before calling your OTel setup, that span is a ghost, and everything descending from it starts a fresh root instead.

Fix: set up the provider first, before any Start. (Obvious in hindsight; invisible while it’s happening, which is kind of the theme here.)

Trap 2 - transport: the silent propagator Link to heading

Between two processes, context.Context doesn’t travel (it’s a local in-memory thing). So you serialise the trace context into whatever the transport carries, and rebuild it on the other side: Inject and Extract.

What actually crosses the wire is tiny: a single traceparent string:

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
|  |                                |                |
|  trace_id (16 bytes)              span_id (8 bytes)|
version                                              flags

That’s the whole thing. Version, trace, span, flags.

The trap: Inject needs a propagator to be registered, and if you forget, it writes nothing into the carrier. No warning, no error. Extract on the other side finds nothing and starts a fresh root.

otel.SetTextMapPropagator(propagation.TraceContext{})

One line, once, per binary. Miss it and the wire carries an empty map, silently.

(If you read my Victoria* setup post: same failure mode as the swallowed 404s. The OTel SDK’s whole philosophy is that telemetry must never break your app — which is right, and which means it will never tell you it’s broken.)

Trap 3 - parent: passing the wrong context Link to heading

This one’s subtle because the tree looks almost right.

I had a job span, a phase span under it, and I was starting each task from the job context:

ctx, span := tracer.Start(c.JobCtx, "task")   // parent = job, skips the phase

One trace, correct-ish parent, no error. But my phase span had zero children — every task hung directly off the job, and the phase sat there empty and pointless. The tree was valid and wrong at the same time.

ctx, span := tracer.Start(c.PhaseCtx, "task")   // parent = phase

One word. The lesson: it’s not enough for the context to be a real span context — it has to be the right one. The link is only ever as correct as the context you hand to Start.

(Propagating context across the RPC boundary to the workers is a whole other story, with its own good traps — that one’s going in a longer piece.)

Trap 4 - death: the span that never ends Link to heading

A span is only exported when you call End() on it. A span you never end is never sent — so if it’s a parent, its children point at a node that isn’t there. A ghost parent, and a tree that looks broken in the UI even though every link is correct.

The task spans were easy: they end when the task completes or when the reaper kills them. But the long-lived ones — job, phase — had no obvious end, so I’d left them open. They never shipped.

Fix: end them deliberately, when the work they represent is actually over.

func (c *Coordinator) Finish() {
    c.phaseSpan.End()
    c.jobSpan.End()
}
// called once Done() is true, before shutting the providers down

And mind the order: end your spans, then flush the providers. The batcher can only ship what’s been ended.

Bonus trap: there’s no tree view where you’d expect one Link to heading

Once all four were fixed, my data was correct: one trace, three levels, retried tasks as siblings, and yet no tree in vmui 🤯. I was surprised, but it’s actually expected. VictoriaTraces’ built-in UI (/select/vmui), has only Group / Table / JSON modes. It shows spans as flat rows. It does not draw waterfalls.

VictoriaTraces stores and queries traces but delegates visualisation to Jaeger or Grafana, via its Jaeger-compatible API. Point a Grafana Jaeger datasource at http://<host>:10428/select/jaeger/, paste the trace_id, and there’s your waterfall. Really, do it, after going through all the trouble, you want a proper visualization.

You can sanity-check the tree without any of that, though: in the Table view, show trace_id and parent_span_id. One trace_id, every task pointing at the phase’s span_id. That’s the tree, in text.

The pattern Link to heading

Every one of these failed the same way: no error, wrong tree. No-op tracer, empty propagator, wrong parent context, unended parent: four different bugs, one identical symptom. Telemetry is built to never crash your app, which is great actually, but as a consequence it will never tell you it’s misconfigured. The only feedback loop is looking at the data and asking “is this shaped the way I meant?”

That’s the real takeaway, more useful than any single fix: when a trace looks wrong, don’t debug the tool. Check, in order: did the provider exist yet, is the propagator set, which context did I pass, did the parent end. It’s almost always one of those four.