The most surprising thing about OpenTelemetry span kinds is that they don’t actually dictate how spans are stored or exported, but rather how they are correlated and visualized by tracing backends.
Let’s see this in action. Imagine a simple request to a web service that then publishes a message to a Kafka topic.
// Start a span for the incoming HTTP request
tracer.StartSpan("GET /users/{id}", Trace.SpanKind.SERVER)
// Later, when processing the request, we make a call to another service
// This would be a CLIENT span
tracer.StartSpan("GET /posts/{userId}", Trace.SpanKind.CLIENT)
// And then we publish a message to Kafka
// This would be a PRODUCER span
tracer.StartSpan("publish_user_post", Trace.SpanKind.PRODUCER)
The SERVER span represents an incoming request handled by a service. The CLIENT span represents an outgoing request initiated by a service. The PRODUCER span represents a message sent by a service to a message broker.
The core problem these span kinds solve is attributing work. When a request comes into your system (SERVER span), you want to know what work it triggered. If that work involved calling another service (CLIENT span), or sending a message (PRODUCER span), you want to see that these downstream operations are children of the initial incoming request. The tracing backend uses the parent-child relationships, along with the SPAN_ID and TRACE_ID, to reconstruct the flow.
Internally, when you start a span, you typically pass a parent span context. However, the SpanKind is a separate attribute. The tracing backend receives a list of spans, each with its own trace_id, span_id, parent_id, and kind. It’s the combination of trace_id and parent_id that forms the causal link. The kind is then used to semantically label the role of that specific operation within the trace. For example, a SERVER span is often the root of a trace for an incoming request, and its children might be CLIENT or INTERNAL spans. A CLIENT span’s children are typically INTERNAL spans representing the client’s own processing logic.
Consider a typical web request flow:
- Incoming HTTP Request: A user’s browser (or another service) makes an HTTP request to your
API Gateway. TheAPI Gatewaycreates aSERVERspan for this request. - Route to Microservice: The
API Gatewayforwards the request to yourUser Service. TheAPI Gatewayinjects the trace context into the outgoing HTTP headers. TheUser Servicereceives this context and starts a newSERVERspan, which becomes a child of theAPI Gateway’sSERVERspan. - Call Another Service: The
User Serviceneeds user details, so it makes an HTTP call to aProfile Service. TheUser Servicestarts aCLIENTspan for this outgoing call, which becomes a child of its ownSERVERspan. The trace context is again injected into the headers. - Profile Service Processing: The
Profile Servicereceives the request and starts its ownSERVERspan, which becomes a child of theUser Service’sCLIENTspan. - Publish to Kafka: After fetching the profile, the
User Servicepublishes a message to a Kafka topic about the user being accessed. It starts aPRODUCERspan for this, which becomes a child of itsSERVERspan.
The key is that the tracing backend uses the parent_id to build the hierarchy. The kind is then used to color or categorize these operations in the UI. For example, a tracing UI might visually distinguish SERVER spans (often at the top level of a request trace) from CLIENT spans (representing outgoing calls).
The most common confusion arises with PRODUCER and CONSUMER spans. A PRODUCER span represents the act of sending a message. A CONSUMER span represents the act of receiving and processing that message. The link between them is established by propagating trace context within the message metadata itself. When the User Service publishes its message, it injects the trace context (including its span_id which becomes the parent_id for the consumer) into the Kafka message headers. The Kafka Consumer then starts a CONSUMER span, using the extracted context to link it back to the PRODUCER span.
The next concept you’ll likely encounter is understanding how trace context propagation works across different protocols (HTTP, gRPC, Kafka headers).