OpenTelemetry treats span events and logs as distinct signals, and the choice between them hinges on whether you’re trying to capture a specific moment in a trace or a more general, timestamped record of something happening.
Let’s see this in action. Imagine a simple web request that involves a database query and an external API call.
{
"traceId": "a1b2c3d4e5f67890",
"spanId": "1234567890abcdef",
"name": "HTTP GET /users",
"kind": "SERVER",
"startTimeUnixNano": 1678886400000000000,
"endTimeUnixNano": 1678886401000000000,
"attributes": {
"http.method": "GET",
"http.url": "/users",
"http.status_code": 200
},
"events": [
{
"name": "db.query",
"timeUnixNano": 1678886400500000000,
"attributes": {
"db.system": "postgresql",
"db.statement": "SELECT * FROM users WHERE id = 123;"
}
},
{
"name": "http.request",
"timeUnixNano": 1678886400700000000,
"attributes": {
"http.method": "GET",
"http.url": "https://api.example.com/userinfo",
"http.client_ip": "192.168.1.100"
}
}
],
"status": {
"code": 0
},
"links": [],
"resource": {
"attributes": {
"service.name": "my-web-app"
}
}
}
In this span, the events array contains two entries. The db.query event captures the exact moment and details of the database interaction. The http.request event captures the outbound call to an external API. Both are precisely timed occurrences within the lifespan of the /users request span.
Now, consider logs. Logs are typically generated by applications to record discrete, timestamped events that might not directly map to a specific operation within a trace.
{
"traceId": "a1b2c3d4e5f67890",
"spanId": "1234567890abcdef", // This log is associated with the /users span
"body": {
"stringValue": "User with ID 123 not found in cache, fetching from database."
},
"timeUnixNano": 1678886400450000000,
"attributes": {
"severityText": "INFO",
"severityNumber": 8,
"user.id": "123"
}
}
This log entry, while associated with the same traceId and spanId as the db.query event, provides a more narrative description of an application state or decision. It’s a record of what happened, not necessarily a sub-operation within the trace’s flow.
The fundamental difference lies in their purpose and how they are structured within OpenTelemetry. Spans represent operations with a duration, defined by a start and end time. Span events, on the other hand, are discrete, timestamped occurrences within the context of a parent span. They are often used to mark significant points or actions that happen during an operation, like a cache miss, an error being caught, or a specific step in a process. Logs, however, are independent records of events that have a timestamp and can optionally be correlated to a trace and span. They are generally more free-form and can contain arbitrary key-value pairs, including rich text bodies.
Think of it this way: a span is a journey, span events are specific landmarks or checkpoints you pass on that journey, and logs are diary entries you write along the way, possibly noting something important you saw or did, even if it wasn’t a formal "checkpoint."
The choice between span events and logs depends on what you need to see in your telemetry. If you’re trying to understand the timing and details of a specific, contained operation that is part of a larger trace (like a database query or an external API call), use span events. If you need to record general application events, debug messages, or state changes that might be useful independently or for correlating with a trace, use logs.
Many backends allow you to view logs that are associated with a specific trace. When you’re looking at a trace, you can often expand a span to see its associated events and any logs that were emitted during that span’s lifetime. This provides a comprehensive view of what happened during a particular operation.
The one thing that often trips people up is the optional spanId on a log record. It’s tempting to always associate logs with a span, but this can lead to an overwhelming amount of data in your trace view if not done judiciously. If a log doesn’t directly illuminate a specific part of a traced operation’s execution, it might be better to emit it as an un-traced log. This keeps your traces focused on the flow of requests and operations, while still capturing valuable application-level events.
The next step in understanding OpenTelemetry signals is exploring metrics and how they complement traces and logs for understanding system behavior over time.