OpenTelemetry in Go lets you trace your services without modifying application code, but sometimes you need more control.
Here’s how to manually instrument your Golang services with OpenTelemetry, giving you granular visibility into your application’s internals.
Let’s start with a simple HTTP server.
package main
import (
"fmt"
"log"
"net/http"
"time"
)
func handler(w http.ResponseWriter, r *http.Request) {
time.Sleep(100 * time.Millisecond) // Simulate work
fmt.Fprintf(w, "Hello, World!")
}
func main() {
http.HandleFunc("/", handler)
log.Println("Server starting on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
This is a basic Go web server. It listens on port 8080 and has a single handler that sleeps for 100 milliseconds before responding.
To manually instrument this, we’ll add OpenTelemetry. First, ensure you have the necessary Go modules:
go get go.opentelemetry.io/otel \
go.opentelemetry.io/otel/sdk \
go.opentelemetry.io/otel/exporters/stdout/stdouttrace \
go.opentelemetry.io/otel/trace
Now, let’s modify the main function.
package main
import (
"context"
"fmt"
"log"
"net/http"
"time"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/trace"
// Exporter to print traces to stdout
"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
)
// initTracer initializes a trace provider with an stdout exporter.
func initTracer() (*sdktrace.TracerProvider, error) {
// Exporter to stdout
exporter, err := stdouttrace.New(stdouttrace.WithPrettyPrint())
if err != nil {
return nil, fmt.Errorf("failed to create stdout exporter: %w", err)
}
// Resource defines the service name and other attributes.
res, err := resource.New(context.Background(),
resource.WithAttributes(
// The service name is required.
// You can add other attributes like host.name, os.type, etc.
// See: https://opentelemetry.io/docs/specs/otel/common/#required-attributes
// For example: semconv.ServiceNameKey.String("my-go-app"),
// semconv.HostNameKey.String("my-host"),
),
)
if err != nil {
return nil, fmt.Errorf("failed to create resource: %w", err)
}
// TracerProvider provides the Tracer.
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exporter), // Use batching for efficiency
sdktrace.WithResource(res),
)
otel.SetTracerProvider(tp) // Set the global TracerProvider
return tp, nil
}
func main() {
tp, err := initTracer()
if err != nil {
log.Fatalf("failed to initialize tracer: %v", err)
}
// Shut down the tracer provider when the application exits.
defer func() {
if err := tp.Shutdown(context.Background()); err != nil {
log.Fatalf("failed to shutdown TracerProvider: %v", err)
}
}()
// Get a tracer instance
tracer := otel.Tracer("my-http-server")
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
// Start a new span for the incoming request
ctx, span := tracer.Start(r.Context(), "http-handler")
defer span.End() // Ensure the span is ended when the function returns
// Add attributes to the span
span.SetAttributes(
// Example: trace.String("http.method", r.Method),
// trace.String("http.url", r.URL.String()),
)
log.Println("Handling request...")
time.Sleep(100 * time.Millisecond) // Simulate work
// Start a child span for a specific operation within the handler
_, childSpan := tracer.Start(ctx, "database-query")
time.Sleep(50 * time.Millisecond) // Simulate database call
childSpan.End()
fmt.Fprintf(w, "Hello, World!")
log.Println("Request handled.")
})
log.Println("Server starting on :8080")
log.Fatal(http.ListenAndServe(":8080", nil))
}
The initTracer function sets up the OpenTelemetry SDK. It creates an exporter that prints traces to standard output (stdouttrace) and a Resource that identifies your service. The TracerProvider is then configured with these and set as the global provider.
Inside the HTTP handler, we obtain a tracer using otel.Tracer("my-http-server"). The Start method on the tracer creates a new Span. The first argument is the context.Context, which is crucial for propagating span information. The second argument is the name of the operation being traced. We use defer span.End() to ensure the span is always closed, recording its duration.
We also demonstrate creating a child span using tracer.Start(ctx, "database-query"). Notice how we pass the ctx from the parent span. This links the child span to its parent, forming a trace. Attributes can be added to spans using span.SetAttributes() for more context.
When you run this instrumented server and send requests (e.g., using curl http://localhost:8080), you’ll see detailed trace output on your console. This output includes information about each span, its duration, and its relationship to other spans.
The most surprising thing about manual instrumentation is how little code it actually adds to your core business logic. You’re not scattering complex OpenTelemetry setup everywhere; instead, you’re wrapping critical sections or request handlers with explicit Start and End calls, keeping your application logic clean.
You can inspect the generated output to see the http-handler span and the nested database-query span. Each span will have a TraceID and SpanID, allowing you to reconstruct the entire request flow. The duration of each span tells you exactly where time is being spent.
The context.Context is the invisible thread connecting your spans. When you call tracer.Start(ctx, ...), OpenTelemetry looks for an existing span in that context. If found, it creates a new span as a child of that existing span. If not found, it starts a new root span for that trace. This is how requests that traverse multiple functions or even multiple services (with context propagation) are linked together.
The next step is to export these traces to a dedicated backend like Jaeger or Prometheus for easier querying and visualization.