Integration Testing Sagas: Simulate Failures and Recovery (2026)

Saga orchestration fails because the orchestrator’s state machine gets corrupted, leading to dropped or duplicated messages between services.

Cause 1: Network Partition Between Orchestrator and Participant

Diagnosis: Check network connectivity from the orchestrator’s pod to the participant service’s endpoint.

kubectl exec <orchestrator-pod-name> -n <namespace> -- nc -vz <participant-service-name>.<namespace>.svc.cluster.local <participant-port>

Fix: If nc times out, investigate Kubernetes NetworkPolicies blocking traffic or underlying network issues. Ensure the NetworkPolicy allows egress from the orchestrator’s namespace to the participant’s namespace on the required port. For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: orchestrator-egress
  namespace: saga-orchestrator
spec:
  podSelector:
    matchLabels:
      app: saga-orchestrator
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: payment-service
    ports:
    - protocol: TCP
      port: 8080 # Example port for payment-service

Why it works: NetworkPolicies are Kubernetes’ firewall. If they’re too restrictive, the orchestrator can’t reach the services it needs to send compensation or completion commands to.

Cause 2: Participant Service Unavailability During Command Execution

Diagnosis: Observe the orchestrator’s logs for timeouts or connection refused errors when sending commands to a specific participant. Check the participant service’s deployment status.

kubectl logs <orchestrator-pod-name> -n <namespace> -c <orchestrator-container> | grep "failed to send command to payment-service"
kubectl get pods -n <namespace> -l app=payment-service

Fix: If the participant pod is crashing or not running, debug the participant service itself. Common issues include insufficient resource limits (CPU/memory) or application errors within the participant. Scale up the participant’s replicas if it’s overloaded.

kubectl scale deployment payment-service --replicas=3 -n <namespace>

Why it works: The orchestrator relies on participants responding to commands. If a participant is down, the orchestrator cannot proceed or initiate compensation, leading to a stalled saga.

Cause 3: Orchestrator State Corruption Due to Application Crash/Restart

Diagnosis: Look for evidence of the orchestrator losing its state. This might manifest as duplicate saga instances starting, or existing sagas being replayed from the beginning. Check the orchestrator’s persistent storage (e.g., database, message queue).

# If using a database for state:
kubectl exec <orchestrator-db-pod> -n <namespace> -- psql -U <user> -d <database> -c "SELECT COUNT(*) FROM sagas WHERE status = 'IN_PROGRESS';"

Fix: Ensure the orchestrator’s state is persisted reliably. If using a database, ensure it’s resilient and has appropriate replication and backup strategies. If using a message queue for state transitions, ensure the queue is durable and messages are acknowledged correctly. For example, if using Kafka for state, ensure acks=all and min.insync.replicas=2 are configured for the relevant topics.

Why it works: The orchestrator’s state machine is the saga. If that state is lost or corrupted, the saga execution becomes unpredictable. Durable persistence ensures that even if the orchestrator pod restarts, it can recover its exact position.

Cause 4: Message Duplication or Loss in the Communication Layer (Message Broker)

Diagnosis: Examine message broker logs and consumer group offsets. If using Kafka, check for unusual numbers of fetch requests without corresponding produce requests, or significant lag in consumer offsets for the saga event topics.

# Example Kafka tool to inspect consumer groups and offsets
kafka-consumer-groups.sh --bootstrap-server <kafka-broker-list> --describe --group <saga-consumer-group>

Fix: Configure your message broker for idempotency and durability. For Kafka, this means:

• Producers: Use enable.idempotence=true and max.in.flight.requests.per.connection=5 (or lower).
• Consumers: Implement idempotent consumers. This often involves checking if an event has already been processed before applying its effects. Store processed event IDs in a database or cache. Ensure isolation.level=read_committed if using transactional producers.

Why it works: Sagas rely on reliable, ordered delivery of commands and events. Message duplication can lead to actions being performed twice (e.g., charging a customer twice), while message loss means a step might be skipped entirely. Idempotency ensures that even if a message is delivered multiple times, the action is only performed once.

Cause 5: Incorrectly Implemented Compensating Actions

Diagnosis: Observe the saga’s behavior during a simulated failure. If a saga enters a compensation phase but then gets stuck or enters an unexpected state, it implies the compensating action failed or didn’t complete its rollback logic.

# Trigger a failure scenario manually (e.g., by stopping a participant service)
# Then, tail the orchestrator logs and the logs of the service responsible for the compensating action.
kubectl logs <orchestrator-pod-name> -n <namespace> -f
kubectl logs <compensating-service-pod> -n <namespace> -f

Fix: Ensure each compensating action is a robust, idempotent operation. It should be able to handle being called multiple times. For example, if a cancel_order compensation is triggered, it should first check if the order is already cancelled before attempting to cancel it again. Log the outcome of compensation clearly.

Why it works: The core principle of a saga is that it always reaches a terminal state, either success or a consistent failure state via compensation. If compensation itself fails, the saga is left in an inconsistent, unrecoverable state.

Cause 6: Race Conditions Between Parallel Steps or Compensations

Diagnosis: If your saga has parallel branches, a race condition can occur if two parallel steps try to modify the same resource, or if a compensation for one parallel branch conflicts with the successful completion of another. This is hard to diagnose directly with a command; it’s usually identified by observing inconsistent final states and correlating with specific execution paths.

Fix: Implement proper locking or optimistic concurrency control for shared resources accessed by parallel saga steps. If using a database, use SELECT ... FOR UPDATE or version columns. Ensure compensation logic correctly handles potential overlaps with other concurrently executing branches by checking the current state of affected resources.

Why it works: Without proper synchronization, concurrent operations can interfere with each other, leading to data corruption or unexpected state transitions that violate the saga’s invariants.

The next error you’ll likely hit is a DeadlockDetected if you’re using a database for state and haven’t properly handled concurrent access to shared resources across parallel saga branches.