Skaffold Verify lets you run integration tests against your deployed application before Skaffold declares the deployment successful, ensuring your changes are actually working in the cluster.
Let’s see it in action. Imagine you have a simple web service deployed with Skaffold. After deploying a new version, you want to make sure it responds correctly to a basic HTTP GET request.
apiVersion: skaffold/v2beta26
kind: Config
deploy:
kubectl:
manifests:
- k8s/*
profiles:
- name: dev
deploy:
kubectl:
manifests:
- k8s/dev/*
verify:
- name: check-api-ready
container:
name: my-app-container # The name of the container running your app
image: my-registry/my-app:latest # The image Skaffold is deploying
command: ["curl", "-f", "http://localhost:8080/health"] # The command to run
# Optional: You can also specify a policy for how many times to retry
# retryUntil: "5m" # Retry for up to 5 minutes
# Or a specific number of retries with a delay
# retry: 5
# delayMillis: 1000
In this example, skaffold dev will deploy your application. Once the deployment is running, Skaffold will execute the check-api-ready verification step. It will run the curl -f http://localhost:8080/health command inside the my-app-container of your deployed pod. The -f flag for curl is crucial: it makes curl return a non-zero exit code if the HTTP request fails (e.g., 404, 500) or if the connection times out. If curl exits with 0 (success), Skaffold considers the verification passed and the deployment complete. If it exits with a non-zero code, Skaffold will report the verification failure and roll back the deployment.
The core problem Skaffold Verify solves is the gap between "deployment complete" and "application actually working." In a typical CI/CD pipeline, a deployment might be marked as successful as soon as the Kubernetes objects (Deployments, StatefulSets) are updated and the pods are running. However, the application within those pods might still be crashing, failing to bind to ports, or not responding to health checks. Skaffold Verify bridges this gap by allowing you to define explicit checks that must pass before the deployment is finalized.
Internally, Skaffold uses the kubectl command-line tool to interact with your cluster. For container verification, it essentially executes kubectl exec <pod-name> -- <command> within the context of the deployed application’s pod. It waits for this command to complete and checks its exit code. If you specify retryUntil or retry/delayMillis, Skaffold will repeatedly execute this kubectl exec command until the condition is met or the retry limit is reached. This is incredibly useful for applications that have a startup latency, where the application might not be immediately ready to serve requests.
The image field in the container verification block is important. Skaffold uses this to identify which specific container within the pod to execute the command in. If your pod has multiple containers, you must specify the correct one. The command is the actual shell command that will be run. You can use any executable available within that container’s image. For more complex scenarios, you can even use a script that’s part of your application’s image.
Beyond container verification, Skaffold Verify also supports kubeconfig and http verification types, offering flexibility for different testing needs. kubeconfig allows you to run arbitrary kubectl commands, while http directly probes HTTP endpoints without needing to run a command inside a container.
A common misconception is that skaffold verify is just a fancier version of kubectl rollout status. It’s more proactive. kubectl rollout status primarily checks the readiness of the Kubernetes resource itself (e.g., has the Deployment scaled up to the desired number of replicas?). Skaffold Verify checks the health and functionality of the application inside those replicas. You can have a Deployment that kubectl rollout status considers successful, but your application might be failing its own internal health checks.
When using skaffold dev with verification, Skaffold will watch for deployment events. Once the Kubernetes resources are reported as ready by the cluster, Skaffold will then initiate its verification steps. If any verification fails, Skaffold will mark the deployment as failed, and the skaffold dev process will typically exit with a non-zero status code, signaling the failure to any upstream CI/CD systems. This ensures that you don’t proceed with further stages of your pipeline (like releasing to production) if your integration tests don’t pass.
The next step after mastering basic verification is to integrate more sophisticated testing frameworks or custom scripts that perform deeper integration checks, such as database connectivity, inter-service communication, or specific business logic validation.