This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Health Check Features

Advanced health check capabilities in Krkn

1: Health Check Run Timing (run_during)
2: Object State Health Checks
3: Prometheus Alert Health Checks
4: Kube Virt Checks

Krkn provides comprehensive health check capabilities to monitor your applications and infrastructure during chaos testing.

Health Check Types

HTTP Health Checks - Monitor HTTP/HTTPS endpoints
Prometheus Alert Health Checks - Evaluate PromQL alerts at health-check phases
KubeVirt Health Checks - Monitor VMI SSH connectivity
Object State Health Checks - Monitor Kubernetes resource conditions

Advanced Features

Health Check Run Timing (run_during) - Control when health checks execute (pre, during, post chaos)
Exit on Failure - Control chaos execution based on health check results
Telemetry Integration - Track health metrics and downtime

Quick Reference

Run Timing Options

health_checks:
    run_during: "pre"                    # Before chaos only
    run_during: "during"                 # During chaos only (default)
    run_during: "post"                   # After chaos only
    run_during: ["pre", "post"]          # Before and after (not during)
    run_during: ["pre", "during", "post"]  # All three stages

Health Check Types in Config

# HTTP endpoints
health_checks:
    interval: 5
    run_during: "during"
    config:
        - url: "http://my-app/health"

# VMI SSH connectivity
kubevirt_checks:
    interval: 10
    run_during: "during"
    namespace: "vms"

# Kubernetes object conditions
object_state_checks:
    interval: 5
    run_during: "during"
    config:
        - name: "etcd-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

1 - Health Check Run Timing (run_during)

Control when health checks execute: pre-chaos, during chaos, or post-chaos

Overview

Health checks in Krkn can run at three different timings using the run_during configuration:

Pre-checks (run_during: "pre"): Run once before chaos scenarios to verify baseline health
During checks (run_during: "during"): Run continuously during chaos to monitor degradation
Post-checks (run_during: "post"): Run once after chaos scenarios to verify recovery

You can combine timings using a list: run_during: ["pre", "post"] to run the same health check at multiple points.

Configuration

Configure health checks with the run_during option in your config.yaml:

health_checks:
    interval: 2                      # For continuous checks (run_during: "during")
    run_during: "during"             # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False           # If True: pre-checks block chaos, post-checks fail the run
    config:
        - url: "http://my-app.example.com/health"
          bearer_token: "my-token"

kubevirt_checks:
    interval: 2                      # For continuous checks (run_during: "during")
    run_during: "during"             # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False           # If True: pre-checks block chaos, post-checks fail the run
    namespace: "my-namespace"
    name: "my-vmi-.*"                # Regex pattern for VMI names

object_state_checks:
    interval: 5
    run_during: "during"
    exit_on_failure: False
    config:
        - name: "etcd-pods-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

Supported Values

Value	Behavior
`"pre"`	Run once before chaos scenarios
`"during"`	Run continuously during chaos (default)
`"post"`	Run once after chaos scenarios
`["pre", "post"]`	Run at multiple timings (any combination)

Use Cases

Pre-Check Only: Ensure Clean Baseline

Verify your system is healthy before starting chaos:

health_checks:
    run_during: "pre"
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Post-Check Only: Verify Recovery

Verify your system has recovered after chaos:

health_checks:
    run_during: "post"
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Pre and Post: Baseline and Recovery

Run the same check before and after chaos:

health_checks:
    run_during: ["pre", "post"]
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Complete Monitoring: Pre, During, and Post

Monitor health at all stages:

health_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - url: "http://my-app.example.com/health"

Exit Codes

Code	Meaning
0	Success
1	Post-scenario failure
2	Critical Prometheus alerts
3	Continuous health check failure (`run_during: "during"`)
4	Pre or post health check failure (when `exit_on_failure: True`)

Backward Compatibility

If run_during is not specified, health checks default to "during":

# This (existing configs)
health_checks:
    interval: 5
    config:
        - url: "http://my-app/health"

# Is equivalent to
health_checks:
    interval: 5
    run_during: "during"  # Default
    config:
        - url: "http://my-app/health"

2 - Object State Health Checks

Monitor Kubernetes resource conditions during chaos testing

Overview

The Object State Health Check plugin allows you to monitor the state of any Kubernetes resource by checking conditions on those objects. This is useful for verifying that critical infrastructure components maintain their desired state during chaos testing.

Features

Flexible Resource Monitoring: Check any Kubernetes resource type (Pod, Deployment, StatefulSet, DaemonSet, etc.)
Condition-Based Checks: Monitor specific conditions like Ready, Available, Progressing
Regex Pattern Matching: Use regex patterns to match multiple objects with one check
Label Selector Support: Filter objects by labels
Run Timing Control: Use run_during to run checks at pre, during, or post chaos stages
All Must Pass: When multiple objects match, ALL must be healthy for the check to pass

Configuration

Basic Structure

object_state_checks:
    interval: 5                              # Check interval for continuous monitoring
    run_during: "during"                     # When to run: "pre", "during", "post", or ["pre", "post"]
    exit_on_failure: False                   # Fail on unhealthy objects
    config:
        - name: "check-name"                 # Descriptive name
          kind: "Pod"                        # Resource kind
          object_name: "my-pod-.*"          # Name or regex pattern
          namespace: "default"               # Namespace
          label_selector: ""                 # Optional label selector
          condition:
              type: "Ready"                  # Condition type
              status: "True"                 # Expected status

Examples

Check Pod Readiness

Verify that all etcd pods are Ready:

object_state_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - name: "etcd-pods-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition:
              type: "Ready"
              status: "True"

Check Deployment Availability

Verify a deployment is Available:

object_state_checks:
    interval: 10
    run_during: "post"
    exit_on_failure: True
    config:
        - name: "myapp-deployment-available"
          kind: "Deployment"
          object_name: "myapp"
          namespace: "default"
          condition:
              type: "Available"
              status: "True"

Multiple Checks

Monitor different resource types:

object_state_checks:
    interval: 5
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    config:
        - name: "kube-apiserver-ready"
          kind: "Pod"
          object_name: "kube-apiserver-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

        - name: "etcd-ready"
          kind: "Pod"
          object_name: "etcd-.*"
          namespace: "kube-system"
          condition: {type: "Ready", status: "True"}

        - name: "app-deployment-available"
          kind: "Deployment"
          object_name: "my-app"
          namespace: "production"
          condition: {type: "Available", status: "True"}

Multiple Object Behavior

IMPORTANT: When multiple objects match the pattern/labels, the health check requires ALL of them to pass.

How It Works

- name: "etcd-pods-ready"
  kind: "Pod"
  object_name: "etcd-.*"           # Matches: etcd-0, etcd-1, etcd-2
  namespace: "kube-system"
  condition: {type: "Ready", status: "True"}

Behavior:

✅ Check PASSES if: etcd-0 Ready=True AND etcd-1 Ready=True AND etcd-2 Ready=True
❌ Check FAILS if: ANY pod has Ready=False (e.g., etcd-1 is not ready)

Common Condition Types

Pods

Condition Type	Status	Meaning
`Ready`	`True`	All containers are ready
`PodScheduled`	`True`	Pod has been scheduled to a node
`Initialized`	`True`	Init containers have completed
`ContainersReady`	`True`	All containers are ready

Deployments

Condition Type	Status	Meaning
`Available`	`True`	Minimum availability requirements met
`Progressing`	`True`	Deployment is progressing

StatefulSets / DaemonSets

Condition Type	Status	Meaning
`Ready`	`True`	All replicas/pods are ready
`Available`	`True`	Resource is available

Pattern Matching

Exact Match

object_name: "my-pod"  # Matches only "my-pod"

Regex Pattern

object_name: "etcd-.*"         # Matches etcd-0, etcd-1, etcd-2, etc.
object_name: ".*-worker-.*"    # Matches any name containing "-worker-"
object_name: "app-[0-9]+"      # Matches app-1, app-2, app-123, etc.

Label Selectors

kind: "Pod"
namespace: "production"
label_selector: "app=myapp,version=v2"
# Checks only pods with both labels

Run Timing Examples

Pre-Check: Gate Chaos on Healthy State

object_state_checks:
    run_during: "pre"
    exit_on_failure: True
    config:
        - name: "control-plane-healthy"
          kind: "Pod"
          namespace: "kube-system"
          label_selector: "tier=control-plane"
          condition: {type: "Ready", status: "True"}

During: Continuous Monitoring

object_state_checks:
    interval: 5
    run_during: "during"
    exit_on_failure: False
    config:
        - name: "app-pods-status"
          kind: "Pod"
          namespace: "production"
          label_selector: "app=myapp"
          condition: {type: "Ready", status: "True"}

Post-Check: Verify Recovery

object_state_checks:
    run_during: "post"
    exit_on_failure: True
    config:
        - name: "all-deployments-available"
          kind: "Deployment"
          namespace: "production"
          condition: {type: "Available", status: "True"}

Supported Resource Kinds

The following built-in Kubernetes resource kinds are supported. Namespaced resources require a namespace; cluster-scoped resources do not.

Namespaced Resources

Pod
Deployment
StatefulSet
DaemonSet
ReplicaSet
Service
PersistentVolumeClaim
Job
CronJob

Cluster-Scoped Resources

Node
PersistentVolume

3 - Prometheus Alert Health Checks

Evaluate PromQL alert expressions before, during, and after chaos experiments

Prometheus alert health checks evaluate PromQL expressions from an alert profile during a Krkn run. They let you verify cluster SLOs and automatically identify health regressions caused by a chaos scenario.

Prerequisites

Prometheus must be reachable from the machine running Krkn.
On OpenShift, Krkn can discover the Prometheus route and bearer token. On Kubernetes, set prometheus_url and prometheus_bearer_token.
An alert profile must contain one or more PromQL expressions.

Configuration

Configure alert health checks under performance_monitoring:

performance_monitoring:
    prometheus_url: "http://prometheus.example.com"
    prometheus_bearer_token: ""
    enable_alerts: True
    alert_profile: config/alerts.yaml
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    only_failures: False

Field	Description	Default
`prometheus_url`	Prometheus API URL.	Auto-detected on OpenShift
`prometheus_bearer_token`	Bearer token used to authenticate with Prometheus.	Auto-detected on OpenShift
`enable_alerts`	Enable alert-profile evaluation.	`False`
`alert_profile`	Path or URL to the alert profile.	`config/alerts.yaml`
`run_during`	Phase or phases when expressions are evaluated.	`during`
`exit_on_failure`	Fail the run when a blocking alert fails.	`False`
`only_failures`	Include only failed evaluations in telemetry and reports.	`False`

Alert Profile

Each entry contains a PromQL expression, a description, and an optional severity:

- expr: sum(rate(apiserver_request_total{code=~"5.."}[5m])) > 0
  description: Kubernetes API server returned errors
  severity: critical

- expr: increase(etcd_server_leader_changes_seen_total[2m]) > 0
  description: etcd leader changes observed
  severity: warning

Krkn ships example profiles in the Krkn config directory. Copy and adapt alerts.yaml for the cluster and SLOs being tested.

Evaluation Phases

Set run_during to one phase or a list of phases:

Value	Behavior
`pre`	Evaluate each expression once before chaos starts.
`during`	Evaluate once after chaos and `wait_duration`, using a range covering the chaos and wait window. It is not polled on the regular health-check interval.
`post`	Evaluate each expression once after chaos completes.
`["pre", "during", "post"]`	Evaluate at all three phases.

Use a pre-check to establish a baseline, a post-check to verify recovery, or both:

performance_monitoring:
    enable_alerts: True
    alert_profile: config/alerts.yaml
    run_during: ["pre", "post"]
    exit_on_failure: True

Severity and Failure Behavior

Alert severity controls how a failed expression is reported:

info and warning evaluations are reported but are non-blocking.
error and critical evaluations are blocking when exit_on_failure: True.
With only_failures: True, passing evaluations are omitted from telemetry and reports.

The phase and evaluation result are included in alert telemetry. Alert results are also shown in the generated HTML and PDF reports.

Configuration by Runner

The same alert health-check settings are available through Krkn, Krkn-Hub, and Krknctl:

Configure the performance_monitoring section in config.yaml:

performance_monitoring:
    prometheus_url: "http://prometheus.example.com"
    prometheus_bearer_token: ""
    enable_alerts: True
    alert_profile: config/alerts.yaml
    run_during: ["pre", "during", "post"]
    exit_on_failure: True
    only_failures: False

Set these environment variables before starting the scenario container:

export PROMETHEUS_URL="http://prometheus.example.com"
export PROMETHEUS_TOKEN=""
export ENABLE_ALERTS=True
export ALERTS_PATH=config/alerts.yaml
export ALERTS_RUN_DURING='[pre, during, post]'
export ALERTS_EXIT_ON_FAILURE=True
export ALERTS_ONLY_FAILURES=False

Pass the global alert options to the scenario command:

krknctl run pod-scenarios \
  --prometheus-url http://prometheus.example.com \
  --prometheus-token "" \
  --enable-alerts True \
  --alerts-path config/alerts.yaml \
  --alerts-run-during '["pre", "during", "post"]' \
  --alerts-exit-on-failure True \
  --alerts-only-failures False

The alert profile must be available at the path supplied to the container, or be provided as a URL supported by Krkn.

4 - Kube Virt Checks

Kube Virt Checks to analyze down times of VMIs

Kube Virt Checks

Virt checks provide real-time visibility into the impact of chaos scenarios on VMI ssh connectivity and performance. Virt checks are configured in the config.yaml here

The system periodically checks the VMI’s in the provided namespace based on the defined interval and records the results in Telemetry. The checks will run continuously from the very beginning of krkn until all scenarios are done and wait durations are complete. The telemetry data includes:

Success status True when the VMI is up and running and can form an ssh connection
Failure response False if the VMI experiences downtime or errors.
The VMI Name
The VMI Namespace
The VMI Ip Address and a New IP Address if the VMI is deleted
The time of the start and end of the specific status
The duration the VMI had the specific status
The node the VMI is running on

This helps users quickly identify VMI issues and take necessary actions.

Additional Installation of VirtCtl (If running using Krkn)

It is required to have virtctl or an ssh connection via a bastion host to be able to run this option. We don’t recommend using the krew installation type. This is only required if you are running locally with python Krkn version, the virtctl command will be automatically installed in the krkn-hub and krknctl images

See virtctl installer guide from KubeVirt

VERSION=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
ARCH=$(uname -s | tr A-Z a-z)-$(uname -m | sed 's/x86_64/amd64/') || windows-amd64.exe
echo ${ARCH}
curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/virtctl-${VERSION}-${ARCH}
chmod +x virtctl
sudo install virtctl /usr/local/bin

Sample health check config

kubevirt_checks:                                      # Utilizing virt check endpoints to observe ssh ability to VMI's during chaos injection.
    interval: 2                                       # Interval in seconds to perform virt checks, default value is 2 seconds, required
    namespace: runner                                 # Regex Namespace where to find VMI's, required for checks to be enabled
    name: "^windows-vm-.$"                            # Regex Name style of VMI's to watch, optional, if left blank will find all names in namespace
    only_failures: False                              # Boolean of whether to show all VMI's failures and successful ssh connection (False), or only failure status' (True) 
    disconnected: False                               # Boolean of how to try to connect to the VMIs; if True will use the ip_address to try ssh from within a node, if false will use the name and uses virtctl to try to connect  
    ssh_node: ""                                      # If set, will be a backup way to ssh to a node. Will want to set to a node that isn't targeted in chaos
    node_names: ""                                    # List of node names to further filter down the VM's, will only watch VMs with matching name in the given namespace that are running on node. Can put multiple by separating by a comma
    exit_on_failure:                                        # If value is True and VMI's are failing post chaos returns failure, values can be True/False

Disconnected Environment

The disconnected variable set in the config bypasses the kube-apiserver and SSH’s directly to the worker nodes to test SSH connection to the VM’s IP address.

When using disconnected: true, you must configure SSH authentication to the worker nodes. This requires passing your SSH private key to the container.

Configuration:

disconnected: True                               # Boolean of how to try to connect to the VMIs; if True will use the ip_address to try ssh from within a node, if false will use the name and uses virtctl to try to connect

SSH Key Setup for krkn-hub or krknctl:

You need to mount your SSH private and/or public key into the container to enable SSH connection to the worker nodes. Pass the id_rsa variable with the path to your SSH keys:

# Example with krknctl
krknctl run --config config.yaml -e id_rsa=/path/to/your/id_rsa

# Example with krkn-hub
podman run --name=<container_name> --net=host \
  -v /path/to/your/id_rsa:/home/krkn/.ssh/id_rsa:Z \. # do not change path on right of colon
  -v /path/to/your/id_rsa.pub:/home/krkn/.ssh/id_rsa.pub:Z \. # do not change path on right of colon
  -v /path/to/config.yaml:/root/kraken/config/config.yaml:Z \
  -d quay.io/krkn-chaos/krkn-hub:<scenario_type>

Note: Ensure your SSH key has appropriate permissions (chmod 644 id_rsa) and matches the key authorized on your worker nodes.

Post Virt Checks

After all scenarios have finished executing, krkn will perform a final check on the VMs matching the specified namespace and name. It will attempt to reach each VM and provide a list of any that are still unreachable at the end of the run. The list can be seen in the telemetry details at the end of the run.

Sample virt check telemetry

Notice here that the vm with name windows-vm-1 had a false status (not able to form an ssh connection), for the first 37 seconds (the first item in the list). And at the end of the run the vm was able to for the ssh connection and reports true status for 41 seconds. While the vm with name windows-vm-0 has a true status the whole length of the chaos run (~88 seconds).

"virt_checks": [
      {
          "node_name": "000-000",
          "namespace": "runner",
          "vm_name": "windows-vm-1",
          "ip_address": "0.0.0.0",
          "status": false,
          "start_timestamp": "2025-07-22T13:41:53.461951",
          "end_timestamp": "2025-07-22T13:42:30.696498",
          "duration": 37.234547,
          "new_ip_address": "0.0.0.2",
      },
      {
          "node_name": "000-000",
          "namespace": "runner",
          "vm_name": "windows-vm-0",
          "ip_address": "0.0.0.1",
          "status": true,
          "start_timestamp": "2025-07-22T13:41:49.346861",
          "end_timestamp": "2025-07-22T13:43:17.949613",
          "duration": 88.602752,
          "new_ip_address": ""
      },
      {
          "node_name": "000-000",
          "namespace": "runner",
          "vm_name": "windows-vm-1",
          "ip_address": "0.0.0.2",
          "status": true,
          "start_timestamp": "2025-07-22T13:42:36.260780",
          "end_timestamp": "2025-07-22T13:43:17.949613",
          "duration": 41.688833,
          "new_ip_address": ""
      }
  ],
"post_virt_checks": [
      {
          "node_name": "000-000",
          "namespace": "runner",
          "vm_name": "windows-vm-4",
          "ip_address": "0.0.0.3",
          "status": false,
          "start_timestamp": "2025-07-22T13:43:30.461951",
          "end_timestamp": "2025-07-22T13:43:30.461951",
          "duration": 0.0,
          "new_ip_address": "",
      }
]

Health Check Features

Health Check Types

Advanced Features

Quick Reference

Run Timing Options

Health Check Types in Config

1 - Health Check Run Timing (run_during)

Overview

Configuration

Supported Values

Use Cases

Pre-Check Only: Ensure Clean Baseline

Post-Check Only: Verify Recovery

Pre and Post: Baseline and Recovery

Complete Monitoring: Pre, During, and Post

Exit Codes

Backward Compatibility

2 - Object State Health Checks

Overview

Features

Configuration

Basic Structure

Examples

Check Pod Readiness

Check Deployment Availability

Multiple Checks

Multiple Object Behavior

How It Works

Common Condition Types

Pods

Deployments

StatefulSets / DaemonSets

Pattern Matching

Exact Match

Regex Pattern

Label Selectors

Run Timing Examples

Pre-Check: Gate Chaos on Healthy State

During: Continuous Monitoring

Post-Check: Verify Recovery

Supported Resource Kinds

Namespaced Resources

Cluster-Scoped Resources

See Also

3 - Prometheus Alert Health Checks

Prerequisites

Configuration

Alert Profile

Evaluation Phases

Severity and Failure Behavior

Configuration by Runner

Related Documentation

4 - Kube Virt Checks

Kube Virt Checks

Additional Installation of VirtCtl (If running using Krkn)

Sample health check config

Disconnected Environment

Post Virt Checks

Sample virt check telemetry

See Also