Operator documentation versions

• 1: Krkn Operator
• 1.1: Installation
• 1.2: Administration
• 1.2.1: Cluster Management
• 1.2.2: User Management
• 1.2.3: Registry Management
• 1.2.4: Mirror Signed Images
• 1.2.5: Provider Configuration
• 1.2.6: Elasticsearch Management
• 1.2.7: Cloud Credentials Management
• 1.2.8: Backup and Restore
• 1.3: Usage
• 1.3.1: Jobs
• 1.3.2: Cluster Terminal
• 1.3.3: Run Scenarios
• 1.3.4: Chaos Studio
• 1.3.5: File Management
• 1.3.6: Job Categories & Resiliency Score Report
• 1.4: Releases & Roadmap
• 1.5: Feedback
• 1.6: OCM/ACM Compatibility

1 - Krkn Operator

Krkn Operator is a Kubernetes-native platform to deploy, configure and orchestrate Chaos Engineering experiments through a web console. Instead of manually executing scenarios, the Operator provides a complete platform to manage users, clusters, permissions and experiment execution from a single place.

Multi-Cluster Chaos Testing

Multi-cluster execution is a core capability of the platform. Every chaos scenario can target one or more clusters simultaneously, regardless of how those clusters were registered:

• Manual targets — clusters registered directly by an administrator through the web console
• OCM/ACM discovery — clusters discovered automatically through Open Cluster Management or Red Hat Advanced Cluster Management

Both methods produce the same unified cluster pool. When executing a scenario, users select any combination of available targets and the platform runs the experiment in parallel across all of them. This enables cross-environment resilience testing (dev, staging, production), regional failover validation, and multi-tenant chaos campaigns — all from a single execution.

Personas

The platform defines two roles with different access levels.

Admin

Full access to the platform. Administrators configure the infrastructure that users operate on.

• Register and remove target clusters
• Create groups with cluster access and permissions
• Create users and assign them to groups
• Configure private registries, mirror signed images, and control registry visibility
• Configure cloud credentials and their visibility
• Configure target providers (e.g. ACM integration)
• All operational features available to users

User

Operational access scoped by group membership. Every user must belong to a group — the group determines which clusters, registries, credentials and features are accessible.

• View and manage jobs (based on View/Cancel permissions)
• Use the cluster terminal (requires Run permission)
• Run scenarios on assigned clusters (requires Run permission)
• Design workflows in Chaos Studio (requires Run permission)
• Upload and manage files (scoped by group visibility)
• Select cloud credentials available to their group when running cloud scenarios

Permission Model

        Admin
          │
    Creates Groups
          │
    ┌─────┴──────┐
    │             │
  Group A      Group B
    │             │
  Clusters     Clusters
  Permissions  Permissions
  (View,Run,   (View)
   Cancel)
    │             │
  Users        Users

Permissions are assigned at the group level and determine what users can do on clusters accessible to their group.

View

View

Allows users to see jobs and their execution results. Users with this permission can inspect single run outcomes, console logs, graph run nodes, and Resiliency Scores — but cannot execute, cancel or remove anything.

Applies to: Jobs

Run

Run

Allows users to execute chaos experiments on the clusters assigned to their group. This includes running single scenarios, designing and executing Chaos Studio workflows, and accessing the cluster terminal for read-only inspection.

Applies to: Run Scenarios · Chaos Studio · Cluster Terminal

Cancel

Cancel

Allows users to cancel running scenarios and to remove scenario runs and their execution history. Without this permission, jobs remain visible but cannot be cancelled or removed.

Applies to: Jobs

Key Capabilities

Compatibility

• Kubernetes
• OpenShift
• Open Cluster Management (OCM)
• Red Hat Advanced Cluster Management (ACM)

See the Compatibility Matrix for tested ACM/OCM versions and platform details.

Next Steps

• Installation — Deploy Krkn Operator with Helm
• Administration — Configure clusters, users, registries, credentials and providers
• Usage — Run scenarios, use Chaos Studio, manage jobs
• Compatibility Matrix — Supported platforms and ACM/OCM versions

1.1 - Installation

Installation Admin

Deploy Krkn Operator on Kubernetes or OpenShift using Helm.

Upgrade from 1.0.x to 1.1.0

Upgrade an existing Helm release in place. The 1.1.x chart includes hooks that preserve existing custom resources while the operator updates their Custom Resource Definitions (CRDs) and migrates legacy run fields. Upgrading from 1.0.x to 1.1.x with helm upgrade does not require deleting CRDs or recreating runs, and is designed to preserve existing run data. Back up your configuration before upgrading.

1. Confirm the release name and namespace, and save the current Helm values for reference:

   # Set these to your existing installation.
   RELEASE_NAME=krkn-operator
   RELEASE_NAMESPACE=krkn-operator-system
   helm list --namespace "$RELEASE_NAMESPACE"
   helm get values "$RELEASE_NAME" --namespace "$RELEASE_NAMESPACE" -o yaml > krkn-operator-values-backup.yaml
   

2. Upgrade the existing release to chart version 1.1.0, retaining its configured values:

   helm upgrade "$RELEASE_NAME" oci://quay.io/krkn-chaos/charts/krkn-operator \
     --version 1.1.0 \
     --namespace "$RELEASE_NAMESPACE" \
     --reuse-values \
     --wait \
     --timeout 10m
   

3. Check the release and operator pods:

   helm status "$RELEASE_NAME" --namespace "$RELEASE_NAMESPACE"
   kubectl get pods -n "$RELEASE_NAMESPACE" -l app.kubernetes.io/name=krkn-operator
   

During the upgrade, a pre-upgrade hook saves legacy run fields and pauses run reconciliation. After the new operator starts, a post-upgrade hook updates the CRDs, completes the run migration, and resumes reconciliation. Helm does not upgrade files in a chart’s crds/ directory by itself; the operator’s upgrade hooks handle that step. While the migration guard exists in the release namespace, scenario and graph run creation and mutation requests return HTTP 503. The hook jobs have a six-minute deadline. If a hook fails, the guard remains active: inspect the hook logs, fix the reported cause, then retry the same helm upgrade command. The hooks resume from saved migration annotations. For an individually blocked run, restore its missing current-API scenario or graph identity and, for private scenarios, its registry name before retrying. Do not remove the guard to bypass an incomplete migration.

Prerequisites

• Kubernetes 1.19+ or OpenShift 4.x
• Helm 3.0+

Quick Start

Helm chart version for this release: 1.1.0

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version 1.1.0 \
  --namespace krkn-operator-system \
  --create-namespace

Verify:

kubectl get pods -n krkn-operator-system -l app.kubernetes.io/name=krkn-operator

Access the console (local):

kubectl port-forward -n krkn-operator-system svc/krkn-operator-console 3000:3000

Production Installation

Choose the method that matches your environment:

Install with a custom values.yaml:

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version 1.1.0 \
  --namespace krkn-operator-system \
  --create-namespace \
  -f values.yaml

Kubernetes with Gateway API

console:
  enabled: true
  gateway:
    enabled: true
    gatewayName: krkn-gateway
    hostname: krkn.example.com
    path: /
    pathType: PathPrefix

Kubernetes with Ingress

console:
  enabled: true
  ingress:
    enabled: true
    className: nginx
    hostname: krkn.example.com
    tls:
      - secretName: krkn-tls
        hosts:
          - krkn.example.com

OpenShift with Routes

console:
  enabled: true
  route:
    enabled: true
    hostname: krkn.apps.cluster.example.com
    tls:
      enabled: true
      termination: edge
      insecureEdgeTerminationPolicy: Redirect

OCM/ACM Integration

Enable automatic cluster discovery through Open Cluster Management (OCM) or Red Hat Advanced Cluster Management (ACM):

See the OCM/ACM Compatibility page for tested versions.

helm install krkn-operator oci://quay.io/krkn-chaos/charts/krkn-operator \
  --version 1.1.0 \
  --namespace krkn-operator-system \
  --create-namespace \
  --set acm.enabled=true

Uninstall

helm uninstall krkn-operator --namespace krkn-operator-system

kubectl delete crds -l app.kubernetes.io/name=krkn-operator

Complete values.yaml Reference

# Default values for krkn-operator
# This is a YAML-formatted file.

## Global settings
global:
  # Namespace where to install (if empty, uses Release.Namespace)
  namespaceOverride: ""
## Images configuration
# NOTE: Tags are automatically updated during release by CI/CD workflow
# - Operator & Data Provider: match the release tag (e.g., v0.4.3-beta)
# - ACM: frozen to latest stable ACM tag at release time
# - Console: frozen to latest stable Console tag at release time
# Local development: uses :latest tags for all components
images:
  operator:
    image: krkn-chaos.docker.scarf.sh/krkn-chaos/krkn-operator:v1.1.0-rc7
    pullPolicy: IfNotPresent
  dataProvider:
    image: krkn-chaos.docker.scarf.sh/krkn-chaos/krkn-operator-data-provider:v1.1.0-rc7
    pullPolicy: IfNotPresent
  acm:
    image: krkn-chaos.docker.scarf.sh/krkn-chaos/krkn-operator-acm:v1.1.0-rc3
    pullPolicy: IfNotPresent
  console:
    image: krkn-chaos.docker.scarf.sh/krkn-chaos/krkn-operator-console:v1.1.0-rc8
    pullPolicy: IfNotPresent
  pullSecrets: []
  # - name: my-registry-secret
## Authentication & Security
auth:
  # JWT secret for token signing (base64 encoded)
  # If empty, a random one is generated by Helm
  # To provide your own: echo -n "your-secret-key" | base64
  jwtSecret: ""
  # JWT token expiry in hours (default: 24)
  # Passed to operator as JWT_EXPIRY_HOURS environment variable
  jwtExpiryHours: 24
  # Trusted reverse-proxy / load-balancer networks (comma-separated CIDRs).
  # Passed to the operator as the TRUSTED_PROXY_CIDRS environment variable.
  #
  # The auth endpoints (/api/v1/auth/register, /login, /is-registered) are
  # rate-limited per client IP. When the operator sits behind an ingress,
  # reverse proxy, or load balancer, the direct connection address is the
  # proxy's, so every client would share a single rate-limit bucket. Setting
  # this to the proxy network(s) makes the operator read the real client IP
  # from the X-Forwarded-For / X-Real-IP headers — but ONLY when the request
  # actually arrives from one of these CIDRs.
  #
  # Leave empty (default) when the operator is reached directly: forwarding
  # headers are then ignored and the connection address is used, so clients
  # cannot spoof the rate-limit key. Only set this if you trust the listed
  # networks to set the forwarding headers correctly.
  #
  # Example: "10.0.0.0/8,192.168.0.0/16"
  trustedProxyCIDRs: ""
  # WebSocket Origin enforcement (comma-separated list of allowed origins).
  # Passed to the operator as the WEBSOCKET_ALLOWED_ORIGINS environment variable.
  #
  # There IS an optional Origin-based access control for WebSocket upgrades
  # (cluster terminal, live log streaming), but it is OPT-IN and OFF by default:
  #
  #   - Empty (default): all origins are accepted. This is safe because
  #     WebSocket connections authenticate with a JWT sent in the
  #     Sec-WebSocket-Protocol subprotocol (not browser cookies), so cross-site
  #     WebSocket hijacking (CSWSH) is not exploitable — a malicious page cannot
  #     read the origin-scoped token and therefore cannot connect.
  #   - Non-empty: enables enforcement. Only same-origin requests and the listed
  #     origins may open WebSocket connections. Use this as optional
  #     defense-in-depth when you want to pin the allowed browser origins.
  #
  # Note: enabling enforcement can reject legitimate traffic behind proxies that
  # rewrite the Host header, so only turn it on if you understand your ingress.
  #
  # Example: "https://console.example.com,http://localhost:3000"
  websocketAllowedOrigins: ""
## Core Operator
operator:
  enabled: true
  replicaCount: 1
  olmBootstrap: false
  olmOpenShift: false
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
  # Data provider sidecar resources
  dataProvider:
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 200m
        memory: 256Mi
  service:
    type: ClusterIP
    port: 8080
    grpcPort: 50051
  logging:
    level: info # debug, info, warn, error
    format: json # json or text
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  nodeSelector: {}
  tolerations: []
  affinity: {}
  extraEnv: []
## ACM Integration (optional)
acm:
  enabled: false
  replicaCount: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 512Mi
  service:
    type: ClusterIP
    port: 8080
    grpcPort: 50051
  config:
    # Secret name for ACM managed clusters (must exist - created by OpenShift ACM)
    secretName: "application-manager"
  logging:
    level: info
    format: json
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  nodeSelector: {}
  tolerations: []
  affinity: {}
  extraEnv: []
## Console (web UI)
console:
  enabled: true
  replicaCount: 1
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 256Mi
  service:
    type: ClusterIP
    port: 3000
    # NodePort (only used if type: NodePort)
    nodePort: 30080
  # Ingress configuration (Kubernetes - legacy)
  ingress:
    enabled: false
    className: "nginx"
    hostname: "krkn-operator.example.com"
    hosts: [] # Legacy list of host/path mappings; takes precedence when non-empty.
    tls:
      enabled: false
      secretName: "" # If empty, uses hostname to generate name
    # Annotations (for cert-manager, AWS ALB, etc.)
    annotations: {}
    # cert-manager.io/cluster-issuer: letsencrypt-prod
    # kubernetes.io/ingress.class: alb
  # Gateway API configuration (Kubernetes - recommended)
  # Requires Gateway API CRDs installed in the cluster
  # See: https://gateway-api.sigs.k8s.io/
  gateway:
    enabled: false
    # Name of the existing Gateway resource to attach to
    gatewayName: "krkn-gateway"
    # Namespace of the Gateway (if different from release namespace)
    # Leave empty to use the same namespace
    gatewayNamespace: ""
    # Optional: Listener/Section name on the Gateway to attach to
    # Useful when Gateway has multiple listeners
    sectionName: ""
    # Hostname for the HTTPRoute
    hostname: "krkn-operator.example.com"
    # Path configuration
    path: "/"
    pathType: "PathPrefix" # PathPrefix, Exact, or ImplementationSpecific
    # Annotations for the HTTPRoute resource
    annotations: {}
  # OpenShift Route configuration
  route:
    enabled: false
    # Leave empty to let OpenShift generate the Route hostname. Set this
    # explicitly only when the cluster has a configured DNS name for it.
    hostname: ""
    host: "" # Legacy hostname fallback.
    tls:
      enabled: true
      termination: edge # edge, passthrough, reencrypt
      insecureEdgeTerminationPolicy: Redirect
    annotations: {}
  logging:
    level: info
    format: json
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  nodeSelector: {}
  tolerations: []
  affinity: {}
  extraEnv: []
## RBAC
rbac:
  create: true
  # OLM creates the scenario-runner resources during bootstrap because the
  # ServiceAccount namespace is chosen at installation time.
  scenarioRunner:
    create: true
## ServiceAccount
serviceAccount:
  create: true
  name: ""
  annotations: {}
## CRDs
crds:
  # Keep CRDs on chart uninstall
  keep: true
## Monitoring (optional)
monitoring:
  enabled: false # SAFE DEFAULT for Kind/Minikube
  service:
    port: 8443
    annotations: {}
  serviceMonitor:
    enabled: false
    interval: 30s
    scrapeTimeout: 10s
    labels: {}
## Network Policy (optional)
networkPolicy:
  enabled: false
  ingress: []
  egress: []
## Update strategy
updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
## Pod disruption budget (for HA)
podDisruptionBudget:
  enabled: false
  minAvailable: 1
## Common labels/annotations
commonLabels: {}
commonAnnotations: {}
## Name overrides
nameOverride: ""
fullnameOverride: ""

1.2 - Administration

Administration Admin

Administrators configure the platform infrastructure that users operate on. This includes registering clusters, managing users and groups, setting up private registries, cloud credentials, observability endpoints and target providers.

Permission Model

The Admin creates groups that define what users can do on the platform. Each group specifies:

• Which clusters are accessible
• Which permissions are granted (View, Run, Cancel)
• Which registries and cloud credentials are visible

Users inherit all permissions from their assigned group.

Admin creates Group
      │
      ├── Assigns Clusters (target-1, target-2, ...)
      ├── Assigns Permissions (View, Run, Cancel)
      ├── Assigns Registry Visibility
      └── Assigns Cloud Credential Visibility
            │
            └── Users in this group inherit everything

Admin Features

1.2.1 - Cluster Management

Cluster Management Admin

Register target Kubernetes clusters for chaos scenario execution. Krkn Operator runs on a management cluster and executes scenarios against registered targets; it does not run scenarios against itself.

Operations

Once registered, clusters become available for assignment to groups through User Management.

1.2.2 - User Management

User Management Admin

Organize users through groups that define cluster access and permissions. The group is the central unit of the permission model — users inherit all capabilities from their assigned group.

Groups

A group defines what its members can access and do on the platform.

Groups also receive permission flags that control what operations their members can perform:

Users

Users are created and assigned to a group during creation.

Permission Cascade

Group "SRE Team"
  ├── Clusters: production-us, production-eu
  ├── Permissions: View, Run
  └── Members: alice, bob
        │
        ├── alice → can run scenarios on production-us, production-eu
        └── bob   → can run scenarios on production-us, production-eu

1.2.3 - Registry Management

Registry Management Admin

Configure private container registries for chaos scenario images and control which groups can access them. By default, scenarios are pulled from the public Quay.io registry. Private registries enable air-gapped deployments and custom scenario images.

Creating a Private Registry

Configure a private registry by providing its connection settings (URL, credentials, TLS configuration).

Visibility

Each registry has a visibility setting that controls who can use it:

Impact on Scenario Selection

When a user selects a private registry during scenario execution, only the scenarios that have been mirrored to that registry will be available. Scenarios not present in the private registry will not appear in the selection list.

The Operator requires a valid signature for each selected scenario image by default. For the commands and workload-image override, see Mirror Signed Images.

1.2.4 - Mirror Signed Images

Mirror Signed Images Admin

For a private or disconnected environment, mirror both the scenario image and any helper workload image it launches. Copy their Cosign signatures with the images. A normal docker pull/docker push, podman pull/podman push, or image-only mirror can leave the signatures behind.

The source repository is quay.io/krkn-chaos/krkn-hub-multiarch. Its tags include scenarios such as node-cpu-hog and pod-scenarios, and helper images such as workload-krkn-hog and workload-krkn-tools. Choose the tags required by the scenarios you plan to run.

Mirror directly between registries

On a machine that can reach Quay.io and your private registry, install Cosign and authenticate to the destination registry. The example mirrors node-cpu-hog and its workload-krkn-hog helper into the same repository. Replace the destination with your own registry host and repository.

SOURCE=quay.io/krkn-chaos/krkn-hub-multiarch
MIRROR=registry.example.com/chaos/krkn-hub-multiarch

cosign login registry.example.com
cosign copy "$SOURCE:node-cpu-hog" "$MIRROR:node-cpu-hog"
cosign copy "$SOURCE:workload-krkn-hog" "$MIRROR:workload-krkn-hog"

cosign copy carries the image and its associated signature artifacts. Repeat it for every scenario and workload tag you need. Preserve the tag names so the private registry’s scenario list and the IMAGE override use the same names as the source. Do not use an image-only copy as a substitute.

For a fully disconnected registry, save the signed images on a connected machine, transfer the resulting directories through your approved transfer path, and load them into the destination registry:

# Connected machine
SOURCE=quay.io/krkn-chaos/krkn-hub-multiarch
cosign save --dir ./node-cpu-hog "$SOURCE:node-cpu-hog"
cosign save --dir ./workload-krkn-hog "$SOURCE:workload-krkn-hog"

# After transferring both directories to a machine with access to the private registry
MIRROR=registry.example.com/chaos/krkn-hub-multiarch
cosign login registry.example.com
cosign load --dir ./node-cpu-hog "$MIRROR:node-cpu-hog"
cosign load --dir ./workload-krkn-hog "$MIRROR:workload-krkn-hog"

cosign save and cosign load include the associated signatures; a container image tarball alone does not.

Verify the mirror

On a connected staging machine, obtain the Krkn Cosign public key from the Krknctl v0.14.2-beta revision used by Operator 1.1.0 through a trusted channel, then verify both destination images:

MIRROR=registry.example.com/chaos/krkn-hub-multiarch
curl -fsSLo cosign.pub https://raw.githubusercontent.com/krkn-chaos/krknctl/991cbce5383836e40f2134ef456dd71c0b40ceec/pkg/verify/cosign.pub
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$MIRROR:node-cpu-hog"
cosign verify --key cosign.pub --insecure-ignore-tlog=true "$MIRROR:workload-krkn-hog"

Krkn uses key-based signatures, so the --insecure-ignore-tlog=true option skips the transparency-log check while Cosign still checks the public key and signed image digest. If verification fails, inspect the destination registry’s support for Cosign signature artifacts and copy the signed image again. Do not re-sign a changed image with an unrelated key: the Operator trusts the Krkn key.

Configure the Operator and workload override

1. In Registry Management, create a private registry with Registry URL registry.example.com and Scenario Repository chaos/krkn-hub-multiarch. Configure credentials, TLS, and group visibility as needed.
2. In Run Scenarios, select that registry and choose node-cpu-hog. The Operator resolves and verifies the mirrored scenario image before running it.
3. In the scenario’s parameter form, override IMAGE with registry.example.com/chaos/krkn-hub-multiarch:workload-krkn-hog. This is the helper pod image used by node-cpu-hog; selecting the private scenario registry does not rewrite this parameter automatically.

Other scenarios may launch different helper images. For example, network-chaos uses workload-krkn-tools, and pod-network-filter uses workload-krkn-network-chaos. Mirror the matching workload-* tag with Cosign and set that scenario’s IMAGE parameter to its private URL. Consult the selected scenario’s parameters and the source tag list for the exact image.

If a run does not start

• Scenario missing from the list: confirm its tag exists in the configured scenario repository and the user can access that registry.
• Unsigned or untrusted scenario: confirm the image and signature were copied together, then verify the destination tag with the Krkn public key. A changed image digest invalidates the original signature.
• Workload pod cannot pull its image: check the IMAGE override, the target cluster’s access to the private registry, and its pull credentials and TLS configuration.

1.2.5 - Provider Configuration

Provider Configuration Admin

Configure target providers that integrate with external cluster management platforms. The provider configuration interface adapts dynamically based on the selected provider.

Supported Providers

Cloud Provider Configuration

Krkn Operator supports cloud scenarios that use saved provider credentials. Administrators create and manage these credentials through Cloud Credentials Management. Users can select credentials their group is allowed to access.

The following scenarios interact with provider infrastructure and need the corresponding cloud or bare-metal credentials:

1.2.6 - Elasticsearch Management

Elasticsearch Management Admin

Administrators can configure and save Elasticsearch endpoints that users can reference when executing chaos scenarios. This enables centralized observability configuration without requiring users to manually enter connection details.

Overview

Elasticsearch integration allows krkn scenarios to index execution metrics, logs, and telemetry data for analysis and historical tracking. Instead of requiring each user to configure Elasticsearch details manually, admins can:

• Pre-configure Elasticsearch endpoints with connection details
• Save named configurations for reuse across all scenarios
• Control which Elasticsearch endpoints are available platform-wide

Users can then select from the saved Elasticsearch configurations when setting global parameters during scenario execution.

Adding an Elasticsearch Endpoint

1. Navigate to Administration > Elasticsearch Management
2. Click Add Elasticsearch
3. Configure the following details:

4. Click Save to add the configuration to the saved list

Managing Saved Configurations

View Saved Elasticsearch Endpoints

The Elasticsearch Management page displays all saved configurations:

• Name: Friendly identifier shown to users
• URL: Elasticsearch endpoint
• Index: Target index pattern
• Actions: Edit or Delete options

Edit an Endpoint

1. Click the Edit icon next to the saved configuration
2. Modify any field (Name, URL, Index, credentials, SSL verification)
3. Click Save to update

Delete an Endpoint

1. Click the Delete icon next to the saved configuration
2. Confirm the deletion

How Users Select Elasticsearch

When users configure global parameters for a scenario or workflow, they can:

1. Expand the Global parameters section
2. Locate the Elasticsearch parameter
3. Click the dropdown to view all saved Elasticsearch configurations
4. Select an Elasticsearch endpoint by name (e.g., “Production ES”)

The selected configuration automatically applies the admin-defined connection details (URL, index, credentials) to the scenario execution.

Users cannot:

• Add new Elasticsearch endpoints from the scenario configuration interface
• Edit existing saved configurations
• View sensitive details like passwords (only admins can see/edit credentials)

Users can:

• Choose from the dropdown list of saved Elasticsearch endpoints
• Opt not to use Elasticsearch (by leaving the parameter unset)

Best Practices

Naming Conventions

Use descriptive names that indicate the purpose or environment:

• ✅ Production Observability
• ✅ Staging Metrics
• ✅ Dev Team Shared ES
• ❌ ES1, Elasticsearch, Test

Security

• Use dedicated credentials for krkn with limited write permissions to specific indices
• Enable SSL verification for production endpoints
• Restrict index patterns to prevent accidental data overwrites (e.g., krkn-* not *)

Index Patterns

Use time-based or scenario-based index naming:

• krkn-chaos-{YYYY-MM-DD} — daily indices for easy retention management
• krkn-scenarios-* — scenario-specific index pattern
• krkn-{environment}-* — environment-specific indices

Verification

After adding an Elasticsearch endpoint, verify it works:

1. Run a simple chaos scenario with the Run Scenarios feature
2. Select the saved Elasticsearch configuration in Global parameters
3. Execute the scenario
4. Check the Elasticsearch index for new documents:

   curl -X GET "https://elasticsearch.example.com:9200/krkn-chaos-*/_search?pretty" \
     -u username:password
   

Successful indexing confirms the configuration is correct and accessible from the krkn-operator platform.

1.2.7 - Cloud Credentials Management

Cloud Credentials Management Admin

Administrators can configure and save cloud provider credentials that users select when running cloud-dependent chaos scenarios. Credentials are stored as Kubernetes Secrets and injected into scenario pods via SecretKeyRef — plaintext cloud secrets never appear in the Custom Resource or pod environment values.

Overview

Many krkn scenarios need cloud API access (for example node stop/start, zone outages, or power outages). Instead of asking every user to paste access keys into scenario parameters, admins can:

• Pre-configure named credentials for each supported cloud provider
• Control visibility so credentials are available to everyone or only to selected groups
• Reuse the same credential across single runs and Chaos Studio workflows

Users then select a saved credential when configuring a scenario. Matching cloud parameter fields are filled from the Secret and locked in the UI.

Supported Providers

When a credential is applied, the platform also sets CLOUD_TYPE to the value expected by krkn-hub for that provider:

Adding a Cloud Credential

1. Navigate to Settings > Cloud Credentials
2. Click Create Cloud Credential (or the add action)
3. Configure the common fields, then the provider-specific fields below
4. Click Save to create the credential

Common fields

Provider-specific fields

AWS

GCP

GCP credentials are mounted into the scenario pod as a file. The operator also sets GOOGLE_APPLICATION_CREDENTIALS to that mount path.

Azure

OpenStack

Baremetal (IPMI/BMC)

VMware

IBM Cloud

Visibility

Each credential has a visibility setting that controls who can use it:

Managing Saved Credentials

View Credentials

The Cloud Credentials page lists saved configurations with:

• Name: Identifier shown in scenario selection
• Provider: Cloud provider type
• Access: Everyone or assigned groups
• Actions: Edit or Delete

Admins can filter the list by name, provider, and access type.

Edit a Credential

1. Click Edit next to the credential
2. Update description, visibility, groups, or provider secret fields
3. Click Save

The provider cannot be changed after creation — create a new credential if you need a different provider. Secret values are never shown after creation. Leave a secret field empty to keep the existing value; provide a new value only when rotating credentials.

Delete a Credential

1. Click Delete next to the credential
2. Confirm the deletion

How Credentials Are Injected

When a run references a cloud credential:

1. The console (and API) store only the credential name on the run (cloudCredentialRef)
2. Cloud-related environment variables from the request are stripped so plaintext keys cannot land in the Custom Resource
3. The operator loads the credential Secret and injects values into the scenario pod using SecretKeyRef (and file mounts where required, such as GCP service-account JSON)

Users cannot:

• Create or edit credentials from the scenario wizard
• View secret values for saved credentials
• Select credentials outside their visibility (group membership)

Users can:

• Choose from the credentials available to them
• Clear the selection and enter parameters manually when no credential is needed

How Users Select Credentials

When configuring parameters for a single scenario or a Chaos Studio node:

1. Open the Load Cloud Credential section (shown when the scenario has cloud-related fields)
2. Select a saved credential from the dropdown (only credentials your group can access)
3. Matching cloud fields become read-only and show masked placeholders (••••••••)
4. Fields for other cloud providers are hidden so the form only shows the applied provider
5. Continue configuring non-cloud parameters and run the scenario

In Chaos Studio, set the credential per node when configuring that node’s scenario. Each cloud-dependent node can use a different credential.

Best Practices

Naming

Use names that encode provider and environment:

• ✅ aws-prod-ci
• ✅ azure-staging-sre
• ✅ bmc-lab-rack-a
• ❌ creds, test, key1

Access control

• Prefer group-based visibility for production credentials
• Use Everyone only for shared non-production accounts
• Rotate keys by editing the credential rather than creating duplicates with the same name

Security

• Never paste long-lived cloud secrets into scenario parameter fields when a saved credential exists
• Treat credential Secrets like any other cluster secret: limit RBAC outside the operator namespace
• Review group membership before granting access to production cloud accounts

Verification

After adding a credential, verify injection works:

1. Run a cloud-dependent scenario with Run Scenarios
2. Select the saved credential under Load Cloud Credential
3. Confirm cloud fields are disabled and show masked placeholders
4. Execute the run and confirm the job starts successfully
5. Optionally inspect the scenario pod: cloud env vars should reference the credential Secret by name, not plaintext values

1.2.8 - Backup and Restore

Backup and Restore Admin

The Krkn Operator includes a built-in backup and restore feature accessible from the web console. Backups contain portable operator configuration for:

• Admin settings
• Cluster targets
• Users and groups
• Private registries
• Elasticsearch configuration

The backup also includes the operator-managed secrets required for these settings.

This feature is not a complete Kubernetes or cluster backup. It does not include runtime registration objects, ACM integration data, uploaded files, file metadata, or the JWT signing key.

Creating a Backup

1. Navigate to the Backup & Restore card in the console.
2. Click Download Backup.
3. The operator creates a compressed .tar.gz archive and saves it to your local machine.

Restoring from a Backup

1. Open the Backup & Restore card in the console.
2. Click Upload & Restore.
3. Select a previously downloaded .tar.gz backup archive and confirm the restore.
4. Monitor the restore progress from the same page.

Only one restore can run at a time. The maximum upload size is 100 MB.

The restore runs asynchronously. Existing resources are updated with the corresponding configuration from the backup, while resources that do not exist are created. Resources absent from the backup are not deleted.

After the restore completes, the operator refreshes the connectivity status of restored target clusters.

Restore Status

Restore status is available from the same page and can be:

• In progress
• Completed
• Failed

Restore job status is held in memory and is not persisted across operator restarts.

1.3 - Usage

Usage

The operational features of Krkn Operator are available to all users, scoped by the permissions assigned to their group. This section covers the day-to-day workflow: running scenarios, designing workflows, monitoring jobs and managing files.

Permission Summary

Features

• Jobs — Monitor and inspect scenario execution results
• Cluster Terminal — Explore target clusters with read-only commands
• Run Scenarios — Execute single chaos scenarios (including selecting saved cloud credentials)
• Chaos Studio — Design visual workflows with serial and parallel execution
• File Management — Upload and manage configuration files
• Job Categories & Resiliency Score Report — Group scenario runs and compare resiliency scores

Cloud credentials themselves are configured by administrators — see Cloud Credentials Management.

1.3.1 - Jobs

Jobs View Cancel

The Jobs page is the platform home screen. It lists scenario executions available to your group, with status updates and access to logs and results.

Run Types

Single Run

A single scenario execution on one or more target clusters. For each run you can inspect:

• Execution outcome (success, failure)
• Console logs of the krkn pod that executed the scenario

Replay & Edit

Single runs can be replayed to re-execute the same scenario with the same configuration. You can also edit the details before replaying to modify:

• Target cluster(s)
• Scenario parameters
• Configuration values
• Run Name

This allows you to quickly iterate on chaos experiments or run the same test across different environments.

Graph Run (Chaos Studio)

A workflow execution created through Chaos Studio. The execution is displayed as a graph where each node represents an individual krkn scenario. For each node you can inspect the outcome and logs independently.

Graph runs can include a Resiliency Score — a calculated metric based on PromQL queries executed during the workflow. This score measures how well the target application withstood the chaos experiment.

Visibility & Permissions

• Only jobs from your own group are displayed
• A cluster-wide tip shows the names of scenarios currently running across all groups (no details)
• Cancelling or removing jobs requires Cancel permission

1.3.2 - Cluster Terminal

Cluster Terminal Run

A built-in terminal for inspecting target clusters with read-only oc and kubectl commands before, during, or after chaos experiments.

Details

• Read-only: only read commands are available — no modifications to the cluster are possible
• Supported commands: oc and kubectl
• Target selection: only clusters accessible to your group are available
• Requires Run permission

1.3.3 - Run Scenarios

Run Scenarios Run

Use the guided wizard to execute a chaos scenario on one or more target clusters in parallel.

Execution Flow

1. Select Target Clusters

Choose one or more clusters to run the scenario on. All selected clusters execute the scenario simultaneously.

2. Select Registry

Choose the container registry for scenario images:

For a disconnected target, follow Mirror Signed Images before selecting a private registry. Scenario signatures and any helper workload images must be available in the mirror.

3. Select Scenario

Browse and select a chaos scenario from the chosen registry.

4. Configure Parameters

Each scenario defines its own parameter set, divided into three categories:

When a scenario starts a helper workload, set its IMAGE parameter to the mirrored workload-* image URL. Selecting a private scenario registry does not change the workload image automatically. See Mirror Signed Images for an example.

Elasticsearch in Global Parameters

When configuring Global parameters, you can select a saved Elasticsearch endpoint to index scenario metrics and telemetry:

1. Expand the Global parameters section
2. Locate the Elasticsearch dropdown
3. Select from available saved configurations (configured by your admin)

Selecting an Elasticsearch configuration automatically applies connection details (URL, index, credentials) without requiring manual input.

Load Cloud Credential

When a scenario declares cloud-related fields (CLOUD_TYPE, AWS_*, AZURE_*, and similar), a Load Cloud Credential section appears:

1. Open Load Cloud Credential
2. Select a saved credential from the dropdown (configured by your admin)
3. Matching cloud fields become read-only and show masked placeholders (••••••••)
4. Fields belonging to other cloud providers are hidden from the form
5. Continue configuring non-cloud parameters as usual

The dropdown only lists credentials your group can access.

Selecting a credential stores only the credential name on the run (cloudCredentialRef). The console and API strip plaintext cloud environment variables from the payload so they cannot land in the Custom Resource. The operator injects secret values into the scenario pod via SecretKeyRef.

5. Mount Files

Optionally attach configuration files previously uploaded through File Management. Files are available via a select dropdown.

6. Preview

Review a summary of all configured parameters before execution.

7. Run

Launch the scenario. You are redirected to the Jobs page where you can monitor the execution in real time.

1.3.4 - Chaos Studio

Chaos Studio Run

A visual drag-and-drop editor for designing complex chaos workflows. Build graphs of scenarios that execute in series or parallel, and measure application resilience with the Resiliency Score.

Creating and Saving Workflows

Creating a New Workflow

1. Navigate to Chaos Studio from the main navigation
2. Click the “New Workflow” button to start a blank canvas
3. Give your workflow a meaningful name that describes its purpose

Adding Nodes

1. Click the “Add Node” button or drag from the node palette
2. Position nodes on the canvas by dragging them
3. Configure each node by clicking on it and selecting:
   • Registry: Choose your scenario source
   • Scenario: Select the chaos scenario to run
   • Parameters: Set mandatory and optional parameters
   • Cloud credential: Optionally load a saved cloud credential for cloud-dependent scenarios
   • Files: Mount any required configuration files

Connecting Nodes

• Click and drag from a node’s output port to another node’s input port to create dependencies
• Serial execution: connect nodes in a chain (A → B → C)
• Parallel execution: connect one node to multiple targets (A → B and A → C)

Saving Your Workflow

1. Click the “Save” button in the top toolbar
2. The workflow is automatically saved to your workspace
3. Saved workflows appear in the Chaos Studio workflow list
4. You can edit saved workflows by selecting them from the list

Visual Editor

• Nodes represent individual krkn scenarios
• Arrows define execution order — connect nodes to create dependencies
• Serial execution: chain nodes sequentially with single arrows
• Parallel execution: branch from one node to multiple targets
• Drag and drop to arrange the workflow visually

Node Configuration

Each node is configured exactly like a single scenario: select a registry, choose a scenario, set mandatory/optional/global parameters, optionally load a cloud credential, and mount files.

Global Parameters in Workflows

When configuring global parameters for workflow nodes:

• Elasticsearch: Select from saved admin-configured endpoints to index metrics across the entire workflow
• Prometheus: Configure Prometheus integration for metric collection
• Cerberus: Enable cluster health monitoring during scenario execution

Cloud Credentials in Workflows

Cloud-dependent nodes can use saved credentials configured under Cloud Credentials Management.

When configuring a node:

1. Open Load Cloud Credential
2. Select a credential available to your group
3. Cloud fields for that provider become masked; other providers’ fields are hidden

Each node can select its own credential (for example one AWS credential on a zone-outage node and a BMC credential on a power-outage node).

Clone: duplicate a configured node to repeat the same scenario multiple times within the workflow.

Validation

The editor validates the workflow before execution:

• Configured nodes: green arrows and connections — ready to execute
• Unconfigured nodes: red arrows and red node border — must be configured before the workflow can run

Only fully configured nodes can be connected and executed.

Resiliency Score

Enable the Resiliency Score on a workflow to measure application resilience:

1. Enable the resiliency score option on the workflow
2. Mount a PromQL file — a special file containing PromQL queries uploaded via File Management
3. Queries execute during the workflow and their results contribute to the final score
4. View the score in the Jobs page alongside the graph run results

The PromQL file defines which metrics to evaluate (latency, error rate, throughput, etc.) and their expected thresholds. The score reflects how closely the application maintained its expected behavior under chaos.

1.3.5 - File Management

File Management

Create and manage text-based configuration files for scenario executions and Chaos Studio workflows.

Adding Files

Create a file by pasting its contents into the text input box. The console does not upload files from your device.

Where Files Are Used

1.3.6 - Job Categories & Resiliency Score Report

Job Categories & Resiliency Score Report

Use categories to organize scenario runs, then compare their resiliency scores over time in the report.

Create a Category

1. Open Categories from the left navigation menu.
2. Select Create Category.
3. Enter a name and choose a color in the color picker.
4. Set the category visibility and create it.

Categories help you group related runs for reporting. Choose a color to identify the category in the console.

Assign a Category to Runs

1. Open Jobs and locate a scenario run.
2. Open the run’s Actions menu and select Category.
3. Select one or more categories to assign them to the run.
4. Repeat for other runs you want to compare.

The category is saved on the run and can be used as a filter in the report.

For example, assign score-category to pod-scenarios-99629e65 and pod-scenarios-aedd3317 to compare their resiliency scores together.

View the Resiliency Score Report

1. Open Resiliency History from the left navigation menu.
2. Select one or more categories and target clusters.
3. Select Apply filters to display the report.

The report plots resiliency scores over time. In Separate configuration groups mode, runs with the same settings appear in the same chart; runs with different settings appear in separate charts. This makes it easier to compare like-for-like runs while keeping different configurations distinct.

Combine Configurations by Category

Select Combine configurations by category to put runs from the selected category into a single chart, including runs with different settings. This helps compare scores across heterogeneous configurations.

When the report includes more than one cluster, each cluster is shown as a separate line so you can compare results across clusters. The report can also be exported as a PDF.

1.4 - Releases & Roadmap

This document outlines the Krkn Operator release strategy, roadmap, and how the community prioritizes features, bug fixes, and security updates.

Roadmap

Release Cadence

Krkn Operator aims to deliver feature releases every 6–8 weeks. These releases may include new features, enhancements, usability improvements, and other roadmap items driven by community feedback and contributor activity.

The release cadence is intended to balance rapid delivery of value to users while maintaining stability and quality across supported environments.

Feature Requests and Roadmap

Community feedback plays a key role in shaping the future of Krkn Operator.

Feature requests submitted through GitHub issues, enhancement proposals, community discussions, and feedback forms are reviewed regularly and considered as part of the project roadmap.

Features are prioritized based on:

• Community interest
• User impact
• Project goals
• Contributor capacity

Planned enhancements and major initiatives will be communicated through project documentation, GitHub, community meetings, and release notes.

Bug Fixes and Security Updates

Bug fixes and security updates are prioritized independently of the feature release cadence.

Critical security vulnerabilities and high-impact issues may be addressed through patch or minor releases outside the normal release cycle when necessary.

Community contributions for bug fixes, testing, and validation are always welcome.

Release Notes

Each release includes documentation for:

• New features and enhancements
• Bug fixes
• Security updates
• Compatibility updates
• Breaking changes and deprecations
• Upgrade or migration guidance, when applicable

Any compatibility changes, breaking changes, or upgrade considerations will be documented as part of the release notes to help users plan upgrades and understand changes between releases.

Getting Involved

The Krkn community welcomes feedback, feature requests, bug reports, testing, and code contributions. Community participation helps shape the roadmap and improve the overall user experience.

For feature requests or bug reports, please open an issue in the Krkn-operator GitHub repository or participate in community discussions.

1.5 - Feedback

Feedback options for this archived release are available through the current Operator feedback page or the Krkn Operator issue tracker.

1.6 - OCM/ACM Compatibility

OCM/ACM Compatibility

Krkn Operator supports multi-cluster chaos testing on Kubernetes and OpenShift. Cluster discovery can be manual or automated through ACM/OCM integration.

Platform Support

ACM Compatibility

The following versions of Red Hat Advanced Cluster Management have been tested and validated with Krkn Operator:

OCM Compatibility

The following versions of Open Cluster Management have been tested and validated:

How ACM Integration Works

When ACM is enabled (--set acm.enabled=true at install time), Krkn Operator deploys a dedicated component that connects to the ACM hub and automatically discovers managed clusters. Discovered clusters appear in the same cluster pool as manually registered targets — users select from both without distinction.

See Provider Configuration for setup details and Installation for the Helm flag.

Reference

• Open Cluster Management (OCM)
• OCM Releases
• Red Hat ACM Documentation