Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Vigil is a single-binary Kubernetes controller built with controller-runtime. It runs one instance per cluster with leader election for high availability.

Components

Component	Purpose
Node Informer	Watches nodes for startup taint changes
Pod Informer	Watches pods for DaemonSet readiness transitions
DaemonSet Cache	Caches DaemonSet list for scheduling rule evaluation
Node Reconciler	Core logic: discover expected DaemonSets, check readiness, remove taint

Data Flow

A new node appears with the configured startup taint
The Node Informer triggers a reconciliation
The reconciler evaluates all DaemonSets against the node’s labels, affinity, and taints
For each expected DaemonSet, the reconciler checks if a Ready pod exists on the node
If all expected DaemonSet pods are Ready, the taint is removed
If not, the node is requeued for re-evaluation

Safe Taint Removal

Vigil uses fresh API server reads (not informer cache) when removing taints, with optimistic concurrency via resourceVersion. This avoids the stale-cache bug that affected Istio’s untaint controller.

DaemonSet Discovery

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Vigil auto-discovers which DaemonSets should run on each tainted node by replicating the kube-scheduler’s DaemonSet scheduling logic using the upstream k8s.io/component-helpers package.

Algorithm

For each DaemonSet in the cluster:

Build a synthetic Pod from the DaemonSet’s pod template
Evaluate nodeSelector and nodeAffinity against the node
Compute the node’s “steady-state” taints by stripping all configured startup taint keys
Check that the DaemonSet tolerates all remaining (steady-state) taints
If all checks pass, the DaemonSet is expected on this node

Why Strip Startup Taints

A brand-new node has multiple startup taints (Vigil’s, Istio CNI’s, EBS CSI’s, EFS CSI’s). These are all temporary. We want to answer “will this DaemonSet run in steady state?” not “can this DaemonSet tolerate the node right now?”

Concepts on Vigil Controller

Architecture

Overview

Components

Data Flow

Safe Taint Removal

DaemonSet Discovery

Overview

Algorithm

Why Strip Startup Taints