Reference on Vigil Controller

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Configuration File

Vigil reads configuration from a YAML file (default: /etc/vigil/config/config.yaml).

# The taint key to watch for and remove
taintKey: "node.example.com/initializing"

# The taint effect to match
taintEffect: "NoSchedule"

# All known temporary startup taint keys in the cluster.
# Used for DaemonSet discovery only — Vigil does not remove these taints.
# Their respective controllers (CSI drivers, CNI plugins) handle removal.
knownStartupTaintKeys:
 - "node.example.com/initializing"
 - "cni.istio.io/not-ready"
 - "ebs.csi.aws.com/agent-not-ready"
 - "efs.csi.aws.com/agent-not-ready"

# Maximum time to wait before removing taint anyway (seconds)
timeoutSeconds: 120

# DaemonSets to exclude from readiness checks
excludeDaemonSets:
 byName:
 - namespace: kube-system
 name: slow-daemonset

CLI Flags

Flag	Default	Description
`--config`	`/etc/vigil/config/config.yaml`	Config file path
`--metrics-bind-address`	`:8080`	Metrics endpoint
`--health-probe-bind-address`	`:8081`	Health/readiness probes
`--leader-elect`	`false`	Enable leader election
`--zap-log-level`	(default)	Log verbosity

Configuration Fields

Field	Type	Default	Description
`taintKey`	string	`node.nextdoor.com/initializing`	The taint key to watch and remove
`taintEffect`	string	`NoSchedule`	The taint effect to match
`knownStartupTaintKeys`	[]string	`[taintKey]`	All temporary startup taint keys in the cluster (used for discovery only — Vigil does not remove these)
`timeoutSeconds`	int	`120`	Max wait time before forced taint removal
`excludeDaemonSets.byName`	[]object	`[]`	DaemonSets to exclude by namespace/name

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Metrics

Metric	Type	Description
`vigil_tainted_nodes`	Gauge	Nodes currently waiting for DaemonSet readiness
`vigil_taint_removal_duration_seconds`	Histogram	Time from node creation to taint removal
`vigil_successful_removals_total`	Counter	Taint removals after all DaemonSets Ready
`vigil_timeout_removals_total`	Counter	Taint removals due to timeout
`vigil_expected_daemonsets`	Gauge (by node)	Expected DaemonSets per node
`vigil_ready_daemonsets`	Gauge (by node)	Ready DaemonSet pods per node
`vigil_reconcile_errors_total`	Counter	Reconciliation errors
`vigil_discovery_duration_seconds`	Histogram	Time to evaluate scheduling rules
`vigil_timeout_blocking_daemonset_total`	Counter (by ds)	Which DaemonSet blocked at timeout

Alerting

Recommended alert rules:

Helm Chart

Mon, 01 Jan 0001 00:00:00 +0000

Installation

helm repo add vigil https://oss.nextdoor.com/vigil
helm repo update
helm install vigil vigil/vigil-controller \
 --namespace vigil-system \
 --create-namespace

Key Values

Value	Default	Description
`replicaCount`	`2`	Number of controller replicas
`image.repository`	`ghcr.io/nextdoor/vigil`	Container image
`image.tag`	Chart appVersion	Image tag
`controllerManager.leaderElection.enabled`	`true`	Leader election
`controllerManager.logLevel`	`info`	Log level
`config.taintKey`	`node.nextdoor.com/initializing`	Taint key to watch
`config.taintEffect`	`NoSchedule`	Taint effect
`config.timeoutSeconds`	`120`	Timeout before forced removal
`config.knownStartupTaintKeys`	See values.yaml	Temporary startup taints to ignore during discovery
`serviceMonitor.enabled`	`false`	Create ServiceMonitor
`resources.requests.cpu`	`100m`	CPU request
`resources.requests.memory`	`64Mi`	Memory request
`resources.limits.cpu`	`500m`	CPU limit
`resources.limits.memory`	`128Mi`	Memory limit

See the full values.yaml for all options.

Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Latest Benchmark Results

Run date: 2026-03-28T21:50:52Z Git SHA: 1577d67 Node count: 10,000

Latency

The stress test measures three distinct latency phases:

Pod startup — time from node creation until all expected DaemonSet pods become Ready. This is dominated by simulated pod startup delays and reflects real-world DaemonSet boot time. Excludes never-ready nodes.
Vigil reaction — time from all pods becoming Ready until the taint is removed. This isolates Vigil’s controller overhead: reconcile loop detection, readiness verification, and the taint removal API call.
End-to-end — total time from node creation to taint removal. For never-ready nodes, this equals the controller timeout.

Pod Startup Latency

Percentile	Latency
p50	3,761ms
p95	38,220ms
p99	51,901ms

Vigil Reaction Time

Percentile	Latency
p50	1,000ms
p95	1,901ms
p99	1,981ms

End-to-End Latency

Percentile	Latency
p50	5,023ms
p95	30,206ms
p99	30,416ms

Outcomes

Outcome	Count	Percentage
Success	9,500	95.0%
Timeout	500	5.0%
Pending	0	0.0%

Resource Utilization

Metric	Value
Peak Heap	275.8 MB
Final Heap	224.1 MB
System Memory	315.2 MB
GC Cycles	229
GC CPU Fraction	0.03%

Resource Utilization Over Time

Heap Memory

xychart-beta
 title "Heap Allocation Over Time"
 x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
 y-axis "Heap (MB)" 0 --> 280
 line [8, 11, 24, 31, 47, 41, 60, 76, 86, 73, 86, 114, 117, 88, 99, 97, 157, 128, 163, 187, 194, 190, 165, 227, 174, 223, 140, 165, 186, 214]

Goroutines

xychart-beta
 title "Active Goroutines Over Time"
 x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
 y-axis "Goroutines" 0 --> 250
 line [140, 185, 197, 196, 195, 194, 192, 199, 199, 199, 190, 192, 192, 194, 196, 194, 195, 199, 199, 192, 195, 199, 194, 204, 194, 194, 192, 190, 195, 133]

Memory grows linearly with node count (10 nodes/sec) as the informer cache accumulates Node and Pod objects. The sawtooth pattern in heap allocation shows the Go garbage collector reclaiming memory between allocation bursts. Goroutine count remains stable (~190-200), confirming no goroutine leaks.