Benchmarks
Performance benchmarks from stress testing
Latest Benchmark Results
Run date: 2026-03-28T21:50:52Z
Git SHA: 1577d67
Node count: 10,000
Latency
The stress test measures three distinct latency phases:
- Pod startup — time from node creation until all expected DaemonSet pods
become Ready. This is dominated by simulated pod startup delays and reflects
real-world DaemonSet boot time. Excludes never-ready nodes.
- Vigil reaction — time from all pods becoming Ready until the taint is
removed. This isolates Vigil’s controller overhead: reconcile loop detection,
readiness verification, and the taint removal API call.
- End-to-end — total time from node creation to taint removal. For
never-ready nodes, this equals the controller timeout.
Pod Startup Latency
| Percentile | Latency |
|---|
| p50 | 3,761ms |
| p95 | 38,220ms |
| p99 | 51,901ms |
Vigil Reaction Time
| Percentile | Latency |
|---|
| p50 | 1,000ms |
| p95 | 1,901ms |
| p99 | 1,981ms |
End-to-End Latency
| Percentile | Latency |
|---|
| p50 | 5,023ms |
| p95 | 30,206ms |
| p99 | 30,416ms |
Outcomes
| Outcome | Count | Percentage |
|---|
| Success | 9,500 | 95.0% |
| Timeout | 500 | 5.0% |
| Pending | 0 | 0.0% |
Resource Utilization
| Metric | Value |
|---|
| Peak Heap | 275.8 MB |
| Final Heap | 224.1 MB |
| System Memory | 315.2 MB |
| GC Cycles | 229 |
| GC CPU Fraction | 0.03% |
Resource Utilization Over Time
Heap Memory
xychart-beta
title "Heap Allocation Over Time"
x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
y-axis "Heap (MB)" 0 --> 280
line [8, 11, 24, 31, 47, 41, 60, 76, 86, 73, 86, 114, 117, 88, 99, 97, 157, 128, 163, 187, 194, 190, 165, 227, 174, 223, 140, 165, 186, 214]Goroutines
xychart-beta
title "Active Goroutines Over Time"
x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
y-axis "Goroutines" 0 --> 250
line [140, 185, 197, 196, 195, 194, 192, 199, 199, 199, 190, 192, 192, 194, 196, 194, 195, 199, 199, 192, 195, 199, 194, 204, 194, 194, 192, 190, 195, 133]Memory grows linearly with node count (10 nodes/sec) as the informer cache
accumulates Node and Pod objects. The sawtooth pattern in heap allocation shows
the Go garbage collector reclaiming memory between allocation bursts. Goroutine
count remains stable (~190-200), confirming no goroutine leaks.
Raw data (210 samples)
| Elapsed (s) | Heap (MB) | System (MB) | Goroutines | GC Cycles |
|---|
| 5.0 | 7.9 | 23.1 | 140 | 11 |
| 10.0 | 7.8 | 27.9 | 152 | 20 |
| 15.0 | 6.6 | 27.9 | 164 | 28 |
| 20.0 | 8.7 | 32.1 | 176 | 35 |
| 25.0 | 10.0 | 32.4 | 184 | 42 |
| 30.0 | 9.8 | 32.4 | 184 | 48 |
| 35.0 | 13.2 | 36.4 | 184 | 53 |
| 40.0 | 10.9 | 36.4 | 185 | 58 |
| 45.0 | 15.4 | 36.4 | 191 | 62 |
| 50.0 | 17.0 | 40.4 | 198 | 66 |
| 55.0 | 17.6 | 40.4 | 205 | 70 |
| 60.0 | 20.4 | 44.6 | 201 | 74 |
| 65.0 | 16.1 | 44.6 | 196 | 79 |
| 70.0 | 20.1 | 44.6 | 192 | 82 |
| 75.0 | 23.5 | 44.7 | 197 | 85 |
| 80.0 | 17.7 | 48.7 | 196 | 89 |
| 85.0 | 22.8 | 48.8 | 193 | 92 |
| 90.0 | 27.7 | 48.8 | 191 | 95 |
| 95.0 | 23.9 | 53.1 | 195 | 98 |
| 100.0 | 31.4 | 53.1 | 199 | 100 |
| 105.0 | 24.2 | 57.1 | 198 | 103 |
| 110.0 | 30.6 | 57.1 | 196 | 105 |
| 115.0 | 36.6 | 61.1 | 194 | 107 |
| 120.0 | 40.5 | 61.1 | 193 | 109 |
| 125.0 | 25.2 | 65.1 | 199 | 112 |
| 130.0 | 29.7 | 65.1 | 198 | 114 |
| 135.0 | 30.7 | 65.1 | 196 | 116 |
| 140.0 | 30.0 | 69.1 | 191 | 118 |
| 145.0 | 47.1 | 69.1 | 195 | 119 |
| 150.0 | 40.5 | 69.1 | 197 | 121 |
| 155.0 | 34.2 | 73.3 | 198 | 123 |
| 160.0 | 51.2 | 73.3 | 194 | 124 |
| 165.0 | 42.0 | 73.3 | 198 | 126 |
| 170.0 | 34.4 | 77.4 | 195 | 128 |
| 175.0 | 50.6 | 77.4 | 193 | 129 |
| 180.0 | 40.7 | 77.5 | 194 | 131 |
| 185.0 | 33.6 | 81.5 | 194 | 133 |
| 190.0 | 47.2 | 81.5 | 198 | 134 |
| 195.0 | 39.1 | 81.6 | 201 | 136 |
| 200.0 | 56.4 | 81.6 | 192 | 137 |
| 205.0 | 37.5 | 85.6 | 196 | 139 |
| 210.0 | 47.7 | 85.6 | 197 | 140 |
| 215.0 | 59.5 | 85.9 | 192 | 141 |
| 220.0 | 67.6 | 89.9 | 194 | 142 |
| 225.0 | 47.2 | 89.9 | 193 | 144 |
| 230.0 | 53.9 | 94.1 | 194 | 145 |
| 235.0 | 60.8 | 94.1 | 197 | 146 |
| 240.0 | 67.3 | 98.1 | 194 | 147 |
| 245.0 | 72.1 | 98.1 | 196 | 148 |
| 250.0 | 75.7 | 98.4 | 199 | 149 |
| 255.0 | 80.4 | 102.4 | 200 | 150 |
| 260.0 | 46.4 | 106.4 | 192 | 152 |
| 265.0 | 49.4 | 106.4 | 190 | 153 |
| 270.0 | 47.5 | 110.4 | 200 | 154 |
| 275.0 | 49.4 | 110.5 | 195 | 155 |
| 280.0 | 88.0 | 110.6 | 196 | 155 |
| 285.0 | 86.0 | 110.6 | 199 | 156 |
| 290.0 | 85.1 | 110.6 | 195 | 157 |
| 295.0 | 82.6 | 114.6 | 198 | 158 |
| 300.0 | 81.1 | 114.6 | 198 | 159 |
| 305.0 | 80.0 | 114.6 | 200 | 160 |
| 310.0 | 79.1 | 114.6 | 197 | 161 |
| 315.0 | 77.4 | 114.8 | 197 | 162 |
| 320.0 | 73.5 | 118.8 | 199 | 163 |
| 325.0 | 70.2 | 122.8 | 198 | 164 |
| 330.0 | 67.0 | 122.8 | 191 | 165 |
| 335.0 | 62.9 | 122.8 | 191 | 166 |
| 340.0 | 57.0 | 122.8 | 191 | 167 |
| 345.0 | 96.8 | 122.8 | 192 | 167 |
| 350.0 | 91.2 | 126.8 | 189 | 168 |
| 355.0 | 86.0 | 130.8 | 190 | 169 |
| 360.0 | 74.8 | 130.8 | 195 | 170 |
| 365.0 | 68.6 | 134.8 | 189 | 171 |
| 370.0 | 107.3 | 134.8 | 194 | 171 |
| 375.0 | 97.3 | 135.1 | 194 | 172 |
| 380.0 | 85.6 | 139.2 | 199 | 173 |
| 385.0 | 74.0 | 143.5 | 196 | 174 |
| 390.0 | 114.4 | 143.6 | 192 | 174 |
| 395.0 | 97.0 | 147.6 | 197 | 175 |
| 400.0 | 83.0 | 147.6 | 197 | 176 |
| 405.0 | 69.3 | 147.6 | 195 | 177 |
| 410.0 | 107.4 | 147.6 | 199 | 177 |
| 415.0 | 92.3 | 147.6 | 199 | 178 |
| 420.0 | 75.7 | 151.6 | 197 | 179 |
| 425.0 | 116.8 | 151.6 | 192 | 179 |
| 430.0 | 97.4 | 151.6 | 196 | 180 |
| 435.0 | 78.6 | 159.6 | 188 | 181 |
| 440.0 | 117.6 | 159.6 | 190 | 181 |
| 445.0 | 95.2 | 159.6 | 194 | 182 |
| 450.0 | 134.9 | 159.6 | 193 | 182 |
| 455.0 | 113.5 | 163.6 | 189 | 183 |
| 460.0 | 88.0 | 163.6 | 194 | 184 |
| 465.0 | 128.2 | 163.6 | 193 | 184 |
| 470.0 | 102.5 | 163.6 | 195 | 185 |
| 475.0 | 142.3 | 163.6 | 198 | 185 |
| 480.0 | 117.1 | 167.7 | 191 | 186 |
| 485.0 | 89.1 | 171.8 | 193 | 187 |
| 490.0 | 128.6 | 171.8 | 195 | 187 |
| 495.0 | 99.2 | 175.9 | 196 | 188 |
| 500.0 | 138.5 | 175.9 | 198 | 188 |
| 505.0 | 110.6 | 175.9 | 195 | 189 |
| 510.0 | 148.3 | 175.9 | 202 | 189 |
| 515.0 | 119.7 | 179.9 | 198 | 190 |
| 520.0 | 86.0 | 179.9 | 199 | 191 |
| 525.0 | 127.1 | 179.9 | 199 | 191 |
| 530.0 | 96.5 | 183.9 | 194 | 192 |
| 535.0 | 137.7 | 183.9 | 193 | 192 |
| 540.0 | 104.1 | 187.9 | 196 | 193 |
| 545.0 | 145.1 | 187.9 | 192 | 193 |
| 550.0 | 109.3 | 187.9 | 197 | 194 |
| 555.0 | 152.5 | 187.9 | 190 | 194 |
| 560.0 | 117.1 | 187.9 | 193 | 195 |
| 565.0 | 156.5 | 187.9 | 195 | 195 |
| 570.0 | 120.6 | 191.9 | 198 | 196 |
| 575.0 | 160.9 | 191.9 | 196 | 196 |
| 580.0 | 124.6 | 195.9 | 196 | 197 |
| 585.0 | 164.8 | 195.9 | 198 | 197 |
| 590.0 | 127.1 | 196.0 | 194 | 198 |
| 595.0 | 167.1 | 196.0 | 196 | 198 |
| 600.0 | 127.5 | 196.0 | 199 | 199 |
| 605.0 | 168.1 | 196.0 | 198 | 199 |
| 610.0 | 127.5 | 204.0 | 198 | 200 |
| 615.0 | 170.1 | 204.0 | 196 | 200 |
| 620.0 | 128.9 | 204.0 | 190 | 201 |
| 625.0 | 169.0 | 204.0 | 192 | 201 |
| 630.0 | 122.8 | 212.1 | 198 | 202 |
| 635.0 | 163.0 | 212.1 | 199 | 202 |
| 640.0 | 118.4 | 212.1 | 199 | 203 |
| 645.0 | 158.1 | 212.1 | 201 | 203 |
| 650.0 | 111.0 | 212.1 | 202 | 204 |
| 655.0 | 150.7 | 212.1 | 203 | 204 |
| 660.0 | 104.0 | 216.1 | 200 | 205 |
| 665.0 | 144.5 | 216.1 | 201 | 205 |
| 670.0 | 186.5 | 216.1 | 192 | 205 |
| 675.0 | 135.1 | 220.6 | 197 | 206 |
| 680.0 | 176.2 | 220.6 | 195 | 206 |
| 685.0 | 127.4 | 224.7 | 191 | 207 |
| 690.0 | 166.9 | 224.7 | 188 | 207 |
| 695.0 | 116.5 | 228.7 | 187 | 208 |
| 700.0 | 154.6 | 228.7 | 191 | 208 |
| 705.0 | 193.8 | 228.7 | 195 | 208 |
| 710.0 | 138.9 | 228.8 | 197 | 209 |
| 715.0 | 181.7 | 228.8 | 193 | 209 |
| 720.0 | 125.4 | 232.8 | 199 | 210 |
| 725.0 | 166.8 | 232.8 | 195 | 210 |
| 730.0 | 204.8 | 232.8 | 202 | 210 |
| 735.0 | 147.1 | 237.0 | 207 | 211 |
| 740.0 | 190.0 | 237.0 | 199 | 211 |
| 745.0 | 135.8 | 241.3 | 187 | 212 |
| 750.0 | 174.1 | 241.3 | 191 | 212 |
| 755.0 | 213.1 | 241.3 | 194 | 212 |
| 760.0 | 150.4 | 249.4 | 192 | 213 |
| 765.0 | 190.6 | 249.4 | 191 | 213 |
| 770.0 | 124.6 | 249.4 | 194 | 214 |
| 775.0 | 164.9 | 249.4 | 194 | 214 |
| 780.0 | 204.2 | 249.4 | 198 | 214 |
| 785.0 | 138.1 | 257.7 | 195 | 215 |
| 790.0 | 177.8 | 257.7 | 198 | 215 |
| 795.0 | 218.6 | 257.7 | 194 | 215 |
| 800.0 | 148.7 | 261.8 | 196 | 216 |
| 805.0 | 189.1 | 261.8 | 198 | 216 |
| 810.0 | 226.8 | 261.8 | 204 | 216 |
| 815.0 | 157.8 | 266.1 | 199 | 217 |
| 820.0 | 198.0 | 266.1 | 192 | 217 |
| 825.0 | 126.8 | 266.1 | 192 | 218 |
| 830.0 | 167.4 | 266.1 | 187 | 218 |
| 835.0 | 205.1 | 266.1 | 195 | 218 |
| 840.0 | 133.0 | 270.3 | 193 | 219 |
| 845.0 | 173.5 | 270.3 | 194 | 219 |
| 850.0 | 213.0 | 270.3 | 194 | 219 |
| 855.0 | 140.3 | 274.7 | 193 | 220 |
| 860.0 | 179.4 | 274.7 | 195 | 220 |
| 865.0 | 218.9 | 274.7 | 200 | 220 |
| 870.0 | 143.4 | 278.7 | 193 | 221 |
| 875.0 | 183.9 | 278.7 | 192 | 221 |
| 880.0 | 222.8 | 278.7 | 194 | 221 |
| 885.0 | 144.7 | 282.7 | 199 | 222 |
| 890.0 | 187.6 | 282.7 | 189 | 222 |
| 895.0 | 226.5 | 282.7 | 195 | 222 |
| 900.0 | 141.2 | 291.0 | 199 | 223 |
| 905.0 | 181.2 | 291.0 | 201 | 223 |
| 910.0 | 220.7 | 291.0 | 201 | 223 |
| 915.0 | 139.8 | 295.0 | 192 | 224 |
| 920.0 | 179.1 | 295.0 | 193 | 224 |
| 925.0 | 219.2 | 295.0 | 193 | 224 |
| 930.0 | 258.0 | 295.1 | 193 | 224 |
| 935.0 | 174.7 | 299.1 | 188 | 225 |
| 940.0 | 213.9 | 299.1 | 192 | 225 |
| 945.0 | 252.0 | 299.1 | 195 | 225 |
| 950.0 | 165.3 | 303.1 | 190 | 226 |
| 955.0 | 205.2 | 303.1 | 194 | 226 |
| 960.0 | 244.9 | 303.1 | 194 | 226 |
| 965.0 | 155.9 | 311.1 | 193 | 227 |
| 970.0 | 197.6 | 311.1 | 187 | 227 |
| 975.0 | 236.4 | 311.1 | 191 | 227 |
| 980.0 | 275.8 | 311.1 | 196 | 227 |
| 985.0 | 186.0 | 315.1 | 195 | 228 |
| 990.0 | 224.2 | 315.1 | 200 | 228 |
| 995.0 | 266.7 | 315.1 | 195 | 228 |
| 1,000.0 | 172.1 | 315.2 | 199 | 229 |
| 1,005.0 | 193.1 | 315.2 | 164 | 229 |
| 1,010.0 | 201.7 | 315.2 | 153 | 229 |
| 1,015.0 | 208.4 | 315.2 | 146 | 229 |
| 1,020.0 | 214.4 | 315.2 | 133 | 229 |
| 1,025.0 | 218.0 | 315.2 | 128 | 229 |
| 1,030.0 | 220.3 | 315.2 | 125 | 229 |
| 1,035.0 | 221.4 | 315.2 | 119 | 229 |
| 1,040.0 | 222.1 | 315.2 | 114 | 229 |
| 1,045.0 | 222.7 | 315.2 | 110 | 229 |
| 1,050.0 | 222.7 | 315.2 | 110 | 229 |
Test Configuration
| Parameter | Value |
|---|
| Node Count | 10,000 |
| Node Rate | 10/sec |
| Controller Timeout | 30s |
| Max Concurrent Reconciles | 50 |
| API Concurrency | 150 |
| DaemonSet Count | 3 |
| Total Duration | 1,051.5s |
Profile Distribution
| Profile | Count | Percentage |
|---|
| immediate | 7,000 | 70.0% |
| delayed | 1,500 | 15.0% |
| crash-recover | 1,000 | 10.0% |
| never-ready | 500 | 5.0% |