Benchmarks

Performance benchmarks from stress testing

Latest Benchmark Results

Run date: 2026-03-28T21:50:52Z Git SHA: 1577d67 Node count: 10,000

Latency

The stress test measures three distinct latency phases:

  • Pod startup — time from node creation until all expected DaemonSet pods become Ready. This is dominated by simulated pod startup delays and reflects real-world DaemonSet boot time. Excludes never-ready nodes.
  • Vigil reaction — time from all pods becoming Ready until the taint is removed. This isolates Vigil’s controller overhead: reconcile loop detection, readiness verification, and the taint removal API call.
  • End-to-end — total time from node creation to taint removal. For never-ready nodes, this equals the controller timeout.

Pod Startup Latency

PercentileLatency
p503,761ms
p9538,220ms
p9951,901ms

Vigil Reaction Time

PercentileLatency
p501,000ms
p951,901ms
p991,981ms

End-to-End Latency

PercentileLatency
p505,023ms
p9530,206ms
p9930,416ms

Outcomes

OutcomeCountPercentage
Success9,50095.0%
Timeout5005.0%
Pending00.0%

Resource Utilization

MetricValue
Peak Heap275.8 MB
Final Heap224.1 MB
System Memory315.2 MB
GC Cycles229
GC CPU Fraction0.03%

Resource Utilization Over Time

Heap Memory

xychart-beta
    title "Heap Allocation Over Time"
    x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
    y-axis "Heap (MB)" 0 --> 280
    line [8, 11, 24, 31, 47, 41, 60, 76, 86, 73, 86, 114, 117, 88, 99, 97, 157, 128, 163, 187, 194, 190, 165, 227, 174, 223, 140, 165, 186, 214]

Goroutines

xychart-beta
    title "Active Goroutines Over Time"
    x-axis "Elapsed (s)" [5, 40, 75, 110, 145, 180, 215, 250, 285, 320, 355, 390, 425, 460, 495, 530, 565, 600, 635, 670, 705, 740, 775, 810, 845, 880, 915, 950, 985, 1020]
    y-axis "Goroutines" 0 --> 250
    line [140, 185, 197, 196, 195, 194, 192, 199, 199, 199, 190, 192, 192, 194, 196, 194, 195, 199, 199, 192, 195, 199, 194, 204, 194, 194, 192, 190, 195, 133]

Memory grows linearly with node count (10 nodes/sec) as the informer cache accumulates Node and Pod objects. The sawtooth pattern in heap allocation shows the Go garbage collector reclaiming memory between allocation bursts. Goroutine count remains stable (~190-200), confirming no goroutine leaks.

Raw data (210 samples)
Elapsed (s)Heap (MB)System (MB)GoroutinesGC Cycles
5.07.923.114011
10.07.827.915220
15.06.627.916428
20.08.732.117635
25.010.032.418442
30.09.832.418448
35.013.236.418453
40.010.936.418558
45.015.436.419162
50.017.040.419866
55.017.640.420570
60.020.444.620174
65.016.144.619679
70.020.144.619282
75.023.544.719785
80.017.748.719689
85.022.848.819392
90.027.748.819195
95.023.953.119598
100.031.453.1199100
105.024.257.1198103
110.030.657.1196105
115.036.661.1194107
120.040.561.1193109
125.025.265.1199112
130.029.765.1198114
135.030.765.1196116
140.030.069.1191118
145.047.169.1195119
150.040.569.1197121
155.034.273.3198123
160.051.273.3194124
165.042.073.3198126
170.034.477.4195128
175.050.677.4193129
180.040.777.5194131
185.033.681.5194133
190.047.281.5198134
195.039.181.6201136
200.056.481.6192137
205.037.585.6196139
210.047.785.6197140
215.059.585.9192141
220.067.689.9194142
225.047.289.9193144
230.053.994.1194145
235.060.894.1197146
240.067.398.1194147
245.072.198.1196148
250.075.798.4199149
255.080.4102.4200150
260.046.4106.4192152
265.049.4106.4190153
270.047.5110.4200154
275.049.4110.5195155
280.088.0110.6196155
285.086.0110.6199156
290.085.1110.6195157
295.082.6114.6198158
300.081.1114.6198159
305.080.0114.6200160
310.079.1114.6197161
315.077.4114.8197162
320.073.5118.8199163
325.070.2122.8198164
330.067.0122.8191165
335.062.9122.8191166
340.057.0122.8191167
345.096.8122.8192167
350.091.2126.8189168
355.086.0130.8190169
360.074.8130.8195170
365.068.6134.8189171
370.0107.3134.8194171
375.097.3135.1194172
380.085.6139.2199173
385.074.0143.5196174
390.0114.4143.6192174
395.097.0147.6197175
400.083.0147.6197176
405.069.3147.6195177
410.0107.4147.6199177
415.092.3147.6199178
420.075.7151.6197179
425.0116.8151.6192179
430.097.4151.6196180
435.078.6159.6188181
440.0117.6159.6190181
445.095.2159.6194182
450.0134.9159.6193182
455.0113.5163.6189183
460.088.0163.6194184
465.0128.2163.6193184
470.0102.5163.6195185
475.0142.3163.6198185
480.0117.1167.7191186
485.089.1171.8193187
490.0128.6171.8195187
495.099.2175.9196188
500.0138.5175.9198188
505.0110.6175.9195189
510.0148.3175.9202189
515.0119.7179.9198190
520.086.0179.9199191
525.0127.1179.9199191
530.096.5183.9194192
535.0137.7183.9193192
540.0104.1187.9196193
545.0145.1187.9192193
550.0109.3187.9197194
555.0152.5187.9190194
560.0117.1187.9193195
565.0156.5187.9195195
570.0120.6191.9198196
575.0160.9191.9196196
580.0124.6195.9196197
585.0164.8195.9198197
590.0127.1196.0194198
595.0167.1196.0196198
600.0127.5196.0199199
605.0168.1196.0198199
610.0127.5204.0198200
615.0170.1204.0196200
620.0128.9204.0190201
625.0169.0204.0192201
630.0122.8212.1198202
635.0163.0212.1199202
640.0118.4212.1199203
645.0158.1212.1201203
650.0111.0212.1202204
655.0150.7212.1203204
660.0104.0216.1200205
665.0144.5216.1201205
670.0186.5216.1192205
675.0135.1220.6197206
680.0176.2220.6195206
685.0127.4224.7191207
690.0166.9224.7188207
695.0116.5228.7187208
700.0154.6228.7191208
705.0193.8228.7195208
710.0138.9228.8197209
715.0181.7228.8193209
720.0125.4232.8199210
725.0166.8232.8195210
730.0204.8232.8202210
735.0147.1237.0207211
740.0190.0237.0199211
745.0135.8241.3187212
750.0174.1241.3191212
755.0213.1241.3194212
760.0150.4249.4192213
765.0190.6249.4191213
770.0124.6249.4194214
775.0164.9249.4194214
780.0204.2249.4198214
785.0138.1257.7195215
790.0177.8257.7198215
795.0218.6257.7194215
800.0148.7261.8196216
805.0189.1261.8198216
810.0226.8261.8204216
815.0157.8266.1199217
820.0198.0266.1192217
825.0126.8266.1192218
830.0167.4266.1187218
835.0205.1266.1195218
840.0133.0270.3193219
845.0173.5270.3194219
850.0213.0270.3194219
855.0140.3274.7193220
860.0179.4274.7195220
865.0218.9274.7200220
870.0143.4278.7193221
875.0183.9278.7192221
880.0222.8278.7194221
885.0144.7282.7199222
890.0187.6282.7189222
895.0226.5282.7195222
900.0141.2291.0199223
905.0181.2291.0201223
910.0220.7291.0201223
915.0139.8295.0192224
920.0179.1295.0193224
925.0219.2295.0193224
930.0258.0295.1193224
935.0174.7299.1188225
940.0213.9299.1192225
945.0252.0299.1195225
950.0165.3303.1190226
955.0205.2303.1194226
960.0244.9303.1194226
965.0155.9311.1193227
970.0197.6311.1187227
975.0236.4311.1191227
980.0275.8311.1196227
985.0186.0315.1195228
990.0224.2315.1200228
995.0266.7315.1195228
1,000.0172.1315.2199229
1,005.0193.1315.2164229
1,010.0201.7315.2153229
1,015.0208.4315.2146229
1,020.0214.4315.2133229
1,025.0218.0315.2128229
1,030.0220.3315.2125229
1,035.0221.4315.2119229
1,040.0222.1315.2114229
1,045.0222.7315.2110229
1,050.0222.7315.2110229

Test Configuration

ParameterValue
Node Count10,000
Node Rate10/sec
Controller Timeout30s
Max Concurrent Reconciles50
API Concurrency150
DaemonSet Count3
Total Duration1,051.5s

Profile Distribution

ProfileCountPercentage
immediate7,00070.0%
delayed1,50015.0%
crash-recover1,00010.0%
never-ready5005.0%