Concepts on Veneer

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Veneer is a Kubernetes controller that bridges the gap between AWS cost data and Karpenter provisioning decisions. It continuously monitors Savings Plans and Reserved Instance utilization via Lumina metrics in Prometheus, then manages NodeOverlay custom resources to steer Karpenter toward cost-optimal instance types.

Data Flow

flowchart LR
 Lumina["Lumina"]
 Prom["Prometheus"]
 Veneer["Veneer"]
 NO["NodeOverlays"]
 Karpenter["Karpenter"]
 Fleet["AWS CreateFleet"]

 Lumina -->|"expose SP/RI metrics"| Prom
 Prom -->|"query cost data"| Veneer
 Veneer -->|"create/update/delete"| NO
 NO -->|"adjust pricing"| Karpenter
 Karpenter -->|"Priority values"| Fleet

 style Lumina fill:#e3f2fd,stroke:#1565c0,color:#1565c0
 style Prom fill:#fbe9e7,stroke:#bf360c,color:#bf360c
 style Veneer fill:#e0f2f1,stroke:#00695c,color:#00695c
 style NO fill:#f1f8e9,stroke:#33691e,color:#33691e
 style Karpenter fill:#ede7f6,stroke:#4527a0,color:#4527a0
 style Fleet fill:#fff3e0,stroke:#e65100,color:#e65100

Lumina discovers AWS Savings Plans, Reserved Instances, and running EC2 instances. It computes utilization and remaining capacity, then exposes these as Prometheus metrics.
Veneer queries Prometheus on a 5-minute interval (matching Lumina’s refresh cycle). The decision engine analyzes capacity data and determines which NodeOverlays should exist.
Karpenter reads NodeOverlay resources and applies price adjustments to its instance type offerings. Adjusted prices become Priority values in the AWS CreateFleet API call.
AWS selects instances based on the allocation strategy and Priority values. See Instance Selection Deep Dive for details.

Two Reconcilers

Veneer runs two independent reconciliation loops:

Instance Selection Deep Dive

Mon, 01 Jan 0001 00:00:00 +0000

This document provides a detailed technical explanation of how Karpenter selects EC2 instances, from pod scheduling through the AWS CreateFleet API call. Understanding this flow is essential for configuring Veneer’s NodeOverlay feature to influence instance selection.

Overview

Karpenter does not make the final instance type selection. Instead, it:

Filters instance types based on NodePool requirements and pod constraints
Sorts instance types by adjusted price
Truncates to a maximum of 60 instance types
Delegates the final selection to AWS via the CreateFleet API

AWS CreateFleet makes the ultimate decision based on:

Bin-Packing and NodeOverlay

Mon, 01 Jan 0001 00:00:00 +0000

This document explains how Karpenter’s bin-packing algorithm can affect – and sometimes bypass – NodeOverlay price adjustments, leading to unexpected instance selection behavior.

Overview

Veneer’s NodeOverlay feature influences instance selection by adjusting prices, which become Priority values in AWS CreateFleet requests. However, this influence only works when multiple instance types are eligible candidates.

The key insight: Karpenter’s bin-packing algorithm filters instance types before NodeOverlay can influence selection. If bin-packing eliminates all instances of a particular architecture, NodeOverlay has nothing to prefer.

Instance Preferences

Mon, 01 Jan 0001 00:00:00 +0000

Instance preferences allow you to express instance type preferences directly on Karpenter NodePools using annotations. Veneer watches NodePools and generates NodeOverlay resources for each preference, influencing Karpenter’s provisioning decisions.

NodeOverlays are preferences, not rules. When Veneer creates a NodeOverlay with a price adjustment, it influences but does not guarantee instance selection. See Instance Selection Deep Dive for how AWS makes the final decision.

Annotation Format

veneer.io/preference.N: "<matcher> [<matcher>...] adjust=[+-]N%"

Where:

N is a positive integer (1-9 recommended) that determines overlay weight/priority
<matcher> is key=value1,value2 or key!=value or key>value or key<value
adjust specifies the price adjustment percentage

Example NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
 name: my-workload
 annotations:
 # Prefer c7a/c7g families with 20% discount
 veneer.io/preference.1: "karpenter.k8s.aws/instance-family=c7a,c7g adjust=-20%"
 # Prefer ARM64 architecture with 30% discount
 veneer.io/preference.2: "kubernetes.io/arch=arm64 adjust=-30%"
 # Combined matcher: m7g on ARM64 with 40% discount
 veneer.io/preference.3: "karpenter.k8s.aws/instance-family=m7g kubernetes.io/arch=arm64 adjust=-40%"

Generated NodeOverlay

For preference veneer.io/preference.1 on the NodePool above, Veneer generates: