NodeOverlay CRD

NodeOverlay is a Karpenter custom resource (karpenter.sh/v1alpha1) that allows adjusting instance type pricing used during provisioning. Veneer creates and manages NodeOverlay resources to influence which instances Karpenter selects.

API Definition

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: <overlay-name>
  labels:
    app.kubernetes.io/managed-by: veneer
    veneer.io/type: <cost-aware|preference>
    veneer.io/source-nodepool: <nodepool-name>  # For preference overlays
spec:
  # Requirements that an instance must match for this overlay to apply
  requirements:
    - key: <label-key>
      operator: <In|NotIn|Gt|Lt|Exists|DoesNotExist>
      values: [<value1>, <value2>, ...]

  # Price adjustment (percentage string)
  priceAdjustment: "<adjustment>"

  # Weight for overlay precedence (higher = higher priority)
  weight: <integer>

Spec Fields

spec.requirements

A list of node selector requirements that an instance type must match for this overlay’s price adjustment to apply. Uses the same requirement format as Karpenter NodePool requirements.

All requirements must be satisfied for the overlay to apply to an instance type.

spec.priceAdjustment

A string representing the price adjustment to apply. Veneer uses percentage adjustments:

• "-30%" – Reduce the effective price by 30% (makes the instance more preferred)

Cost-aware overlays use configurable discounts strictly between -100% and 0%; the default is -50%. Relative discounts retain price differences between instance types and avoid the arbitrary tie-breaking caused by assigning every matching offering the same absolute price.

The adjusted price becomes the Priority value in the AWS CreateFleet API call. Lower Priority = higher preference.

spec.weight

An integer that determines overlay precedence. When multiple overlays match the same instance type, the overlay with the highest weight takes effect.

Weight System

Veneer uses a tiered weight system to ensure the most specific capacity data takes precedence:

This hierarchy ensures that when both an RI and a Compute SP apply to the same instance type, the more specific RI overlay (weight 30) takes precedence over the general Compute SP overlay (weight 10).

Naming Conventions

Cost-Aware Overlays

Veneer generates overlay names from the capacity type prefix and identifying attributes:

Naming prefixes are configurable via the overlays.naming configuration. See Configuration Reference.

Preference Overlays

Preference overlays use the NodePool name and preference number:

Labels

All Veneer-managed overlays carry these labels:

These labels are used for:

• Listing all Veneer-managed overlays: kubectl get nodeoverlays -l app.kubernetes.io/managed-by=veneer
• Filtering by type: kubectl get nodeoverlays -l veneer.io/type=preference
• Finding overlays for a NodePool: kubectl get nodeoverlays -l veneer.io/source-nodepool=my-workload

Examples

Cost-Aware: Reserved Instance Overlay

Created when Lumina detects active Reserved Instances for m5.xlarge in us-west-2. The deeper default discount is intended to keep covered on-demand capacity competitive with Spot while preserving instance-price ordering:

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: cost-aware-ri-m5-xlarge-us-west-2
  labels:
    app.kubernetes.io/managed-by: veneer
    veneer.io/type: cost-aware
spec:
  requirements:
    - key: node.kubernetes.io/instance-type
      operator: In
      values: ["m5.xlarge"]
    - key: topology.kubernetes.io/region
      operator: In
      values: ["us-west-2"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
  priceAdjustment: "-90%"
  weight: 30

Cost-Aware: EC2 Instance Savings Plan Overlay

Created when Lumina detects an EC2 Instance Savings Plan covering the m5 family in us-west-2 with remaining capacity. Its deeper default discount keeps covered on-demand capacity competitive with Spot without flattening family prices:

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: cost-aware-ec2-sp-m5-us-west-2
  labels:
    app.kubernetes.io/managed-by: veneer
    veneer.io/type: cost-aware
spec:
  requirements:
    - key: karpenter.k8s.aws/instance-family
      operator: In
      values: ["m5"]
    - key: topology.kubernetes.io/region
      operator: In
      values: ["us-west-2"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
  priceAdjustment: "-90%"
  weight: 20

Cost-Aware: Compute Savings Plan Overlay

Created when Lumina detects a Compute Savings Plan with remaining capacity. This example scopes the overlay to an on-demand-only NodePool:

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: cost-aware-compute-sp-global
  labels:
    app.kubernetes.io/managed-by: veneer
    veneer.io/type: cost-aware
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["on-demand"]
    - key: karpenter.sh/nodepool
      operator: In
      values: ["on-demand-general"]
  priceAdjustment: "-50%"
  weight: 10

Compute Savings Plan Safety and Scope

Compute Savings Plans are broader than RI and EC2 Instance SP coverage. An unscoped Compute SP overlay can match every on-demand offering and can change purchasing behavior in NodePools that permit both Spot and On-Demand capacity.

Use one of these deployment patterns for mixed-capacity clusters:

• Set overlays.computeSavingsPlan.enabled: false while leaving RI, EC2 Instance SP, and preference overlays enabled.
• Set overlays.computeSavingsPlan.nodePoolSelector.names to NodePools that are constrained to On-Demand capacity.

When names are configured, Veneer adds a karpenter.sh/nodepool In [...] requirement. Scope supports explicit NodePool names only; it does not support metadata label selectors. An empty name list leaves the Compute SP overlay unscoped.

The Compute SP overlay also requires minRemainingCapacityDollars of remaining hourly commitment for minBelowThresholdDuration before creation. These defaults are $50/hour and 15m. This hysteresis prevents a short utilization dip from immediately creating a broad overlay; unsafe conditions remove an existing overlay immediately.

Preference Overlay

Created from a veneer.io/preference.2 annotation on a NodePool:

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: pref-my-workload-2
  labels:
    app.kubernetes.io/managed-by: veneer
    veneer.io/type: preference
    veneer.io/source-nodepool: my-workload
spec:
  requirements:
    - key: karpenter.sh/nodepool
      operator: In
      values: ["my-workload"]
    - key: kubernetes.io/arch
      operator: In
      values: ["arm64"]
  priceAdjustment: "-30%"
  weight: 2

Disabled Mode Overlay

When overlays.disabled: true, overlays include an impossible requirement:

apiVersion: karpenter.sh/v1alpha1
kind: NodeOverlay
metadata:
  name: cost-aware-ri-m5-xlarge-us-west-2
spec:
  requirements:
    - key: node.kubernetes.io/instance-type
      operator: In
      values: ["m5.xlarge"]
    - key: veneer.io/disabled
      operator: In
      values: ["true"]  # No node will ever have this label
  priceAdjustment: "-90%"
  weight: 30

How NodeOverlay Affects Karpenter

When a NodeOverlay exists, Karpenter changes its behavior in two ways:

1. Price Adjustment: The overlay’s priceAdjustment modifies the effective price of matching instance types. This adjusted price is used for sorting and as the Priority value in the CreateFleet API call.

2. Allocation Strategy Change: The presence of any NodeOverlay switches the allocation strategy:

   • Spot: price-capacity-optimized becomes capacity-optimized-prioritized
   • On-Demand: lowest-price becomes prioritized

   This ensures AWS uses the Priority values (adjusted prices) when selecting instances.

See Instance Selection Deep Dive for the full technical explanation.