Nomad v2.1.0 Release Notes: Experimental Batch Queue, ACL Breaking Changes, and Upgrade Guide

Introduction

HashiCorp released Nomad v2.1.0, following v2.0.7. This release introduces an experimental batch job queue for node pools, tightens the ACL permissions required to manage server membership, and removes configuration fields that have been deprecated for several releases. Two breaking changes require action before upgrading production clusters. Full notes: github.com/hashicorp/nomad/releases/tag/v2.1.0.

What Is New

Experimental: Batch Job Queue on Node Pools (GH-28674)

Node pools can now be configured with a batch job queue that controls how pending batch jobs are scheduled when resources are constrained. Two queuing strategies are available:

• FIFO — jobs are placed strictly in the order they were submitted.
• Dynamic Priority — jobs are prioritized by age and by "fairness" of resources requested/placed, so long-waiting or resource-light jobs are not starved by large, recently submitted ones.

The release notes mark this feature as experimental and do not publish the final HCL schema. Treat the following as illustrative only and confirm exact field names against the official node pool documentation before relying on it in production:

# Illustrative only — verify exact schema in Nomad docs before use
node_pool "batch-workers" {
  description = "Pool dedicated to batch workloads"
 
  # Experimental batch queue configuration (GH-28674)
  # queue_algorithm = "fifo"            # or "dynamic_priority"
}

If batch workloads are latency-sensitive or resource contention is already a problem on shared pools, this is worth testing in a non-production namespace once the schema is finalized.

Jobspec2: optional() on Object-Typed Variables (GH-28424)

Nomad's HCL2 jobspec now supports the optional() modifier on attributes inside object-typed variables, matching the Terraform HCL optional object attributes pattern. This lets job authors define variables with partial defaults:

variable "resources" {
  type = object({
    cpu    = optional(number, 100)
    memory = optional(number, 256)
  })
  default = {}
}
 
job "web" {
  group "app" {
    task "server" {
      resources {
        cpu    = var.resources.cpu
        memory = var.resources.memory
      }
    }
  }
}

Callers can now omit cpu or memory from the object they pass without Nomad rejecting the variable, reducing boilerplate in shared job templates.

New CLI: operator snapshot filter (GH-28580)

A new subcommand allows editing a Raft snapshot file directly, which is useful for removing or filtering state before restoring a snapshot into a cluster:

nomad operator snapshot filter \
  -job-filter='my-old-job' \
  input.snap output.snap

Consult nomad operator snapshot filter -h for the full set of supported filters before using this against a production snapshot.

TLS Error Message Changes (GH-28579)

Code quality improvements to TLS configuration handling mean TLS handshake failure error messages have changed. This is not a functional break, but any log parsing, alerting, or dashboards that match on exact TLS error strings should be reviewed after the upgrade.

Breaking Changes

agent:write Required for Server Join (GH-28553)

Joining or managing server membership via the CLI (nomad server join) or the HTTP API now requires the agent:write ACL capability. Tokens that previously relied on broader or implicit permissions for this operation will be rejected. Update ACL policies accordingly:

# acl-agent-write.policy.hcl
agent {
  policy = "write"
}

nomad acl policy apply -description "Agent write for server join" agent-write acl-agent-write.policy.hcl

Verify any automation (configuration management, join scripts, operators dashboards) uses a token with this capability before upgrading servers.

Removal of Legacy Retry-Join Config (GH-28554)

The deprecated top-level server fields retry_join, retry_max, retry_interval, and start_join have been removed. Configuration using these fields will fail to start after the upgrade. Migrate them into the server_join block:

server {
  enabled = true
 
  server_join {
    retry_join     = ["10.0.1.10", "10.0.1.11", "10.0.1.12"]
    retry_max      = 3
    retry_interval = "15s"
  }
}

Audit every server.hcl / agent config file in the fleet for the old top-level keys before rolling out v2.1.0.

Deprecations

• API: Allocation.Resources and Allocation.TaskResources fields are deprecated in the Go API client and HTTP responses. Switch consumers to Allocation.AllocatedResources (GH-28486).
• Client state: client state databases restored from backups predating Nomad 0.9 are no longer compatible and will cause client startup to fail (GH-28486). Confirm backup/restore tooling isn't still pulling from pre-0.9 archives.

Bug Fixes of Note

• Fixed node heartbeats that could be sent from other nodes, a correctness/security-relevant fix (GH-28598).
• Fixed the plan applier never refreshing its optimistic snapshot under sustained saturation, which could cause scheduling stalls (GH-28664).
• Fixed incorrect CPU fingerprinting on Apple M4 chips (GH-28627).
• Fixed a task-state overwrite bug on clients (GH-27944).
• Fixed a var put CLI panic on invalid input (GH-28630).
• Fixed CSI plugin status -verbose reporting capabilities not actually supported by the plugin (GH-28626).
• UI fixes for CPU utilization graphs on cores-reserved allocations and for pagination bugs on the jobs list page (GH-28569, GH-28178).
• Go updated to 1.27.3 and the busybox base image used for container final images bumped to 1.38.0 (GH-28647, GH-28679, GH-28622).

Upgrade Steps

1. Audit ACL policies and automation for the new agent:write requirement on server join (GH-28553).
2. Grep agent configs for the removed retry_join, retry_max, retry_interval, and start_join top-level fields and move them into server_join blocks (GH-28554).
3. Review any log parsing or alerting rules that match specific TLS handshake error strings (GH-28579).
4. If managing Nomad resources (ACL policies, namespaces, node pools) from Terraform via the hashicorp/nomad provider, pin and refresh the provider separately from the Nomad binary version:

terraform {
  required_providers {
    nomad = {
      source  = "hashicorp/nomad"
      version = "~> 2.0"
    }
  }
}

terraform init -upgrade
terraform plan

5. Perform a standard rolling upgrade: upgrade server agents first (one at a time, confirming cluster leadership and Raft health between each), then upgrade client agents.

# On each server, after replacing the binary
systemctl restart nomad
nomad server members

Verify

After upgrading, confirm the cluster is on v2.1.0 and healthy:

nomad version
nomad server members
nomad node status

Confirm ACL enforcement for server join if ACLs are enabled:

nomad acl token self
nomad server join -h

If using the experimental batch queue, validate node pool configuration parses correctly:

nomad node pool status batch-workers

Conclusion

Nomad v2.1.0 is primarily an operational hygiene release: it closes out a long-deprecated config path (retry_join and friends), tightens ACL requirements around server membership, and ships a number of targeted scheduler and UI bug fixes. The experimental batch job queue is worth tracking for teams running mixed batch/service workloads, but its HCL schema is not yet finalized in the release notes — wait for official docs before committing production configuration to it. The two breaking changes are small in scope but will cause startup failures or permission errors if skipped, so audit agent configs and ACL policies before rolling this out.