Kueue Sentinel Pipeline
The problem
The ODH Dashboard calls Kubernetes APIs to manage Kueue and Kubeflow Trainer resources. The dashboard team does not control those upstream operators. When they change — and they do, without warning — the dashboard breaks.
Until recently, the team found out the same way every time: a nightly E2E test failed, someone filed a Jira ticket, and the fix happened after the fact. There was no proactive detection. No early warning. Two Jira blockers confirmed the pattern — RHOAIENG-58488 (Kueue API version change) and RHOAIENG-88520 (Trainer immutability change) — both discovered in production.
The missing piece was noticed by multiple engineers independently:
"I think we should be notified/informed about any upcoming Kueue version upgrades so we could track these and keep an eye out for them."
"I see this was done at very last moment… way after code freeze for 3.5 GA. Is there any channel which communicates this?"
"I have some concerns on how we will handle the fact RHBoK can be lifecycled independent of RHOAI — if RHBoK 2.0 suddenly becomes available, how is Dashboard intended to react on an unsupported API?"
Three separate engineers. Three separate messages. All asking for the same thing. This pipeline is the answer.
How it works
The pipeline runs every night. It fetches the latest releases of Kueue and Kubeflow Trainer from GitHub, spins up a real Kubernetes cluster, and runs three layers of tests against it. If anything breaks, a Slack notification goes to the team before anyone starts their day.
The three layers each catch a different class of problem:
kubectl --dry-run=server. Webhook rejections and label errors surface here.The Kueue and Trainer versions tested are always the latest from GitHub releases — whatever upstream shipped that week. Layer 1 runs on every PR; Layers 2 and 3 run nightly on a fresh Kind cluster.
The incidents
Four specific incidents drove the design of the pipeline. Each one had already caused a production problem before the sentinel existed. Each maps to one of the layers above.
API version mismatch RHOAIENG-58488
What happened: Kueue promoted its storage API from v1beta1 to v1beta2. The dashboard was still calling v1beta1 — pausing and resuming RayJobs failed silently on any cluster running Kueue 1.3.x. No error surfaced in the UI. The workloads just never transitioned.
Steps taken:
- Updated TypeScript types from
v1beta1tov1beta2across the codebase - Added contract tests that validate the correct API version on every PR
- Pinned CRD YAML fixtures so schema drift is caught before code merges
✓ Covered by Layer 1 — Contract Tests
Immutable field RHOAIENG-88520
What happened: Kubeflow Trainer made spec.trainer immutable after job creation in v2.3.0. The dashboard's scale feature was PATCHing that field. The cluster silently rejected the request. The scale operation appeared to succeed in the UI but nothing changed on the cluster.
Steps taken:
- Removed the PATCH-based scale path from the dashboard entirely
- Updated TypeScript types and all related mock factories
- Layer 3 now explicitly asserts the field is immutable — any future change in either direction will trip an alert
✓ Covered by Layer 2 + Layer 3
Webhook label enforcement
What happened: Kueue's admission webhook requires the kueue.x-k8s.io/queue-name label on all managed workloads. If the label is absent, or the webhook changes how it validates label format, workloads are silently rejected at the cluster. The dashboard shows the resource as created; it never actually runs.
Steps taken:
- Added dry-run validation for every workload type the dashboard creates
- Replaced static
kubeconformchecks with livekubectl --dry-run=serveragainst a real cluster - Manifests used in tests are derived from the actual dashboard YAML factories — not hand-written test files
✓ Covered by Layer 2 — API Dry-Run
HardwareProfile race condition RHOAIENG-85382
What happened: A timing issue between Kueue's admission webhook and the scheduler caused priority classes to not always be applied on HardwareProfile-backed workloads. The failure was intermittent and cluster-configuration-dependent — hard to reproduce locally, invisible in unit tests.
Steps taken:
- Layer 3 integration smoke now validates the full admission lifecycle including priority class application
- Every nightly run checks end-to-end: workload created, admitted, priority class applied, job completes
✓ Covered by Layer 3 — Integration Smoke
It's running
GitHub Actions workflow in odh-automations, scheduled at 06:30 UTC daily. Three jobs: Schema Check, API & Integration Tests, Report & Notify. Each job writes a per-test summary table to the workflow run page — pass/fail, version tested, detail text. No log digging required.
🔗 Sample pipeline run — Kueue v0.19.3 + Trainer v2.3.0, all layers green
Every run posts to Slack. Pass looks like this:
Numbers
- 4 incidents that drove the design
- 3 test layers (contract tests, API dry-run, integration smoke)
- ~15 min per nightly run end-to-end
- 6 API dry-run checks + 7 integration smoke checks = 13 automated assertions per nightly run
- 2 upstream repos monitored: Kueue and Kubeflow Trainer