Add first-class KubernetesPersistentVolumeResource (and fix related YAML serializer bugs)#16929
Conversation
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16929Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16929" |
There was a problem hiding this comment.
Pull request overview
This PR aims to make the Kubernetes publisher emit deployable YAML by preventing empty {} mappings in generated PersistentVolume / PersistentVolumeClaim manifests (notably for DefaultStorageType = "pvc"), and adds regression tests/snapshots to cover the affected paths.
Changes:
- Make several V1 spec sub-properties nullable (removing eager
= new()) so YAML serialization omits them when empty. - Add regression tests and Verify snapshots for
pvcandhostpathstorage type publishing, including an explicit{}-substring invariant. - Update XML docs/remarks on the affected spec properties to explain the null-default behavior and why it matters for Kubernetes apply.
Show a summary per file
| File | Description |
|---|---|
| tests/Aspire.Hosting.Kubernetes.Tests/KubernetesPublisherTests.cs | Adds regression tests for pvc and hostpath storage type publishing and asserts no empty {} mappings appear in emitted YAML. |
| tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithPvcStorageType_GeneratesValidPersistentVolumeAndClaim#00.verified.yaml | New snapshot covering the generated Deployment for pvc storage type. |
| tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithPvcStorageType_GeneratesValidPersistentVolumeAndClaim#01.verified.yaml | New snapshot covering the generated PersistentVolume for pvc storage type. |
| tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithPvcStorageType_GeneratesValidPersistentVolumeAndClaim#02.verified.yaml | New snapshot covering the generated PersistentVolumeClaim for pvc storage type. |
| tests/Aspire.Hosting.Kubernetes.Tests/Snapshots/KubernetesPublisherTests.PublishAsync_WithHostPathStorageType_GeneratesValidPersistentVolumeAndClaim.verified.yaml | New snapshot covering the generated Deployment for hostpath storage type. |
| src/Aspire.Hosting.Kubernetes/Resources/PersistentVolumeSpecV1.cs | Makes ClaimRef, HostPath, Local, and NodeAffinity nullable to avoid emitting empty mappings. |
| src/Aspire.Hosting.Kubernetes/Resources/PersistentVolumeClaimSpecV1.cs | Makes DataSource, DataSourceRef, and Selector nullable to avoid emitting empty mappings. |
| src/Aspire.Hosting.Kubernetes/Resources/VolumeNodeAffinityV1.cs | Makes Required nullable to avoid emitting required: {} when unset. |
Copilot's findings
- Files reviewed: 8/8 changed files
- Comments generated: 2
mitchdenny
changed the title
|
Tracking issue: #16999 (created retroactively to capture the design decisions and context for this work). Now in the 13.4 milestone alongside the future-work items called out in that issue. |
7d0624d to
485e96f
Compare
|
Re-running the failed jobs in the CI workflow for this pull request because 4 jobs were identified as retry-safe transient failures in the CI run attempt.
Matched test failure patterns (1 test)
|
4746841 to
d830d5b
Compare
d830d5b to
f1c4555
Compare
|
Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
|
|
❌ CLI E2E Tests failed — 98 passed, 1 failed, 5 unknown (commit Failed Tests
View all recordings
📹 Recordings uploaded automatically from CI run #26351772900 |
|
Not for 13.4, moving. |
f1c4555 to
e886f5b
Compare
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
/deployment-test |
|
🚀 Deployment tests starting on PR #16929... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
The <em> HTML tag isn't supported in C# XML documentation and won't render in IntelliSense or the published API docs. Only <c>, <code>, and <para> are permitted for inline emphasis. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
PR Testing ReportPR Information
Artifact Version Verification
Changes AnalyzedKubernetes hosting integration only:
Test Scenarios ExecutedScenario 1 — Name-match PV binding + StatefulSet auto-promotionObjective: AppHost: var k8s = builder.AddKubernetesEnvironment("k8s");
var pgData = k8s.AddPersistentVolume("pg-data")
.WithStorageClass("fast-ssd")
.WithCapacity("20Gi");
builder.AddPostgres("pg")
.WithDataVolume("pg-data")
.WithPersistentVolume(pgData);Emitted:
Scenario 2 — Project mount-path overload (#9430)Objective: AppHost (starter template + PV binding on the apiservice project): var k8s = builder.AddKubernetesEnvironment("k8s");
var apiData = k8s.AddPersistentVolume("api-data")
.WithStorageClass("standard").WithCapacity("5Gi");
var apiService = builder.AddProject<Projects.PvProj_ApiService>("apiservice")
.WithHttpHealthCheck("/health")
.WithPersistentVolume(apiData, "/srv/data");Emitted:
Scenario 3 — Fallthrough for unbound volume with env
|
| Scenario | Status | Notes |
|---|---|---|
| 1. Name-match binding + StatefulSet promotion | ✅ Passed | All invariants hold, no empty YAML blocks |
| 2. Project mount-path overload | ✅ Passed | #9430 fixed |
3. Fallthrough with DefaultStorageType = "pvc" |
✅ Passed | #14096 fixed, valid PVC emitted |
| 4. Duplicate binding | ✅ Passed | Safe silent dedupe |
| 5. Cross-environment binding | Latent gap, not merge-blocking, follow-up recommended |
Overall Result
✅ PR VERIFIED for its stated scope.
The three named issues (#9430 project mount-path, #14096 PVC fallthrough, and the serializer emit-empty-blocks bug) are all fixed by this PR against a real emitted Helm chart. StatefulSet auto-promotion works cleanly for both container and project workloads, PVC names match consistently, and no invalid empty spec blocks appear anywhere in the emitted YAML.
Follow-up recommendation (not blocking)
- Add publish-time validation that a
KubernetesPersistentVolumeBindingAnnotationrefers to a PV owned by the sameKubernetesEnvironmentResourceas the bound workload. Today this misuse silently produces YAML that won't apply.
A workload assigned to Kubernetes environment B cannot bind a persistent volume declared on environment A: the two charts render into separate namespaces/clusters, so the workload's claimName reference would resolve to a PVC that does not exist alongside it and Kubernetes would fail to schedule the pod with 'persistentvolumeclaim not found'. Validate at publish time in the per-env compute-resource loop, where both the resolved workload compute environment and the annotated PV are known, and throw with a clear message pointing at the offending resource, both environments, and the fix. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
PR #16929 Testing Report (follow-up: cross-env validation)PR Information
Artifact Version Verification
What Changed Since the Previous Test Run (
|
| Scenario | Status | Notes |
|---|---|---|
| S1: Name-match + StatefulSet promotion | ✅ Passed | Correct PVC + claimName |
S2: Mount-path overload (#9430) |
✅ Passed | Works for ProjectResource-style compute |
S3: Fallthrough with DefaultStorageType=pvc (#14096) |
✅ Passed | Emits per-workload PVC |
| S4: Duplicate binding | ✅ Passed | Safe silent dedupe |
| S5: Cross-env binding (new validation) | ✅ Passed | Throws InvalidOperationException with clear message |
Overall Result
✅ PR VERIFIED — all 5 scenarios pass on b8c21ba6, including the new cross-environment validation.
Recommendations
- No blocking issues found.
- The error message in S5 is developer-friendly and self-explanatory.
- Ready for merge once required CI checks complete.
- KubernetesPublisherTests.cs: replace bare Assert.DoesNotContain("{}") with
targeted AssertNoBuggyEmptyMappings helper. The bare check would false-positive
on valid Kubernetes YAML shorthand (emptyDir: {}, resources: {}).
- KubernetesPersistentVolumeExtensions.cs: XML doc example now calls
WithDataVolume("pg-data") explicitly. The parameterless form auto-generates
a name via VolumeNameGenerator.Generate, which would not match a PV named
"pg-data", so the documented name-match binding would not fire.
- KubernetesDeployWithPersistentVolumeTests.cs: rewrite the DNS_LABEL comment
to explain the constraint (and cite the Kubernetes docs) instead of implying
a tracking issue exists.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
|
TS code sample? |
|
@davidfowl good call — added a TypeScript example to the PR description right after the C# one, showing the same shape via the auto-generated SDK bindings ( |
Covers WithStorageClass(ParameterResource), WithCapacity(ParameterResource),
and WithVolumeAnnotation(key, ParameterResource) — the overloads that go
through the Helm placeholder / value-capture path. Verifies each parameter
renders as a per-PV-scoped Helm template reference in the PVC manifest
({{ .Values.parameters.<pv-name>.<param> }}) and lands in values.yaml so
consumers can supply the value via --set / values overrides.
The literal-value overloads never exercise the placeholder path, so a
regression there could silently break generated PVC YAML or values.yaml
without any other existing test failing.
Addresses PR review feedback from @davidfowl.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
The mount-path overload of WithPersistentVolume accepted an isReadOnly
parameter but silently dropped it: WithContainerVolumes constructed a
new VolumeMountV1 with only Name/MountPath (ignoring the source
KubernetesResource.Volumes[i].ReadOnly), and the bound-volume branch
in WithPodSpecVolumes set PersistentVolumeClaim without ReadOnly. Net
result: isReadOnly: true never reached the rendered manifest, so the
XML doc promise ("When true, mounts the volume read-only") was unfulfilled.
Now propagate ReadOnly on both emission paths, gated on == true so
existing manifests that never set the flag don't gain a redundant
readOnly: false and churn every snapshot. Cover both the container
volumeMount side and the pod-volume-source side (belt-and-braces:
if a co-scheduled container in the same pod forgets its mount flag,
the PVC source still forces read-only).
Adds PublishAsync_WithFirstClassPersistentVolume_ReadOnlyFlag_PropagatesToVolumeMountAndPodVolume
which YAML-parses the emitted StatefulSet and asserts the exact fields
(not substring matches, so an unrelated readOnly: on a securityContext
can't satisfy the check).
Addresses PR review feedback from copilot-pull-request-reviewer.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tests selector (audit mode)The full test matrix and all jobs still run in audit mode. The tests and jobs below are what selective CI would run under enforcement. 4 / 99 test projects · 4 jobs, from 26 changed files. Selected test projects (4 / 99)
Selected jobs (4)
How these were chosen — grouped by what changed🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🔧 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 🧪 Job reasons
Selection computed for commit |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Documentation was authored and committed locally ( Triggered signals: Documentation written but not pushed:
See the workflow run for details: https://github.com/microsoft/aspire/actions/runs/28758907542 |
Tracking issue: #16999
Summary
Adds first-class app-model support for Kubernetes persistent volumes, plus the underlying YAML serializer bug fixes that the new code path exposes. The change is best read as two layers stacked on top of each other:
KubernetesPersistentVolumeResource— a new first-class resource for describing a volume once in the app model and binding it to workloads.Together they let AppHost authors say "here's a durable disk with this shape, mount it here" once, in real C# / TypeScript, and get correct, cluster-friendly Kubernetes output.
Layer 1 — YAML serializer bug fixes
The serializer is configured with
OmitNull, but several V1 spec types eagerly initialized complex sub-properties tonew(). Non-null empty objects don't get omitted, so we shipped invalid manifests like:The fix, applied uniformly to
PersistentVolumeSpecV1,PersistentVolumeClaimSpecV1,StatefulSetSpecV1,StatefulSetUpdateStrategyV1, andVolumeNodeAffinityV1, is to make those complex sub-properties nullable so the existingOmitNullfilters do the right thing.This matters far more after Layer 2, because auto-promoting bound workloads to
StatefulSet(below) triggers the previously-rare stateful-set code paths on nearly every PV scenario.Closes:
DefaultStorageType = "pvc"produces invalid PV YAMLdataSource/dataSourceRef/selectorblocksLayer 2 — First-class
KubernetesPersistentVolumeResourceToday's
WithVolume("name", "/path")on a container resource leans entirely on the environment-wideDefaultStorageType/DefaultStorageClassName/DefaultStorageSizeknobs. Every volume in the env is forced to share one shape, which doesn't match real clusters where different workloads want different storage classes, sizes, or CSI parameters.A first-class
KubernetesPersistentVolumeResourcemirrors howKubernetesIngressResourceandKubernetesGatewayResourcealready work: the volume is a realIResourceyou configure once, then bind to workloads.New public API surface
All new APIs are marked
[Experimental("ASPIRECOMPUTE002")], matching the package's existing experimental surface.Resource
Carries the storage class, capacity, access modes, and
metadata.annotationsneeded to render a singlev1.PersistentVolumeClaimat publish time.Environment builder
Fluent configuration
Binding workloads to a volume
End-to-end example
The same shape works from a TypeScript polyglot AppHost via the auto-generated SDK bindings —
addPersistentVolume,withStorageClass,withCapacity, and the mount-path overload projected aswithKubernetesPersistentVolumeMount:How it hangs together at publish time
The two paths that touch a bound volume run in different phases of the publish pipeline. To keep them in lockstep, name derivation is centralized on the resource:
WithPodSpecVolumesinResourceExtensions)KubernetesPersistentVolumeBindingAnnotation.PersistentVolumeClaimvolume source on the pod and setclaimName = binding.Volume.GetClaimName().emptydir/hostpath/pvc).ProcessPersistentVolumeResourcesinKubernetesEnvironmentResource)KubernetesPersistentVolumeResource, build aPersistentVolumeClaimwithmetadata.name = volumeResource.GetClaimName(), using the resource's own storage class / capacity / access modes (falling back to the env-wide defaults when not set).GeneratedClaimso the publishing context can emit it into the Helm chart output.KubernetesPublishingContext)KubernetesPersistentVolumeResourcewhoseParent == environment.Because both the pod's
claimNameand the PVC'smetadata.nameroute throughKubernetesPersistentVolumeResource.GetClaimName(), the two phases can never drift.Behavior change: workloads with a bound PV are promoted to
StatefulSetAny workload that binds to a
KubernetesPersistentVolumeResourceis rendered as aStatefulSetregardless of its default workload kind. There is intentionally no opt-out.Why: a
Deploymentwith aReadWriteOncePVC andreplicas > 1is broken by design — the second pod cannot mount the same volume. The whole point of the abstraction is to encode the correct defaults, so binding to a durable volume flips you onto theStatefulSetpath automatically. This is also what turns Layer 1'sStatefulSetserializer fixes from a rare edge case into a routine one.Fallthrough behavior for unbound volumes
Volumes on a workload that do not have a binding continue to use the environment's
DefaultStorageType(emptydir/hostpath/pvc). This is fully covered by tests, so mixing "one first-class PV + one throwaway ephemeral volume" on the same workload works correctly.The
pvcfallthrough path previously emitted a barePersistentVolumealongside itsPersistentVolumeClaim, which was invalid (noPersistentVolumeSource). It now emits only the PVC and lets the cluster'sStorageClassdynamically provision the backing PV, matching the documented Kubernetes model.Issues closed
DefaultStorageType = "pvc"produces invalid PV YAML.dataSource/dataSourceRef/selectorblocks.WithVolumeon aProjectResource.Related work
AddPersistentVolume(...).WithStorageClass(...)emits a PVC and lets the cluster'sStorageClassdynamically provision the PV. The per-workloadWithDynamicProvisioning(bool)API from Added .WithDynamicProvisioning method and default for specifying PV and PVC publishing behavior #16503 does not survive this redesign, but the same scenario is covered.volume.v0discussion (Volume resource types? #1521).Tests
Aspire.Hosting.Kubernetes.Tests(unit / snapshot):StatefulSetpromotion.ProjectResourcemount-path binding.DefaultStorageType = "pvc"andhostpathpaths.Aspire.Cli.EndToEnd.Tests(KinD durability E2E):KubernetesDeployWithPersistentVolumeTests— C# AppHost, container workload, verify data survives pod recreation.KubernetesDeployWithProjectPersistentVolumeTests— C# AppHost, project workload, mount-path overload.KubernetesDeployTypeScriptWithPersistentVolumeTests— TypeScript polyglot AppHost, verifying the polyglot binding path emits equivalent output.All 261
Aspire.Hosting.Kubernetes.Testspass locally; full PR CI is green.