Fix AKS deployment test timeouts and ACR token expiration#14591
Merged
Conversation
- Increase az aks update --attach-acr timeout from 3 to 5 minutes (ReconcilingAddons phase can take several minutes) - Increase kubectl wait pod readiness timeout from 120s to 300s (pods need time to pull images from ACR and start) - Add ACR re-login step after AKS creation to refresh Docker credentials that may have expired during 10-15 min provisioning (OIDC federated tokens expire after ~5 minutes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/deployment-test |
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 14591Or
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 14591" |
Contributor
|
🚀 Deployment tests starting on PR #14591... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes intermittent test failures in AKS deployment end-to-end tests by addressing three root causes: insufficient timeouts for AKS-ACR integration updates, insufficient pod readiness wait times, and expired ACR authentication tokens during long-running AKS provisioning.
Changes:
- Increased
az aks update --attach-acrtimeout from 3 to 5 minutes to accommodate the ReconcilingAddons phase - Increased pod readiness wait timeout from 120s to 300s (and corresponding WaitForSuccessPrompt from 3 to 6 minutes)
- Added ACR re-login step after AKS provisioning to refresh Docker credentials that expire during the 10-15 minute AKS creation
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| tests/Aspire.Deployment.EndToEnd.Tests/AksStarterDeploymentTests.cs | Applied timeout increases and ACR re-login for basic AKS starter deployment test |
| tests/Aspire.Deployment.EndToEnd.Tests/AksStarterWithRedisDeploymentTests.cs | Applied identical timeout increases and ACR re-login for AKS starter with Redis deployment test |
Contributor
🎬 CLI E2E Test RecordingsThe following terminal recordings are available for commit
📹 Recordings uploaded automatically from CI run #22292193918 |
Contributor
|
✅ Deployment E2E Tests passed Summary: 21 passed, 0 failed, 0 cancelled Passed Tests
🎬 Terminal Recordings
|
Member
Author
|
/deployment-test |
Contributor
|
🚀 Deployment tests starting on PR #14591... This will deploy to real Azure infrastructure. Results will be posted here when complete. |
Contributor
|
✅ Deployment E2E Tests passed Summary: 21 passed, 0 failed, 0 cancelled Passed Tests
🎬 Terminal Recordings
|
Copilot AI
pushed a commit
that referenced
this pull request
Mar 10, 2026
* Fix AKS deployment test timeouts and ACR token expiration - Increase az aks update --attach-acr timeout from 3 to 5 minutes (ReconcilingAddons phase can take several minutes) - Increase kubectl wait pod readiness timeout from 120s to 300s (pods need time to pull images from ACR and start) - Add ACR re-login step after AKS creation to refresh Docker credentials that may have expired during 10-15 min provisioning (OIDC federated tokens expire after ~5 minutes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix WaitForSuccessPrompt timeout for chained kubectl waits The Redis test chains 3 kubectl wait commands (300s each), so the worst-case total is 900s (15 min). Increase WaitForSuccessPrompt from 6 min to 16 min to provide adequate buffer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Mitch Denny <mitch@mitchdeny.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes intermittent failures in
AksStarterDeploymentTestsandAksStarterWithRedisDeploymentTests.Root Causes Identified
az aks update --attach-acrtimeout too short (3 min): The ReconcilingAddons phase after updating ACR role assignments can take several minutes. Increased to 5 minutes.Pod readiness timeout too short (120s): After Helm deployment, pods need time to pull images from ACR and start. Increased
kubectl waittimeout from 120s to 300s.ACR Docker credentials expire during AKS provisioning: The initial ACR login (Step 4b) happens before AKS creation which takes 10-15 minutes. OIDC federated tokens expire after ~5 minutes, so Docker credentials in
~/.docker/config.jsonmay be stale by the time images are pushed. Added a re-login step after AKS provisioning completes.Changes
az aks updatetimeout from 3→5 minutesEvidence
Recent failures from workflow run #22208855323:
AksStarterDeploymentTests:kubectl waittimed out - pods not ready within 120sAksStarterWithRedisDeploymentTests:WaitUntil timed out after 00:03:00onaz aks update --attach-acr