fix(deploy,sync): ensure canister is Running before asset sync

[lwshang]

Problem

icp deploy/icp sync against the asset canister intermittently failed at the sync step, when the sync plugin's first canister call reached the target while it was not serving:

... Canister <id> is stopped and therefore does not have a CallContextManager, error code Some("IC0508")

Investigation found two distinct failure modes:

• Mode A — canister genuinely not Running before sync. install_code is status-preserving, so a canister that enters the command already Stopped/Stopping (handed out Stopped from a canister pool, or left Stopped by a deploy that died in the stop→install→start window) is installed but never started, and the plugin's first call hits IC0508. Persistent — reruns don't help. (Why developer-docs preview deploys failed repeatedly.)
• Mode B — query path transiently stale right after a restart. When the canister was Running, deploy correctly does stop→upgrade→start, but start_canister only makes it Running in the subnet's certified state. IC query calls are eventually-consistent reads served by a single replica that may still lag the restart's commit height; the plugin's first calls are queries, so they can momentarily observe the just-vacated Stopped state. Transient — a rerun "fixes" it. (Why the always-running docs canister failed intermittently.)

Fix

Mode A — commit e0797fe3:

• icp deploy owns the lifecycle (it just installed): after install and before sync it starts every sync-target canister, then leaves it Running. start_canister is idempotent and synchronous per the IC spec, so no status poll is needed. Scoped to canisters with sync steps.
• icp sync does NOT own the lifecycle: it prechecks run status and aborts with an actionable error if not Running, replacing the cryptic IC0508. It does not auto-start.
• install.rs is deliberately unchanged — it stays status-preserving.

Mode B — commit eb367aa2:

• In icp deploy, after starting the canister and before sync, wait until the query path consistently observes the canister Running. Probe with a query for a method no canister exports and classify the reject: "is stopped/stopping" (IC0508/IC0509) means the replica still lags; any other reject (e.g. "no query method") or a reply means it is serving. Require a few consecutive ready observations, spaced so they may land on different replicas.
• Confidence-raising, not a hard guarantee — query reads are per-node and boundary nodes load-balance across replicas, an inherent property of the decentralized read path — but it makes the post-restart race rare.

Tests

• deploy_starts_stopped_canister_before_sync — deploy, stop, redeploy; deploy starts the canister before the plugin runs and ends Running with synced data. Reproduces Mode A pre-fix.
• sync_aborts_when_canister_not_running — icp sync against a stopped canister aborts with the actionable message; the sync step never runs.

Mode B is a timing/replica-lag race that a single-node local replica doesn't exhibit, so it has no e2e test. cargo build, cargo clippy --tests, cargo fmt --check clean.

Out of scope (follow-ups)

• Clearer plugin-side error than raw IC0508.
• Whether resolve_install_mode_and_status should repair a Stopping canister.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR hardens icp deploy/icp sync asset synchronization by ensuring the target canister is Running (and query-serving) before sync operations begin, preventing intermittent IC0508 failures during the sync plugin’s first calls.

Changes:

• icp sync: prechecks canister status and aborts early with an actionable error if the canister is not Running.
• icp deploy: starts any sync-target canisters post-install and waits until the query path consistently observes them as Running before running sync steps.
• Adds e2e regression tests covering deploy-starts-before-sync and sync-aborts-when-stopped.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
crates/icp-cli/src/commands/sync.rs	Adds a canister-status precheck to abort sync early when the canister isn’t Running.
crates/icp-cli/src/commands/deploy.rs	Starts sync-target canisters after install and adds a query-path readiness wait to reduce post-restart IC0508 races.
crates/icp-cli/tests/sync_tests.rs	Adds a test ensuring icp sync aborts (and does not run sync steps) when the canister is stopped.
crates/icp-cli/tests/deploy_tests.rs	Adds a regression test ensuring icp deploy starts a stopped canister before running plugin-based sync.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.