Skip to content

fix(push): prevent hang and tag race in apify push (#1131)#1134

Open
l2ysho wants to merge 18 commits into
masterfrom
1131-apify-push-can-hang-indefinitely-and-can-exit-0-before-the-latest-tag-is-applied
Open

fix(push): prevent hang and tag race in apify push (#1131)#1134
l2ysho wants to merge 18 commits into
masterfrom
1131-apify-push-can-hang-indefinitely-and-can-exit-0-before-the-latest-tag-is-applied

Conversation

@l2ysho
Copy link
Copy Markdown
Contributor

@l2ysho l2ysho commented May 14, 2026

Fixes #1131.

apify push had two reliability issues:

  1. Hang after build stream ends. If the build log stream closed before the build reached a terminal status, push waited forever. Now polls build
    status with a cap from --wait-for-finish (defaults to no cap when unset, but exits cleanly on terminal status).
  2. Tag race. Push could exit 0 before the latest tag was applied, so a following actor.start({ build: 'latest' }) would race. Now polls for the
    tag to point at the new build (30s budget) before exiting.

Also:

  • parseWaitForFinishMillis extracted and unit-tested; treats negative/NaN as "no cap".
  • --wait-for-finish 0 now means "don't wait" (was previously equivalent to unset).
  • Exit code is BuildTimedOut when build is still READY/RUNNING at the cap.
  • using _signalHandler moved to cover the polling loops, so Ctrl+C aborts the build platform-side.

@l2ysho l2ysho self-assigned this May 14, 2026
@l2ysho l2ysho added the t-dx Issues owned by the DX team. label May 14, 2026
@github-actions github-actions Bot added this to the 140th sprint - Tooling team milestone May 14, 2026
@github-actions github-actions Bot added the tested Temporary label used only programatically for some analytics. label May 14, 2026
Resolves an issue where waitForFinishMillis could be NaN for invalid flag inputs, leading to unexpected behavior.
@l2ysho l2ysho force-pushed the 1131-apify-push-can-hang-indefinitely-and-can-exit-0-before-the-latest-tag-is-applied branch from fa017eb to 07e801a Compare May 14, 2026 22:00
@l2ysho l2ysho marked this pull request as ready for review May 18, 2026 12:11
@l2ysho l2ysho requested a review from vladfrangu as a code owner May 18, 2026 12:11
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this test is kinda useless :D

Comment thread src/commands/actors/push.ts Outdated
// @ts-expect-error FIX THESE TYPES 😢
} else if (build.status === ACTOR_JOB_STATUSES.READY) {
warning({ message: 'Build is waiting for allocation.' });
process.exitCode = CommandExitCodes.BuildTimedOut;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is technically wrong, if I pass in apify push --waitForFinish=5 it should not give me a non-0 exit code when the timer ran out

Comment thread src/commands/actors/push.ts Outdated
// ended early, timeout hit). Poll so the status branches below see the
// real outcome. With no --wait-for-finish, the flag documents "waits
// forever", so poll without a deadline.
const deadline = waitForFinishMillis === undefined ? Infinity : Date.now() + waitForFinishMillis;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the deadline should be computed right before starting the build, not after waiting for the outputLog to finalizr/crash.

also, won't this cause a double-wait? Imagine an actor that takes 4minutes to build, if --waitForFinish is set to 30s, we print for 29s then connection dies for the log stream, we wait another 30s after? Ideally we calculate a delta between build start, and log end, and wait out the remaining time, if any, right?

Comment thread src/commands/actors/push.ts Outdated
if (build.status === ACTOR_JOB_STATUSES.SUCCEEDED && buildTag) {
// 30s budget is independent of --wait-for-finish: the build is already
// done, we're only waiting on the platform to update the tag pointer.
const tagDeadline = Date.now() + 30_000;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

30s is wayy too much, i'd bail out after 5 max. Also lets print out a message that we are "applying the build tag" (even tho the console does it) just so users know why their terminal is hanging

Comment thread src/lib/utils.ts
export function parseWaitForFinishMillis(flag: string | undefined): number | undefined {
if (flag === undefined) return undefined;
const parsed = Number.parseInt(flag, 10);
if (!Number.isFinite(parsed) || parsed < 0) return undefined;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mmmm, can you compare what happens right now with --waitForFinish=0 vs this PR? It's a use case we should support imo (fire-and-forget)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in actual version it is ignored, it just show you whole build log to the end. In PR it ends with

Actor build detail https://console.apify.com/actors/k3Ew4ttdaIThULhmR#/builds/0.25.6
Actor detail https://console.apify.com/actors/k3Ew4ttdaIThULhmR
Warning: Build is still running.

l2ysho added 5 commits May 25, 2026 16:30
When --wait-for-finish is passed explicitly, hitting the cap with the
build still in READY/RUNNING is the user's chosen bounded wait, not a
failure. Only surface BuildTimedOut for these states when the flag was
omitted.
Compute a single deadline before the build call and share it between log
streaming and the status poll loop. Previously the poll loop started a
fresh --wait-for-finish budget after outputJobLog returned, so a log
stream that died near the cap could double the user's wait.
Cut the post-build tag-apply budget from 30s to 5s, print a status line
so users see why push is still waiting, and drop the non-zero exit code
on tag-miss — the build itself succeeded, a lagging tag pointer is a
warning, not a failure.
Copy link
Copy Markdown
Contributor

@patrikbraborec patrikbraborec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the current behavior is better, if I understand it correctly:

After the build log stream ends, push now polls the build's actual status until it reaches a
terminal state, instead of trusting the stream (which could close early and leave push waiting forever).

Also, this will help with follow ups: #1135

LGTM

Skip the post-build tag-apply wait when --wait-for-finish=0 so the flag
is truly fire-and-forget regardless of how fast the build completed.
Document the 0 semantics in the flag help text.

Also picks up stale doc regeneration from #1128 (--describe / --search
wording).
@l2ysho l2ysho requested a review from szaganek as a code owner May 27, 2026 08:46
Comment thread src/commands/actors/push.ts Outdated
l2ysho and others added 2 commits May 27, 2026 09:13
The previous guard was unreachable: when --wait-for-finish is unset, the
poll loop only exits on terminal status, so READY/RUNNING branches were
dead. Switch the guard to `waitForFinishMillis !== 0` so finite caps that
hit the deadline surface a non-zero exit code (useful for scripts), while
fire-and-forget (--wait-for-finish=0) still returns 0 on READY.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Edyta <142720610+szaganek@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-dx Issues owned by the DX team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apify push can hang indefinitely and can exit 0 before the latest tag is applied

5 participants