Skip to content

Rootless artifact permission repair never runs: agent image ENTRYPOINT hijacks the repair command and adds an exact 30 s failure per directory #6334

Description

@mavam

Versions: AWF v0.27.35 (latest; also reproduced by inspection on main), gh-aw v0.82.11 workflow lock, Copilot CLI 1.0.70, GitHub-hosted ubuntu-24.04 runner, rootless network-isolation topology mode (--rootless, network.isolation: true).

Observed

After the agent completes and containers stop, cleanup emits three warnings, each preceded by an exactly ~30.5 s stall (total ~92 s of idle time per run):

06:59:46.394 [INFO] Audit artifacts available at: /tmp/gh-aw/sandbox/firewall/audit
07:00:16.952 [WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/logs (exit 1)
07:00:47.521 [WARN] Rootless artifact permission repair failed for /tmp/gh-aw/sandbox/firewall/audit (exit 1)
07:01:18.069 [WARN] Rootless artifact permission repair failed for /tmp/awf-1784271472136-chroot-home (exit 1)

Note there is no stderr detail even though #6072 added stderr capture. This is not B10 (#6069/#6025): the run already used the tag-only image ref, and agent:0.27.35 was tagged locally by the pre-download step, so no registry I/O occurred.

Root cause

fixArtifactPermissionsForRootless() (src/artifact-permissions.ts) runs docker run ... <registry>/agent:<tag> sh -c 'chown -R "$TUID:$TGID" /fix 2>/dev/null; chmod -R a+rwX /fix' without --entrypoint. The agent image declares ENTRYPOINT ["/usr/local/bin/entrypoint.sh"] (containers/agent/Dockerfile), so sh -c ... becomes arguments to the entrypoint instead of the container command. The repair container's environment contains only TUID/TGID, so in entrypoint.sh wait_for_iptables() the AWF_NETWORK_ISOLATION guard is unset and it polls for /tmp/awf-init/ready for INIT_TIMEOUT=300 iterations of 0.1 s (exactly 30 s), then exit 1. All its output goes to stdout, which is why the warning has no stderr.

The chown/chmod never executes, so the repair has been a silent 30-second-per-directory no-op since the entrypoint gained the iptables-init handshake. Even if AWF_NETWORK_ISOLATION=1 were injected, the entrypoint would drop privileges to awfuser via gosu before running the command, so the chown -R could never succeed through the entrypoint path.

Reproduction

On any host with docker, as a non-root user:

docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.35
mkdir -p /tmp/fixme
time docker run --rm --pull never --network none \
  --cap-drop ALL --cap-add CHOWN --cap-add DAC_OVERRIDE --cap-add FOWNER \
  -e TUID=$(id -u) -e TGID=$(id -g) -v /tmp/fixme:/fix:rw \
  ghcr.io/github/gh-aw-firewall/agent:0.27.35 \
  sh -c 'chown -R "$TUID:$TGID" /fix 2>/dev/null; chmod -R a+rwX /fix'

This prints "[entrypoint] Waiting for iptables initialization from init container...", stalls 30 s, prints "[ERROR] Timed out waiting for iptables init container after 30s", and exits 1. Adding --entrypoint sh (and dropping the extra sh argument) completes in under a second.

Any gh-aw workflow using the Copilot engine with the default rootless network-isolation sandbox on a GitHub-hosted runner reproduces it in CI (example: tenzir/content run 29561530034, job 87824891997).

Suggested remediation

In fixArtifactPermissionsForRootless(), bypass the agent entrypoint so the repair runs as container root with the granted CHOWN/DAC_OVERRIDE/FOWNER caps: add --entrypoint sh and pass ['-c', 'chown -R "$TUID:$TGID" /fix 2>/dev/null; chmod -R a+rwX /fix'] as the command (or use a minimal dedicated image). Consider also failing the repair fast and surfacing entrypoint stdout in the warning, and extending the B10 catalog entry, since the "exactly ~30 s per directory" signature persists post-#6025 via this second mechanism.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions