`remember` hangs indefinitely on Windows: `_git_root`'s subprocess timeout is defeated by `subprocess.run`'s post-kill `communicate()` (pipe handle-inheritance race)

[mbe14]

Environment

• Cortex plugin 4.3.0 (Claude Code marketplace install), PostgreSQL backend (pgvector/pg17, Docker)
• Windows 11 Pro (26200), Python 3.13.13
• Client: Claude Code desktop; reproduced on FastMCP 3.2.4 and 3.4.4 (version-independent)

Symptom

Calling the remember tool over a live MCP connection never returns — the call sits 10–30 minutes until the client's idle timeout aborts it (or the user cancels). Meanwhile:

• memory_stats on the same connection responds instantly, before and after the hang.
• The DB write itself sometimes lands (row appears), sometimes not — but the tool response never arrives.
• The auto-capture hooks are unaffected (separate short-lived processes).
• The hang does not reproduce when driving the same server binary out-of-band (raw JSON-RPC over stdio, same env, same protocol version, same arguments, with or without _meta.progressToken) — only on a real long-running client connection.

Diagnosis

I added a temporary watchdog to mcp_server/__main__.py (a daemon thread dumping all thread stacks every 30 s). During a live 10-minute hang it captured the handler wedged here, on every tick:

[thread asyncio_0]
  ...
  tool_error_handler.py:107  _run_coroutine_on_thread → loop.run_until_complete(handler_fn(args))
  handlers/remember.py:345   _handler_impl → _resolve_domain(directory, ...)
  handlers/remember.py:246   _resolve_domain → resolve_cwd(directory)
  shared/domain_mapping.py:404  resolve_cwd → _git_root(cwd)
  shared/domain_mapping.py:236  _git_root → subprocess.check_output(["git", "-C", path, "rev-parse", "--show-toplevel"], stderr=DEVNULL, timeout=3)
  subprocess.py:472  check_output → run(...)
  subprocess.py:565  run → exc.stdout, exc.stderr = process.communicate()     ← inside the TimeoutExpired handler
  subprocess.py:1222 communicate → self._communicate(input, endtime, timeout)
  subprocess.py:1663 _communicate → self.stdout_thread.join(...)              ← blocks forever

[thread Thread-1 (_readerthread)]
  subprocess.py:1615 _readerthread → buffer.append(fh.read())                 ← never sees EOF

Note what this means: the timeout=3 did fire and git was killed. The hang is after that. On Windows, subprocess.run's TimeoutExpired handler calls process.communicate() a second time — without a timeout — to collect post-kill output (CPython 3.13 subprocess.py:565):

except TimeoutExpired as exc:
    process.kill()
    if _mswindows:
        # Windows accumulates the output in a single blocking
        # read() call run on child threads, ...
        exc.stdout, exc.stderr = process.communicate()

That second communicate() joins the stdout reader thread with no deadline. The reader thread is blocked in fh.read() and never sees EOF even though git is dead, because the pipe's write end is still open in another process: with the classic Windows handle-inheritance race, a child spawned concurrently from another thread of the same parent inherits the (temporarily inheritable) stdout pipe write-handle of the git spawn. Cortex spawns other children from the server process (e.g. the AP upstream bridge — CORTEX_MEMORY_AP_ENABLED defaults to 1), so a long-lived sibling child can end up holding the duplicated write handle → EOF never arrives → communicate() blocks forever → the tool never responds.

This also explains every secondary observation:

• memory_stats never touches domain resolution → no git spawn → instant.
• Out-of-band repros don't hang: without the concurrent sibling spawn there is no handle leak, git returns in milliseconds and the timeout never even fires.
• In one instance the pending write flushed exactly when the client reconnected and killed the server — pipe teardown finally unblocked the wedged call.

_get_remote_url (domain_mapping.py:36) has the identical pattern (check_output(["git", ...], timeout=3)) and runs inside _build_registry(), which is also on the first-request path via resolve_cwd. Other subprocess-with-PIPE call sites (git_diff*.py, hooks) share the risk class, though the hooks live in short-lived processes where the damage is bounded.

Fix I applied locally (validated)

Both functions can avoid subprocess entirely, which sidesteps the whole class:

def _git_root(path: str) -> str | None:
    """Find the git repo root for a path. Returns None if not in a repo."""
    try:
        p = Path(path).resolve()
        for candidate in (p, *p.parents):
            if (candidate / ".git").exists():   # dir (normal) or file (worktree)
                return str(candidate).replace("\\", "/")  # match git's forward-slash output
        return None
    except Exception:
        return None


def _get_remote_url(repo_path: Path) -> str:
    """Get git remote origin URL. Returns '' if no remote."""
    try:
        cfg = repo_path / ".git" / "config"
        if not cfg.is_file():
            return ""
        section = ""
        for raw in cfg.read_text(encoding="utf-8", errors="replace").splitlines():
            line = raw.strip()
            if line.startswith("["):
                section = line.lower()
            elif section == '[remote "origin"]' and line.lower().startswith("url"):
                _, _, value = line.partition("=")
                if value.strip():
                    return value.strip()
        return ""
    except Exception:
        return ""

With these two changes (plus CORTEX_MEMORY_AP_ENABLED=0, see below) the hang is gone: remember over the live Claude Code connection returns in ~1 s warm / ~6 s on first call (model load), verified repeatedly.

If you prefer keeping git for correctness in exotic setups (GIT_DIR overrides, bare repos), the minimal alternative is Popen + communicate(timeout=3) + kill() and never calling communicate() again — return None and let the fallback path handle it. But note the pure-Python walk matches rev-parse --show-toplevel for the setups the registry can represent anyway (it only scans .git directories).

Suggestions

1. Replace _git_root / _get_remote_url with the non-subprocess versions (or the kill-without-recollect pattern).
2. Audit the remaining subprocess-with-PIPE call sites reachable from tool handlers on Windows for the same post-timeout communicate() trap.
3. Consider whether the AP upstream spawn can use non-inheritable handles / close_fds-equivalent hygiene, since it is the long-lived sibling that keeps leaked pipe handles alive.

Happy to provide the full watchdog dumps if useful.

[cdeust]

Thank you for an outstanding report — the watchdog stack trace made the mechanism unambiguous, and your validated fix was applied nearly verbatim. Credit for both diagnosis and fix is yours.

Fixed in 68c90835, merged to main (aeef9cc3):

• _git_root now walks up from the path to the first .git (directory, or worktree gitdir: file) in pure Python — no subprocess, no pipes, so the post-timeout communicate() trap is gone by construction.
• _get_remote_url parses .git/config directly, as you suggested — it sat on the same first-request path via _build_registry().

What we verified (macOS): byte-identical outputs vs git rev-parse --show-toplevel / git remote get-url origin, including inside a linked worktree; 14 new tests in tests_py/shared/test_domain_mapping.py (normal repo, subdir, worktree .git file, nested repos, non-repo, missing path, backslash normalization, .git/config parsing edge cases). One note on the worktree edge of _get_remote_url: its only caller filters on .git being a directory before calling it, so the config-file-absent case degrades to "" explicitly rather than mis-resolving.

What we could NOT verify — being honest about it: the hang itself cannot reproduce on macOS by construction (no handle-inheritance race in these conditions). The real proof that remember responds over a live Claude Code connection on Windows has to come from your setup — a confirmation with your watchdog would be very valuable.

Your other suggestions are tracked, not dropped:

• Suggestion 2 (audit of remaining subprocess-with-PIPE sites): done — two sites share this exact risk profile (auto_task_record_writer via the live record_session_end tool, and git_diff_exec). Tracked in Windows: remaining subprocess-with-PIPE call sites reachable from live handlers share the #91 post-timeout communicate() trap #94.
• Suggestion 3 (AP bridge handle hygiene): evaluated but not implemented blindly — CPython's close_fds=True default should already restrict inheritance, so either that doesn't hold for asyncio.create_subprocess_exec on Windows or the race is narrower. Without a Windows machine we can't tell which; tracked in Windows: remaining subprocess-with-PIPE call sites reachable from live handlers share the #91 post-timeout communicate() trap #94 as an investigation item. If you can share the watchdog dumps you mentioned, they would help settle it.
• While in there we also found a pre-existing (non-regression) Windows path-keying bug that defeats the registry fast path: Windows: domain registry fast-path never hits — path_to_repo keyed with backslash paths while _git_root returns forward slashes #93.

The fix ships in the next plugin release; until then it is on main.

[cdeust]

Released in v4.4.0 (tag v4.4.0, main @ 82ee1e6c) — see the CHANGELOG entry crediting your report and fix. A confirmation on your Windows setup (ideally with your watchdog) would let us close this.

[mbe14]

Confirmed fixed on the original box — Windows 11 Pro (26200), Python 3.13.13, plugin updated to v4.5.0 through the marketplace.

Test conditions were deliberately the hostile ones from the report: pristine upstream code on the #91 paths (no local patches) and CORTEX_MEMORY_AP_ENABLED=1 (now the explicit default in plugin.json).

Results over a live Claude Code connection:

• First remember on a freshly spawned server: full round trip, response received. The client's own debug log times it: Tool 'remember' completed successfully in 3s (force-store exercising the whole write pipeline); a gate-rejected remember returned in about a second. Repeated calls stay instant. Previously this sat 10–30 minutes.
• Watchdog: same faulthandler.dump_traceback_later instrumentation as the original diagnosis, armed at the top of mcp_server/__main__.py. All-thread dumps during a held-open run of the same build show only healthy stacks — the anyio worker in run and the proactor loop idle in windows_events.py:_poll / run_forever. No communicate() / stdout_thread.join() frames anywhere. (One logging note: Claude Code's MCP jsonl only retains stderr around startup, so the periodic dumps were captured by driving the same build out-of-band with stdin held open.)

_git_root / _get_remote_url outputs also spot-checked on this box: repo root and origin URL resolve identically to the git CLI, including with a backslash input path.

As far as I'm concerned this can be closed. The #92 first-call scenario passed on the same fresh server (confirming there separately), and the remaining AP-bridge question has new data in #94.

[cdeust]

Confirmed fixed on the reporter's original box with v4.5.0, exercised under the original hostile conditions (AP bridge enabled, watchdog armed — no communicate()/stdout_thread.join frames anywhere). Thanks again @mbe14 for a model bug report: diagnosis, validated fix, and verification all came from you. Closing.