Skip to content

[codex] Centralize Plugin Analytics Metadata#27102

Merged
jameswt-oai merged 5 commits into
mainfrom
jameswt/plugin-telemetry-metadata-refactor
Jun 22, 2026
Merged

[codex] Centralize Plugin Analytics Metadata#27102
jameswt-oai merged 5 commits into
mainfrom
jameswt/plugin-telemetry-metadata-refactor

Conversation

@jameswt-oai

@jameswt-oai jameswt-oai commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

This PR moves construction of PluginTelemetryMetadata from loader and model helpers into PluginsManager, which already owns installed plugin state and will eventually perform remote identity enrichment. The metadata type remains in codex-plugin, and serialized analytics events remain unchanged.

Before

flowchart LR
    subgraph Events["Analytics event paths"]
        direction TB
        Lifecycle["Local install / uninstall"]
        Config["Enable / disable"]
        Remote["Remote install"]
        Used["Plugin used"]
    end

    subgraph Construction["Metadata construction"]
        direction TB
        Loader["Loader telemetry helpers"]
        Summary["PluginCapabilitySummary::telemetry_metadata"]
        Override["Caller adds remote_plugin_id"]
    end

    Metadata["PluginTelemetryMetadata"]

    Lifecycle --> Loader
    Config --> Loader
    Remote --> Loader
    Loader -->|"local events"| Metadata
    Loader -->|"remote install"| Override
    Override --> Metadata
    Used --> Summary
    Summary --> Metadata
Loading

Telemetry metadata was constructed through loader helpers, a capability-summary method, and a remote-install call-site override.

After

flowchart LR
    subgraph Events["Analytics event paths"]
        direction TB
        Lifecycle["Local install / uninstall"]
        Config["Enable / disable"]
        Remote["Remote install"]
        Used["Plugin used"]
    end

    Manager["PluginsManager — single construction owner"]
    Metadata["PluginTelemetryMetadata"]

    Lifecycle --> Manager
    Config --> Manager
    Remote -->|"authoritative remote ID"| Manager
    Used -->|"capability summary"| Manager
    Manager --> Metadata
Loading

Every analytics path delegates metadata construction to PluginsManager. Remote install still supplies its authoritative backend ID explicitly.

What Changes

  • Make loader code return a focused plugin capability summary instead of constructing analytics metadata.
  • Centralize immutable plugin telemetry metadata construction in PluginsManager.
  • Route local install/uninstall, remote install, enable/disable, and plugin-used emitters through the manager.
  • Preserve the current serialized analytics contract exactly.

Normal metadata still has no remote override. Remote install continues to provide its authoritative backend ID explicitly, so the existing serializer continues reporting that ID through plugin_id. Snapshot-based enrichment is intentionally deferred to the final PR.

Testing

  • just test -p codex-core-plugins (238 tests passed)
  • just test -p codex-plugin (3 tests passed)
  • Scoped Clippy/compile checks passed for codex-plugin, codex-core-plugins, codex-app-server, and codex-core.

Split Overview

main
├── #27093  Debug analytics capture                 (merged)
├── #27099  Non-mutating plugin smoke               (merged)
├── #27100  Remote install/uninstall smoke          (merged)
└── #27102  Plugin telemetry metadata refactor      ← you are here
    └── #27669  Persist remote plugin identity

After #27102 and #27669 merge:
└── Final PR: add explicit local and remote IDs to plugin analytics

Review order and dependencies:

  1. #27093 Add debug-only analytics event capture (merged)
  2. #27099 Add a plugin analytics smoke workflow (merged)
  3. #27100 Add a remote plugin analytics mutation smoke workflow (merged)
  4. This metadata refactor, independent and based on main
  5. #27669 Persist remote plugin identity, stacked on this PR
  6. Final remote-ID behavior PR, created after the prerequisites merge

The original #26281 remains open as the aggregate reference until the final replacement PR is published.

@jameswt-oai jameswt-oai marked this pull request as ready for review June 9, 2026 01:31
@jameswt-oai jameswt-oai requested a review from a team as a code owner June 9, 2026 01:31
@jameswt-oai jameswt-oai changed the title Centralize plugin telemetry metadata construction [codex] Centralize plugin telemetry metadata construction Jun 11, 2026
…try-metadata-refactor

# Conflicts:
#	codex-rs/app-server/src/request_processors/plugins.rs
#	codex-rs/core-plugins/src/loader.rs
#	codex-rs/core-plugins/src/manager.rs
@jameswt-oai jameswt-oai changed the title [codex] Centralize plugin telemetry metadata construction [codex] Centralize Plugin Analytics Metadata Jun 11, 2026
jameswt-oai added a commit that referenced this pull request Jun 15, 2026
## This PR

The original [combined remote plugin analytics PR
#26281](#26281) mixed reusable
analytics test infrastructure, two manual smoke workflows, a metadata
refactor, and the final identity behavior. This PR isolates the generic
capture mechanism so it can be reviewed and landed before any
plugin-specific behavior.

- Add a debug-only analytics destination that writes final request
payloads as JSONL.
- Suppress HTTP delivery whenever capture mode is selected, including
after capture write failures.
- Keep release behavior unchanged even when the capture environment
variable is present.
- Keep the mechanism generic; this PR contains no plugin-specific
behavior.

Set `CODEX_ANALYTICS_EVENTS_CAPTURE_FILE=/path/events.jsonl` when
running a debug Codex binary to inspect the exact batched payload that
would otherwise be sent to the analytics endpoint.

## Testing

- `just test -p codex-analytics` (76 passed)
- `just test --release -p codex-analytics` (73 passed)
- CI is green across the required platform matrix.

## Split Overview

```text
main
├── #27093  Debug analytics capture                 ← you are here
│   └── #27099  Non-mutating plugin smoke
│       └── #27100  Remote install/uninstall smoke
└── #27102  Plugin telemetry metadata refactor

After #27093, #27099, #27100, and #27102 merge:
└── Final PR: add remote_plugin_id to plugin analytics
```

Review order and dependencies:

1. [#27093 Add debug-only analytics event
capture](#27093) **(this PR, based
on `main`)**
2. [#27099 Add a plugin analytics smoke
workflow](#27099) (stacked on
#27093)
3. [#27100 Add a remote plugin analytics mutation smoke
workflow](#27100) (stacked on
#27099)
4. [#27102 Centralize plugin telemetry metadata
construction](#27102) (independent,
based on `main`)
5. Final remote-ID behavior PR (created after PRs 1-4 merge)

The original [#26281](#26281)
remains open as the green aggregate reference until the final PR is
published.
jameswt-oai added a commit that referenced this pull request Jun 16, 2026
…ics Smoke Test (#27099)

## This PR

The original [combined remote plugin analytics PR
#26281](#26281) mixed reusable
analytics test infrastructure, two manual smoke workflows, a metadata
refactor, and the final identity behavior. This PR establishes a
non-mutating end-to-end plugin smoke workflow before any analytics
identity semantics change.

- Add `plugin-analytics-smoke` to the existing app-server test client.
- Exercise plugin disable, enable, and use through production app-server
RPC paths.
- Isolate config writes in a temporary file and use a loopback Responses
API server.
- Capture analytics without sending them to the production analytics
backend.
- Validate the current local `plugin_id`, names, capability metadata,
thread, turn, and model fields.

This is intentionally a baseline smoke workflow. It does not assert
`remote_plugin_id`; the final PR will update it when that field exists.
Review this PR as the net diff against #27093.

## Testing

- The test-client target compiles successfully.
- The combined reference branch exercised the manual smoke against the
live remote plugin service.
- CI is green across the required platform matrix.

## Split Overview

```text
main
├── #27093  Debug analytics capture
│   └── #27099  Non-mutating plugin smoke           ← you are here
│       └── #27100  Remote install/uninstall smoke
└── #27102  Plugin telemetry metadata refactor

After #27093, #27099, #27100, and #27102 merge:
└── Final PR: add remote_plugin_id to plugin analytics
```

Review order and dependencies:

1. [#27093 Add debug-only analytics event
capture](#27093) (based on `main`)
2. [#27099 Add a plugin analytics smoke
workflow](#27099) **(this PR,
stacked on #27093)**
3. [#27100 Add a remote plugin analytics mutation smoke
workflow](#27100) (stacked on this
PR)
4. [#27102 Centralize plugin telemetry metadata
construction](#27102) (independent,
based on `main`)
5. Final remote-ID behavior PR (created after PRs 1-4 merge)

The original [#26281](#26281)
remains open as the green aggregate reference until the final PR is
published.
jameswt-oai added a commit that referenced this pull request Jun 16, 2026
…ke Test (#27100)

## This PR

The original [combined remote plugin analytics PR
#26281](#26281) mixed reusable
analytics test infrastructure, two manual smoke workflows, a metadata
refactor, and the final identity behavior. This PR adds the
account-mutating validation workflow separately so its cleanup and
recovery guarantees can be reviewed without the final analytics behavior
change.

- Add a manually invoked remote plugin install/uninstall smoke workflow.
- Require explicit account-mutation confirmation and an initially
uninstalled plugin.
- Validate the current `codex_plugin_installed` contract, where
`plugin_id` is the backend ID.
- Restore and verify the original uninstalled state, with a dedicated
recovery command.

This baseline intentionally does not require `codex_plugin_uninstalled`,
because production does not emit that event yet. The final PR will
update this smoke to require local `plugin_id`, `remote_plugin_id`, and
uninstall emission. Review this PR as the net diff against #27099.

## Testing

- `just test -p codex-app-server-test-client` (3 focused
capture/validation tests passed)
- The live workflow was previously exercised on the green combined
reference branch, and the original uninstalled account state was
restored.
- CI is green across the required platform matrix.

## Split Overview

```text
main
├── #27093  Debug analytics capture
│   └── #27099  Non-mutating plugin smoke
│       └── #27100  Remote install/uninstall smoke  ← you are here
└── #27102  Plugin telemetry metadata refactor

After #27093, #27099, #27100, and #27102 merge:
└── Final PR: add remote_plugin_id to plugin analytics
```

Review order and dependencies:

1. [#27093 Add debug-only analytics event
capture](#27093) (based on `main`)
2. [#27099 Add a plugin analytics smoke
workflow](#27099) (stacked on
#27093)
3. [#27100 Add a remote plugin analytics mutation smoke
workflow](#27100) **(this PR,
stacked on #27099)**
4. [#27102 Centralize plugin telemetry metadata
construction](#27102) (independent,
based on `main`)
5. Final remote-ID behavior PR (created after PRs 1-4 merge)

The original [#26281](#26281)
remains open as the green aggregate reference until the final PR is
published.
…try-metadata-refactor

# Conflicts:
#	codex-rs/core-plugins/src/loader.rs
…try-metadata-refactor

# Conflicts:
#	codex-rs/core-plugins/src/manager.rs
@jameswt-oai jameswt-oai merged commit 44dbae9 into main Jun 22, 2026
31 checks passed
@jameswt-oai jameswt-oai deleted the jameswt/plugin-telemetry-metadata-refactor branch June 22, 2026 17:27
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants