feat: add PostHog telemetry for wrapper usage tracking by mldangelo · Pull Request #19 · promptfoo/promptfoo-python

mldangelo · 2026-01-11T06:44:43Z

Summary

Add anonymous telemetry to track Python wrapper usage patterns
Match telemetry implementation in the main TypeScript promptfoo project
Track wrapper_used event with execution method (global, npx, or error)

Details

Event properties tracked:

method - How promptfoo was invoked (global install vs npx fallback)
wrapperType - Always "python"
packageVersion - Python wrapper version
pythonVersion - Python runtime version
platform - OS platform (darwin, linux, win32)
isRunningInCi - CI environment detection

Shared infrastructure:

Same PostHog endpoint as TypeScript (https://a.promptfoo.app)
Same user ID from ~/.promptfoo/promptfoo.yaml
Same opt-out mechanism (PROMPTFOO_DISABLE_TELEMETRY=1)

Dependencies added:

posthog>=3.0.0 - PostHog Python SDK
pyyaml>=6.0.0 - YAML parsing for config file

Test plan

All 106 existing tests pass
Linting (ruff) passes
Type checking (mypy) passes
Manual verification: telemetry disabled when PROMPTFOO_DISABLE_TELEMETRY=1
Manual verification: telemetry disabled when IS_TESTING=1

🤖 Generated with Claude Code

Add anonymous telemetry to track Python wrapper usage patterns, matching the telemetry implementation in the main TypeScript project. - Track `wrapper_used` event with method (global/npx/error) - Share user ID with TypeScript version (~/.promptfoo/promptfoo.yaml) - Use same PostHog endpoint (https://a.promptfoo.app) - Respect PROMPTFOO_DISABLE_TELEMETRY=1 opt-out - Non-blocking, error-swallowing design Dependencies added: posthog, pyyaml Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR adds anonymous telemetry tracking to the Python wrapper using PostHog analytics, matching the implementation in the main TypeScript promptfoo project. The telemetry tracks wrapper usage patterns to help improve the product while respecting user privacy through opt-out mechanisms.

Changes:

Implemented PostHog-based telemetry system with wrapper_used event tracking
Added privacy controls via PROMPTFOO_DISABLE_TELEMETRY and IS_TESTING environment variables
Integrated telemetry calls into CLI execution paths to track usage method (global, npx, or error)

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

File	Description
src/promptfoo/telemetry.py	New module implementing PostHog telemetry with user ID management, CI detection, and event recording
src/promptfoo/cli.py	Added telemetry tracking calls at three execution points to record wrapper usage method
pyproject.toml	Added posthog and pyyaml dependencies, plus types-pyyaml for development
uv.lock	Dependency lock file updates for posthog (with backoff, distro, requests, etc.) and pyyaml packages

Comments suppressed due to low confidence (1)

src/promptfoo/cli.py:205

The error path records telemetry with method="error" after failing to find promptfoo or npx, but this telemetry call happens before the error messages are printed. If the telemetry call somehow delays or interferes with the error output, it could impact user experience. Consider moving the telemetry call after the error messages are printed, or ensure it's non-blocking and completes quickly.

        else:
            record_wrapper_used("error")
            print("ERROR: Neither promptfoo nor npx is available.", file=sys.stderr)
            print("Please install promptfoo: npm install -g promptfoo", file=sys.stderr)
            print("Or ensure Node.js is properly installed.", file=sys.stderr)
            sys.exit(1)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-11T06:54:43Z

src/promptfoo/telemetry.py

+            with open(config_file) as f:
+                config = yaml.safe_load(f)
+                return config if isinstance(config, dict) else {}
+        except Exception:


The _read_global_config function catches all exceptions silently and returns an empty dict. This could hide legitimate errors like permission issues or corrupted YAML files. Consider logging these errors (if a logging mechanism is available) or at least distinguishing between "file doesn't exist" (expected) and "file exists but can't be read" (unexpected error that might warrant user notification).

Suggested change

except Exception:

except (OSError, yaml.YAMLError) as exc:

# Log unexpected errors but do not let telemetry break the CLI

sys.stderr.write(

f"Warning: unable to read telemetry config file {config_file}: {exc}\n"

)

Copilot · 2026-01-11T06:54:44Z

src/promptfoo/telemetry.py

+"""
+Telemetry module for the promptfoo Python wrapper.
+
+Sends anonymous usage analytics to PostHog to help improve promptfoo.
+Telemetry can be disabled by setting PROMPTFOO_DISABLE_TELEMETRY=1.
+"""
+
+import atexit
+import os
+import platform
+import sys
+import uuid
+from pathlib import Path
+from typing import Any, Optional
+
+import yaml
+from posthog import Posthog
+
+from . import __version__
+
+# PostHog configuration - same as the main promptfoo TypeScript project
+_POSTHOG_HOST = "https://a.promptfoo.app"
+_POSTHOG_KEY = "phc_E5n5uHnDo2eREJL1uqX1cIlbkoRby4yFWt3V94HqRRg"
+
+
+def _get_env_bool(name: str) -> bool:
+    """Check if an environment variable is set to a truthy value."""
+    value = os.environ.get(name, "").lower()
+    return value in ("1", "true", "yes", "on")
+
+
+def _is_ci() -> bool:
+    """Detect if running in a CI environment."""
+    ci_env_vars = [
+        "CI",
+        "CONTINUOUS_INTEGRATION",
+        "GITHUB_ACTIONS",
+        "GITLAB_CI",
+        "CIRCLECI",
+        "TRAVIS",
+        "JENKINS_URL",
+        "BUILDKITE",
+        "TEAMCITY_VERSION",
+        "TF_BUILD",  # Azure Pipelines
+    ]
+    return any(os.environ.get(var) for var in ci_env_vars)
+
+
+def _get_config_dir() -> Path:
+    """Get the promptfoo config directory path."""
+    return Path.home() / ".promptfoo"
+
+
+def _read_global_config() -> dict[str, Any]:
+    """Read the global promptfoo config from ~/.promptfoo/promptfoo.yaml."""
+    config_file = _get_config_dir() / "promptfoo.yaml"
+    if config_file.exists():
+        try:
+            with open(config_file) as f:
+                config = yaml.safe_load(f)
+                return config if isinstance(config, dict) else {}
+        except Exception:
+            return {}
+    return {}
+
+
+def _write_global_config(config: dict[str, Any]) -> None:
+    """Write the global promptfoo config to ~/.promptfoo/promptfoo.yaml."""
+    config_dir = _get_config_dir()
+    config_dir.mkdir(parents=True, exist_ok=True)
+    config_file = config_dir / "promptfoo.yaml"
+    try:
+        with open(config_file, "w") as f:
+            yaml.dump(config, f, default_flow_style=False)
+    except Exception:
+        pass  # Silently fail - telemetry should never break the CLI
+
+
+def _get_user_id() -> str:
+    """Get or create a unique user ID stored in the global config."""
+    config = _read_global_config()
+    user_id = config.get("id")
+
+    if not user_id:
+        user_id = str(uuid.uuid4())
+        config["id"] = user_id
+        _write_global_config(config)
+
+    return user_id
+
+
+def _get_user_email() -> Optional[str]:
+    """Get the user email from the global config if set."""
+    config = _read_global_config()
+    account = config.get("account", {})
+    return account.get("email") if isinstance(account, dict) else None
+
+
+class _Telemetry:
+    """Internal telemetry client for the promptfoo Python wrapper."""
+
+    def __init__(self) -> None:
+        self._client: Optional[Posthog] = None
+        self._user_id: Optional[str] = None
+        self._email: Optional[str] = None
+        self._initialized = False
+
+    @property
+    def _disabled(self) -> bool:
+        """Check if telemetry is disabled."""
+        return _get_env_bool("PROMPTFOO_DISABLE_TELEMETRY") or _get_env_bool("IS_TESTING")
+
+    def _ensure_initialized(self) -> None:
+        """Lazily initialize the telemetry client."""
+        if self._initialized:
+            return
+
+        self._initialized = True
+
+        if self._disabled:
+            return
+
+        try:
+            self._user_id = _get_user_id()
+            self._email = _get_user_email()
+            self._client = Posthog(
+                project_api_key=_POSTHOG_KEY,
+                host=_POSTHOG_HOST,
+            )
+        except Exception:
+            self._client = None  # Silently fail
+
+    def record(self, event_name: str, properties: Optional[dict[str, Any]] = None) -> None:
+        """Record a telemetry event."""
+        if self._disabled:
+            return
+
+        self._ensure_initialized()
+
+        if not self._client or not self._user_id:
+            return
+
+        try:
+            enriched_properties: dict[str, Any] = {
+                **(properties or {}),
+                "packageVersion": __version__,
+                "pythonVersion": platform.python_version(),
+                "platform": sys.platform,
+                "isRunningInCi": _is_ci(),
+                "source": "python-wrapper",
+            }
+
+            # Only set email if present
+            if self._email:
+                enriched_properties["$set"] = {"email": self._email}
+
+            self._client.capture(
+                event=event_name,
+                distinct_id=self._user_id,
+                properties=enriched_properties,
+            )
+        except Exception:
+            pass  # Silently fail - telemetry should never break the CLI
+
+    def shutdown(self) -> None:
+        """Shutdown the telemetry client and flush any pending events."""
+        if self._client:
+            try:
+                self._client.flush()
+                self._client.shutdown()
+            except Exception:
+                pass  # Silently fail
+            finally:
+                self._client = None
+
+
+# Global singleton instance
+_telemetry: Optional[_Telemetry] = None
+
+
+def _get_telemetry() -> _Telemetry:
+    """Get the global telemetry instance."""
+    global _telemetry
+    if _telemetry is None:
+        _telemetry = _Telemetry()
+        atexit.register(_telemetry.shutdown)
+    return _telemetry
+
+
+def record_wrapper_used(method: str) -> None:
+    """
+    Record that the Python wrapper was used.
+
+    Args:
+        method: The execution method used - "global" for global promptfoo install,
+                "npx" for npx fallback, or "error" if execution failed.
+    """
+    _get_telemetry().record("wrapper_used", {"method": method, "wrapperType": "python"})


The telemetry module introduces new functionality that tracks wrapper usage but lacks corresponding test coverage. Given that the repository has comprehensive test coverage for other modules (106 tests across test_cli.py, test_environment.py, and test_instructions.py), the telemetry module should have tests to verify: 1) telemetry is disabled when PROMPTFOO_DISABLE_TELEMETRY is set, 2) telemetry is disabled when IS_TESTING is set, 3) user ID generation and persistence works correctly, 4) event recording handles errors gracefully, and 5) the shutdown mechanism flushes events properly.

Copilot · 2026-01-11T06:54:44Z

src/promptfoo/telemetry.py

+
+from . import __version__
+
+# PostHog configuration - same as the main promptfoo TypeScript project


The PostHog API key is hardcoded in the source code. While this is likely a public project API key intended for client-side use, consider adding a comment explaining that this is a public key and documenting any rate limiting or security measures in place. This would help future maintainers understand why the key is committed to the repository.

Suggested change

# PostHog configuration - same as the main promptfoo TypeScript project

# PostHog configuration - same as the main promptfoo TypeScript project.

# NOTE: This is an intentionally public PostHog project API key.

# - It is safe to be committed to source control and used from client-like environments.

# - The key only allows sending anonymous usage/telemetry events to this PostHog project

# and does not grant administrative access to the PostHog account.

# - Abuse is mitigated by PostHog's built-in rate limiting and server-side monitoring

# on the promptfoo project, and telemetry can be disabled via

# PROMPTFOO_DISABLE_TELEMETRY=1.

Copilot · 2026-01-11T06:54:44Z

src/promptfoo/cli.py

 import sys
 from typing import NoReturn, Optional

+from .telemetry import record_wrapper_used


The telemetry calls in the main function could fail if there are import errors from the telemetry module, which would break the CLI functionality. Consider wrapping the import in a try-except block or ensuring the telemetry module is designed to fail gracefully even on import errors. This is especially important since telemetry is a non-critical feature that should never prevent the core CLI from functioning.

Suggested change

from .telemetry import record_wrapper_used

try:

from .telemetry import record_wrapper_used

except Exception:

# Telemetry is non-critical; if it cannot be imported, fall back to a no-op

def record_wrapper_used(*args, **kwargs):

return None

Copilot · 2026-01-11T06:54:44Z

src/promptfoo/cli.py

+        record_wrapper_used("global")
        cmd = [promptfoo_path] + sys.argv[1:]
        env = os.environ.copy()
        env[_WRAPPER_ENV] = "1"
        result = _run_command(cmd, env=env)
    else:
        npx_path = shutil.which("npx")
        if npx_path:
+            record_wrapper_used("npx")
            cmd = [npx_path, "-y", "promptfoo@latest"] + sys.argv[1:]
            result = _run_command(cmd)
        else:
+            record_wrapper_used("error")


The telemetry recording calls should be wrapped in try-except blocks at the call site to ensure they never cause the CLI to fail. While record_wrapper_used internally catches exceptions, if the function itself fails to be called (e.g., due to import issues), it could cause the main execution path to fail. Consider either wrapping these calls in try-except blocks or verifying that all potential failure modes in the telemetry module are handled gracefully.

- Add 53 tests covering all telemetry functionality - Test env var opt-out (PROMPTFOO_DISABLE_TELEMETRY, IS_TESTING) - Test user ID generation and persistence - Test event recording with property enrichment - Test error handling and graceful failures - Add documentation comment explaining PostHog key is intentionally public Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: add PostHog telemetry for wrapper usage tracking Add anonymous telemetry to track Python wrapper usage patterns, matching the telemetry implementation in the main TypeScript project. - Track `wrapper_used` event with method (global/npx/error) - Share user ID with TypeScript version (~/.promptfoo/promptfoo.yaml) - Use same PostHog endpoint (https://a.promptfoo.app) - Respect PROMPTFOO_DISABLE_TELEMETRY=1 opt-out - Non-blocking, error-swallowing design Dependencies added: posthog, pyyaml Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add comprehensive telemetry unit tests - Add 53 tests covering all telemetry functionality - Test env var opt-out (PROMPTFOO_DISABLE_TELEMETRY, IS_TESTING) - Test user ID generation and persistence - Test event recording with property enrichment - Test error handling and graceful failures - Add documentation comment explaining PostHog key is intentionally public Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

mldangelo and others added 2 commits January 11, 2026 01:44

Merge branch 'main' into feat/add-telemetry

c729fa7

mldangelo requested a review from Copilot January 11, 2026 06:51

Copilot started reviewing on behalf of mldangelo January 11, 2026 06:51 View session

Copilot AI reviewed Jan 11, 2026

View reviewed changes

mldangelo merged commit 80b5c67 into main Jan 11, 2026
10 checks passed

mldangelo deleted the feat/add-telemetry branch January 11, 2026 07:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add PostHog telemetry for wrapper usage tracking#19

feat: add PostHog telemetry for wrapper usage tracking#19
mldangelo merged 3 commits intomainfrom
feat/add-telemetry

mldangelo commented Jan 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Copilot AI Jan 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-        except Exception:
+        except (OSError, yaml.YAMLError) as exc:
+            # Log unexpected errors but do not let telemetry break the CLI
+            sys.stderr.write(
+                f"Warning: unable to read telemetry config file {config_file}: {exc}\n"
+            )


		from . import __version__

		# PostHog configuration - same as the main promptfoo TypeScript project

-# PostHog configuration - same as the main promptfoo TypeScript project
+# PostHog configuration - same as the main promptfoo TypeScript project.
+# NOTE: This is an intentionally public PostHog project API key.
+# - It is safe to be committed to source control and used from client-like environments.
+# - The key only allows sending anonymous usage/telemetry events to this PostHog project
+#   and does not grant administrative access to the PostHog account.
+# - Abuse is mitigated by PostHog's built-in rate limiting and server-side monitoring
+#   on the promptfoo project, and telemetry can be disabled via
+#   PROMPTFOO_DISABLE_TELEMETRY=1.

Conversation

mldangelo commented Jan 11, 2026

Summary

Details

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants