feat: Health Score v2 (Akrites) scoring pipeline, lifecycle, and impact score (CM-IN-1196)

[gaspergrom]

Summary

Adds the Health Score v2 (Akrites methodology) scoring pipeline: per-repo maintainer health, security/supply-chain, and development-activity category scores, plus a lifecycle classifier and an impact score, rolled up to project level. Also fixes several correctness bugs found during data verification against production before rollout.

• Split Health Score v2 computation into 5 independently-materialized category pipes (maintainer/security/development/lifecycle/impact) to keep copy-job runtime down — a single combined pipe was re-scanning base tables per category and had grown from ~13min to 20-28+ min per run.
• healthScoreV2 applies spec-defined graceful degradation: if 1-2 of 3 categories are unavailable, the score rescales across only the available categories rather than defaulting to a fabricated 0; NULL only when all three are unavailable.
• Bus factor fix: the maintainer-health bus-factor score counted maintainer role records purely by an unexpired endDate (a manually-curated field that stays at a "still active" sentinel indefinitely unless someone explicitly closes the role). This let archived/dead repos keep a maxed-out bus-factor score from maintainers with no real activity in years, which combined with the graceful-degradation rescale to produce healthScoreV2 = 100 on dead repos. Bus factor now also requires the maintainer to have actual activityRelations activity on that repo within the same trailing 12-month window. Verified against production: Xamarin.Forms and SpaceVim (archived) drop from a maxed bus-factor score to 0; kubernetes/kubernetes stays healthy and unaffected.
• Lifecycle fallback fix: the lifecycle decision tree's final ELSE branch always resolved to 'active' whenever a repo failed every other branch, including repos with zero commits, zero issues, and zero PRs in every window checked (confirmed live on a fully dormant, non-archived repo with no commit-timeline data and no signal at all). The chain now returns NULL for that case instead of a fabricated 'active', propagated correctly through:
  • health_score_v2_lifecycle_ds — widened lifecycleLabelV2 from String to Nullable(String) (was going to hard-fail the copy job on any NULL insert).
  • health_score_v2.pipe — removed a coalesce(lifecycleLabelV2, 'active') that would have silently erased the NULL back into 'active' one hop downstream.
  • project_insights_copy.pipe — the project-level "best-state-wins" rollup already drops NULL entries for free via groupArray, but an all-NULL or fully-excluded project produces an empty array, and arrayElement([], 1) on an empty array silently returns '' rather than NULL. Added an explicit guard. While tracing this, found the actual root cause of 5 pre-existing blank-lifecycleLabel projects (each project's sole repo has repositories.excluded = true) — a separate, pre-existing data situation, not fixed here, but no longer surfaces as a silent empty string.

All fixes verified against live production Tinybird data before merge, including full-chain propagation checks and regression checks against known-healthy projects (kubernetes, projects with real but thin activity signal).

Test plan

• tb check passes on all changed pipes/datasources
• Verified bus-factor fix against production data: archived repos (Xamarin.Forms, SpaceVim, HPCToolkit, Ursa) drop from healthScoreV2=100 to realistic scores (22-62); active repos (kubernetes, GravityView, Hitomi-Downloader, binutils-gdb) unaffected
• Verified lifecycle NULL fix against production data: a genuinely zero-signal repo (blocknetdx/blocknet) resolves to NULL at both repo and project level; archived repos still correctly resolve to 'archived' even with zero activity; repos with any real signal still resolve to a real state, not NULL
• Verified datasource schema widening (String → Nullable(String)) is a non-breaking, metadata-only ClickHouse ALTER
• Deploy to staging, confirm pipes deploy cleanly (staging data is known to be sparse, so this validates deployability not data correctness)
• Deploy to production, re-validate full chain against live data post-deploy

Deploy notes

Deploy order matters — datasources before pipes, then pipes in dependency order:

1. health_score_v2_lifecycle_ds.datasource, health_score_v2_repo_copy_ds.datasource (schema changes)
2. health_score_v2_lifecycle.pipe, health_score_v2_maintainer.pipe (independent, both feed health_score_v2)
3. health_score_v2.pipe (reads both category datasources above)
4. project_insights_copy.pipe (reads health_score_v2_repo_copy_ds)

After pushing pipes, the copy jobs need a forced run (tbc <pipe> --yes) rather than waiting for the nightly cron, in the same order, so the NULL/bus-factor fixes propagate through the full chain before this is user-visible.

[cursor]

PR Summary

High Risk
Changes user-visible health, lifecycle, and impact metrics across repos/projects and alters maintainer bus-factor logic; deploy order and copy-job backfills matter, though filtered collection dependency behavior is unchanged.

Overview
Introduces Health Score v2 (Akrites) end-to-end in Tinybird: five scheduled category COPY pipes (maintainer, security, development, lifecycle, impact) materialize per-repo inputs, health_score_v2.pipe joins them into health_score_v2_repo_copy_ds with Layer 2 rescaling, and project/repo insights surfaces healthScoreV2, labels, lifecycle, and impact via project_insights_copy.pipe plus the project_insights / project_repo_insights endpoints.

Correctness fixes tighten scoring: maintainer bus factor now requires recent activityRelations (not stale role rows), lifecycle returns NULL when there is no activity signal (schema widened to Nullable(String)), health_score_v2.pipe stops coalescing lifecycle to 'active', and project lifecycle rollup guards empty groupArray so it does not become ''.

Separately, collection contributor dependency adds a daily collection_contributor_dependency_copy materialization and routes the default unfiltered widget load (limit=100, no filters) to a cheap precomputed lookup instead of the live aggregation that was timing out at collection scale; filtered requests still use the live path.

^{Reviewed by Cursor Bugbot for commit a3e0359. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

PR titles must follow Conventional Commits. Love from, Your reviewers ❤️.

[cursor]

Bus factor counts non-contribution activity

Medium Severity

The bus-factor fix requires recent activityRelations rows on the repo, but the subquery does not restrict activity type. Stars, forks, and other non-contribution events can satisfy the join, so maintainers with no real maintenance work in the window may still inflate bus factor—similar to the stale role-record issue this change was meant to fix.

 

^{Triggered by learned rule: Tinybird pipes counting contributors must filter by activityTypes}

^{Reviewed by Cursor Bugbot for commit a20d1bc. Configure here.}

[cursor]

NULL health score becomes zero

High Severity

Project and repo rows set healthScoreV2 with toUInt8(round(hv2.healthScoreV2)) without a NULL guard, unlike upstream pipes that use toNullable. Missing or unavailable scores can become 0, which is a valid “critical” score and contradicts healthLabel when that field is NULL. When every included repo has NULL healthScoreV2, avg can yield NaN; NaN is not treated as NULL in the label multiIf, so the project can show healthLabel critical while the score is wrong.

Additional Locations (1)

• services/libs/tinybird/pipes/project_insights_copy.pipe#L288-L289

 

^{Reviewed by Cursor Bugbot for commit a20d1bc. Configure here.}

[Copilot]

Pull request overview

Adds the Akrites Health Score v2 pipeline and exposes repository/project health, lifecycle, and impact metrics.

Changes:

• Materializes five independent scoring categories and combines available health signals.
• Adds lifecycle and impact scoring with project-level rollups.
• Exposes the new metrics through insights endpoints and datasources.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
pipes/project_repo_insights.pipe	Exposes Akrites fields for projects and repositories.
pipes/project_insights.pipe	Exposes Akrites project fields.
pipes/project_insights_copy.pipe	Rolls up and labels repository scores.
pipes/health_score_v2.pipe	Combines category scores with graceful degradation.
pipes/health_score_v2_security.pipe	Computes security and supply-chain health.
pipes/health_score_v2_maintainer.pipe	Computes maintainer health.
pipes/health_score_v2_lifecycle.pipe	Classifies repository lifecycle.
pipes/health_score_v2_impact.pipe	Computes package-mediated impact.
pipes/health_score_v2_development.pipe	Computes development activity health.
datasources/project_insights_copy_ds.datasource	Adds Akrites insight columns.
datasources/health_score_v2_security_ds.datasource	Stores security scores.
datasources/health_score_v2_repo_copy_ds.datasource	Stores combined repository results.
datasources/health_score_v2_maintainer_ds.datasource	Stores maintainer scores.
datasources/health_score_v2_lifecycle_ds.datasource	Stores lifecycle classifications.
datasources/health_score_v2_impact_ds.datasource	Stores raw impact scores.
datasources/health_score_v2_development_ds.datasource	Stores development scores.

Comments suppressed due to low confidence (3)

services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe:85

• All three values only use issues opened in the last 18 months, but the aggregation currently reads every partition in issues_analyzed. Add the 18-month predicate to WHERE so ClickHouse can prune old yearly partitions.

        FROM issues_analyzed
        GROUP BY channel

services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe:90

• The metric only counts PRs from the last 18 months, yet this scans all yearly partitions in pull_requests_analyzed. Push the same time condition into WHERE to prune historical partitions.

        FROM pull_requests_analyzed
        GROUP BY channel

services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource:24

• This statement contradicts the implemented Layer 2 graceful degradation in health_score_v2.pipe: unavailable categories are excluded and remaining category scores are rescaled, while the result is NULL only when all categories are unavailable. The earlier “sum” description is also incomplete for that case.

    - No graceful-degradation/signal-coverage redistribution yet — a repo with zero signal in a category
      scores 0 there rather than having points redistributed from available categories.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

There are 5 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a3e0359. Configure here.}

[cursor]

NULL lifecycle shadows abandoned

High Severity

The new no-signal NULL branch requires zero commits in the 12-month activityRelations windows and zero issues/PRs, but does not consider lastCommitAt. Repos with an old lastCommitAt and no issues/PRs — the normal abandoned case — also have zero ingested commits in those windows, so they hit NULL before the abandoned branch can run. In consistent data, abandoned becomes effectively unreachable.

 

^{Reviewed by Cursor Bugbot for commit a3e0359. Configure here.}

[cursor]

Raw activityRelations in maintainer score

Medium Severity

Bus-factor and org-diversity joins read raw activityRelations, while the sibling Health Score v2 pipes and the datasource docs require activityRelations_deduplicated_cleaned_bucket_union for analytics. Uncleaned rows (duplicates, rows meant to be filtered out) can inflate recent-activity maintainer and org counts and skew Maintainer Health.

 

^{Reviewed by Cursor Bugbot for commit a3e0359. Configure here.}

[cursor]

Roles not scoped to collection

Medium Severity

The precomputed copy aggregates maintainers_roles_copy_ds roles by memberId only, with no collection/insightsProjectId filter. The live path uses member_roles, which restricts roles to projects in the collection. Precomputed responses can show maintainer roles from outside the collection, so default (unfiltered) widget loads disagree with the filtered live path.

 

^{Reviewed by Cursor Bugbot for commit a3e0359. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (8)

services/libs/tinybird/pipes/health_score_v2_maintainer.pipe:81

• repositories is a versioned ReplacingMergeTree, but this query filters deletedAt and reads mutable excluded before collapsing versions. A repository whose exclusion flag changed can therefore produce two rows with the same URL and duplicate the category output. Read FINAL, as established in project_insights_copy.pipe:131 and repositories_populated_copy.pipe:7.

                FROM repositories WHERE deletedAt IS NULL

services/libs/tinybird/pipes/health_score_v2.pipe:52

• Because repositories is a versioned ReplacingMergeTree, filtering before FINAL leaves old non-deleted versions visible and can retain soft-deleted repositories in the final score datasource. Use the current version before applying deletedAt, consistent with repositories_populated_copy.pipe:7.

        FROM (SELECT DISTINCT url AS repoUrl FROM repositories WHERE deletedAt IS NULL) AS base

services/libs/tinybird/pipes/health_score_v2_lifecycle.pipe:60

• This reads mutable archived values from every physical version of the repositories ReplacingMergeTree. If a repository was archived or unarchived, DISTINCT url, archived emits both states and materializes conflicting lifecycle rows for one URL. Collapse with FINAL before filtering/selecting mutable fields.

        FROM (SELECT DISTINCT url, archived FROM repositories WHERE deletedAt IS NULL) AS base

services/libs/tinybird/pipes/project_insights_copy.pipe:176

• The displayed project score is rounded to an integer, but the label thresholds use the unrounded average. For example, 84.6 is exposed as healthScoreV2 = 85 while receiving healthLabel = 'healthy', contradicting the documented 85+ “excellent” band. Apply the thresholds to the same rounded value.

        toUInt8(round(hv2.healthScoreV2)) AS healthScoreV2,
        multiIf(
            hv2.healthScoreV2 IS NULL,
            NULL,
            hv2.healthScoreV2 >= 85,

services/libs/tinybird/datasources/project_insights_copy_ds.datasource:39

• These datasource descriptions document the previous package-derived calculation, but the new pipe computes health from per-repository category scores, lifecycle across repository labels, and impact as a maximum rather than an average. This is the contract consumers will consult, so update it to match the implemented aggregation and null conditions.

    - `healthScoreV2` column is the Akrites-methodology composite health score (0-100), averaged from `ossPackages_enriched_ds.healthScore` across the project's linked packages. Distinct from the legacy `healthScore` column (community/contributor-based); null when the project has no linked package data.
    - `healthLabel` column buckets `healthScoreV2`: 'excellent' (85+), 'healthy' (70-84), 'fair' (50-69), 'concerning' (30-49), 'critical' (<30). Lowercase, matches `ossPackages_enriched_ds.healthLabel` convention.
    - `lifecycleLabel` column is the project's aggregated maintenance state across its packages, using best-state-wins precedence (active > stable > declining > abandoned > archived) over `ossPackages_enriched_ds.lifecycleLabel`.
    - `impactScore` column is the average Osprey criticality `impact` (0-100) across the project's packages. Null when the project has no linked package data.
    - `impactLabel` column buckets `impactScore`: 'foundational' (85-100), 'major' (60-84), 'moderate' (30-59), 'minor' (0-29). Null when `impactScore` is null.

services/libs/tinybird/pipes/project_insights.pipe:15

• This public endpoint description ties all five fields to linked package data, but health and lifecycle are repository-derived and become null based on signal coverage; only impact is package-mediated. Update the nullability note so API consumers do not infer the wrong prerequisite.

    - `healthScoreV2`, `healthLabel`, `lifecycleLabel`, `impactScore`, `impactLabel` are the Akrites-methodology fields (see `project_insights_copy_ds` for band definitions); may be null when the project has no linked package data. `healthScoreV2` is distinct from the legacy `healthScore` field.

services/libs/tinybird/pipes/project_repo_insights.pipe:14

• This endpoint description says the Akrites fields may be null due to missing package data, but repository health and lifecycle do not depend on packages. Only impact has that prerequisite; health/lifecycle nullability reflects unavailable repository signals.

    - `healthScoreV2`, `healthLabel`, `lifecycleLabel`, `impactScore`, `impactLabel` are the Akrites-methodology fields (see `project_insights_copy_ds` for band definitions); may be null when the record has no linked package data. `healthScoreV2` is distinct from the legacy `healthScore` field.

services/libs/tinybird/datasources/health_score_v2_repo_copy_ds.datasource:24

• This states that graceful degradation is not implemented, directly contradicting health_score_v2.pipe:33-49, which rescales over available category weights. The earlier “sum” description is consequently also inaccurate whenever a category is unavailable. Document the actual rescaling behavior in one place.

    - No graceful-degradation/signal-coverage redistribution yet — a repo with zero signal in a category
      scores 0 there rather than having points redistributed from available categories.