Scoring methodology

Architecture overview

Every merged PR produces three numbers:

The core design decision: the LLM never produces an effort number directly. LLMs are unreliable at absolute magnitude but good at classification. The size/effort base is computed deterministically from lines changed, and the LLM only answers two questions: "how much of this diff is real work vs. churn?" and "is it good?"

Step 1 - Exclusions

Before any counting, these files are stripped and do not contribute to any score:

• -Test files: paths containing /tests/, /test/, /__tests__/, /__mocks__/; files ending in Test.php, .spec.ts, .test.ts, etc.
• -Vendored/generated: paths containing /vendor/, /node_modules/, /site-packages/, /boto3/, etc.
• -Dependency lockfiles: package-lock.json, composer.lock, yarn.lock, poetry.lock, symfony.lock.

These same exclusion rules apply to both velocity scoring and AI attribution share percentages, so "non-excluded lines" means the same thing in both contexts.

Step 2 - Non-code weighting (1/4)

After exclusions, remaining files are classified as non-code if they match any of these rules. Non-code lines count at 1/4 weight when computing effective line counts.

Formula: effective_lines = code_lines + round(non_code_lines / 4)

Step 3 - Per-module grouping

Files are grouped by their top-level directory. Files at repository root go into a _root group. Each group is scored independently, and the group scores are summed.

Rationale: two developers sharing one branch score the same total as two separate PRs. The logarithmic compression applies per module, not globally, preventing one massive directory from drowning everything else.

Step 4 - Base velocity step table

Each group's effective line count maps to a base score through this step table. Above 3000 lines, scoring grows logarithmically - it does not hit a flat cap.

Step 5 - Per-group modifiers

Effort multiplier (LLM)

An LLM classifies each file in the diff into a churn bucket, then estimates what fraction of total lines represent genuine engineering effort. The multiplier is a number from 0.0 to 1.0.

Large feature PRs that introduce substantial new functionality score high effort even when they include some boilerplate or config files. The multiplier only drops significantly when changes are predominantly auto-generated, copy-pasted, or mechanical.

For very large PRs (diff exceeding ~250K chars), the diff is chunked and each chunk analyzed independently. A second LLM pass reads all chunk summaries to emit a single unified effort_multiplier for the PR.

Final velocity: velocity_score = base_velocity x effort_multiplier

Quality score (LLM, 0-100)

Produced by the same LLM call as the effort multiplier. Uses banded anchors to keep scores comparable across PRs and reviewers.

Auditability

Every scored PR stores:

• -Typed score columns (quality_score, velocity_score) for fast queries and dashboards.
• -A full JSON blob of the LLM response - summaries, strengths, risks, review focus, confidence, and the raw file classifications.
• -A breakdown of the base velocity computation: which files were excluded and why, which group each file landed in, the effective line count, the step-table base, and each modifier applied.

This means you can always answer "why did this PR score X?" with a specific, auditable trail - not a black box number.