A Living Engineering Knowledge Graph, Refreshed From Every PR

Start with a merged PR, because that is where the graph wakes up. Every hour a listener scans merged PRs through the GitHub GraphQL API and writes a small PR-history document: what changed, where, and by whom. It makes no model calls at all. An allowlist decides which repos it watches. This is the cheapest thing in the whole system, pure metadata capture at essentially zero marginal spend, and it runs constantly. The point of running it that often is that recording that something changed should never be the thing you ration. That is the first half of a deliberate split we will come back to.

To see why the split matters, follow what has to happen for that change to actually mean something in the graph.

Building the map before reading the territory

For each repository we first build a code-structure graph: nodes for classes and methods, edges for “contains” and for call relationships within the service. We do this before we let a model near the source, and the reason is economic. Walking that graph lets us select the salient nodes instead of dumping the whole tree at an LLM. Cost then scales with how much code is actually worth documenting rather than with raw repo size, which for a multi-repo company is the difference between a system you can afford to run and one you can’t.

The extraction itself runs as distinct phases:

Phase A  select      walk the code graph, pick salient leaf nodes
Phase B  extract      sample real source, produce grounded leaf docs
Phase C  synthesize   roll leaves up into parent (module) docs
Phase D  overview     synthesize one service-level README
Phase E  publish      write to the graph store + git

The ordering carries a single load-bearing rule, borrowed openly from the GraphRAG and CodeWiki bottom-up patterns: leaf documents are the only ones that read source. A leaf samples real code and carries full grounding with it. Verbatim quotes, exact line references, the specific line where a gotcha lives. Phases C and D never reopen a file. Parent docs and the one service overview are produced by synthesis, inheriting grounded claims from their children rather than re-deriving them.

That inheritance is what keeps a high-level summary honest. A service README in this system is not free-floating prose someone hopes is still true; it is a roll-up of leaf claims, each of which traces down to a real line. Top-down summaries float free of the code almost immediately. Bottom-up ones can’t, because there is nowhere for an unsourced claim to enter above the leaves. The accuracy property and the cost property come from the same decision: synthesis works over already-distilled text, so the layers above the leaves are both cheaper and more trustworthy.

Extraction runs on a managed inference platform, AWS Bedrock, with mixed-model routing. A faster, cheaper model does the recall-heavy extraction in Phase B, where there is a lot of work and it is mostly “what does this code say.” A stronger model handles the synthesis phases, where reasoning matters more than volume. Matching model strength to the job is the single decision that most shapes the cost-and-quality trade-off.

Where the model is wrong, and what catches it

Pointing an LLM at a repo and asking for documentation is easy. Getting documentation you would let an autonomous agent act on is the hard part, and the failure mode is not blank pages. It is plausible, confident, wrong prose. A model will assert a race condition in code that is single-threaded, or cite a line number that drifted three commits ago. Hand an agent a stale or fabricated claim and it will build on the fiction without hesitating.

So the LLM output is necessary but not sufficient, and the trust comes from deterministic checks wrapped around it:

• A line-ref snapper. Models quote code well and count lines badly, so we don’t trust the cited numbers. We snap every cited range back to the nearest real function, class, or comment block in the actual file. The claim stays; the anchor gets corrected to ground truth.
• A speculative-claim challenger. A second pass interrogates assertions that sound right but may not survive contact with the source, the “race condition in single-threaded code” class of error. If a claim can’t be defended against the code, it gets cut or flagged.
• Per-claim evidence markers. Every claim carries a marker tying it to its grounding, so a downstream reader, human or agent, can tell a sourced fact from an inference.
• Alias canonicalization. The same concept gets called five things across five teams. We canonicalize against a taxonomy so the graph doesn’t fragment into synonyms.

We checked whether any of this works on a single representative service: a Spring Boot Kotlin codebase of roughly 10k lines, about a thousand graph nodes, a hundred selected leaves. On a hand-checked sample of 19 claims across 6 files we measured zero hallucinated claims, roughly 90% of statements substantively accurate, and, after shipping the snapper and challenger, roughly 90% of line references landing within two lines of the cited construct. The line-ref number is the one to dwell on. The raw model output was nowhere near that precise. The biggest accuracy win in the project wasn’t a better prompt; it was a few hundred lines of boring deterministic code that corrects the one thing models reliably get wrong.

Why the freshness listener earns its keep

Now the other half of the split makes sense. Re-running Phase B against a repo is the expensive operation: it reads source, burns the recall model across many leaves, and we only want to pay for it on code that actually moved. So we re-synthesize leaf documents per repo on a schedule, letting the grounded layer catch up deliberately. The hourly PR listener, by contrast, gives the graph a continuously updated record of change for free.

Splitting “record that something changed” (constant, free) from “re-understand what it means” (occasional, paid) is what keeps the system from going broke or going stale. Conflate the two and you pay full extraction cost on every commit, or you skip it and let the graph rot by Friday. A knowledge graph that’s accurate on Monday and wrong three days later isn’t worth maintaining, and neither is one that bankrupts you to keep current.

Teaching the graph your vocabulary

Code structure tells you how the system is wired. It doesn’t tell you what a “registered agent” or a “compliance filing” is in our business. So alongside the code graph we mine a ubiquitous-language layer: a two-tier model of curated, high-trust business concepts plus lower-trust candidate concepts, governed by a taxonomy of around a hundred-plus domain nouns drawn from team glossaries. The first concept to graduate into a shared, cross-repo definition came out of a compliance glossary. A graph that knows your domain nouns is far more useful to an agent than one that only knows your method signatures, because it carries meaning rather than just call edges.

Two stores, one truth

Phase E writes the output twice. It goes to a Postgres-backed graph with a search index behind an HTTP API, and it is mirrored to git as markdown under a knowledge/auto/<repo>/ convention. The git copy means the knowledge lives next to the code and is reviewable like code; the indexed store means it’s queryable in milliseconds. We serve it back to AI workflows through a tooling layer, so agents pull grounded, current context across repos instead of guessing.

That first milestone extracted thousands of documents across more than a dozen repositories, and the graph has grown well past that since. We treated it like a system, not a script: telemetry, concurrency limits, token budgets, and a test suite gating the pipeline.

The honest gap today is that grounding is text-and-line-based, not yet AST-verified. Snapping to the nearest construct is a strong heuristic, but parsing the tree with tree-sitter would let us assert structural facts with certainty rather than confidence, and that’s the next layer we’re building toward. The larger bet is simpler. As more of our delivery becomes AI-assisted, the bottleneck shifts from writing code to grounding the systems that write it. The most important knowledge about a codebase used to live in a handful of people’s heads: why this service retries on a 409 but not a 408, which queue is load-bearing, the one method everyone is afraid to touch. Tribal knowledge stops being a bottleneck when the tribe’s knowledge is written down, kept current from every PR, and tied to the line of code it came from.