Case study

An earlier version of this project claimed GraphRAG beat plain RAG on PubMedQA. An audit uncovered three confounders: the GraphRAG arm used a reranker missing from the baseline, retrieved from a different corpus, and stored question-derived metadata and the correct answer in a form that could reach the LLM prompt. The original comparison could not isolate the effect of graph traversal.

This repo is the fixed version, built to answer the question honestly: with everything else held constant, what does a knowledge graph actually add?

A controlled 4-arm ablation on PubMedQA, each arm adding exactly one component on top of the last:

Corpus, chunking, embedder, prompt, LLM, random seed, sample size, and top-k are controlled across the 4 arms. The reranker is introduced separately in plain_rr and then held constant for the graph and graph-concept comparison. Graph context uses generic === STUDY n === labels with no titles and no gold label anywhere near the prompt; a test asserts the question string never appears in retrieved context, closing off the exact leak that broke the original comparison. Significance between adjacent arms is a paired McNemar test on the same questions, not a raw accuracy comparison.

The win is one specific, isolated mechanism — parent-document expansion: once a chunk is retrieved, walk back to its parent paper and hand the LLM the full reconstructed abstract instead of an isolated fragment.

The further MeSH concept-hop arm doesn’t clear significance against the plain graph arm (p=0.6889) and costs roughly 5.4x the latency — so it's not the arm this project ships. See RESULTS.md for the full breakdown, or the live benchmark page for the numbers behind this page.

• The historical result needs replication. The graph used for the published run likely had duplicate HAS_CONTEXT links. A clean rerun and retained per-question outputs are required before treating this as reproducible evidence.
• The graph is not isolated from context length. A flat parent-document baseline given the same full abstracts and token budget has not been tested. The measured improvement may come from more complete context rather than graph structure itself.
• Absolute accuracy remains limited. A majority-yes baseline scored 56.5% on the historical sample, close to the reported graph accuracy. Performance should be interpreted with that baseline and the class-level metrics in mind.
• The hosted demo is not this benchmark. Its retrieval corpus, graph backend, LLM, and reranker configuration differ. Benchmark accuracy cannot be inferred from the demo.

For an interactive explanation that works even when the demo sleeps, visit the no-backend walkthrough. Open limitations are tracked in GAPS.md.

The benchmark pipeline runs on ArangoDB and a local Ollama model, which produced every number above but can't be hosted for free. Rather than rewrite that pipeline (which would require re-running the GPU benchmark to re-validate the published numbers), the winning graph retrieval logic was extracted behind a stable answer(question, graph_id) service boundary and given a second, parallel backend: a small Neo4j AuraDB graph (the 1,000-paper labeled split, not the full 62k-paper benchmark corpus) plus a cloud LLM provider chain (Gemini, falling back to local Ollama) so the demo needs no GPU. FastAPI on Render serves it; this Next.js app and a reasoning-path graph visualization are the frontend.

Three real bugs surfaced only once this was live, not in tests:

• Free-tier OOM. Loading both the sentence encoder and the cross-encoder reranker (two full torch models) killed the 512MB Render instance. Fix: skip the reranker on this tier — every retriever already degrades to raw top-k without one by design, so this was a config flag, not a code change.
• A lazy-loader race condition. Concurrent first requests could each start building the shared encoder/graph connection independently. Fixed with a lock around the check-then-create path.
• A wedged service from a corrupted cache. An instance restart mid-write left a truncated vector cache on disk; every subsequent query then hit the same EOFError until the disk was reset. Fixed by treating an unreadable cache as a miss (rebuild, don't crash) and writing the cache atomically (temp file + rename) so a mid-write kill can never leave a partial file again — caught and fixed by firing a real test question at the deployed backend, not by unit tests alone.

The same answer() call behind the chat UI and /query is also exposed as an MCP tool at /mcp — any MCP client (Claude Code, Claude Desktop, Cursor) can use this agent by adding one URL, with no cloning and no API key:

Same free-tier cold-start caveat applies as the chat UI: the first call after an idle period can take 60-120s. See backend/README.md for the Claude Desktop / Cursor config JSON.

• Live chat demo
• Interactive benchmark dashboard
• Source on GitHub
• Full results write-up