Retrospective benchmark

We tested Robertium against 50 known drug repurposings spanning 60 years of clinical history. The result was 0 hits — and that result is the most informative finding of the benchmark.

Robertium is a novelty discovery system. By construction it filters out the well-cited repurposings that make up any reasonable gold standard. The architecturally-correct answer for such a benchmark is therefore zero — and what matters is what the diagnostic mode finds when the novelty filter is disabled.

Why 0 / 50 is the right answer

Robertium implements Don R. Swanson's 1986 literature-based discovery framework. The central premise of Swanson ABC reasoning is to surface drug → mediator → outcome chains whose endpoints are under-cited together in the published literature. The production pipeline operationalises this with a strict novelty constraint: any candidate hypothesis where the drug → outcome connection already appears as a direct claim more than once in the corpus is filtered out (max_direct_evidence = 1, see src/robertium/graph/repurposing.py:192).

The 50 gold-standard cases are, by definition, the opposite of what this filter retains: verified drug repurposings whose drug → indication link is supported by multiple direct publications accumulated over decades. They sit on the wrong side of the novelty filter by construction.

Running them through the production pipeline therefore returns zero hits at any rank for the full 50 — not because the methodology fails, but because the methodology explicitly excludes them. This is a design feature, not an implementation defect. A search over PubMed will return papers about ketamine and treatment-resistant depression because such papers exist by the hundreds. Robertium's job is to surface connections that no single paper makes explicit yet.

The diagnostic — filter disabled

3 / 14
in-corpus chains surfaced
in no-filter mode
21.4%
per-drug recall@1000
on in-corpus cases
Top 2%
fingolimod's rank within
its candidate pool (5 / 249)

To distinguish "filtered by design" from "the underlying ABC chain is absent from our corpus," we re-ran discovery on the 14 in-corpus + borderline gold-standard cases with the novelty filter disabled. Three cases surfaced with biologically plausible chains supported by verifiable PubMed evidence. The two-named-domain filter was retained; only the novelty constraint was lifted.

The per-drug rank reports the position of the correct outcome within just that drug's no-filter candidate pool. We treat it as the primary metric here: "fingolimod's correct chain ranked 5 of 249 within its own candidate pool" is more directly interpretable than its position in a 1,329-entry universe partitioned across 14 drugs.

The three surfaced chains

Each card shows the chain Robertium constructed, its rank within the drug's no-filter pool, a one-sentence biological note, and the PubMed evidence for both legs of the chain.

fingolimod → VEGFR2 → ALS
multiple_sclerosis → als per-drug rank 5 of 249

The curator-supplied mechanism for fingolimod → ALS is S1P-receptor modulation. Robertium surfaced an alternative chain via VEGFR2 — the VEGF signalling axis is implicated in motor-neuron degeneration through vascular contribution to ALS pathology, and fingolimod's effect on VEGFR2 is documented in the retinopathy context. This is the kind of non-canonical mediator a literature-based discovery system is expected to produce.

A → B evidence 1 paper · MS domain

  • PMID 34603029 (2021) — Fingolimod and Diabetic Retinopathy: A Drug Repurposing Study

B → C evidence 3 papers · ALS domain

metformin → α-synuclein → Alzheimer's disease pathophysiology
epilepsy → alzheimer per-drug rank 115 of 159

The curator's mechanism is AMPK activation. Robertium surfaced an alternative chain via α-synuclein, a protein conventionally associated with Parkinson's disease but with a documented role in AD pathophysiology through CSF biomarker work. The AMPK route fails because the corresponding B → C edge lives in an ALS-domain paper, not an Alzheimer-domain paper.

A → B evidence 1 paper · epilepsy domain

  • PMID 30308130 (2018) — Effects of metformin on apoptosis and α-synuclein in a rat model of pentylenetetrazole-induced epilepsy

B → C evidence 1 paper · alzheimer domain

  • PMID 30477568 (2018) — CSF α-synuclein levels in sporadic and familial Alzheimer's disease
valproic acid → BDNF → bipolar disorder
epilepsy → depression per-drug rank 455 of 510

This chain has 2 direct citations connecting valproate to bipolar disorder, meaning the production pipeline would drop it under the novelty filter. This case is the clearest example of a hypothesis the filter intentionally excludes — and demonstrates that the novelty-aware scoring correctly down-ranks the chain once direct evidence accumulates.

A → B evidence 2 papers · epilepsy domain

B → C evidence 1 paper · depression domain

  • PMID 36700317 (2023) — BDNF in bipolar vs unipolar depression

Why the remaining 11 in-corpus cases miss

Eleven of the 14 in-corpus + borderline cases do not surface even with the novelty filter disabled. The misses break down into three structural categories, all reflecting corpus coverage rather than methodology behaviour.

Reason Count Examples
Drug essentially absent from corpus 2 exenatide, onabotulinumtoxinA
Outcome absent from B → C edges 5 treatment-resistant depression, migraine, mania in bipolar disorder, postherpetic neuralgia, familial adenomatous polyposis
Drug too thinly represented 4 mebendazole, amantadine, imatinib, disulfiram

These are corpus coverage gaps, not methodology failures. Two paths address them: corpus expansion into pre-2015 historical literature (the current corpus runs 2015–2027 only, which does not overlap the gold-standard's 1955–2018 first-proof years), and extraction improvements for less-common diseases and drugs — particularly better outcome normalisation so specific labels like "treatment-resistant depression" or "familial adenomatous polyposis" map onto the broader disease claims that the corpus does contain.

What an appropriate benchmark would look like

The gold-standard known-repurposings benchmark answers the question "would Robertium have re-discovered the existing literature?" That is not the right question for a novelty-discovery system. A defensible benchmark for Robertium has three components:

  • Prospective tracking. Persist the top-k hypotheses output at a fixed snapshot, then revisit one to three years later to measure how many acquired direct experimental or clinical evidence. This is the only measurement that directly tests "would Robertium have predicted future repurposings?" without circular dependence on the existing literature.
  • Expert review for biological plausibility. Domain specialists rate the top-k hypotheses on a small ordinal scale (implausible / plausible / known). This produces a precision-style metric for the top of the ranked list, which is the part of the output actually used.
  • Comparison with prior LBD systems. Run the same input corpus through SemMedDB-based discovery tools and a faithful reimplementation of Swanson 1986 against an identical input. Report relative precision on the same expert-graded top-k. This frames Robertium's contribution against established baselines rather than against a benchmark designed for different system semantics.

The known-repurposings benchmark retains diagnostic value: it confirms that the extraction and graph-construction stages can reach correct ABC chains when the novelty filter is bypassed, as the no-filter results above demonstrate for 3 of 14 cases. It is unsuitable as a primary recall metric for the production system.

Limitations

  • Reproductive, not predictive. The benchmark runs against the full current 2015–2027 corpus with no date cutoff. The corpus year range does not overlap the gold standard's 1955–2018 first-proof years, so genuine retrospective discovery with a date cutoff (Option D in the script) cannot be evaluated until the corpus is extended backward.
  • Matcher uses normalised text + substring fallback. Outcome matching relies on normalised string comparison; drugs use salt-suffix-aware exact matching. MeSH IDs are not yet used (planned v1.1). Drug substring matches that would touch combination products are deliberately refused to avoid false positives.
  • In-corpus categorisation is hand-curated. The split into 6 in-corpus / 8 borderline / 36 out-of-corpus cases is curator judgment for the 50 cases (e.g., GI cancers treated as adjacent to pancreatic cancer; mood disorders as adjacent to depression). Documented in scripts/benchmark/analyze_results.py.
  • No-filter diagnostic recomputes from claims, not via the Kuzu graph. Edge counts and the scoring formula are identical to the production pipeline. Ordering of ties may differ between graph traversal and SQL recomputation.

Data and reproduction

Reproduction:

cd ~/Documents/robertium

# Baseline (50 cases, novelty filter on)
uv run python scripts/benchmark/run_benchmark.py --restart

# No-filter diagnostic (14 in-corpus + borderline cases)
uv run python scripts/benchmark/run_benchmark.py --disable-novelty-filter --restart

# Aggregate metrics + this report
uv run python scripts/benchmark/analyze_results.py

Both runs are read-only against Postgres. No INSERT, UPDATE, or DELETE on production tables. The full 50-case baseline completes in ~10 seconds; the 14-case no-filter run in ~3 seconds. Benchmark scripts are MIT-licensed alongside the rest of the project; the gold-standard dataset is CC-BY-4.0.