Updates

Benchmark v1.0 — architectural finding from 0 / 50 result

Today we ran the first formal retrospective benchmark of Robertium against a curated gold standard of 50 known drug repurposings spanning 60 years of clinical history. The result was 0 hits at any rank.

This is not a methodology failure. It is the architecturally-correct answer for a system implementing Swanson 1986 literature-based discovery. The novelty filter (max_direct_evidence = 1) is the central design choice that distinguishes Robertium from a literature retrieval system — it explicitly excludes hypotheses where the drug → outcome connection is already directly published. The 50 gold-standard cases are, by definition, well-cited repurposings; they sit on the wrong side of the novelty filter by construction.

To distinguish "filtered by design" from "the chain is absent from our corpus," we ran a diagnostic with the novelty filter disabled on the 14 in-corpus cases. Three surfaced with biologically plausible chains, each supported by verified PMID evidence:

• fingolimod → VEGFR2 → ALS (per-drug rank 5 of 249)
• metformin → α-synuclein → Alzheimer's disease pathophysiology (115 of 159)
• valproic acid → BDNF → bipolar disorder (455 of 510)

The remaining 11 misses break down as 2 drugs absent from corpus, 5 specific outcomes absent from BC edges, and 4 drugs too thinly represented to form chains. These are corpus coverage gaps, not methodology failures.

The takeaway: Robertium's value claim is novelty precision (best measured by prospective tracking and expert review), not known-repurposing reproduction. A new page at /benchmark documents the full finding with citations.

Classifier v2: tighter drug-type filtering after manual review of the top 20 candidates

The catalog's top-20 high-confidence cross-domain hypotheses underwent manual review against PubMed and ClinicalTrials.gov to confirm that the strongest candidates are actionable for researchers. The review produced a concrete improvement to the quality classifier.

Three patterns from the manual review:

• Pipeline reproduces real repurposing directions. erlotinib → EGFR → glioblastoma matches Phase II trials NCT00720356, NCT00124657, NCT00045110. metformin → α-synuclein → Parkinson's matches active trials NCT07055958, NCT03685357, NCT05781711. When the pipeline assigns "reproduces_known" status, the underlying NCT IDs check out.
• Genuine novel cross-domain signal exists in the catalog. The clearest example is carbamazepine → KRAS → pancreatic ductal adenocarcinoma. No direct PubMed studies of carbamazepine in pancreatic cancer exist, but the carbamazepine ↔ KRAS pathway link is documented in colorectal cancer (Shaykevich et al. 2024, Invest New Drugs, PMC10944448, on SMARCA4/BRG1 modulation by KRAS status). KRAS drives over 90% of PDAC. The cross-domain inference (epilepsy literature → KRAS regulation → pancreatic application) is exactly the signal Swanson ABC reasoning is designed to surface.
• Several compound categories were passing the outreach-quality filter despite not being actionable repurposing candidates. The v1 classifier filtered for chemical-name patterns but did not separate therapeutic drugs from imaging tracers, biological samples, surgical procedures, internal pharma codes, or peptide constructs.

Compound categories the v1 classifier did not separate:

• PET imaging tracers (PBB3, 18F-PI-2620, APN-1607). Diagnostic compounds that bind disease markers, not therapeutics.
• Biological samples (ALS-CSF). Research material, not a drug.
• Surgical or device interventions (deep brain stimulation, focused ultrasound, transcranial magnetic stimulation). Procedures, not pharmaceuticals.
• Internal pharma research codes without an approved common name (WSD-0922, ABBV-221, NI113). The compound exists, but a researcher cannot obtain it.
• Vague natural-product mixtures ("evodiae fructus active components", "ALA/AUR combination"). Not a specific molecule.
• Peptide constructs identified by Greek letters or LIR-style notation (1D9-LIRΔTP53INP2). Research-stage constructs.

None of these are extraction errors — the LLM correctly identified the drug mention in the source paper. They are gaps in downstream classification.

Six new exclusion categories added to the quality classifier: imaging_compound, biological_sample, procedure_not_drug, research_code, vague_compound, peptide_construct. A whitelist of approved drugs that look like research codes (osimertinib = AZD9291, abemaciclib = LY2835219, tacrolimus = FK506, plerixafor = AMD3100, and others) prevents over-exclusion of FDA-approved compounds that retain their development codes in the literature.

Totals after re-classification: high-confidence 62 → 59, mid-confidence 610 → 574, outreach-quality total 672 → 633, novel-signal with no existing trial 354 → 334 (homepage updated).

Five candidates worth domain-expert attention. All have novel-signal status, no existing clinical trial, and survived the v2 classifier:

• carbamazepine → KRAS → pancreatic ductal adenocarcinoma. Cross-domain bridge from epilepsy to oncology through a validated PDAC driver.
• valproic acid → HDAC2 → cholinergic dysfunction in Alzheimer's. HDAC inhibition is established mechanistically; the link to nucleus basalis cholinergic neurons in AD is not a documented repurposing direction.
• osimertinib → EGFR → Alzheimer's disease. EGFR signaling in neurodegeneration is an emerging research area. Third-generation EGFR-TKIs have not been investigated for AD.
• ketamine / esketamine → NMDA receptor → ALS excitotoxicity. NMDA-mediated excitotoxicity in ALS is well-studied; ketamine class drugs as a repurposing direction are not.
• valproic acid → PKM2 → poor overall survival in pancreatic cancer. VPA's metabolic effects via PKM2 are described; the specific PDAC survival association is not.

If you work in any of these areas, feedback (including disagreement) is the most useful thing you can send. Open an issue at github.com/routewise96/robertium or email daniel@robertium.com. The dataset is archived on Zenodo (10.5281/zenodo.20110977) so specific candidates can be cited directly.

The remaining 15 hypotheses from the original top-20 will be reviewed under the same protocol, with full results published as a follow-up. Beyond that, the obvious next improvement is integrating curated drug-class metadata (DrugBank, ChEMBL, RxNorm) so the rule-based blacklist becomes a derived classifier driven by ontology.

Top-20 manual validation: full breakdown

Earlier today I published a self-assessment post about reviewing the top-5 high-confidence cross-domain hypotheses by hand. The exercise produced classifier v2 with six new exclusion categories and a refined 633-candidate catalog. As promised, here is the breakdown for the remaining hypotheses in the top-20.

After classifier v2 cleanup (PBB3, 1D9-LIRΔTP53INP2, ALS-CSF, and similar artifacts removed), the current top-20 by outreach score collapsed into about 10 distinct biological hypotheses. The rest are variant phrasings of the same drug-mediator-outcome chain surfaced from different domain pair directions, or different outcome wording (e.g. "Alzheimer's disease" vs "Alzheimer disease", "pancreatic ductal adenocarcinoma" vs "pancreatic adenocarcinoma"). This is the next obvious improvement on the roadmap: outcome normalization via UMLS or similar.

Distribution across the 10 distinct hypotheses:

• Reproduces FDA-approved indication: 1. erlotinib + EGFR + pancreatic cancer. FDA-approved in 2005 in combination with gemcitabine (NCIC-CTG PA.3 trial, Moore et al. 2005). Strong sanity check that the pipeline surfaces real repurposing directions.
• Reproduces ongoing or completed clinical investigation: 3. erlotinib + EGFR + glioblastoma (multiple Phase II trials, largely negative); metformin + α-synuclein + Parkinson's (active trials NCT07055958, NCT03685357, NCT05781711); valproic acid + HDAC + ALS (NCT00136110, Phase III negative per Piepers et al., Ann Neurol 2009).
• Mechanistically plausible, novel: 4. carbamazepine + KRAS + pancreatic ductal adenocarcinoma (KRAS-carbamazepine link documented in colorectal cancer (Shaykevich et al. 2024, Invest New Drugs, PMC10944448)); osimertinib + EGFR + Alzheimer's disease (drug with documented superior BBB penetrance, Kpuu 0.21 vs ≤0.12 for other EGFR-TKIs); valproic acid + PKM2 + pancreatic cancer survival (VPA has documented anti-PDAC activity through other pathways but PKM2-specific link is novel); osimertinib + EGFR + pancreatic adenocarcinoma (basket trial NCT02465060, direct PDAC indication not established).
• Preclinical evidence, no human trials: 2. auraptene + MMP-9 + epilepsy (citrus-derived coumarin with documented PTZ-kindling protection in mice (Iran J Pharm Res 2019, PMC6934955); MMP-9 implicated in epileptic focus formation per Pijet et al. 2018); apigenin + GFAP + Alzheimer's disease (reproduces known preclinical direction — apigenin/GFAP/AD studied in GFAP-IL6 mouse models, this is sanity check not novel).
• Variants through entity-resolution gaps: ~10 hypotheses. Same biology, different outcome phrasing or domain-pair routing. UMLS or RxNorm normalization would collapse these.

What this tells us:

• The pipeline reproduces real repurposing directions. 4 of 10 distinct hypotheses match documented clinical or FDA-approved use.
• Genuinely novel mechanistically-plausible candidates exist. 4 of 10. Each has the mediator pathway documented in one literature, the drug action in another, and no direct study connecting them.
• Entity normalization is the next improvement. ~10 of 20 top-quality hypotheses being variants of the same biological idea is a meaningful signal that outcome and drug normalization would improve catalog clarity.

Refined top-5 for researcher attention:

• carbamazepine → KRAS → pancreatic ductal adenocarcinoma. Strongest example of cross-domain discovery in the catalog.
• osimertinib → EGFR → Alzheimer's disease. Drug with proven BBB penetrance, target with emerging neurodegeneration relevance.
• valproic acid → PKM2 → pancreatic cancer survival. Established HDAC inhibitor, specific PKM2-PDAC survival link is novel.
• osimertinib → EGFR → pancreatic adenocarcinoma. Refinement of a known direction with a newer drug.
• auraptene → MMP-9 → epilepsy. Natural product with documented PTZ-kindling protection in mice.

If you work in any of these areas, feedback (including disagreement) is the most useful response. Full evidence chains: /hypotheses.

The catalog is mostly EGFR

Stepping back from the top-20 review: about half of the high-confidence list is variations on erlotinib or osimertinib hitting EGFR in some context. This is not a bug — EGFR is overexpressed in glioblastoma, the canonical driver of NSCLC, present in pancreatic adenocarcinoma, and increasingly implicated in Alzheimer's. The mediator is real and the cross-domain bridges through it are real.

But it does mean the catalog's diversity story depends on the long tail. Carbamazepine through KRAS, valproic acid through PKM2, ketamine through NMDA — these are the candidates that justify the cross-domain framing. The EGFR cluster is the sanity check.

Validation enrichment: clinical-trial and literature status for every outreach-quality hypothesis

Each of the 672 outreach-quality cross-domain hypotheses is now annotated with two external signals so a researcher can see at a glance what kind of candidate they are looking at: whether a clinical trial exists for the drug–outcome pair (via the public ClinicalTrials.gov v2 API), and whether the connection is already documented in the literature (via PubMed E-utilities).

Distribution after enrichment:

• Clinical trials: 50 active, 55 completed, 8 preclinical-only, 544 with no registered trial, 15 unknown (drug name not searchable as a single intervention).
• Literature: 361 novel signals (no PubMed paper proposes the drug as treatment/repurposing for this outcome), 117 partial evidence, 179 reproduces a documented association, 15 unknown.

The most useful combination for outreach is novel literature signal together with no clinical trial — those are candidates that Robertium surfaced and that nobody appears to have followed up on yet. The catalog at /hypotheses now shows badges on every row and a "Show novel signals only" button that filters the table to those candidates.

Implementation: scripts/enrich_clinical_trials.py and scripts/enrich_literature_status.py. Both scripts are rate-limited (5 req/s for ClinicalTrials.gov, ~2.5 req/s for PubMed without an API key), idempotent, and run only against outreach-quality hypotheses to keep the API budget bounded. Drug names that are combinations or identifiers (for example "ALA/AUR combination" or "1D9-LIRΔTP53INP2") are flagged unknown without making a request. Trials enrichment took ~4 minutes; literature enrichment took ~10 minutes.

Corpus doubled: 99,132 papers, 7,957 cross-domain hypotheses

Re-ingest expansion across all six domains is complete. size_target was raised to 15,000–18,000 per domain and the is_oa: "true" OpenAlex filter was removed (closed-access papers still ship metadata + abstracts, which is all the pipeline needs). The corpus roughly doubled and the cross-domain orchestrator was re-run on the rebuilt knowledge graph.

Before → after, in raw counts:

• Papers ingested: 47,370 → 99,132
• L1-passed (PubTator biomedical filter): 27,177 → 47,255
• Structured claims: 82,708 → 140,178
• Cross-domain hypotheses (raw): 4,593 → 7,957
• Outreach-quality hypotheses: 328 → 672 (62 high-confidence + 610 mid-confidence)

For researchers: more papers per domain produces stronger cross-domain bridges. New high-confidence candidates surfaced in the top: erlotinib → EGFR → glioblastoma (now #1 at score 0.457), metformin → α-synuclein → Parkinson's disease (a neuroprotective signal via the ALS–Alzheimer's mediator chain), and Riluzole → MMP-9 → epilepsy (an ALS drug bridging into seizure literature). The full ranked catalog is on /hypotheses.

Pipeline ran end-to-end across all six domains. Cross-domain orchestrator completed in under an hour with no quality-gate failures.

All six domains live: 4,593 cross-domain hypotheses now searchable

Major depressive disorder is now ingested and processed — the last of the six target domains. With depression in place, the cross-domain orchestrator ran the full 30 directional pairs (each domain as drug source × each other domain as outcome) and produced 4,593 cross-domain hypotheses with complete evidence chains, all browsable on the hypotheses page.

Final corpus numbers: 47,370 papers ingested, 27,177 passed the L1 biomedical-relevance filter, 82,708 structured claims extracted, six therapeutic domains in the knowledge graph (glioblastoma, epilepsy, ALS, Alzheimer's disease, pancreatic cancer, major depressive disorder).

For researchers: the catalog now lets you ask, for any pair of these diseases, which drugs from one literature might modulate outcomes in the other through a shared molecular mediator — even when no paper has ever co-mentioned the drug and the target disease. Filter by drug domain, outcome domain, or search by gene/protein/drug name; each hypothesis links to the underlying PubMed evidence on both legs of the chain.

Six therapeutic domains, 82,000+ claims

Robertium pipeline now covers six therapeutic domains: glioblastoma, epilepsy, ALS, Alzheimer's disease, pancreatic cancer, and major depressive disorder. Total: 47,370 papers ingested, 27,177 passed biomedical filtering, 82,708 structured claims extracted across all six domains in the knowledge graph.

The expansion validates that the methodology works across both oncology and neurology, with cross-domain hypothesis discovery now possible across 30 directional pairs of domains.

Didn't expect ketamine in the ALS list

Running the depression × ALS pair surfaced ketamine and esketamine through NMDA receptor signaling in ALS excitotoxicity. NMDA-mediated excitotoxicity in ALS is well-established but I had not seen ketamine class drugs framed as a repurposing direction in the ALS literature. Riluzole, yes — that's the existing standard of care via glutamatergic modulation. But ketamine specifically, no.

Worth a closer look. Either the connection is genuinely under-explored, or there's a reason ALS researchers haven't pursued it that I'm not aware of yet. The pipeline is finding things; whether each finding is real is a separate question that requires domain knowledge I don't have.

Cross-domain validation: epilepsy ↔ glioblastoma

The first cross-domain analysis between two diseases — epilepsy and glioblastoma — yielded 383 hypotheses with full evidence chains. Top candidates included pairs through MMP-9, P2X7R, EGFR, and ferroptosis-related mediators — including several that were later confirmed in the post-expansion catalog.

Notable: vortioxetine, an antidepressant published in Nature Medicine 2024 (Lee et al.) as showing anti-glioblastoma activity, automatically surfaces in our cross-domain catalog when depression is added — providing independent validation of the methodology.

Methodology decisions: why we chose Swanson's ABC model

There are several approaches to literature-based discovery. We chose Don R. Swanson's ABC model from the 1980s — surfacing hypotheses where drug A and disease C share a mediator B but have no direct co-mention.

Why this approach:

• Interpretable: every hypothesis is a chain of evidence with PMIDs
• Conservative: minimal hallucination compared to free-form LLM generation
• Reproducible: deterministic graph algorithm on structured claims
• Validated: Swanson's original work demonstrated successful repurposing predictions (fish oil → Raynaud's, magnesium → migraine)

We use modern LLMs only for the claim extraction step, keeping hypothesis generation in deterministic graph queries.

Open data philosophy

Robertium is open-source (MIT) and open-data (CC-BY-4.0 on first preprint release). This is intentional.

Drug discovery infrastructure is too important to be locked behind proprietary platforms. We believe:

• The biomedical literature is a public good. Tools to navigate it should be public too.
• Reproducibility is fundamental to science. Closed systems break that.
• Researchers should inspect, modify, and improve the methods that surface hypotheses they spend years validating.

Sustainability comes from collaboration with researchers, not lock-in.