How Source Check works

Source Check answers certamen questions from the books the NJCL lists as sources, and cites the printed page every claim comes from. This page explains how it is built, from the PDFs to the answer on your screen.

Overview

You ask a question, or paste a tossup with its answer. Source Check searches the textbooks for the passages that bear on it, then has a language model write an answer that may use only those passages. Every claim carries a citation marker like [1, 2]. Clicking one shows the exact sentence it came from, with the book and page.

When you paste a tossup, the answer starts with a verdict (Supported, Contradicted, Partly supported or Not found in the sources) and then checks each clue separately.

The sources

The library is the set of books the NJCL lists for certamen (see the NJCL source list): thirteen books covering Roman history, mythology, Latin literature and daily life, about 7,000 pages in all.

Pipeline at a glance

1. 1ExtractPDFs → text, one record per printed page
2. 2Indexpages → passages, embeddings and a keyword index
3. 3Retrievequestion → the 12 most relevant passages
4. 4Answerpassages → a cited answer, quotes checked

Steps 1 and 2 run once, offline, whenever the books change. Steps 3 and 4 run for every question.

1. Extracting pages

The obvious approach, converting each PDF to one long text file, throws away the one thing a citation needs: the page. So the text is pulled out one page at a time, and each page keeps the number printed on it in the book. That number is often not the PDF's page index. Grimal's page 298 is the 307th page of the PDF, because of the front matter.

Printed page numbers come from two places:

• Embedded page labels. Ten of the thirteen PDFs carry the book's own numbering, which the extractor reads directly.
• The page headers. For the other three, the number is read from the running header or footer. The scans' text layers often drop digits ("105" comes out as "5"), so each page's number is decided by a vote of the pages around it rather than trusted on its own.

Pages without a usable text layer are run through OCR (Tesseract). Chronicle of the Roman Republic was OCR'd from scratch: half of it had no text at all. The extractor also strips running heads and page numbers, rejoins words hyphenated across lines, and keeps two-column dictionary pages in reading order.

Each page becomes one JSON record:

{
  "source_id": "grimal",
  "cite": "Grimal, Dictionary of Classical Mythology",
  "pdf_page": 307,
  "page": "298",
  "ocr": false,
  "text": "Myceneus  The hero who founded the city of Mycene and gave it his name. …"
}

2. Building the index

Pages are split into passages of about 250 words. A passage never crosses a page boundary, so every search hit maps to exactly one page. The three mythology dictionaries use smaller passages (about 100 words), roughly one entry each; otherwise a short entry gets lost among its neighbours on the page.

Each passage is stored twice:

• as an embedding, a 768-number vector from Google's gemini-embedding-001 that captures what the passage means, and
• in a keyword index (SQLite FTS5, ranked with BM25) for exact words and names.

The finished index is a single SQLite file: 23,855 passages, about 127 MB. There's no database server to run.

3. Finding passages

Certamen questions hinge on exact names (Myceneus, lex Hortensia, Brixia), and that's where meaning-based search is weakest. So retrieval combines several searches:

1. Query rewriting. A quick model call turns the question into two or three search queries and adds alternate spellings: Mycenae/Mycene, Heracles/Hercules, Odysseus/Ulysses. In a conversation it also resolves follow-ups like "who was his father?"
2. Meaning search. Each query is embedded and compared against every passage.
3. Keyword search. All the words together, plus a separate search for each rare word. Otherwise a page that repeats "Latin", "author" and "born" outranks the one page that says "Brescia".
4. Stem matching. A word with almost no exact hits is also searched by its stem, so contubernalis finds contubernium, contubernia, and OCR slips like contubernah.

The ranked lists are merged with reciprocal rank fusion: a passage scores well if it ranks high in any list, and better if it shows up in several. Hits on rare words count double.

score(passage) = Σ over lists  weight / (60 + rank in that list)

The top 12 passages go on to the next step.

4. Writing the answer

Gemini 3.8 Flash receives the numbered passages and the question, with instructions to use nothing else. It doesn't write citation numbers itself. It returns the answer as a list of claims, and for each claim the exact sentence, word for word, from the passage that supports it.

{ "segments": [
    { "text": "Commodus was born on 31 August 161.",
      "sources": [ { "passage": 3, "quote": "Born 31 August 161 at Lanuvium" } ] }
] }

The server then checks every quote against the passage text. Matching ignores spacing, punctuation and case. A quote that isn't really there is replaced by the closest matching sentence and flagged as such. Finally the server numbers the citations in order and attaches the book and page to each. Because the numbering happens in code, a marker can't point at the wrong sentence.

Alternate versions of myths

Myths come in versions, and certamen questions usually follow the standard one. Before the model reads a passage, the server marks wording that signals a variant ("some say", "another version", "according to some", "in a later version") and wording that signals the usual account ("the usual version", "most authors").

The answer gives the standard version first and lists the others under Alternate versions, quoting each source's own framing. A tossup whose clue matches only a variant is marked Supported (alternate version only). For example, Paris killing Achilles from behind a statue in the temple of Thymbraean Apollo is a later version tied to Polyxena; the standard version is an arrow guided by Apollo at the Scaean Gate.

How it runs on the site

The textbooks and their index live in a separate Cloud Run service. The site's server is the only thing that can call it, using a shared secret. The site's code is on GitHub but the books are not: the index ships inside the sources service's container image, which is built directly from local files.

On the site, Source Check is for signed-in users, with a limit of 60 questions per hour each.

Copyright

Most of these books are under copyright, so Source Check is built to quote them, not republish them. It shows only the cited sentence plus one sentence on each side, with the book and page, so you can check a claim. It never serves whole passages or the PDFs, and it is limited to signed-in users so it can't be used to copy a book out bit by bit.

Limits and costs

• OCR noise. Some books' text layers are rough. Quotes from Morford & Lenardon can contain garbled words, and words printed in Greek letters in the dictionaries come out as junk.
• Answers can be wrong. Retrieval can miss the passage that matters, and the model can misread one. That's why every claim is cited: read the sentence.
• Speed. An answer takes roughly 10–17 seconds, most of it the model thinking before it writes.
• Cost. About 1¢ per question. Building the index cost under $1 in embeddings.