The Product Journal by Nova Skye Nova Skye · Technical product manager Contact

Companion · 03 · The knowledge pipeline

  1. Problem
  2. Rules
  3. Evidence
  4. Decisions
  5. Result

The pipeline that builds the chatbot's knowledge base

The SettleSMART chatbot uses retrieval-augmented generation (RAG): it answers only from a knowledge base. This is the system that collected and cleaned the knowledge of more than sixty settlement organizations, and the rules I set before the first line of code.

In short

10,000+
cleaned pages
500+
PDFs
60+
settlement organizations
The crawl console dashboard in July 2026, built for verification: tiles for Fetched, Included, Manual review, Duplicate and Failed, then tables of engineer labels such as supporting_context, kb_candidate and manual_review The same dashboard in November 2026: one heading, What happens to each page, four outcomes named Included, Duplicate, Left out and Not reached, and a sentence saying 96 pages were left out on purpose, with the diagnostics folded away
The console in July and in November, on the same crawl. The first showed every decision in the engineer's terms, which I needed while I learned what the crawler did. The second asks one question and answers it in a sentence. Tap to enlarge.Fig. 01
Stage
In use · builds the SettleSMART knowledge base
My role
The problem, the rules, the scope, the quality bar, the trade-offs, the stakeholders, and the final call on what "done" means
The bet
Rules with tests, not a model, will give the chatbot a knowledge base that can be audited line by line
Measure
10,000+ cleaned pages and 500+ PDFs from 60+ organizations
Technology
Python · FastAPI · React console · SQLite
Section The story Marginalia · the PM layer
1

The problem

The knowledge newcomers need lives on the websites of more than sixty settlement organizations across Alberta: program pages, eligibility rules in PDFs, office hours, a phone number on a contact page.

2

Two rules before any code

Archive first, judge later. Rules, not a model, decide what is knowledge.

Archive first. Fetch every page a site offers and keep the raw text. Decide afterwards, at export time, what belongs. A wrong judgement before the fetch costs a second visit to the site; a wrong judgement afterwards costs a rule change and a re-export.

Rules, not a model. No language model anywhere in the pipeline decides what is true. Every decision is a deterministic rule with a test that reproduces the real page it came from. Budget: at about 17,000 pages per run, paying AI credits to classify them made no sense. Trust: a rule makes the same mistake every time, so an audit can find it and a test can fix it. A model's answer changes between runs and cannot be audited line by line.

The new-crawl form in November 2026, with the first two rules written on the screen: fetch the whole site first, decide later; no AI, no external calls beyond fetching the site
The new-crawl form carries the first two rules in its own words: "Fetch the whole site first, decide later" and "No AI, no external calls beyond fetching the site." Tap to enlarge.Fig. 02
3

Checking the export against the live sites

A junk scan only reports patterns present in a file, so it cannot catch a deletion. I asked for comparisons against the live sites, and then for something simpler: pick files at random and check them against the real page. Seven rounds in two days. Each round: read every sampled file by eye, trace each oddity to the raw text, count how many files across the whole export carry the same pattern, and only then write a rule with a test.

The eight issue types, by files affected
Issue found, rounds six and sevenFiles affected, counted before fixing
A bold marker glued after a colon1,129
A line with an unpaired bold marker, a run split across a line break559
Chevron breadcrumbs left in the text443
"here" and "Learn more" links to other sites dropped316 pages
Indented paragraphs that render as code blocks233
Two clock times with no dash ("9:00 a.m. 11:30 a.m.")216
Breadcrumbs whose chevron was an icon glyph170
Sentences glued together ("assessment.If you")126

The live comparison on 29 October caught five issues the scans could not see. Round seven caught an office-hours line on a contact page being judged "garbled", because "8:30am–4:30pm" looks like nonsense to a detector built for broken PDF fonts. Each one became a rule with a test.

4

The decision log

No.WhenDecisionWhyWhat followed
P-01Jul 2026Archive first, judge laterThe raw text is kept, so a wrong rule costs a re-export.Rule changes are applied by re-exporting the kept text, with no second visit to the site.
P-02Jul 2026Rules with tests decide what is knowledge, never a modelBudget and trust. A rulebook with tests is the simplest way to show a funder or a reviewer how every decision was made.Every cleaning rule is tested against the real page it came from.
P-03Aug 2026Two-tier fetching: plain HTTP first, a headless browser only when a page needs itA 2.7× per-page speed-up on the benchmark. I asked what would break if the fast tier guessed wrong: a page with too little text, which the quality check already flags.The crawl engine shipped with both fetch tiers.
P-04Sep 2026Explain every number on the console in plain language; fold diagnostics awayThe staff who run this tool should not need me in the room.Every default got a reason written next to it.
P-05Oct 2026Shape the export to the chatbot team's two constraintsThe chatbot's search reads text, so a header in every file tells the model which organization a page came from. Its uploader matches on file name, so names are stable.The exporter ships a standard header in every file and stable names.
P-06Oct 2026Hunt for losses by random sample against the live page, not by junk scan aloneA junk scan cannot catch a deletion.The office-hours line was caught this way.
The live monitor in November 2026 adding up the pages: 243 found equals 147 downloaded plus 0 could not download plus 72 left out on purpose plus 24 PDF or Word files plus 0 not reached The Borderline pages queue in November 2026, with the rule written on the screen: the rules decide, and a human decision always wins over the automatic one
The live monitor adds up every page found. The review queue became "Borderline pages", with its rule on the screen: the rules decide, and a human decision always wins. Tap to enlarge.Fig. 03
5

The result

Outputs: what shipped

  • A crawl engine with two fetch tiers, PDF and Word reading, and a named, polite identity
  • A console in plain language: jobs, live monitor, page explorer, borderline queue, cleaning rules, export, audit
  • An exporter shaped to the chatbot team's brief: a standard header in every file, stable names, one folder per organization, a manifest and a loss check per run

Outcomes: what changed

10,000+
cleaned pages to date, plus 500+ PDFs, from 60+ organizations
6

What I took from it

The pipeline taught me that a RAG system is only as good as what goes into it.

I learned how to track every file, and the source behind each piece of generated content.