The problem
The knowledge newcomers need lives on the websites of more than sixty settlement organizations across Alberta: program pages, eligibility rules in PDFs, office hours, a phone number on a contact page.
The SettleSMART chatbot uses retrieval-augmented generation (RAG): it answers only from a knowledge base. This is the system that collected and cleaned the knowledge of more than sixty settlement organizations, and the rules I set before the first line of code.
In short
The knowledge newcomers need lives on the websites of more than sixty settlement organizations across Alberta: program pages, eligibility rules in PDFs, office hours, a phone number on a contact page.
Archive first, judge later. Rules, not a model, decide what is knowledge.
Archive first. Fetch every page a site offers and keep the raw text. Decide afterwards, at export time, what belongs. A wrong judgement before the fetch costs a second visit to the site; a wrong judgement afterwards costs a rule change and a re-export.
Rules, not a model. No language model anywhere in the pipeline decides what is true. Every decision is a deterministic rule with a test that reproduces the real page it came from. Budget: at about 17,000 pages per run, paying AI credits to classify them made no sense. Trust: a rule makes the same mistake every time, so an audit can find it and a test can fix it. A model's answer changes between runs and cannot be audited line by line.
A junk scan only reports patterns present in a file, so it cannot catch a deletion. I asked for comparisons against the live sites, and then for something simpler: pick files at random and check them against the real page. Seven rounds in two days. Each round: read every sampled file by eye, trace each oddity to the raw text, count how many files across the whole export carry the same pattern, and only then write a rule with a test.
| Issue found, rounds six and seven | Files affected, counted before fixing |
|---|---|
| A bold marker glued after a colon | 1,129 |
| A line with an unpaired bold marker, a run split across a line break | 559 |
| Chevron breadcrumbs left in the text | 443 |
| "here" and "Learn more" links to other sites dropped | 316 pages |
| Indented paragraphs that render as code blocks | 233 |
| Two clock times with no dash ("9:00 a.m. 11:30 a.m.") | 216 |
| Breadcrumbs whose chevron was an icon glyph | 170 |
| Sentences glued together ("assessment.If you") | 126 |
The live comparison on 29 October caught five issues the scans could not see. Round seven caught an office-hours line on a contact page being judged "garbled", because "8:30am–4:30pm" looks like nonsense to a detector built for broken PDF fonts. Each one became a rule with a test.
The pipeline taught me that a RAG system is only as good as what goes into it.
I learned how to track every file, and the source behind each piece of generated content.