Record 01Seven versions rebuilt: the tool was fine, the way of working was not
Record 01 / 2026-10-30
Seven versions rebuilt: the tool was fine, the way of working was not
Seven versions in one day, all seven rebuilt. The HTML-to-PDF tool never failed once — and because it never failed, the mistakes piled up just as fast.
01 What happened
- I took "the workflow" to mean only the output machinery: split, render, check overflow, a few minutes per loop. But nothing guarded the question "what goes on this page and why", and I built and showed all seven pages at once. One wrong call meant seven wrong pages; the faster the tool, the faster the mistakes.
- I acted on the words and never checked the intent. Asked for "rules", I wrote twelve and then swapped out better material to obey them. Told the text was "final", I dropped it verbatim into bullet points. Pages with no fitting material got a table of numbers or a box diagram anyway, although I had been told to leave them empty. All of it is one failure: never asking what the reader should take from this page.
- The text had grown one sentence per review comment, never rewritten for the flow between pages. My caution about numbers leaked into the layout as grey caveats on every page. And three parties were producing — the author, me, another AI — with no one owning the final version, so versions multiplied without improving.
02 Human correction
- Fix the order and show one thing per step: the author's skeleton → prose that reads straight through → cut (keep only what the last page pays off) → one page or one figure to settle the direction → then the rest. Tool checks are floors only — no overflow, no tiny type, no per-page caveats — never ceilings like "every page needs a figure". A page without material stays empty and says so.
- Before starting, confirm: who reads, for how long, and what one sentence they should leave with; what "good" looks like as two or three real examples, not adjectives; which materials are allowed, banned, or may be left out; and who decides the final version. Put these questions before executing any instruction.
Evidence課題選考デッキ v3〜v7 の作業記録（非公開）process · design · communication
Record 02 / 2026-09-26 / J-Grad Admission RAG
Stop sending the AI on errands; hand it the whole folder once
"Does someone currently employed need an English score for this program?" — one question, and the page waited 30 seconds before saying "timed out".
01 What happened
- The first build split one question into several smaller ones, asked the AI each separately, and stitched the replies into a template. Like sending someone on errand after errand, every trip added waiting.
- Timing each stage showed the question was understood fine; writing the answer hit the external service's limit.
- Raising the limit only makes the failure arrive later.
02 Human correction
- Reordered the work. Once the question is understood, everything that can be settled locally — finding passages, narrowing to the school, checking rules — runs in our own code; the evidence goes to the AI as one bundle, and it writes once. A repeated question gets the previous answer back.
- At most two AI calls per question. What got faster was not the AI, but how often we asked it.
EvidenceIssue #171 ↗Issue #180 ↗ADR 0006 ↗RAG · latency · design
Record 03 / 2026-09-25 / J-Grad Admission RAG
The more checks we added, the more correct answers failed
"English is worth 100 points; the interview is online" — the first half is in the document, the second is not. This answer passed every check we had.
01 What happened
- To keep the AI from saying things not in the document, I wrote into the acceptance criteria: "the program must catch any unsupported statement in the answer."
- Version one compared words; version two matched meaning word by word. Every counterexample added a rule, until the correct paraphrase "up to 100 points" was rejected too.
- When version three started parsing subjects and objects, I noticed I could no longer explain what the check did. The automated tests had stayed green the whole time.
02 Human correction
- We dropped the direction. Questions like "am I eligible" or "what do I submit" are answered by code following human-reviewed rules; the AI only turns the result into prose, labelled "reference".
- Rules since: a counterexample is a clue to a cause, not a new rule. And a mechanism I cannot explain does not ship, however green the tests.
EvidencePR #176 ↗PR #178 ↗PR #179 ↗LLM · review · scope
Record 04 / 2026-10-24
Prohibition-heavy instructions came back as audit-speak
"All three execution authorizations are exhausted. There is no authorization for a fourth. Unknown usage remains fully reserved." — I could not tell what it had understood.
01 What happened
- The cause was my own instruction file: prohibitions and definitions, not one line about what the work was for.
- An AI mirrors the register of its instructions. More prohibitions, more "X is not to be regarded as Y".
- Adding requirement-probing tools did not help. The problem was not the requirements but the missing purpose.
02 Human correction
- Instructions now open with three sentences: what for, who uses it, what counts as done. Only irreversible actions remain prohibited.
- The same AI, next day: "I understand you want me to compare Japanese AI job requirements with your work, find the gaps, write short briefs, and stop once delivered."
EvidenceCOLLABORATION.md（内部）agents · prompting
Record 05 / 2026-09-28 / J-Grad Admission RAG
"Unknown" must never become "no"
For someone who had not said whether they were employed, the system answered "no employment certificate needed". The right answer was "please confirm whether you are employed".
01 What happened
- Most programs make decisions yes-or-no, and an empty field counts as no. So the rule "employed applicants must submit" read as "no need to submit" for anyone who had not answered.
- This is the most dangerous kind of error at an admissions desk: a student told "not needed" does not check again.
- One such case surfaced while adding the University of Tokyo documents.
02 Human correction
- Every decision now has three outcomes: applies, does not apply, not enough information. When information is missing, the answer says what is missing.
- There is one such decision engine, shared by both universities. Not building one per school means the same mistake never needs fixing twice.
EvidencePR #215 ↗ADR 0012 ↗rules · design · safety
Record 06 / 2026-10-01 / J-Grad Admission RAG
I had become a messenger
Paste the development AI's summary to the design AI, paste the reply back. I realised I was shuttling small fixes and no longer looking at what had been built.
01 What happened
- Each new document meant writing the same content in separate places: screens, report, source notes. The more places, the more likely one goes stale.
- I had become a messenger because the work ran as one question, one reply, with no point where a stage's results were reviewed together.
- Adding a document had become transcription, not judgement.
02 Human correction
- Two changes. Questions go out in one batch and fixes come back as one package at the end of a stage; screens, report and configuration are all generated from a single human-checked table.
- No before-and-after timing, so no "X percent faster". What I can say: writing the same thing in five places is gone, and what I review is only whether the content is right.
EvidencePR #277 ↗PR #289 ↗ADR 0017 ↗agents · process
Record 07 / 2026-09-27 / J-Grad Admission RAG
Adding the second school, the first school's knowledge came out of the code
With nothing as input, the program "found" 25 Science Tokyo entity names. The knowledge was coming from the code, not the PDF.
01 What happened
- Code written against the first school's documents had knowledge baked in: "page 7 is the common section", "the 2027 clause", "this college has this name". With one school, nobody notices.
- Run the University of Tokyo documents through the same code and Science Tokyo's page layout and college names leak in.
- This does not prove that automatically extracting names and relations cannot work; it was a construction problem. But it showed that what ran correctly for school one is not thereby correct for school two.
02 Human correction
- The old code was labelled "first school only" and isolated so no other school can use it. A new school's knowledge enters only through human-reviewed excerpts and relations.
- One line was added to the checklist for every new school: is the previous school's knowledge hiding somewhere in the code?
EvidencePR #208 ↗ADR 0011 ↗data · multi-school · testing
Record 08 / 2026-09-01 / J-Grad Admission RAG
We promised page numbers, and could not name the page for more than half
On the first real guideline, more than half the extracted facts could not say which page they came from. Every test had passed.
01 What happened
- The AI's code discarded the PDF's page information when converting to text, then guessed pages afterwards by searching the text for "Page N".
- Paragraphs without a heading had no such marker and lost their page. Tests used synthetic data that always had headings, so this path was never exercised.
- I noticed by counting how many facts built from the real PDF had an empty page field.
02 Human correction
- Page numbers are attached to each paragraph the moment the PDF is read and carried to the end, never reverse-engineered from display text.
- Since then the first checkpoint of every stage is one end-to-end run on the real PDF. Passing on synthetic data is only the beginning.
EvidenceIssue #3 ↗PR #10 ↗RAG · data · testing
Record 09 / 2026-09-14 / J-Grad Admission RAG
All tests green. A maths-only exception had become "university-wide"
The sentence "mathematics handles English scores differently" had been registered as a university-wide rule. The neighbouring sentence had a test, and that one really was university-wide.
01 What happened
- The step that tags each sentence with "who does this apply to" looked at the big section heading first and the department name second.
- This kind of error happens before retrieval or answering. Get "who does this apply to" wrong, and every citation looks right while applying another department's rule to this student.
- The test protected only the one sentence someone had wanted fixed.
02 Human correction
- Only paragraphs that genuinely sit in a shared position in the document's structure can be judged university-wide. No special-casing of particular sentences or words.
- Adjacent correct and incorrect examples from the same real PDF were both added to the tests. When one sentence is fixed, its neighbour is checked too.
EvidencePR #108 ↗RAG · testing · scope
Record 10 / 2026-10-24 / J-Grad Admission RAG
The number in the README had no record behind it
The retrieval score on the public page and the only saved result file in the repository did not match.
01 What happened
- A different AI noticed. Reading the public repository, it recomputed the score from the saved file and reported a mismatch.
- A repository-wide search found the published number only in prose; the result file had been saved exactly once.
- Most likely a trial run in an unsaved working folder left its number in the text and nowhere else.
02 Human correction
- Three places were corrected to the saved file's value, with "when, how many queries, which data" written beside the number, and a one-line correction left in the old document.
- Since then every published number binds to a saved record. A number without a record is not written, however well I remember it.
EvidencePR #300 ↗保存レポート ↗evaluation · honesty
Record 11 / 2026-09-26 / J-Grad Admission RAG
The same index already existed in five places
Every acceptance run began with about seven minutes of waiting while the index was rebuilt. But the index should have existed long ago.
01 What happened
- "Load the index" and "build the index" lived in the same function; if the location could not be read, it decided the index did not exist and rebuilt it.
- After a permission error, the AI's fix was to create a new working folder, which rebuilt the index as well.
- A count found identical indexes in five places.
02 Human correction
- Three rules: feature code uses existing data read-only; rebuilds, copies and downloads each need approval; lacking permission is not the same as the data not existing.
- "Count what exists before building" became a fixed item in my reviews.
EvidenceIssue #183 ↗PR #184 ↗data · ownership
Record 12 / 2026-10-24 / Japanese Speaking Coach
Grammatically perfect — and the wrong question for this listener
To a Japanese colleague, the app suggested asking back "how long have you been in Japan?". Format, grammar and politeness checks had all passed.
01 What happened
- Asking "how are things over there?" when both are in the same room; not returning a greeting the other person offered first. Every sentence was correct, at the right politeness level.
- Automatic checks scored full marks. The generating AI's own "check yourself" step never caught one.
- A separately prepared judge AI noticed, reading from the listener's side, and raised the same point twice in a row.
02 Human correction
- One rule added: return questions and greetings must also make sense from the listener's side. Then one more round of checks.
- The judge is also an AI and can share the same blind spot. The final word rests with a native speaker's review, which has not happened yet.
Evidencestatus.md ↗D-005 ↗evaluation · LLM-as-judge
Record 13 / 2026-09-27 / J-Grad Admission RAG
An inconclusive experiment, closed as inconclusive
In a comparison trial of a new PDF parser, the pages processed exceeded the budget more than twofold. After repeated stops and restarts, nobody could say which result belonged to which run.
01 What happened
- The design fixed a page budget and a pass threshold in advance. The budget exists to prevent "run until something looks good".
- A "adopt as fallback" proposal appeared mid-way and nearly became the conclusion. When another configuration stalled, it was tempting to call the tool slow.
- Neither had enough evidence. The stall came from our own monitoring program.
02 Human correction
- No reruns to make the numbers look better; closed as "not adopted". Every run record was kept and the unverified items were listed.
- An experiment that exceeds its budget does not count, however good its numbers. That is now a commitment made before an experiment starts.
EvidenceIssue #177 ↗ADR 0009 ↗evaluation · decision