Remembering everything

For nineteen months I have kept a transcript of every substantive conversation in my life. The corpus, as measured: 420 dated transcript files, 93 people with files of their own, 23 MB on disk, earliest capture January 2025. When I need to know what someone told me on a specific call, an agent finds the answer, reads the evidence, and cites the source file in about fourteen seconds. This post is about the pipeline behind that, told mostly through the parts that broke.

The measurement

The benchmark is question in, answer out. An agent with a shell and file-search tools, pointed at the corpus root, gets a question whose answer it has to locate and then actually read:

Search the transcripts/ directory. What have I been told about managed agents, and on what date? Answer in two sentences with the source filename.

run 1    14.0s
run 2    14.0s
run 3    14.9s

median   14.0s

All three runs returned a correct, sourced answer with the right file. I ran a harder variant first, naming a specific person alongside the topic, and it came back slower and far less consistent: 15.0, 23.1, 16.3 seconds. Naming someone makes the agent disambiguate a speaker as well as a subject, and on the slow run it kept reading until it had found three separate calls instead of stopping at one. I am not publishing that query, because it names a real person and describes what he said to me in private, and the published query has to be the exact string that was timed. So the number above comes from a query I can print in full.

I have no "before" time to put next to it, and I will not fabricate one. Before this existed, the answer to that question was for practical purposes unretrievable: the original recording would have to be found and re-listened to, assuming it still existed anywhere. Fourteen seconds against probably-never is the honest comparison, and it beats any invented afternoon.

Words and voices are separate problems

Audio reaches the corpus from two directions: recordings I make on my phone and meetings captured by a notetaker. Recordings go through two APIs in parallel. Deepgram Nova-3 produces the words with per-word timestamps and nothing else. pyannote produces the diarization: which voice was active during which time ranges, matched against a small library of enrolled voiceprints. The merge is one rule: each word is assigned to whichever speaker pyannote says was talking at that word's timestamp. Neither tool is asked to do the other's job, which means when attribution goes wrong I know which of the two to blame.

Two knobs that used to be one

Speaker identification carries the first scar. Until 2026-07-30 a single variable did two jobs: it went to pyannote as the matching threshold, and it doubled as my gate for accepting the identity pyannote returned. Those look like the same number. They are two numbers, because pyannote in exclusive mode force-assigns enrolled identities to leftover clusters at arbitrarily low confidence; its threshold shapes the matching without rejecting anything. So loosening the matcher to catch a quiet speaker silently loosened the acceptance gate too, and a weak match could walk into the corpus wearing a real person's name.

The fix split them. --threshold goes to pyannote. --match-floor is mine, applied to the confidence that comes back: anything under it is demoted to an anonymous SPEAKER_XX label and routed to a reconciliation step where a human confirms the identity. A silently wrong name poisons that person's relationship record and every future answer built on it. An anonymous label costs one question. Erring toward "ask" wins on arithmetic.

The floor is calibrated from the pipeline's own history. A report script replays the cached diarization results and histograms every match: at the baseline, 44 runs and 136 clusters, true matches sat between 70 and 100 while false positives topped out at 29. A gap of 22 points, with the floor inside it. The script warns when the gap narrows to ten, because it narrows as the voiceprint library grows and the calibration has to be redone.

The default you cannot walk back

The speaker-count flag defaults to auto-detect, even though most of my calls have two people, because the two failure directions cost different amounts. If auto-detect over-splits and invents an extra cluster, that cluster comes back at low confidence, the floor demotes it, and fixing the record is a relabel. If I force two clusters onto audio that had three people, the third person's lines scatter across both labels. No relabeling separates them again; the audio has to be re-run from scratch. When one mistake is a rename and the other is unrecoverable, the default belongs on the side you can undo. Asymmetric failure costs decide more of this pipeline's settings than accuracy does.

Build on the surface the vendor promises to keep

Meetings come from Granola, a notetaking app that records calls. The first version of my sync did something clever: it scraped an internal session token out of the desktop app's local files and used it against the internal API. Then version 7.255.6 moved the token store to encrypted-only storage, the plaintext file my code read stopped being refreshed, and the whole channel went dark on an update I had no say in. The replacement uses the public API with a permanent key created in the app's own settings. The key never expires and knows nothing about the app's internals, so it survives every release. Boring beat clever the moment the vendor shipped an update.

Granola also has a defect I design around rather than fix. Its speaker separation is exactly two channels: microphone audio is me, system audio is everyone else. On a two-person call that is a correct diarization. On a call with three or more people, every voice other than mine collapses into one name. The pipeline cannot repair that, so it stops pretending: in group-call transcripts, quotes from anyone but me are attributed as "likely" with the name, never as certain, unless something outside the transcript confirms it. In-person captures are worse, since every voice in the room hits my mic; those files carry a speakers_unreliable: true flag and the verbatim opens with a note saying the labels below cannot be trusted. It costs me something to write that flag into my own archive, but a record that states its unreliability stays usable years later.

The meeting that never happened

The strangest failure produced a rule I now consider load-bearing. The pipeline once picked up the recording of an interview that never took place. The transcript body was effectively empty, stray fragments and nothing more, and the tool generated its notes layer anyway: a plausible, well-structured writeup of a meeting nobody attended. Had that file been processed normally, my corpus would hold a confident record of a conversation that never occurred, and some future agent would cite it. The rule since: an empty or near-empty transcript body means no conversation happened, whatever the generated notes claim, and the file gets flagged for human context instead of written up. A summary is not evidence that a conversation occurred.

Six gates and no model

Text messages are the third channel, and their ingestion is the most conservative code I own, because it reads words other people wrote on a machine that holds disk access and push rights to my own repos. No model sits in the routing loop. A conversation is ingested only after it passes six deterministic gates: the sender must be shaped like a real phone number or email address; it must carry at least seven digits, which kills shortcodes; I must have sent at least one message in the thread; the thread must span at least eight messages across at least two active days; the content must be less than half OTP-and-notification-shaped; and the handle must be explicitly mapped, by me, to a person I already keep a file on. Unmapped handles are skipped and reported, never guessed.

The best gate is the cheapest: outbound greater than zero. You never reply to your bank. A counter of sent messages separates humans from noise more reliably than any classifier I could run, and it is free.

macOS Contacts is deliberately never consulted for identity. A contact name is attacker-settable: anyone who texts me can ask to be saved under any name they choose, and I would be the one saving it. Trusting that field would mean letting a stranger pick the label that decides whether their messages get ingested. Identity comes only from my own explicit mapping, and the mapping fails closed.

Two layers

Everything lands in one folder of markdown files, one per conversation, plus one file per person. Every transcript has two layers. The top is curated: a short frontmatter summary saying who was involved and why the conversation mattered, then key topics, quotes with stated attribution confidence, decisions, action items. The bottom is the verbatim layer: the full speaker-labeled transcript, appended by a script and never touched by hand. That rule is mechanical on purpose. The moment a model is allowed to clean up the verbatim, the verbatim stops being evidence, so the append is idempotent and there is no other way to write that section.

The two layers are why the fourteen seconds holds as the corpus grows. Full-text search across all 23 MB runs in under a second and always did; grep was never anyone's bottleneck. The scan that matters reads only the top of every file: the summary layer across all 420 transcripts comes back in 0.04 seconds, and that figure stays flat as individual transcripts get long, because the verbatim bodies sit below the fold. An agent scans every summary, shortlists the two or three files worth reading, and spends its time inside those. Cheap wide scan, expensive narrow read. File size stopped being a design constraint the day the layers were split.

The per-person files close the loop. Each carries a table of every transcript that person appears in, each transcript names its participants, and a validator walks the graph in both directions and reports anything dangling. Ask about a person and the agent gets their full history; ask about a topic and it lands on the people who carried it.

If I rebuilt this from zero, I would keep one design idea before any component: the system is only as good as its refusals. The matcher refuses to name a speaker below the floor. The ingester refuses to guess who a handle belongs to. The transcript refuses to hide that its labels are unreliable. The empty recording refuses to become a meeting. Each refusal was bought with a specific failure, and each one converts a silent wrong answer into a loud question. Memory that admits what it does not know is the only kind worth querying.

A memory you can trust is a stack of refusals with a search box on top.