Rules, Not Ratings: A Weekly Job That Learns My Taste (and Can't Be Trusted)

Part 5 of a series on building Wardrobe AI, and the last of the core arc. Part 1 set up the guiding principle and Part 2 the recommendation pipeline; Part 3 built a per-item feedback loop; Part 4 introduced LangGraph with a trip planner. This post combines the two threads: a second LangGraph that learns my taste, and the guardrails it needs because, unlike everything before it, it rewrites the app’s own behavior with no one watching. All the code in this post is public, in github.com/JiamanBettyWu/mise; snippets link to their source.

Why thumbs-on-items wasn’t enough

The feedback loop from Part 3 learns which items I like. It works, but I kept feeling it was learning the wrong thing, and it took me a while to see why.

My wardrobe is not Netflix. Netflix recommends from a vast catalog of films I mostly haven’t seen; figuring out “will she like this title” is most of the job. But every item in my closet is there because I already chose it. I saw it in a store, liked it enough to buy it, and brought it home. The catalog is pre-filtered by my own taste. So asking “does she like this jacket?” is a solved question; the answer is almost always yes, that’s why I own it.

That makes the per-item signal weak. The Beta-Bernoulli multiplier can tell a beloved scarf from a meh one, but the spread is narrow, because there are no truly bad items to find. The real quality problem lives in the combinations and the patterns. I might love a blazer and love sneakers and still not want them together. I might lean toward neutral palettes, or avoid layering in summer, or always want one statement piece. Those are rules, not ratings, and no amount of per-item scoring discovers a rule.

So I wanted a second kind of learning: something that reads my whole history of reactions and distills the durable preferences behind them. “You tend to avoid bright colors.” “You rate monochrome outfits higher.” Statements that could feed back into the prompt and shape future outfits at the level where my taste actually operates.

The catch: this thing edits its own brain

Here’s what makes this different from anything earlier in the series. A learned preference doesn’t just nudge one item’s odds; it goes into the generation prompt as a standing instruction and shapes every future outfit. And as I argued in Part 3, that kind of signal is a rumor, not a thermostat: if the per-item loop guesses wrong, future feedback corrects it automatically; if an inferred rule is wrong, nothing self-corrects it. It just quietly biases everything until I notice and delete it.

So I’m building a process that runs on its own every week, with no one watching, and rewrites the app’s understanding of my taste. That’s a risky thing to automate. Most of the work here isn’t the inference; it’s the guardrails that make a self-modifying system safe to leave running.

It’s the second LangGraph in the project (after the trip planner), and I picked it for that on purpose: a failed weekly run costs nothing (it just tries again next week), so it’s the lowest-stakes place in the codebase to take agent-building reps. Here’s the graph:

Walking the graph (and the rules baked into each step)

The shape is simple: fetch, decide, infer, validate, write. The interesting part is the guardrail living in each node.

Fetch, then check the evidence. The first node pulls my entire verdict history (with the attribution, weather, and notes captured at recommendation time, the why behind each thumb, from Part 3). The next is a gate: if there aren’t at least a handful of verdicts, the graph exits immediately and does nothing. “Insufficient evidence” is a valid, successful outcome. Early on, the honest answer is “I don’t know your taste yet,” and the worst thing the system could do is invent rules from three data points. The model is even told that returning an empty list is success, not failure.

Infer, and cite your sources or it doesn’t count. The model gets the history and proposes candidate preference statements. The non-negotiable rule: every statement must cite the specific outfits that justify it. Because a wrong inference can’t self-correct, the only real safety net is that I can read it, see the evidence, and decide it’s wrong, so an uncited, vague “you like nice clothes” is worthless. A statement must point at the actual verdicts behind it, and any statement backed by fewer than three distinct outfits is dropped as noise rather than a pattern.

There’s a small, fun engineering detail here. The model cites evidence by position (“outfits [2], [7], [9]”) rather than by database ID. UUIDs are long, token-expensive, and language models transcribe them wrong constantly. So the prompt numbers each verdict [1], [2], [3]…, the model cites those integers, and the validation step maps them back to real IDs. Cheaper, and far more reliable.

Validate, then re-derive from scratch. The validation node turns the model’s indices into real evidence IDs, enforces the three-outfit floor, and throws out anything that collides with a rule I previously dismissed (more on that below). Critically, each weekly run regenerates the entire inferred set from scratch; it doesn’t append to last week’s guesses. Appending would let errors compound silently over months; re-deriving means every rule, every week, has to re-earn its place from the current evidence.

The two guardrails I’m proudest of

Two of these are worth dwelling on, because they’re the kind of thing you only appreciate after it bites you.

Atomicity without transactions. Supabase (via its REST layer) doesn’t give me real transactions; I can’t wrap “delete the old rules, write the new ones” in an all-or-nothing block. If the process died between those two steps, I could be left with no preferences at all. The fix is almost embarrassingly simple: change the order.

# No transactions available, so ORDER is the only atomicity lever.
supabase().table("preferences").insert(fresh_rows).execute()        # new set lands first
supabase().table("preferences").delete().in_("id", prior_ids).execute()  # then drop the old

^{→ full context: upsert_node in backend/services/preference_inference.py}

Insert the fresh set first, then delete the old one. A crash in the middle leaves both sets briefly present, momentarily redundant, but never empty. And the graph only ever reaches this step if the model call and parsing both succeeded, so a failed inference aborts with the table completely untouched. “Worst case is harmless redundancy” is the whole design goal.

Health you can see, because no one is watching. This runs on a schedule with nobody monitoring it. How would I even know if it silently stopped? Rather than build alerting, I made health a visible part of the product: every successful run stamps a “last reviewed” timestamp, and the profile page just shows it in plain language: “preferences reviewed 3 days ago.” If that number starts climbing, something’s wrong, and I see it the next time I open the app.

And the human override threaded through all of it: anything I edit gets promoted to a user-authored rule the job will never touch again, and anything I dismiss becomes a tombstone the job is forbidden from re-suggesting. The automation stays firmly in the passenger seat. It proposes; I dispose.

What the first runs actually did

Two things happened the first few times this ran, and both made me happier about the design than any test had.

The very first live run failed, and it was the good kind of failure. I’d told the model to “return only JSON,” and it didn’t: being a reasoning model, it narrated its analysis first (“I’m looking for patterns across these outfits…”) and tucked the actual JSON in a fenced block at the end, which my too-strict parser choked on. The lesson there is worth keeping: “return only JSON” is a request, not a guarantee. The durable fix is a forgiving parser at the point where output gets read, not a sterner instruction the model can still ignore. But the part I cared about is what the failure cost: nothing. The graph only reaches the write step on success, so the inference aborted with the preferences table completely untouched.

Once the parser was fixed, the second thing: it ran clean and wrote exactly one preference from my entire history (“prefers sandals or open footwear over sneakers in athleisure outfits,” drawn from six outfits) and declined to invent any others. That restraint is the whole design working. Faced with thin evidence, it would rather say one true thing than five shaky ones, because the floor of three backing outfits quietly dropped everything weaker. A more eager system would have handed me a confident list of half-imagined rules; this one shrugged and gave me the single pattern it could actually defend.

(Here is a later one as it renders: the statement, its evidence count, and the two controls that keep me in charge.)

Still on the workbench

This isn’t a finished product; it’s a project I keep living in and extending, which is half the fun. Two things I’m actively sketching:

• A shopping assistant. Today the app only shops for me reactively, when the trip planner spots a gap. I want to flip that around: let me just say what I’m after (“a lightweight neutral jacket for fall”) and have it curate options the way it curates outfits, filtered through everything it’s learned about my taste. The product-search plumbing from Part 4 is already most of the machinery; this is mostly a new front door to it.
• Friends. The whole system is single-user today (my closet, my taste), but almost none of the ideas here are personal to me. Letting a few friends each have their own wardrobe and learned preferences is the bigger lift I’m circling.

So treat this series as a snapshot, not a final word. Both of those are probably their own future posts: the shopping assistant is another agent waiting to happen, and multi-user quietly turns a cozy personal app into a real systems problem. And one follow-up is already written in my head, because the work is done: after shipping all of this, I built an eval loop to measure whether the recommender was actually getting better rather than just feeling better, and it caught things I never would have found by vibes. That’s the next post.

The whole series, in one idea

Five posts, one recurring principle, stated back in Part 1: stochastic weights for preferences, deterministic logic for physics. Everything since has been that idea refracted through different problems.

• The recommender (Part 2) splits hard physical constraints from soft, probabilistic taste.
• The feedback loop (Part 3) splits within the soft world: a numeric loop that self-corrects can be aggressive; an inferred rule that can’t must be kept weak, legible, and overridable.
• The two agents (Part 4 and Part 5) show the same tool, a LangGraph, wearing opposite risk profiles: a friendly, on-demand planner whose worst failure is a missing link, and a background process whose worst failure would be quietly reshaping my taste, fenced in accordingly.

If there’s a single takeaway, it’s that the interesting engineering in an AI-powered app usually isn’t the model call. It’s everything around it: deciding which problems are physics and which are preference, matching how aggressive a mechanism is to how well it can recover from being wrong, and designing, always, for the version of the system that’s confidently mistaken. The model is one step in the pipeline. The judgment is in the rest of it.

Go deeper

Everything in this post is real, running code:

• The whole graph, with the guardrails as docstrings where they live: backend/services/preference_inference.py
• The weekly cron that runs it: jobs/infer_preferences.py and .github/workflows/infer-preferences.yml
• The design doc this post condenses, including the trade-offs I didn’t take: docs/feedback-loop-design.md

Thanks for reading.