One Tap, Four Meanings: What a Thumbs-Down Actually Teaches

Part 3 of a series on building Wardrobe AI. Part 1 set up the guiding principle; Part 2 walked the pipeline that decides which clothes the model sees and promised a deeper look at the one stage it waved past, the feedback signal. This is that look. All the code in this post is public, in github.com/JiamanBettyWu/mise; snippets link to their source.

I wanted it to know my taste

The recommender from Part 2 was good at variety and weather, but it didn’t know me. It didn’t know I reach for neutral colors, that I think most outfits are improved by exactly one interesting piece, or that I quietly hate a particular jacket I keep meaning to donate. I wanted the app to learn my aesthetic over time, to get more me the longer I used it.

There are really only two ways to get taste into the system. The first is to just tell it: type “I prefer neutral basics” into a settings box and have the model treat that as a rule. Easy, and I built it (more on why it’s a special case below). The second is to have it learn from my reactions: let me thumb outfits up or down and infer my taste from the pattern. That second path is where all the interesting problems live, because recording a thumbs-down is trivial and learning the right lesson from it is not.

Here’s the trap. I thumb down today’s outfit. What did I just say? Maybe the jacket is ugly. Maybe the jacket is fine but clashed with those pants. Maybe the whole thing was lovely and it’s just freezing and I wanted something warmer. Maybe it was perfect for a coffee run but I have a meeting. One tap, at least four completely different meanings, and if the system guesses wrong it doesn’t just miss once. It bakes the wrong lesson into every future morning. This post is about making that one tap teach the right thing.

Three doors a preference can walk through

Taste enters this system through three distinct doors, and (this is the whole point of the post) they are deliberately not treated the same:

1. Explicit preferences I write (“prefer neutral colors”). A hard constraint, honored in every outfit, no exceptions.
2. Per-item feedback (thumbs on individual outfits). A soft, self-correcting nudge, expressed as the sampling-weight multiplier from Part 2.
3. Inferred preferences: patterns a weekly “reflection” job distills from my whole feedback history (“you seem to avoid layering in summer”). Also soft, but fundamentally riskier. The machinery of that weekly job is its own story (Part 5); what matters now is how its output is treated.

Doors 2 and 3 are both “learning from feedback,” but they fail in opposite ways. Hold that thought; first, the math behind door 2.

The math: what a thumbs-down does to the odds

Recall from Part 2 that every item carries a feedback multiplier that tips its sampling odds up or down. Here’s how that number is actually computed, and why the obvious version is a trap.

The naive version. Track likes and dislikes per item; the item’s score is its like rate, ups / (ups + downs). Clean, but brittle. Thumb down a shirt exactly once and its score is 0/1 = 0: a single bad day damns it forever. Thumb one up once and it’s 1/1 = 100%: a perfect item on one data point. With the handful of verdicts you actually accumulate day to day, raw rates lurch around wildly.

The fix: smoothing. Instead of trusting the raw count, start every item from a neutral assumption and let evidence move it:

score = (ups + 1) / (ups + downs + 2)      # 0 verdicts -> 0.5 (neutral)

^{→ full context: backend/services/outfit_history.py}

Those +1 and +2 are the whole trick. With no data, the score is 1/2, a shrug, “no opinion yet.” One thumbs-down takes a fresh item to 1/3 ≈ 0.33, not 0: disliked, but not condemned on a sample of one. As real verdicts pile up, those constants matter less and the score converges on the true rate. This is a Beta-Bernoulli estimate, the textbook way to estimate a yes/no rate (here, “will I like it?”) when samples are scarce, with the +1/+2 acting as a prior belief of 50/50. The everyday version: a restaurant with one 1-star review isn’t really a zero-star restaurant. You assume roughly average until enough reviews prove otherwise.

Spreading the blame. A thumbs-down lands on an outfit, but the multiplier lives on items. If I dislike a four-piece outfit, which piece was the problem? Honest answer: I don’t know, and neither does the system. So it refuses to over-blame: one verdict distributes one unit of credit or blame across the items:

credit = 1.0 / len(item_ids)               # a 4-item outfit -> 0.25 each

A single tap is one vote, not four. It nudges everything in the outfit a little, rather than slamming each piece as if it were individually condemned. Over many outfits, the genuinely bad item shows up in more dislikes and accumulates the blame; the innocent bystanders that happened to be in one bad outfit recover.

Bounding it. Finally, that smoothed score (between 0 and 1) is stretched onto a deliberately narrow multiplier:

mult = FLOOR + (CEILING - FLOOR) * score   # [0,1] -> [0.6, 1.4]

^{→ full context: backend/services/outfit_history.py}

The ceiling (1.4) keeps a beloved item from dominating every outfit. The floor (0.6) is the one I want to call out, because it’s the exploration vs. exploitation dial from Parts 1 and 2 made concrete. Exploitation says “show me what I’ve liked.” Exploration says “keep showing me other things anyway.” That floor of 0.6 means even an item I’ve thumbed down repeatedly never drops to zero probability; it just gets quieter. The system never stops exploring, even on the stuff it thinks I dislike, because “I disliked it once in July” is not proof I should never see it in October. The feedback multiplier exploits; the floor guarantees exploration; both are tunable constants.

Disambiguating the tap

The math above quietly assumes the dislike was about the items. But remember the four meanings of a thumbs-down. So a dislike can optionally carry an attribution: was it the items, the combination, the weather, or the occasion? That tag changes who gets blamed.

• Specific items: blame lands only on the pieces I named. High-confidence signal.
• The combination: the items are individually fine; it was the pairing. The items are exonerated (zero blame in the multiplier), and instead that exact set goes on the deterministic block-list from Part 2’s post-generation filter. (This is where that “known-bad combination” tag came from: I promised you in Part 2, and here it is.)
• The weather / the occasion: not the outfit’s fault at all. The items are exonerated; a weather complaint is dropped from learning entirely (it’s feedback on the forecast, not the wardrobe).

Attribution is what stops a single grumpy “too cold today” from quietly penalizing four perfectly good items for the rest of the year. It routes each verdict to the part of the system that can actually act on it: items to the multiplier, bad pairings to the block-list, weather to the bin.

Recent verdicts also go into the prompt as plain context: disliked outfits as an “avoid recombining similar assemblies” list, liked ones explicitly captioned style direction only, do not recreate these. That caption is deliberate. Recency weighting just pushed those exact items down to keep things fresh, and I don’t want the model clawing them back by hunting near-duplicates of something I liked last week.

The real lesson: a loop that self-corrects vs. a bias that doesn’t

Now the payoff, and the reason doors 2 and 3 exist as separate doors.

The per-item feedback loop (door 2) is safe to be aggressive because it self-corrects. Suppose it over-penalizes a shirt after one unlucky dislike. The floor keeps the shirt in rotation; it resurfaces; if I like it next time, its score climbs right back. Wrong guesses get pulled back toward the truth by the next data point. It behaves like a thermostat: it can overshoot, but the feedback that caused the overshoot is the same feedback that corrects it.

An inferred preference (door 3) is a different animal. When the weekly reflection job writes “you avoid bright colors,” that sentence goes into the prompt and shapes every outfit from then on. If it’s right, wonderful. If it’s wrong (maybe I downvoted three bright things for unrelated reasons), nothing in the loop fixes it. A wrong inference is a systematic bias with no floor: it quietly tilts everything, and the only thing that ever corrects it is me noticing and deleting it.

That asymmetry dictates the design. Because a wrong inference can’t self-correct, it has to be legible and overridable: every inferred preference is short, specific, cites the exact feedback that produced it, and shows up in an editable list I can prune in one click. And because it’s soft, a hard preference I wrote myself always wins when the two conflict. The model is told to lean on learned preferences but never to sacrifice weather, coherence, or an explicit rule. So a wrong inference degrades gracefully: it nudges a few outfits until I catch it, rather than quietly overruling everything.

This is the same organizing principle from Part 1, one level deeper. Part 1 split physics (hard) from preference (soft). Here, within the soft world, there’s a second split: a signal that self-corrects can be trusted to be assertive; a signal that can’t must be kept legible, weak, and easy to stop.

Where this goes next

Door 3, the weekly “reflection” job that reads my whole feedback history and distills durable preferences, has been lurking in the background this whole post. It turns out to be a small AI system in its own right, with a surprisingly fussy set of rules to keep it from doing exactly the kind of unrecoverable damage I just described. But it’s also the second of two agentic pieces in this project, so before I get to it I want to introduce the agentic system with a friendlier example: a trip packing planner that can decide its own next move. That’s Part 4.

Go deeper

Everything in this post is real, running code:

• The whole multiplier, smoothing + blame-splitting + attribution routing, is one pure function: _feedback_multipliers in backend/services/outfit_history.py
• The hard-vs-soft split in the prompt (user preferences as constraints, learned ones as nudges): backend/services/claude.py
• The design doc where these rules were actually worked out, trade-offs and rejected alternatives included: docs/feedback-loop-design.md