The Same Outfit, Four Days Straight: Fixing What the Prompt Can't

Part 2 of a series on building Wardrobe AI. Part 1 covered what the app does and the principle behind it: stochastic weights for preferences, deterministic logic for physics. This post is where that principle earns its keep. The feedback math gets a breadth-level mention here and a full treatment in Part 3. All the code in this post is public, in github.com/JiamanBettyWu/mise; snippets link to their source.

The bug that started it: four days, one outfit

The first version of the recommender was about as simple as it sounds: pull all my available clothes from the database, hand the whole list to the model, ask for an outfit. It worked. It also recommended me the same outfit four days in a row.

That’s a special kind of disappointing, because it’s not wrong. The outfit was fine all four days. The trouble is that a stylist who suggests the identical thing every morning isn’t really doing the job. The whole value, as I framed it in Part 1, is the warm start: a fresh push to react to. A stuck recommendation is a cold start wearing a trenchcoat.

Why did it happen? Large language models have a well-known tendency to anchor on whatever comes first in their input. My database returned items in a stable order, so the model kept seeing the same clothes at the top of the list and kept reaching for them. The fix people reach for first is “tell the model to be more varied,” and I did eventually do a version of that, but prompting alone is a soft nudge against a strong bias. The bigger lever turned out to be upstream.

The key idea: the model never sees the whole closet

Here’s the realization that reorganized the whole feature: the quality of a recommendation is decided mostly before the AI is ever called.

Instead of dumping my entire wardrobe on the model, I built a pipeline that curates a smaller candidate pool first, a deliberately chosen, shuffled subset of clothes, and only that goes to the model. Think of the model as the host at a party greeting guests, and the pipeline as the bouncer with the guest list. By the time the host says hello, the bouncer has already decided who’s in the room. Most of the interesting decisions live with the bouncer.

The pipeline is almost entirely deterministic plumbing wrapped around two small bits of randomness. Here’s the whole thing end to end:

Let me walk it stage by stage. I’ll go deep on everything except the feedback math, which has enough subtlety to deserve its own post (Part 3); here it’s just “a number that tips the odds.”

Stage 1: The extremes gate (physics, and it runs first)

Before anything gets weighted or sampled, a deterministic filter drops items that are absurd for today’s weather, like a heavy parka when it’s 30°C (86°F). This is the “physics” side of the principle: there’s nothing fuzzy about “it’s too hot for a parka,” so it’s a hard rule, not a probability.

It runs first, and the ordering matters. If absurd items were still in the pool during sampling, they’d take up slots that should go to wearable clothes. A parka sitting in the candidate pool on a hot day is, at best, an item the model ignores and, at worst, one it accidentally picks. Gate first, then the model only ever reasons about clothes that make sense for today.

Note what the gate does not do: it only removes the absurd extremes, not the merely-suboptimal. Fine-grained “is this warm enough for a cool morning” judgment is handled later, by the model itself. That two-tier split (hard gate for the absurd, model judgment for the nuanced) comes back when we get to the prompt.

Stage 2: Recency decay (so yesterday’s clothes step back)

This is the direct fix for the four-days bug. Every item gets a recency score: the more recently (and more often) it’s been recommended, the higher its score, with the influence fading a little each day. Concretely it’s exponential decay, where each day in the past counts for less:

DAILY_DECAY = 0.85           # per-day multiplier; ~4-day half-life
# an item worn d days ago contributes DAILY_DECAY ** d to its recency score

^{→ full context: backend/services/outfit_history.py}

“Half-life” here means that after about four days, a wear counts for half as much; after eight, a quarter. The analogy is a music shuffle that’s smart about repeats: a song you just heard drops down the queue, but it doesn’t fall off the playlist. Nothing is ever banned for being recent; it’s just made less likely for a while, so the rotation breathes.

One subtlety I got wrong the first time and had to fix: recency pressure only makes sense when you have substitutes. I have exactly one pair of dressy shoes. Penalizing them for “you wore them yesterday” just means tomorrow’s elevated outfit has no shoes at all. So items in a small category (at or below a threshold count of available pieces) are exempt from recency entirely. Rotation is for when there’s something to rotate to. I learned this the way most of these rules got learned: it did something dumb first. (I call it the sandals incident, and it shows up again below.)

Stage 3: The feedback nudge (breadth only; full story in Part 3)

Alongside recency, each item carries a feedback multiplier derived from my thumbs-up / thumbs-down history: liked items get their odds gently boosted, disliked ones gently reduced. Crucially, it’s a bounded nudge (it never drives an item’s probability to zero or to certainty), which keeps a little exploration alive even for things I’ve thumbed down.

That’s all you need for the pipeline view: it’s a number that tips the dice. How that number is computed (it involves a smoothed Bayesian estimate and blame attribution) is the subject of Part 3.

Stage 4: Weighted sampling (where the randomness happens)

Now the two signals combine into a single sampling weight per item:

w = recency_factor * feedback_multiplier

Then I draw a random subset, roughly 70% of the pool, where an item’s weight is its chance of being picked. Higher weight (not worn lately, generally liked) means more likely to make the cut; lower weight means less likely, never impossible.

The neat part is how you sample without replacement when every item has a different weight. There’s an elegant one-liner for it (the Efraimidis–Spirakis algorithm): give each item a random key and sort by it.

key = math.log(random.random()) / w     # bigger weight -> key closer to 0 -> ranks higher
# sort all items by key (descending); the top k are a correct weighted sample of size k

^{→ full context: backend/services/outfit_history.py}

This uses only the standard library: no dependencies, no model call, fully deterministic to debug given a fixed random seed. This is the “stochastic” half of the system, and it’s deliberately small and contained.

This is also where Part 1’s exploration vs. exploitation trade-off stops being an abstraction and becomes literal code. The feedback multiplier is the exploitation lever: it leans on what’s already worked, pulling proven items up. The randomness in the draw is the exploration lever: it keeps putting less-favored and not-recently-worn items in front of me, which is exactly how I stumble onto combinations I’d never have chosen myself. There’s a subtler reason for exploration, too. The feedback multiplier can only learn about items it actually shows me; a piece that never gets sampled generates no verdicts. So rotation isn’t only about variety, it’s what feeds the taste-learning. Kill the randomness and you’d quietly starve the very thing that’s supposed to get smarter over time.

The nice part is that the balance between the two isn’t hard-coded into the logic; it lives in a handful of tunable constants: how hard a thumbs-up boosts an item, what fraction of the wardrobe gets sampled (~70%), how fast recency decays (DAILY_DECAY = 0.85). Nudge them one way and the app plays favorites; nudge them the other and it turns adventurous. How far to push each dial, and why the feedback nudge keeps a hard floor so it never stops exploring entirely, is a Part 3 question.

Stage 5: Category floors (don’t evict all the shoes)

Random sampling has a failure mode: by chance, a draw can wipe out a whole category. Pull 70% at random and you might end up with zero shoes in the pool, and then there’s no possible complete outfit, no matter how clever the model is.

So after sampling, category floors top the pool back up: guarantee a minimum number of tops, bottoms, shoes, and outerwear. If a category fell below its floor, the next-best items (the ones just outside the random cut) get promoted back in until the floor is met. It’s cheap insurance (having an item in the pool costs nothing; the model can always ignore it), and it’s the final piece of the sandals incident: not just “don’t rotate away the only shoes,” but “never let the dice remove an entire category.”

Stage 6: The candidate pool meets the model

After the floors, what’s left is the candidate pool, the curated, shuffled rack the model finally gets to see. Now the model gets called. One detail matters for what comes next: it doesn’t return a single outfit per mode, it returns three candidates, ordered best-first. Hold onto that, because it’s what makes the last filter possible. First, how the prompt steers the call itself.

One line of logging makes the whole pipeline debuggable. Every run prints the pool it built, which answers the question I ask most often when an outfit surprises me: was that item sampled out, or did the model just ignore it?

INFO wardrobe.recommend: candidate pool (52 of 73 after gate + sampling):
'90s Vintage Maxi Denim Skirt front slit, Aritzia Wilfred black suit vest,
Black cushioned recovery flip flop, Black long-sleeve merino crew tee,
Black ribbed boat-neck tee, Blue satin strapless ruched tube top,
Cream collarless ultra light down jacket, ...

That’s a real run: 73 items available, 21 gated or sampled out today, 52 in the room when the model starts picking.

The other half: steering the model with the prompt

Everything above curates which clothes the model considers. The system prompt handles how it assembles them. The two are complementary levers, and I tuned the prompt the same way I tuned the pipeline: by watching it fail and writing a rule.

Three passes are worth sharing.

Warmth, the nuanced tier. Remember the extremes gate only removes the absurd. Everyday “is this actually warm enough for a 9°C (48°F) morning” is a judgment call, so I gave it to the model: every item has a warmth rating from 1 to 5, and the prompt asks it to reason about the outfit’s combined, layered warmth against today’s high and low (light pieces for heat; for cold, either a high-warmth piece or several lighter layers). Hard gate for the absurd, model judgment for the nuance: the same physics-versus-context split.

Omit a slot rather than force a bad one. My favorite failure. The model kept pairing sports sandals with an elegant dress, repeatedly. The logic was understandable: an outfit needs shoes, and after sampling the sandals were sometimes the only footwear left, so it dutifully jammed them in. The fix was to make “complete” optional:

An outfit missing shoes beats an outfit with off-mode shoes.

The prompt now tells the model that if nothing fits a slot for the occasion, it should leave that slot empty and note the gap in its reasoning, rather than forcing something off-key. It’s a small example of treating the prompt as policy: an incomplete-but-coherent outfit is more useful than a complete-but-absurd one.

Explore the full set. Finally, the prompt closes the loop on the original bug. It tells the model explicitly that the inventory order is randomized and implies no preference, so explore the whole set rather than defaulting to the first few items. On its own this would be a weak nudge against the anchoring bias. But it’s not on its own: the sampler already shuffles the pool every time, so the prompt and the pipeline push the same direction from two sides. The randomized order removes the stable thing the model was anchoring to; the instruction tells it not to anchor on whatever it gets handed instead.

An honest epilogue to the four-days bug, though: these two forces killed the worst of the repetition, but months later I caught a sneakier version of the same disease. The model had developed a fondness for alternating between two similar favorites. The fix was a third force: the prompt now also receives a short list of what it recommended in the past week, with an instruction to prefer pieces that aren’t on it. How I found that pattern, proved the fix actually worked with numbers instead of vibes, and then verified it in production is a story about building an eval loop, and it gets its own post later in the series.

Stage 7: The post-generation filter (known-bad and exact repeats)

This is the one stage that runs after the model, on its output rather than its input, which is exactly why the model returns three candidates instead of one. A final deterministic filter walks the candidates and takes the first one that survives two hard checks:

• Known-bad combinations. If I’ve thumbed down an outfit and flagged that the problem was the combination specifically (not the weather, not the occasion), that exact set of items is rejected permanently. This isn’t a soft preference to whisper to the model; it’s a recorded fact, so it’s enforced in code. (Where that “it was the combination” tag comes from is a Part 3 story.)
• Exact recent repeats. Any exact outfit recommended in the last 7 days is also rejected, so the same assembly can’t reappear within the week. This is set-level dedup; item-level rotation is recency’s job back in Stage 2.

Now the three-candidates choice pays off: if the best candidate is rejected, the filter falls through to the next acceptable one instead of coming back empty-handed, and only if every candidate is a known-bad combo does it skip the mode. The prompt asks for variety; this filter enforces it.

Stepping back

The shape of this whole feature is the Part 1 principle made concrete. The parts that should be reliable (don’t suggest a parka in July, never empty a category, never repeat a known-bad combo) are deterministic code I can step through and debug. The parts that should be lively are a small, bounded dose of randomness. The model sits at the end, doing the hard part of assembling a coherent outfit, but only over a set the pipeline has already made safe and varied.

The one stage I kept waving past, that feedback multiplier, turns out to be the most subtle of all. Recording a thumbs-down is easy; learning from it correctly is surprisingly hard, and getting it wrong quietly poisons everything downstream.

That’s Part 3.

Go deeper

Everything in this post is real, running code:

• The sampler (recency decay, weighted sampling, category floors): backend/services/outfit_history.py
• The extremes gate: backend/services/weather_gate.py
• The system prompt with the warmth, slot-omission, and explore rules: backend/services/claude.py
• The full algorithm reference, with formulas and worked examples: docs/recommendation-algorithm.md