HabitLoop

The model does not count

Language models are bad at arithmetic and confident about it. A habit tracker is almost entirely arithmetic — streaks, completion rates, weekday patterns — so the design rule was simple: the model never touches a number it has to compute.

Streaks, rates, per-habit weekday and weekend totals, and best-to-worst rankings are pure Python, unit-tested, including the cases that are easy to get wrong: a miss logged today ends the streak, and a habit started three days ago is measured over three days, not thirty. The model receives the finished summary and does the one thing it is good at: interpreting it.

Checking what it says

Handing the model true numbers is not enough on its own. Testing against a local model turned up three failures that every number-matching check would pass: the right numbers attached to the wrong "worst" habit, the longest streak quoted as the current one, and a Sunday-only rate quoted as the weekend rate. So every answer now goes through a validator that checks each number against the habit and the part of the week it is attached to, keeps current and longest streaks apart, and requires any "best" or "worst" claim to agree with the computed ranking. Each of those three failures is now a test.

Measured, not assumed

Every expected answer is computed by the same Python statistics, and every coach answer runs through the app's own code path on a local Qwen2.5-7B, scored by fixed rules rather than another model.

• A held-out set, written and committed before the latest fixes (43 questions over three new logs, 129 answers, run once): 50.4% correct (95% interval 41.9–58.9%). Single figures, streaks and weekday-versus-weekend answers were right 35 times out of 39.
• Comparisons are the measured weak spot. Best and worst habit: 4 of 30. Most misses name the right habit, then add a wrong one.
• Refusals didn't carry over. The first set's 24 unanswerable questions were all declined; on held-out, 12 of 15 got an answer anyway.
• The validator caught 40% of the held-out wrong answers and flagged 19% of the right ones.
• On the first 153-answer set the fixes were designed from, accuracy went from 77.8% to 88.2%. The gap between that and the held-out number is why the held-out set exists.

Next

Refusal rules for questions the log can't answer, comparison answers computed in Python rather than phrased by the model, and the same evaluation against the hosted Llama 3.3 70B default.