Bayesian Knowledge Tracing, honestly

Bayesian Knowledge Tracing has a reputation it only half deserves. People talk about it as if it were either a magic personalization engine or a discredited relic. It is neither. It is four numbers and one strong assumption, and almost everything that goes wrong with it goes wrong at the assumption.

The four numbers are familiar: prior knowledge, learn rate, and the two noise terms — slip (you knew it but answered wrong) and guess (you didn’t but answered right). The assumption is that a skill, once learned, stays learned. That single modeling choice is where honesty lives or dies.

Where it earns its keep

Used well, BKT is a discipline for not throwing away evidence. A student’s performance on Tuesday should inform what you believe on Thursday. A quiz score shouldn’t reset belief to zero. Carrying a posterior forward across encounters is the entire point, and most simple gradebooks fail to do it.

Slip and guess are not nuisance parameters. They are the model admitting, out loud, that a single answer is weak evidence.

Where it quietly lies

The failure mode is treating the parameters as fixed truths rather than fitted estimates. A high learn rate on sparse data is a guess dressed as a measurement. A model that never forgets will happily report mastery a month after the student last touched the skill. If your product surfaces a confident number on top of that, you have laundered uncertainty into authority.

The fix is not a fancier model. It is showing the confidence alongside the belief, and refusing to report a posterior the data cannot support. The same restraint that makes mastery a claim rather than a sequence applies here: measure fewer things, and say how sure you are.

A working rule

If you cannot state, in a sentence, why the model believes what it believes about a given student and skill, do not put that belief in front of a teacher. Legibility is not a feature you add later; it is a property you design the model to have.