decoding
Everything on Ground Truth tagged “decoding” — 2 items.
Temperature and top-p: how an AI actually picks its next word Lesson
A language model does not know its next word - it produces a list of odds and then rolls dice. The rules of that dice roll are why the same prompt gives you a boring answer one day and a wild one the next.
Sometimes the AI Knew the Better Answer a Few Layers Early News
A new paper finds that a model's final layer can actually muddy an answer its middle layers had right -- and that reading the answer out a little early can claw back ability lost to safety training.