decoding
Everything on Ground Truth tagged “decoding” — 3 items.
Beam search: keeping several drafts alive to find a likelier sentence Lesson
Beam search is a decoding method that keeps the few highest-scoring partial outputs at every step instead of committing to the single likeliest word, which finds more probable sentences than greedy decoding and exposes why the most probable text is often not the best text.
Temperature and top-p: how an AI actually picks its next word Lesson
A language model does not know its next word - it produces a list of odds and then rolls dice. The rules of that dice roll are why the same prompt gives you a boring answer one day and a wild one the next.
Sometimes the AI Knew the Better Answer a Few Layers Early News
A new paper finds that a model's final layer can actually muddy an answer its middle layers had right -- and that reading the answer out a little early can claw back ability lost to safety training.