sequence-models
Everything on Ground Truth tagged “sequence-models” — 3 items.
Teacher forcing and exposure bias Lesson
Teacher forcing trains a sequence model by always feeding it the correct previous token instead of its own output, which makes training fast and stable but leaves the model unprepared for its own mistakes at generation time -- a mismatch called exposure bias.
Genome language models: predicting DNA the way a chatbot predicts text Lesson
Genome language models are trained to predict the next base in a DNA sequence, the same objective that produces chatbots, and they turn out to learn enough biology to write working viral genomes from scratch.
RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers Lesson
Recurrent neural networks process sequences one step at a time while carrying a memory, and LSTMs added gates to fix their forgetfulness; together they powered a decade of sequence modeling and set up exactly the problem transformers later solved.