architectures
Spiking neural networks: computing with events and time Lesson
Spiking neural networks communicate through discrete events and evolving internal states, making timing part of computation without guaranteeing biological fidelity or efficiency.
Hypernetworks: a neural network that writes part of another network Lesson
A hypernetwork is a model that generates weights or weight changes for another model, making one fixed system behave like many specialized systems without storing every specialist separately.
Graph neural networks: learning from things defined by their connections Lesson
A graph neural network learns representations of data whose meaning lies in how its parts connect — molecules, road networks, social graphs — by repeatedly having each node exchange messages with its neighbours.
Linear Attention Lesson
Linear attention rewrites the transformer's attention math so its cost grows in a straight line with sequence length instead of exploding as the square, letting models handle very long inputs cheaply, at the price of a compressed memory of the past.
RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers Lesson
Recurrent neural networks process sequences one step at a time while carrying a memory, and LSTMs added gates to fix their forgetfulness; together they powered a decade of sequence modeling and set up exactly the problem transformers later solved.