EAGLE and EAGLE-2 make large language model inference up to four times faster without changing outputs.
Eagle and Eagle 2 are two recent works from the speaker's group whose purpose is lossless inference acceleration for large language models. They keep the original model's output distribution while turning the sequential autoregressive decoding of a large language model into parallel computation, using a speculative sampling framework with a small trained draft model.
In the vanilla autoregressive pipeline each token is generated one by one: the second token must be generated before the third can be obtained, so inference is sequential. Modern GPUs are good at parallel computing, so the talk asks whether this sequential inference can be transferred into a parallel computing pipeline.
Speculative sampling, which some people call speculative decoding, has two modes. In the draft mode a tiny model that approximates the large model runs vanilla autoregressive inference and produces a collection of drafted tokens. In the checking mode the original large language model checks the correctness of all those tokens in parallel.
Checking follows a two-step procedure. A value r is sampled uniformly from the interval zero to one, and if r is less than min one, p_t divided by q_t, for the current token, that token is accepted. This min one, p_t over q_t quantity is called the accepted rate.
A theorem in the two papers says the above PQ sampling procedure is equivalent to sampling from P, the distribution of the large language model, so the text distribution is preserved. When P and Q are closer, the draft guesses more accurately, giving a higher accepted rate and a higher speedup ratio.
Instead of next token prediction, Eagle does next feature prediction. The evolution path between word embeddings such as e-how to e-can is complicated and needs a super large model to overfit, while the path between top-layer feature vectors f-how to f-can is very simple, so only a tiny model is needed.
Next feature prediction alone ignores which token was actually sampled, so the model cannot tell which branch it took. Eagle also involves the sampling information: it concatenates the feature vector with the word embedding of the token sampled in the last round, for example e-always, into a longer vector predicting the next feature.
When the original large language model is 7 billion parameters, the draft model has 0.24 billion, counting for only 3.4% of it; when the original model is 70 billion, the draft model is roughly 1 billion, or 1.4%. Everything in Eagle is trainable on cheap GPUs such as RTX 3090 within one or two days.
Eagle relies on a static tree structure and always generates two branches, wasting computation: given the prompt ten plus two equal, it drafts one and three although three is impossible. Eagle 2 builds a context-aware dynamic tree instead, keeping a single branch for one and then following it with two.
The accepted rate needs P from the large model, which the draft mode cannot access, so Eagle 2 approximates it with the confidence of the draft model Q. An experiment plotting draft confidence against the accepted rate shows a strong linear correlation, so confidence decides which nodes to prune.
“each token is generated one by one.”
“So we can see this is a sequential inference.”
“However, we also notice that the tiny model is very small.”
“You only need a tiny model to do that.”
“let's try to do next feature prediction.”
“Eagle is three times faster than Wadena decoding,”
“Ego and Ego2 for Lossless Inference Acceleration.” https://www.youtube.com/watch?v=oXRSorx-Llg. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/ego-and-ego2-for-lossless-inference-acceleration.html.