Recording intelligenceAI-generated brief · check the source for context

Ego and Ego2 for Lossless Inference Acceleration

48:25 recording · AUTO · 1 speaker

Watch the original

Executive Summary AI
  • The speaker presents two papers from his group, EAGLE and EAGLE-2, whose goal is lossless inference acceleration for large language models, and starts by showing that vanilla autoregressive decoding generates tokens strictly one at a time while modern GPUs are built for parallel computing.
  • The proposed remedy is speculative sampling, a two-mode framework in which a tiny draft model guesses several tokens cheaply and the original large model then checks all of them in parallel, accepting a token when a uniform random value r is below min(1, p_t/q_t) and otherwise resampling from the normalised max(0, p - q) correction distribution.
  • EAGLE's core observation is that the evolution path at the feature level (f_how to f_can to f_I) is far simpler than at the embedding level, so instead of next-token prediction the draft model does next-feature prediction with a single trainable transformer layer on top of a frozen embedding layer, transformer stack and LM head.
  • To fix the remaining feature uncertainty, the predicted feature is concatenated with the word embedding of the token actually sampled in the previous round, lifting acceptance to roughly 0.8 and speed-up to roughly three times, with draft models of 0.24 billion parameters (3.4%) for a 7 billion model and about one billion (1.4%) for 70 billion, trainable on an RTX 3090 in one or two days.
  • EAGLE-2 replaces the static draft tree with a context-aware dynamic one, ranking nodes by draft-model confidence (shown to correlate linearly with the true acceptance rate) and keeping the top eight, which raises MT-bench speed-up from three times to about 4.2 times, reaches 4.6 average accepted tokens, and lets an RTX 3060 costing $600 hit 40 tokens per second — faster than vanilla decoding on a

Brief overview

EAGLE and EAGLE-2 make large language model inference up to four times faster without changing outputs.

  1. Predict the next feature, not the next tokenThe top-layer feature path is simple enough for a single transformer layer, while the embedding path needs a super large model.
  2. Feed the sampled token back to kill uncertaintyConcatenating the predicted feature with the embedding of the token just sampled pushed acceptance to roughly 0.8 and speed-up to three times.
  3. Dynamic trees beat static treesPosition one has the highest acceptance and position six the lowest, so EAGLE-2 prunes by draft confidence and reaches about 4.2 times speed-up.

Questions this recording answers

10 questions, each answered where it is said
What are Eagle and Eagle 2 designed to do?

Eagle and Eagle 2 are two recent works from the speaker's group whose purpose is lossless inference acceleration for large language models. They keep the original model's output distribution while turning the sequential autoregressive decoding of a large language model into parallel computation, using a speculative sampling framework with a small trained draft model.

Why is vanilla autoregressive inference slow on modern GPUs?

In the vanilla autoregressive pipeline each token is generated one by one: the second token must be generated before the third can be obtained, so inference is sequential. Modern GPUs are good at parallel computing, so the talk asks whether this sequential inference can be transferred into a parallel computing pipeline.

How does the speculative sampling framework split the work?

Speculative sampling, which some people call speculative decoding, has two modes. In the draft mode a tiny model that approximates the large model runs vanilla autoregressive inference and produces a collection of drafted tokens. In the checking mode the original large language model checks the correctness of all those tokens in parallel.

How is a drafted token accepted or rejected?

Checking follows a two-step procedure. A value r is sampled uniformly from the interval zero to one, and if r is less than min one, p_t divided by q_t, for the current token, that token is accepted. This min one, p_t over q_t quantity is called the accepted rate.

Does speculative sampling change the quality of the output?

A theorem in the two papers says the above PQ sampling procedure is equivalent to sampling from P, the distribution of the large language model, so the text distribution is preserved. When P and Q are closer, the draft guesses more accurately, giving a higher accepted rate and a higher speedup ratio.

What is Eagle's core idea for building the draft model?

Instead of next token prediction, Eagle does next feature prediction. The evolution path between word embeddings such as e-how to e-can is complicated and needs a super large model to overfit, while the path between top-layer feature vectors f-how to f-can is very simple, so only a tiny model is needed.

How does Eagle solve the feature uncertainty issue?

Next feature prediction alone ignores which token was actually sampled, so the model cannot tell which branch it took. Eagle also involves the sampling information: it concatenates the feature vector with the word embedding of the token sampled in the last round, for example e-always, into a longer vector predicting the next feature.

How large is Eagle's draft model and how is it trained?

When the original large language model is 7 billion parameters, the draft model has 0.24 billion, counting for only 3.4% of it; when the original model is 70 billion, the draft model is roughly 1 billion, or 1.4%. Everything in Eagle is trainable on cheap GPUs such as RTX 3090 within one or two days.

What does Eagle 2 change compared with Eagle 1?

Eagle relies on a static tree structure and always generates two branches, wasting computation: given the prompt ten plus two equal, it drafts one and three although three is impossible. Eagle 2 builds a context-aware dynamic tree instead, keeping a single branch for one and then following it with two.

How does Eagle 2 decide which draft tree nodes to keep?

The accepted rate needs P from the large model, which the draft mode cannot access, so Eagle 2 approximates it with the confidence of the draft model Q. An experiment plotting draft confidence against the accepted rate shows a strong linear correlation, so confidence decides which nodes to prune.

Key Quote

“each token is generated one by one.”

— Ego and Ego2 for Lossless Inference Acceleration, at 3:33▶ watch at 3:33
Key Quote

“So we can see this is a sequential inference.”

— Ego and Ego2 for Lossless Inference Acceleration, at 3:43▶ watch at 3:43
Key Quote

“However, we also notice that the tiny model is very small.”

— Ego and Ego2 for Lossless Inference Acceleration, at 5:56▶ watch at 5:56
Key Quote

“You only need a tiny model to do that.”

— Ego and Ego2 for Lossless Inference Acceleration, at 14:33▶ watch at 14:33
Key Quote

“let's try to do next feature prediction.”

— Ego and Ego2 for Lossless Inference Acceleration, at 14:41▶ watch at 14:41
Key Quote

“Eagle is three times faster than Wadena decoding,”

— Ego and Ego2 for Lossless Inference Acceleration, at 31:12▶ watch at 31:12

Cite this

“Ego and Ego2 for Lossless Inference Acceleration.” https://www.youtube.com/watch?v=oXRSorx-Llg. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/ego-and-ego2-for-lossless-inference-acceleration.html.

Report this page
What is wrong with this page?