weixin_44604698的博客

This paper mainly contributes the following 2 points:So the core logic the algorithm.

I think there are 3 advantages in SGLang. It allows direct programing in python, it suuport RadixAttention to effeicient KVCache reuse, and it used compressed finite state machine to accelerate the structured output.Reuse the KVCache with the same prompts.

I remember that when I started to using LLM tools, like chatgpt, or kimi, which are terrific at handing complex reasoning jobs, such as math, commonsense and other tasks. After a while, these tools can do it. The most important factor is the chain of thoug

The model distillation is a special method used to compress the model for convenient deployment. It involves using a small model that doesn't significantly decline the accurency. The principle behind this method is the use of "soft target" which are differ

The principle is very simple from the above graph. Regarding the conversation from MHA to GQA, we use the mean pool method to do it over the keys and values heads, as shown in the following figure: 1.GQA leads to a higher quality than MQA, but faster than

where:

[ cosθ -sinθsinθ cosθ ]Let us demostrate this using the "complex-number shortcut".1. We denotes 2-D matric [x, y] as the complx number:z = x + y2. If we want to rotate some angle, in complex numbers, we usually multiplys:e^(iθ) = cos(θ) + isin(θ)3. So, we

(Because the formula is so difficult to type, I wrote these by hand.)

BERT(Bidirectional Encoder Representation from Transformers)是2018年10月由Google AI研究院提出的一种预训练模型，论文来自于：BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding。整体结构分为Embedding、Transformer Encoder结构和模型输出。