🧱 Transformer 参数量与 FLOPs 计算器（GQA / MoE / SwiGLU）Transformer Parameter Count & FLOPs Calculator (GQA, MoE, SwiGLU)
输入层数、隐藏维度、注意力头数、KV 头数（GQA/MQA）、FFN 维度、词表大小、MLP 类型（SwiGLU/GELU）、偏置、嵌入共享和 MoE 专家数，逐项精确计算 Transformer 参数量（注意力、MLP、嵌入、输出层分别列出），以及每 token 前向/训练 FLOPs 和总训练算力；Llama 3.1、Qwen2.5、Mistral、Mixtral、GPT-2、BERT、ViT 预设与官方参数量逐位一致。Enter layers, hidden size, heads, KV heads (GQA/MQA), FFN size, vocabulary, MLP type (SwiGLU/GELU), biases, tied embeddings and MoE experts to get an exact Transformer parameter count (attention, MLP, embeddings and head itemised) plus forward and training FLOPs per token and total training compute. Presets for Llama 3.1, Qwen2.5, Mistral, Mixtral, GPT-2, BERT and ViT match the official counts digit for digit.
| 部分Part | 参数量Params | 占比Share |
|---|
参数量按结构逐项精确计算（预设与 Hugging Face 官方模型的参数量逐位一致，如 Llama 3.1 8B = 8,030,261,248）。FLOPs 只计矩阵乘法，每次乘加记 2 FLOPs：前向 ≈ 2 × 激活参数（不含嵌入，含输出层）+ 注意力 4·L·s·d（因果掩码按一半计）；训练 ≈ 3 × 前向（反向约为前向两倍），不含激活重算。ViT 预设不含分类头与 pooler。Parameters are counted term by term (presets match the official Hugging Face counts exactly, e.g. Llama 3.1 8B = 8,030,261,248). FLOPs count matrix multiplies only, 2 FLOPs per multiply-add: forward ≈ 2 × active parameters (excluding embeddings, including the output layer) + attention 4·L·s·d (halved for causal masks); training ≈ 3 × forward, without activation recomputation. The ViT preset excludes the classifier head and pooler.
使用说明与常见问题How-to & FAQ
「Transformer 参数量与 FLOPs 计算器（GQA / MoE / SwiGLU）」是什么？What is the Transformer Parameter Count & FLOPs Calculator (GQA, MoE, SwiGLU)?
输入层数、隐藏维度、注意力头数、KV 头数（GQA/MQA）、FFN 维度、词表大小、MLP 类型（SwiGLU/GELU）、偏置、嵌入共享和 MoE 专家数，逐项精确计算 Transformer 参数量（注意力、MLP、嵌入、输出层分别列出），以及每 token 前向/训练 FLOPs 和总训练算力；Llama 3.1、Qwen2.5、Mistral、Mixtral、GPT-2、BERT、ViT 预设与官方参数量逐位一致。Enter layers, hidden size, heads, KV heads (GQA/MQA), FFN size, vocabulary, MLP type (SwiGLU/GELU), biases, tied embeddings and MoE experts to get an exact Transformer parameter count (attention, MLP, embeddings and head itemised) plus forward and training FLOPs per token and total training compute. Presets for Llama 3.1, Qwen2.5, Mistral, Mixtral, GPT-2, BERT and ViT match the official counts digit for digit.
如何使用「Transformer 参数量与 FLOPs 计算器（GQA / MoE / SwiGLU）」？How do I use the Transformer Parameter Count & FLOPs Calculator (GQA, MoE, SwiGLU)?
选择一个模型预设查看结构，或直接修改任一参数（会自动切换为“自定义”）。上方显示总参数量和每 token 前向 FLOPs，中间是非嵌入参数、训练 FLOPs、6ND 近似等，下方表格是各部分参数占比。MoE 模型会同时显示总参数和每 token 激活参数。Pick a preset or edit any field (the preset switches to Custom). The top line shows total parameters and forward FLOPs per token; below are non-embedding params, training FLOPs and the 6ND estimate, then a breakdown table. MoE models show both total and active parameters.
为什么 Llama 3.1 8B 是 8.03B 而不是 8B？Why is Llama 3.1 8B 8.03B?
模型名是取整后的规模。按结构逐项计算：32 层 × (注意力 41.9M + SwiGLU MLP 176.2M + 归一化) + 嵌入 525M + 输出层 525M = 8,030,261,248，与 Hugging Face 上的官方参数量完全一致。The name is rounded. Counting term by term — 32 layers × (attention 41.9M + SwiGLU MLP 176.2M + norms) + embeddings 525M + output head 525M — gives 8,030,261,248, exactly the official Hugging Face count.
FLOPs 为什么约等于 2 × 参数量？Why are FLOPs about 2 × parameters?
前向传播中每个权重参与一次乘法和一次加法（2 FLOPs），所以每 token 约 2N；再加上注意力分数和加权求和的 4·L·s·d 项。训练时反向传播约为前向两倍，合计约 6N（即 6ND 公式）。In the forward pass each weight does one multiply and one add (2 FLOPs), so about 2N per token, plus the attention-score and weighted-sum term 4·L·s·d. Backward costs about twice the forward, giving ≈6N per token — the 6ND rule.
「Transformer 参数量与 FLOPs 计算器（GQA / MoE / SwiGLU）」免费吗？文件会上传吗？Is the Transformer Parameter Count & FLOPs Calculator (GQA, MoE, SwiGLU) free? Are my files uploaded?
「Transformer 参数量与 FLOPs 计算器（GQA / MoE / SwiGLU）」完全免费，无需注册。你输入的内容只在本机浏览器里处理（并保存在本机方便下次继续），不会上传到任何服务器，适合处理未发表的论文和数据。The Transformer Parameter Count & FLOPs Calculator (GQA, MoE, SwiGLU) is free with no sign-up. Everything you enter is processed in this browser only (and kept locally so you can continue later); nothing is uploaded, so it is safe for unpublished papers and data.