🤗 LLM-Perf Leaderboard 🏋️
0 100
0 81920
Model 🤗 | Experiment 🧪 | Prefill (s) | Decode (tokens/s) | Memory (MB) | Energy (tokens/kWh) | Open LLM Score (%) |
|---|---|---|---|---|---|---|
4bit-gptq-exllama-v2-eager | 0.087 | 108.613 | 15763.828 | 1063016.711 | 27.20* |
👆 Hover over the points 👆 for additional information.
📊 We only show the top 90% LLMs based on latency ⌛
0 100
0 81920
Model 🤗 | Experiment 🧪 | Prefill (s) | Decode (tokens/s) | Memory (MB) | Energy (tokens/kWh) | Open LLM Score (%) |
|---|---|---|---|---|---|---|
4bit-gptq-exllama-v2-eager | 2.486 | 147.371 | 65311.037 | 1212590.352 | 29.56* |
👆 Hover over the points 👆 for additional information.
📊 We only show the top 90% LLMs based on latency ⌛
0 100
0 81920
Model 🤗 | Experiment 🧪 | Prefill (s) | Decode (tokens/s) | Memory (MB) | Energy (tokens/kWh) | Open LLM Score (%) |
|---|---|---|---|---|---|---|
4bit-gptq-exllama-v2-eager | 0.094 | 100.902 | 11624.262 | 1009135.618 | 27.20* |
👆 Hover over the points 👆 for additional information.
📊 We only show the top 90% LLMs based on latency ⌛
We tested the 32vCPU AWS C7i instance for the benchmark. The memory requirement is the max RAM consumption during the decode phase.
0 100
0 81920
Model 🤗 | Experiment 🧪 | Prefill (s) | Decode (tokens/s) | Memory (MB) | Energy (tokens/kWh) | Open LLM Score (%) |
|---|---|---|---|---|---|---|
bfloat16-eager-onnxruntime | 1.149 | 105.028 | 33874.035 | 10315773.503 | 27.20 |
👆 Hover over the points 👆 for additional information.
📊 We only show the top 90% LLMs based on latency ⌛
📝 About
The 🤗 LLM-Perf Leaderboard 🏋️ is a leaderboard at the intersection of quality and performance. Its aim is to benchmark the performance (latency, throughput, memory & energy) of Large Language Models (LLMs) with different hardwares, backends and optimizations using Optimum-Benhcmark.
Anyone from the community can request a new base model or hardware/backend/optimization configuration for automated benchmarking:
- Model evaluation requests should be made in the 🤗 Open LLM Leaderboard 🏅 ; we scrape the list of canonical base models from there.
- Hardware/Backend/Optimization configuration requests should be made in the 🤗 LLM-Perf Leaderboard 🏋️ or Optimum-Benhcmark repository (where the code is hosted).
✍️ Details
- To avoid communication-dependent results, only one GPU is used.
- Score is the average evaluation score obtained from the 🤗 Open LLM Leaderboard
- LLMs are running on a singleton batch with a prompt size of 256 and generating a 64 tokens for at least 10 iterations and 10 seconds.
- Energy consumption is measured in kWh using CodeCarbon and taking into consideration the GPU, CPU, RAM and location of the machine.
- We measure three types of memory: Max Allocated Memory, Max Reserved Memory and Max Used Memory. The first two being reported by PyTorch and the last one being observed using PyNVML.
All of our benchmarks are ran by this single script benchmark_cuda_pytorch.py using the power of Optimum-Benhcmark to garantee reproducibility and consistency.