Skip to content

bench: 2026-07-06 M1 Ultra full refresh (post VLM-port and fix batches) #673

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 1 commit into from
Jul 6, 2026
Merged

bench: 2026-07-06 M1 Ultra full refresh (post VLM-port and fix batches) #673

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
170 changes: 170 additions & 0 deletions benchmarks/metal_m1ultra_2026-07-06.csv
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions benchmarks/metal_m1ultra_2026-07-06_single_qwen3.5-35b-a3b-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt,prompt_target_len
qwen3.5-35b-a3b-4bit,models/qwen3.5-35b-a3b-4bit,19,31,84.67,224.40,376.88,82.25,2026-07-06,Apple_M1_Ultra_128GB,0.3.3,release,100,"Hello, how are you today?",
2 changes: 2 additions & 0 deletions benchmarks/metal_m1ultra_2026-07-06_single_qwen3.6-35b-a3b-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt,prompt_target_len
qwen3.6-35b-a3b-4bit,models/qwen3.6-35b-a3b-4bit,19,27,82.96,229.03,342.45,78.84,2026-07-06,Apple_M1_Ultra_128GB,0.3.3,release,100,"Hello, how are you today?",
170 changes: 170 additions & 0 deletions benchmarks/metal_m1ultra_vlm_2026-07-06.csv
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-06-13_single_LFM2-350M-8bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
LFM2-350M-8bit,./models/LFM2-350M-8bit/,8,100,12.25,653.11,178.43,560.44,2026-06-13,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-06-14_single_bitnet-b1.58-2B-4T-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
bitnet-b1.58-2B-4T-4bit,./models/bitnet-b1.58-2B-4T-4bit/,8,100,47.86,167.17,620.58,161.14,2026-06-14,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-06-14_single_bitnet-b1.58-2B-4T.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
bitnet-b1.58-2B-4T,./models/bitnet-b1.58-2B-4T/,8,100,44.55,179.57,685.03,145.98,2026-06-14,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
20 changes: 20 additions & 0 deletions benchmarks/pylm_m1ultra_2026-06-16_repfresh.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
gemma-2b-4bit,./models/gemma-2b-4bit/,8,32,31.37,254.99,153.44,208.54,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
gemma-3-4b-it-4bit,./models/gemma-3-4b-it-4bit/,8,100,58.96,135.68,909.52,109.95,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
gemma-4-26b-a4b-it-4bit,./models/gemma-4-26b-a4b-it-4bit/,7,100,74.90,93.46,1366.69,73.17,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
gemma2-2b-4bit,./models/gemma2-2b-4bit/,8,20,41.18,194.25,143.69,139.18,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
glm4-flash-4bit,./models/glm4-flash-4bit/,7,100,99.20,70.56,2193.73,45.58,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
gpt-oss-120b-4bit,./models/gpt-oss-120b-4bit/,7,100,98.23,71.26,1759.48,56.83,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
jamba-v0.1-4bit,./models/jamba-v0.1-4bit/,8,77,49.41,161.92,589.95,130.52,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
meta-llama-3.1-8b-instruct-4bit,./models/meta-llama-3.1-8b-instruct-4bit/,7,100,61.14,114.50,925.29,108.07,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
mixtral-8x7b-4bit,./models/mixtral-8x7b-4bit/,8,100,120.15,66.58,1831.55,54.60,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
nemotron-h-30b-4bit,./models/nemotron-h-30b-4bit/,7,100,72.69,96.29,1074.34,93.08,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
phi-3.5-mini-4bit,./models/phi-3.5-mini-4bit/,7,100,39.92,175.34,594.21,168.29,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
qwen2.5-7b-instruct-4bit,./models/qwen2.5-7b-instruct-4bit/,7,100,50.97,137.32,895.66,111.65,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
qwen3-30b-a3b-4bit,./models/qwen3-30b-a3b-4bit/,7,100,80.35,87.12,1458.04,68.59,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
qwen3-vl-30b-a3b-4bit,./models/qwen3-vl-30b-a3b-4bit/,7,62,82.37,84.98,903.18,68.65,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
qwen3-vl-32b-4bit,./models/qwen3-vl-32b-4bit/,7,51,296.00,23.65,2332.10,21.87,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
qwen3.5-0.8b-4bit,./models/qwen3.5-0.8b-4bit/,7,100,21.32,328.40,348.21,287.18,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
smollm-135m-4bit,./models/smollm-135m-4bit/,7,100,13.56,516.33,263.40,379.65,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
solar-open-100b-4bit,./models/solar-open-100b-4bit/,7,100,168.09,41.64,2772.70,36.07,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
starcoder2-3b-4bit,./models/starcoder2-3b-4bit/,7,100,31.38,223.10,592.27,168.84,2026-06-16,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_hunyuan-a13b-instruct-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
hunyuan-a13b-instruct-4bit,./models/hunyuan-a13b-instruct-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_llada2.0-mini-preview-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
llada2.0-mini-preview-4bit,./models/llada2.0-mini-preview-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_llama-3.2-1b-instruct.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
llama-3.2-1b-instruct,./models/llama-3.2-1b-instruct/,8,100,29.44,271.71,532.25,187.88,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_mellum2-12b-a2.5b-base.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
mellum2-12b-a2.5b-base,./models/mellum2-12b-a2.5b-base/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_mistral-small-4-119b-2603-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
mistral-small-4-119b-2603-4bit,./models/mistral-small-4-119b-2603-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_phi-3-small-8k-instruct-aq4_64.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
phi-3-small-8k-instruct-aq4_64,./models/phi-3-small-8k-instruct-aq4_64/,7,33,86.10,81.31,1221.82,27.01,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_2026-07-06_single_qwen3-next-80b-a3b-instruct-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
qwen3-next-80b-a3b-instruct-4bit,./models/qwen3-next-80b-a3b-instruct-4bit/,7,100,102.92,68.02,1933.06,51.73,2026-07-06,Apple_M1_Ultra_128GB,mlx-lm-0.31.3,python,100,"Hello, how are you today?"
10 changes: 10 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-06-16_repfresh.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
gemma-4-26b-a4b-it-4bit,./models/gemma-4-26b-a4b-it-4bit/,277,13,1053.40,262.96,211.64,61.42,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
gemma-4-e2b-it-4bit,./models/gemma-4-e2b-it-4bit/,277,100,420.68,658.46,995.32,100.47,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
gemma3n-e2b-4bit,./models/gemma3n-e2b-4bit/,275,27,522.03,526.79,442.65,61.00,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
internvl3-1b,./models/internvl3-1b/,270,39,162.24,1664.19,145.87,267.35,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
llava-interleave-qwen-0.5b-bf16,./models/llava-interleave-qwen-0.5b-bf16/,744,60,189.77,3920.58,268.81,223.21,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
molmo2-4b,./models/molmo2-4b/,438,47,625.03,700.77,769.94,61.04,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
phi-3.5-vision-4bit,./models/phi-3.5-vision-4bit/,770,26,736.88,1044.95,280.90,92.56,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
qwen3.5-0.8b-4bit,./models/qwen3.5-0.8b-4bit/,,,,,,,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
qwen3.5-35b-a3b-4bit,./models/qwen3.5-35b-a3b-4bit/,,,,,,,2026-06-16,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_deepseek-ocr-2-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
deepseek-ocr-2-4bit,./models/deepseek-ocr-2-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_deepseek-ocr-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
deepseek-ocr-4bit,./models/deepseek-ocr-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_deepseek-vl2-small-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
deepseek-vl2-small-4bit,./models/deepseek-vl2-small-4bit/,436,9,551.19,791.02,98.52,91.35,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_dots.ocr-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
dots.ocr-4bit,./models/dots.ocr-4bit/,74,1,172.89,428.01,0.03,34334.76,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_ernie-4.5-vl-28b-a3b-thinking-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
ernie-4.5-vl-28b-a3b-thinking-4bit,./models/ernie-4.5-vl-28b-a3b-thinking-4bit/,113,100,408.92,276.34,1770.46,56.48,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_fastvlm-0.5b-bf16.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
fastvlm-0.5b-bf16,./models/fastvlm-0.5b-bf16/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_glm-ocr-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
glm-ocr-4bit,./models/glm-ocr-4bit/,78,11,74.50,1047.02,38.11,288.66,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_granite-4.0-3b-vision-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
granite-4.0-3b-vision-4bit,./models/granite-4.0-3b-vision-4bit/,6,11,185.70,32.31,87.01,126.42,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_granite-vision-3.2-2b-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
granite-vision-3.2-2b-4bit,./models/granite-vision-3.2-2b-4bit/,6,100,38.40,156.25,666.52,150.03,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_hunyuanocr-mlx-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
hunyuanocr-mlx-4bit,./models/hunyuanocr-mlx-4bit/,284,11,261.88,1084.46,72.78,151.14,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_idefics2-8b-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
idefics2-8b-4bit,./models/idefics2-8b-4bit/,340,15,972.26,349.70,147.39,101.77,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_idefics3-8b-llama3-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
idefics3-8b-llama3-4bit,./models/idefics3-8b-llama3-4bit/,3041,100,4718.98,644.42,1113.21,89.83,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_lfm2-vl-450m-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
lfm2-vl-450m-4bit,./models/lfm2-vl-450m-4bit/,,,,,,,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?",FAIL:warmup
2 changes: 2 additions & 0 deletions benchmarks/pylm_m1ultra_vlm_2026-07-06_single_qwen3-omni-30b-a3b-instruct-4bit.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlx_version,build_type,max_tokens,prompt
qwen3-omni-30b-a3b-instruct-4bit,./models/qwen3-omni-30b-a3b-instruct-4bit/,65,63,388.87,167.15,1545.63,40.76,2026-07-06,Apple_M1_Ultra_128GB,mlx-vlm-0.4.4,python,100,"What is in this image?"
5 changes: 4 additions & 1 deletion docs/benchmark_results/model_tests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ M5 Max, and mlx-lm / mlx-vlm baselines, see

| Hardware | File | Status | Last Updated |
|----------|------|--------|-------------|
| Mac Studio M1 Ultra 128GB | [model_tests_m1ultra.md](model_tests_m1ultra.md) | Active | 2026-06-15 |
| Mac Studio M1 Ultra 128GB | [model_tests_m1ultra.md](model_tests_m1ultra.md) | Active | 2026-07-06 |
| MacBook Pro M5 Max 128GB | [model_tests_m5max.md](model_tests_m5max.md) | Active | 2026-06-15 |
| NVIDIA GB10 (DGX Spark) | [model_tests_gb10.md](model_tests_gb10.md) | Active | 2026-06-17 |

Expand All @@ -27,6 +27,9 @@ Current source-of-truth data lives in `benchmarks/`:
| `metal_m5max_vlm_2026-05-20.csv` | M5 Max | 2026-05-20 (mlxcel 0.0.28, MLX 0.31.2; Gemma3n + Molmo v1 + Phi-3.5 vision + Gemma3 4B VLM entries) | VLM |
| `pylm_m5max_2026-05-18.csv` | M5 Max | 2026-05-19 benchmark campaign (mlx-lm 0.31.3 baseline; CSV date crossed midnight) | Text |
| `pylm_m5max_vlm_2026-05-18.csv` | M5 Max | 2026-05-19 benchmark campaign (mlx-vlm 0.4.4 baseline; CSV date crossed midnight) | VLM |
| `metal_m1ultra_2026-07-06.csv` | M1 Ultra | 2026-07-06 (mlxcel 0.3.3; full text re-benchmark post VLM-port batch #660-#664 and fixes #666-#668/#671, 169 rows) | Text |
| `metal_m1ultra_vlm_2026-07-06.csv` | M1 Ultra | 2026-07-06 (mlxcel 0.3.3; full VLM re-benchmark, 14 new VLM families measured) | VLM |
| `pylm_m1ultra_*2026-07-06*_single_*.csv` | M1 Ultra | 2026-07-06 (mlx-lm 0.31.3 / mlx-vlm dev; per-model python baselines for 21 newly added models, 13 measured / 8 python-side FAIL) | Baselines |
| `metal_m1ultra_2026-06-15.csv` | M1 Ultra | 2026-06-15 (mlxcel 0.2.1, MLX pin a6ec712; full text re-benchmark post #289 fix, 151 rows) | Text |
| `metal_m1ultra_vlm_2026-06-15.csv` | M1 Ultra | 2026-06-15 (mlxcel 0.2.1, MLX pin a6ec712; full VLM re-benchmark, 55 measured rows) | VLM |
| `metal_m1ultra_2026-06-15_pre289_regressed.csv` | M1 Ultra | 2026-06-15 (mlxcel pre-#290; bf16-scale decode regression evidence sweep) | Text |
Expand Down
Loading