Wow. I wasn't aware that 4090 is so fast.

This is from one/two generations old hardware but it's for 70B model which might be of interest.

CPU: AMD 3995WX, GPU: 2x Nvidia 3090, Ubuntu 23.10, Kernel 6.5.0-14, NV Driver: 545.23.08, CUDA: 12.3.1

For comparison, 7B model on the same hardware

could you try at 7b model for correct comparation? Thanks!

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

what the hell this is lower than m4 max, i mean 2x 3090 whatt..??

Probably because llama.cpp is not well optimized on gpus. You can expect significantly better throughput with sglang and vllm.

I am not sure why, but the results that I get are not very consistent. I suspect that it may due to the cooling limitations of the smaller laptop. I repeated the test now and the results are very similar to yours.

You can compute these. By default, you can use ~75% of the total RAM with the GPU. You can use more if you do some tricks

I'm also using a MBP16 M2Max with the same CPU/GPU specs, but only 32 gb ram and my results are roughly the same:

M2 MAX (MBP 16) 8+4 CPU, 38 GPU, 32 GB RAM ✅

build: 22da055 (1566)

Yes, it is expected that the same cpu/gpu spec will have similar performance values for same models to be compared regardless of RAM, as long as the size of the model to be used can be loaded into memory.
The amount of RAM is a limiting factor in the size of the model that can be loaded, as only 75% (by default) of the unified memory can be used as VRAM on the GPU
https://github.com/ggerganov/llama.cpp#memorydisk-requirements

off topic, but your benchmark output is my desktop rn :D

This one has 150 GB/s memory bandwidth, correct?

Yes, that's correct. (source)

Could it run a Q5 quant of llama3 70b Instruct at ~2 tokens per second?

I'm also interested to see if it can run a bit bigger model with some kind of reasonable outcome. Just don't want to spend MCB Max money

The old models give excellent comparative numbers but I wonder if the benchmark needs to be re-based around the current most popular models at some point.

Not just bigger ones for finding the biggest but popular sets / distillations that go from small to very large.

It looks like 96-128 ish gigs of shared memory will be practical on Apple / AMD / nvidia digits going forward.

The question is not silly - the observation is expected. At large batch size (PP means batch size of 512) the computation is compute bound. I.e. the speed depends on how many FLOPS you can utilize. For quantum models, the existing kernels require extra compute to dequantize the data compared to F16 models where the data is already in F16 format.

@eightpigs can you please confirm that this is indeed a 14'' MBP that you used for testing? The spec you provided is only available for 16'' MBP M4 Pro Max which hase a higher memory bandwidth than 14'' MBP M4 Pro Max model (536gb/s vs 410gb/s).

@maciejjedrzejczyk This is the result from my testing on a 14’’ MBP. The specs I provided are correct, and here are the details:

> system_profiler SPHardwareDataType SPDisplaysDataType 
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: Mac16,6
      Chip: Apple M4 Max
      Total Number of Cores: 16 (12 performance and 4 efficiency)
      Memory: 128 GB
      ...

Graphics/Displays:

    Apple M4 Max:

      Chipset Model: Apple M4 Max
      Type: GPU
      Bus: Built-In
      Total Number of Cores: 40
      ...

The M4 Max memory bandwidth can go up to 546GB/s: https://support.apple.com/en-us/121553

Thank you for confirmation. My confusion was based on the ground that I used apple store configurator specific to my country which only showed a single available spec for 14'' MBP M4 Pro Max with a lower memory bandwidth (36GB RAM). I like the 14'' form factor much more than that of 16'' version but that was the only missing part in my research :) Just a follow-up question - would you consider the thermals (temp while on lap, fan noise, multitasking etc.) on this machine at an acceptable level while using LLM inference?

I also prefer the 14-inch MBP.

As for noise, I mostly run 7B or 14B models, so noise hasn’t been an issue for me. Here’s some data I tracked with my Apple Watch for reference:

• DeepSeek-R1-Distill-Llama-70B-8bit: Fan noise is around 56dB.
• DeepSeek-R1-Distill-Qwen-32B-MLX-8bit: Fan noise is around 48dB.
• DeepSeek-R1-Distill-Qwen-14B-8bit: Fan is almost silent.

On multitasking, I haven’t run into any scenarios where I felt the machine was under pressure. Performance has been more than enough for me.

Can you please test Gemma 3 27B for me?

Depends on how much RAM you want to dedicate to AI inferencing. I think if you tweak your MacOS you might have the option to use up to 12GB for the model. So a model with 20B parameters quantised down to 4bit might just work. If it is a plain M3 you have, inferencing speed might be too slow, so you probably would stay with a smaller model.

Thank you for the thoughtful response. I am keeping an eye out for any tests here that most closely resemble my system.

You can try Deepseek-R1:14b Q4_K_M from ollame, it's only 9.0GB:
ollama run deepseek-r1:14b

I think it's primarily due to memory bandwidth (first column). Where they have the same bandwidth, the results are close. But in cases where the M2 Max or Ultra has substantially higher bandwidth, it outperforms the equivalent M3 or M4.

You know UP TO 16,9 that is marketing - may be Apple run that speed comparison with a model, which did not fit before into the unified memory and now does so. We have to tweak memory settings to make a bigger than default part of the RAM GPU-unified. But never all RAM can be used as unified RAM. See #2182 (comment) and all msg above.

Or Apple might offer a much more performant MLX option with M3 Ultra. Who knows? Benchmarks will tell in a few days I assume.

Will this end up like with NVIDIA's RTX series? Today some M4s (mostly those with maxed RAM) are no more listed on the website for several countries - if you are able to load it at all. Apple seems to have major backend issues since hours. Mmh.

Found some numbers: https://creativestrategies.com/mac-studio-m3-ultra-ai-workstation-review/
Seems to be only tg, no pp.

Nobody seems to have posted any pp so far... and I wonder why.

Edit: https://www.reddit.com/r/LocalLLaMA/comments/1j9jfbt/comment/mhe1ku9/

Nobody seems to have posted any pp so far... and I wonder why.

Considering the article's bias I'm not surprised. The 5090's going to destroy the Mac when the model fully fits in VRAM, so the author uses a 128k context and swapping on the 5090 (not even partial offloading) to make the Mac appear more effective. For prompt processing I think the 5090 might actually beat the Mac even with partial offloading. IMO he should have also done tests with smaller contexts to show the distinction between a model that fits in VRAM (5090 wins) and one that doesn't (Mac wins).

Our llama-bench should be the standard for testing llama.cpp but sadly a lot of people don't know about it.

So identical speeds to the M2 ultra it looks like, because same bandwidth... Not worth buying imo!

Well, entry level model with 256GB of RAM costs exactly the same as 192GB model costed previously, plus (if one's rich) there's 512GB version, so even though speed seems similar, there's still argument to be made in favour of those.

I've already seen several benchmarks that say the M3 only makes sense in terms of memory and not performance. It seems you get better value for money with a M4 Max.

Still I wonder what is the TG performance with a ~500GB model vs a ~50GB model on the same machine.

All else equal - probably 10x, as there is 10x as much weights, no matter if it's memory or compute bound. But it's not that simple: >600B DeepSeek/R1 are MoEs with iirc ~37B parameters per expert, so I'd expect it to work much (2x?) faster than 70B Llama.

Doubtful. M4 Ultra does not exists, at least not yet.