Vitor Cepeda Lopes

The memory unit economics of Apple Silicon are insane. When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference. M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station. I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...) I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory. With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each. The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.