I remembered that I have a favorite programming benchmark, the one that I previously used to evaluate programming languages, that I can now use to evaluate coding models! I'm talking, of course, about ICFP Programming Contest 2006 virtual machine.
So I asked models to read
boundvariable.org/task.shtml and to implement the UM virtual machine capable of running `sandmark.umz`.
I've made a few runs with a few models that are +- in the same price range on Opencode Zen / Go: DeepSeek V4 Pro, GLM-5.1, Kimi-2.6, MiniMax-M3, Claude Haiku 4.5 and GPT-5.4 mini (well, DeepSeek is way cheaper, actually).
So, GLM, DeepSeek, Kimi: usually use C, usually get pretty close, but then invariably are confused about "self-decompressing" wording on the web page, which makes them run `sandmark` not directly on their VM (as they should), but using `
um.um` (UM emulator written in UM) -- which, of course, is terribly slow. Then they get stuck trying random useless ideas to speed things up. I had to interrupt them at that point.
Minimax-M3: usually uses C, is never confused about "self-decompressing", but invariably screws up op 13 every time. Sometimes it manages to dig itself out of that hole and delivers good results; other times it finds a way to dig itself deeper with some other mistakes.
Haiku just did the whole thing in python (with terrible performance) and with some additional error causing `sandmark` to terminate early and without producing full output. Haiku then proudly declared that it has successfully completed its task.
And GPT-5.4 mini just one-shots the whole thing every time, in Go or Rust, pretty fast, too -- definitely making it look too easy. It's a clear winner, and it's not close.
I tried warning DeepSeek and Kimi about `
um.um`; after that, DeepSeek one-shotted the task pretty fast, too, and Kimi decided that it has to ignore my advice, and ended up stuck trying useless performance tricks again.
I tried warning MiniMax-M3 about op 13, but it found some new way to screw up the implementation.