freedom Koan-Sin Tan (@koansin) profile picture

Coder?

Jhubei City, Hsinchu County,
Joined January 2009
Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)Photo by freedom Koan-Sin Tan (@koansin)
freedom Koan-Sin Tan retweeted
I've been traveling the world a bit. One of the most incredible experiences was getting to visit Taiwan w/ @JensenHuang. It was wild 🤯.. huge crowds of engineers, insanely good street food at night markets, and of course a lot of good conversations across many days. I caught some of it on video, see below. Alright, that's it from me, love you all ❤️ Timestamps: 0:00 - Highlights 0:46 - Intro 3:25 - Computex: crowds, engineers, tech, future of AI 9:12 - Chat with Jensen: keynote prep & nerves 13:03 - Chat with Jensen: Taiwan's tech history 19:47 - Raohe night market 29:37 - Chat with Jensen: memories from childhood 33:44 - NVIDIA party 40:29 - Lunch: braised pig's feet 43:09 - Chat with Jensen: GPUs, AI, and competition 55:07 - Ningxia night market 1:10:14 - Taiwanese breakfast 1:29:54 - Outro
495
1,141
242
12,560
977,579
ANE INT8 on A20 Pro is about 105 TOPS for INT8. Measured with my little MPSGraph based app, github.com/freedomtan/measur…. Note that these numbers are likely Winograd 1D F(2, 3) "accelerated".
2
1
3
205
Fixed my app. I didn't have weights designed to avoid sparsity. With carefully designed inputs and weights, I got 85 TOPS. For FP8, I got only 52 TOPS.
2
1
3
181
Dug into ANECompiler.framework of iOS/macOS 27 RC and found “bonded networks” on H18g/M6 and H19/A20 Pro. Before, multi-ANE chips like the M-Ultra just split up the work at the driver level across separate devices. (1/2)
2
2
1
6
255
But with bonded networks, the compiler actually splits a single model across two physical ANE units itself. So, I guess there is relatively high-speed interconnect between the two ANE units. The compiler spits out a .hwx file with both bonded and non-bonded versions. (2/2)
2
1
5
590
freedom Koan-Sin Tan retweeted
MLCommons® just launched MLPerf® Mobile on the Google Play Store! 📱 Benchmark your Android device’s AI performance on real-world ML tasks with this free, open-source app. Try it now: play.google.com/store/apps/d…
2
6
3
8
2,504
freedom Koan-Sin Tan retweeted
Just built an MCP for Ghidra. Now basically any LLM (Claude, Gemini, local...) can Reverse Engineer malware for you. With the right prompting, it automates a *ton* of tedious tasks. One-shot markups of entire binaries with just a click. Open source, on Github now.
83
766
83
4,385
287,002
If you ever wondered how to run Apple's ANE .hwx file on non-jailbroken phones, check my little program github.com/freedomtan/coreml…
1
4
146
It turns out it's possible to retrieve per-op profiling information w/o using `MLComputePlan`. Some undocumented classes and methods are required, though. the output: github.com/freedomtan/coreml… simple objective-c code: github.com/freedomtan/coreml… @fleetwood___: look what I found
1
4
232
freedom Koan-Sin Tan retweeted
You can now use any Hugging Face Hub model with KerasNLP (as long as the corresponding architecture is in KerasNLP)! What's more, you can also upload your own fine-tuned KerasNLP models to Hugging Face in one line. huggingface.co/blog/keras-nl…
7
35
4
175
30,233
freedom Koan-Sin Tan retweeted
Today we’re announcing Meta LLM Compiler, a family of models built on Meta Code Llama with additional code optimization and compiler capabilities. These models can emulate the compiler, predict optimal passes for code size, and disassemble code. They can be fine-tuned for new optimizations and compiler tasks. @HuggingFace repo ➡️ go.fb.me/tdd3dw Research paper ➡️ go.fb.me/85zwgy LLM Compiler achieves state-of-the-art results on code size optimization and disassembly. This work shows that AI is learning to optimize code and can assist compiler experts in identifying opportunities to apply optimizations. We’re releasing LLM Compiler 7B & 13B models under a permissive license for both research and commercial use in the hopes of making it easier for developers and researchers alike to leverage this in their work and carry forward new research in this space.
138
749
192
3,916
511,952
freedom Koan-Sin Tan retweeted
Back on April 1st I posted my three laws of robotics. Here are my three laws of AI. 1. When an AI system performs a task, human observers immediately estimate its general competence in areas that seem related. Usually that estimate is wildly overinflated. 2. Most successful AI deployments have a human somewhere in the loop (perhaps the person they are helping) and their intelligence smooths the edges. 3. Without carefully boxing in how an AI system is deployed there is always a long tail of special cases that take decades to discover and fix. Paradoxically all those fixes are AI-complete themselves.
8
45
5
161
26,633
freedom Koan-Sin Tan retweeted
My @AllwinnerTech Nezha and @StarFiveTech VisionFive 2 @risc_v SBCs run #OpenBSD smoothly. Nezha dmesg: sprunge.us/WicZw7 VF2 dmesg: sprunge.us/2CEWQi #riscv #sbc
3
7
40
2,355
freedom Koan-Sin Tan retweeted
Google announces Stealing Part of a Production Language Model We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under \20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.
40
312
94
1,682
266,200
freedom Koan-Sin Tan retweeted
📢 Announcing the MediaPipe LLM Inference API → goo.gle/3PbybEf Learn to run Gemma & other on-device LLMs with MediaPipe & @TensorFlow Lite, & get updates on: 💡 Performance & optimizations 🛠️ Supported model architectures 🧠 Experimental LLM Inference API 🧡 & more!
13
58
4
228
53,101
According to the name of its firmware, M3's ANE is h15/themis, the one from A16 (A17 Pro's ANE is h16/iaso). So its compute capacity stays roughly the same as A16.
249