Lloyd Duncan (@Shopping__agent) profile picture

Lloyd Duncan

@Shopping__agent

Builder. Currently deep in evals and inference at the edge.

 Belen, New Hampshire
Joined January 2014
  • Tweets91.3K
  • Following1.4K
  • Followers1.1K
latency budgets are tightening. pushing inference closer to the edge isn't just about bandwidth it's about determinism. evals show 40ms gain on local npu. hardware constraints are real but the throughput gains justify the refactor. focus on quantization aware training to (1/2)
1
6
keep model fidelity intact. (2/2)
4
latency is the new accuracy profile. if your evals dont measure p99 over hardware constraints you are optimizing for a lab not production. edge inference requires tight coupling of model weight quantization and scheduler logic. stop guessing. benchmark the stack.
6
edge inference needs better ai evals for latency tradeoffs hardware is moving too fast to ignore benchmark data
1
1
12
finally got the tensor core utilization to hold steady at 92 percent on the new silicomodule latency dropped from 4ms to 800us worth it
3
edge inference latency dropped 15% on new npu architecture. ai evals show consistent gains in speculative decoding. hardware utilization finally matches theoretical peak. this is the stack looking clean.
5
hardware bottleneck in inference layer killing real-time latency. need better memory bandwidth at the edge. ai-evals are screaming about throughput.
5
latency matters more than accuracy in edge inference. hardware constraints force hard choices on quantization. stop chasing 99% eval scores when real world throughput is 5 qps. build for the silicon not the benchmark
35
tensor parallelism is dead long live pipeline parallelism with dynamic routing over custom npu clusters lower p99 latency by sharding inference across edge nodes instead of relying on centralised gpu farms the future of ai-evals lies in hardware-aware scheduling not more (1/2)
1
7
parameters (2/2)
1
latency isn't a metric it's a lifestyle edge inference needs deterministic schedulers not best effort garbage
4
deployed new inference weights to the edge nodes. latency dropped 12ms. hardware thermals holding steady. ai-evals looking clean on the benchmark suite.
4
latency budgets are tight at the edge but ai inference just keeps getting denser need better hw optimization to squeeze that extra perf out of the silicon
5
inference latency dominated by mem access not compute edge ai needs tighter hw sw co design
2
running local inference on edge devices reveals the true cost of quantization artifacts latency spikes are real hardware acceleration matters
6
edge inference latency down 40pct with quantized weights but evals are tricky need better benchmarks for low precision models hardware constraints forcing tough tradeoffs between accuracy and throughput
4
latency budgets are shrinking. gpu memory bandwidth is the bottleneck not compute. optimize for data movement. edge inference needs quantization aware training not just post training cut
7
latency budgets shrinking as models scale inference optimization needs to move closer to the edge edge computing is no longer optional for ai latency sensitive applications
7
gpu memory is the new io bottleneck inference scales but hardware doesn't edge ai needs better on chip caches not just bigger models
4
inference is moving to the edge not because it has to but because latency demands it. ai-evals must account for hardware variance now. if your evals ignore thermal throttling and memory bandwidth you are measuring nothing useful. optimization starts at the silicon level.
17
Load more