Google's newly released Gemini 3.8 Flash shows clear improvement over the previous 3.6 — scores on our internal OpenResty Coding Evals benchmark went up noticeably. But latency also took a significant hit; it's way slower than before 😂 See the two charts below for details. Looks like Google is going all-in on cost-efficient small models now? lol. A Gemini Pro model seems nowhere in sight…

Below are the latest results from our private OpenResty Coding Evals benchmark, now including the new Claude Opus 5 model. It has indeed claimed the throne. We've also added a leaderboard for average end-to-end latency.

After a long stretch of work, we finally have results from running open-weight models against our internal OpenResty Coding Evals benchmark. Getting there meant fixing a ton of issues with opencode, pi, and claude code when driving open-source models — making sure these agent harnesses weren't handicapping the models. GLM 5.2 is genuinely impressive — approaching Claude Sonnet 5 territory — but there's still a noticeable gap against models like Claude Opus 4.8. Note that due to inherent LLM randomness, score differences within 1% should be considered normal variance. First image: quality scores. Second image: average total latency per task across agents. Google and xAI models are definitely the fastest, though their quality scores are lower. Still waiting for Kimi K3, Qwen 3.8 Max, and DeepSeek V4 stable to publicly release all model weights. Once they do, we'll add them to the eval. Over the next few days I'll also run our other, harder benchmark — the OpenResty Bug-Hunt Evals. Stay tuned!

Spent the past few days compiling our hardest real-world bugs from OpenResty Inc's internal codebase into a new benchmark: OpenResty Bug-Hunting Evals. Tested all major frontier models from Anthropic, OpenAI, and Google. Results below. The task: given only a bug's symptom description, autonomously locate the root cause, fix it correctly, without introducing any regressions. All bugs live in our private company repos, completely off the public internet, so no data leakage, no contamination, making it as objective as it gets. Anthropic's mythic Claude Fable 5 leads by a wide margin. Even OpenAI's latest GPT-5.6 Sol still trails noticeably, it hasn't caught up to Claude Opus 4.8 Max yet. Google's frontier models remain at the bottom. This eval burns WAY more tokens than our previous OpenResty Coding Evals btw. One thing worth noting: OpenAI's models + coding agent have noticeably lower end-to-end latency. Anthropic's top models clearly win on quality, but the total wall-clock time is painful, nearly an hour for the full eval run.

We just used OpenResty Inc's internal proprietary software pojects' hardest real algorithm design & impl problems to form an LLM evals suite. This suite is coding & testing from scratch. And it invokes the native CLI agents like claude code, codex, antigravity, and grok build. The suite has never been published so LLMs never saw it before. Below are the quality score ranking results from burning a lot of tokens! Claude Fable 5 and GPT-5.6 Sol are definitely the kings!

The evaluation test set recently developed by Yao Shunyu from Tencent, which assesses large models' in-context learning capabilities, is quite intriguing. In their tests, GPT-5.1 secured the first position, followed by Claude Opus 4.5 in second place. This test specifically challenges large models' ability to learn and apply knowledge within a given context, ensuring the content has not been encountered during the pre-training phase, thus making it very difficult to game the system. However, even the top performer, GPT-5.1, only achieved a success rate of just over 23%. In contrast, GPT-5.2 showed a noticeable decline in performance. #LLMs #Claude #OpenAI

An interesting performance optimization case: A high-throughput core system showed excellent stress test results, yet its CPU utilization was unusually high. This "outwardly healthy, inwardly fragile" system state is often the most challenging to diagnose. By using OpenResty XRay we quickly identified the blind spot everyone had overlooked: an inefficient Gzip compression configuration, which was consuming over half of the CPU resources. We highly recommend reading this case study, as it perfectly demonstrates OpenResty XRay's value: cutting through the ambiguity of surface-level metrics to pinpoint the root cause of problems. blog.openresty.com/en/xray-g…

Any unexplained technical anomaly is an "invisible debt" often overlooked amidst rapid business growth. Recently, we helped a client resolve a challenging Nginx memory leak, whose root cause was a classic C/C++ process memory leak in a production environment. This issue acts like a hidden cost sink, silently draining system resources and team energy in subtle, hard-to-detect ways. Many monitoring tools can tell you "what happened," but they often fall short of explaining "why it happened." This lack of deep diagnostic capability is precisely what allows technical debt to accumulate. So, how does OpenResty XRay empower R&D and Operations teams to see beyond surface-level symptoms and pinpoint the true root cause of problems, transforming reactive post-mortem investigations into predictable, proactive, and definitive solutions? This article will unveil the answer through a concrete customer case: blog.openresty.com/en/mem-le…