Tagged: Benchmark

102 articles

JioStar architect on running AI discovery for 500M users

JioStar's Vijay Seshadri shares how AI-powered discovery works at a scale of 500 million users. He covers lessons on evals, guardrails and moving AI into production. Source: openai.com

Repligate reshares a post without comment

The snippet shows a retweet by Repligate (j⧉nus) of a post from GRITCULT, with no commentary added. Surrounding page text mentions Perplexity adding a model called Gemini 4 Carbon, pitched as lowering benchmark costs, but the content is too fragmentary to confirm details. Source: digg.com

Google launches Multimodal Knowledge Agent benchmark

Google has introduced the MKA benchmark, designed to test how well agents reason through complex visual tasks. The accompanying report shows how Gemini 4 Carbon performs. Source: blog.google

AI Release Analytics: Cadence, Trends and Benchmarks Across Major Labs

An analytics resource tracks how quickly AI models ship, including releases per month, each lab's cadence, and benchmark trajectories. It covers OpenAI, Anthropic, Google, Meta, SpaceXSI, and others. Source: aireleasetracker.com

LLMs find new tradeoffs on the TIS-100 optimization frontier

A benchmark gave 16 large language models the assembly puzzle game TIS-100 and asked them to improve working programs. The models produced 133 new Pareto-optimal solutions. Source: news.lavx.hu

Gemini Omni Flash 1.2 vs Qwen 3.8 Omni Flash: one makes video, one reads it

One model outputs video with audio, while the other returns text from an hour of footage. They are fundamentally different tools with different pricing meters and no shared leaderboard to compare them. Source: orcarouter.ai

New Leaderboard Benchmarks LLMs on Real-World Construction Work

A new Construction AI Leaderboard was created to measure how well LLMs handle real-world construction tasks. It also puts particular emphasis on performance in BIM, an area ONESTRUCTION focuses on. Source: huggingface.co

OpenAI's BioBasicsBench tests AI agents on everyday lab work

OpenAI has introduced BioBasicsBench, a benchmark of 100 tasks. It measures how well AI agents interpret raw experimental data and handle routine wet-lab science. Source: openai.com

Nine LLMs benchmarked on professional Parkinson's disease clinical tasks

A benchmark of nine large language models on 200 professional Parkinson's disease tasks found minimal differences between them in factual accuracy. The models diverged more in other aspects of performance. Source: bioengineer.org

Harness adapters bring safe RL to agentic development

Harness adapters can make local agentic development more efficient and accurate. In smoke training tests on held-out eval tasks, they made fewer tool calls. Source: huggingface.co

Same model, different harness: CUGA, Hermes, Deep Agents compared

The test ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks spanning 9 apps and 457 APIs, including email, payments, and file storage. The comparison shows how much the surrounding harness matters when the underlying model stays the same. Source: huggingface.co

StateBench tests how LLMs track evolving state in long texts

StateBench is a new benchmark for long-context state tracking. It measures how well language models keep tabs on entities whose properties change throughout a long document. Source: huggingface.co

LifeVal benchmark tests whether AI can handle everyday life tasks

LifeVal is a new benchmark that measures how well AI helps people get ordinary tasks done. The scenarios range from finding childcare to managing personal finances and dealing with government paperwork. Source: openai.com

Alphabet lost $500B in a week despite shipping its best model, says Mostaque

Alphabet shed $500 billion in a week even as it released its best model yet. Emad Mostaque argues search ads will peak in Q1 2027, and that the benchmark crown comes with an asterisk. Source: finance.biggo.com

LLMs parse public health policy debates in German news

Researchers show that GPT-5 and Gemini can extract and classify policy-framing quotes from German news articles. Their F1-scores come close to 80 percent when measured against a human benchmark. Source: bioengineer.org

Revamped Artificial Analysis index widens gap with open models

Artificial Analysis has overhauled its Intelligence Index, reshuffling the leaderboard. As a result, Kimi K3.1 Pro slipped five places in the rankings. Source: the-decoder.com

Kernel-Tuned ML Pushes Network Intrusion Detection to Nearly 99% Accuracy

A new study reports that Random Forest and Support Vector Machines with Gaussian and Sigmoid kernels, paired with systematic data preprocessing, reached accuracy of up to 99.87% on benchmark data. The results suggest careful kernel tuning can significantly boost network intrusion detection performance. Source: bioengineer.org

Open-data ML framework nears perfect breast cancer diagnosis on benchmark

Researchers in China have built a fully reproducible, open-data framework that combines interpretable machine learning with quantitative cytopathology. On a benchmark dataset, it reaches near-perfect accuracy in diagnosing breast cancer. Source: bioengineer.org

Anthropic study: AI agents still can't do real-world scientific research

Cutting-edge coding agents are turning machine learning research into automatically executable workflows, but most existing studies focus on narrow benchmark settings. Anthropic's research indicates that AI still falls short of conducting genuine, real-world scientific research. Source: eu.36kr.com

HeyGen's Video 1 takes top spot on silent text-to-video leaderboard

AI video startup HeyGen announced on 22 October 2026 that Video 1, its first general-purpose video model, ranked first for text-to-video on the Artificial Analysis leaderboard for silent video. The result marks a strong debut for the company's move beyond avatar-style content. Source: app.dealroom.co