Tagged: Benchmark

108 articles

MedChemBench tests whether AI can spot promising drug candidates

The new MedChemBench benchmark measures how well AI models read molecular structures, reason from experimental evidence, and choose compounds in early-stage drug discovery. Source: openai.com

Structured prompts lift LLM accuracy in concrete beam design

A benchmark study finds that structured prompting raises large language model accuracy on reinforced concrete beam calculations from 37 percent upward. Source: google.com

OVHcloud debuts Olala mini, its first LLM, without benchmark scores

French cloud provider OVHcloud has unveiled Olala mini 7A1B, an open model with 7 billion parameters of which 1 billion are active. It is due on Hugging Face on 19 November, though no benchmark scores were published. Source: google.com

Google TPUs post new AI training benchmark results

Google's TPUs hit the benchmark's target quality in 1.70 minutes. They scaled up to 4,096 chips while maintaining 96% efficiency. Source: blog.google

FinGPT-Risk tackles multi-task financial risk prediction on seven datasets

Researchers developed FinGPT-Risk, a unified multi-task language modeling framework that combines tabular features, textual profiles and meta-adaptation. It is evaluated on seven benchmark datasets for financial risk prediction. Source: bioengineer.org

XYBench tests whether AI can spot the flawed premise behind a question

XYBench is a new benchmark of more than 8,000 questions that checks how AI assistants handle requests built on misconceptions, such as asking how to thicken frosting. It measures whether a model addresses the underlying problem rather than just answering the question as posed. Source: huggingface.co

JioStar architect on running AI discovery for 500M users

JioStar's Vijay Seshadri shares how AI-powered discovery works at a scale of 500 million users. He covers lessons on evals, guardrails and moving AI into production. Source: openai.com

Repligate reshares a post without comment

The snippet shows a retweet by Repligate (j⧉nus) of a post from GRITCULT, with no commentary added. Surrounding page text mentions Perplexity adding a model called Gemini 4 Carbon, pitched as lowering benchmark costs, but the content is too fragmentary to confirm details. Source: digg.com

AI Release Analytics: Cadence, Trends and Benchmarks Across Major Labs

An analytics resource tracks how quickly AI models ship, including releases per month, each lab's cadence, and benchmark trajectories. It covers OpenAI, Anthropic, Google, Meta, SpaceXSI, and others. Source: aireleasetracker.com

Google launches Multimodal Knowledge Agent benchmark

Google has introduced the MKA benchmark, designed to test how well agents reason through complex visual tasks. The accompanying report shows how Gemini 4 Carbon performs. Source: blog.google

LLMs find new tradeoffs on the TIS-100 optimization frontier

A benchmark gave 16 large language models the assembly puzzle game TIS-100 and asked them to improve working programs. The models produced 133 new Pareto-optimal solutions. Source: news.lavx.hu

Gemini Omni Flash 1.2 vs Qwen 3.8 Omni Flash: one makes video, one reads it

One model outputs video with audio, while the other returns text from an hour of footage. They are fundamentally different tools with different pricing meters and no shared leaderboard to compare them. Source: orcarouter.ai

New Leaderboard Benchmarks LLMs on Real-World Construction Work

A new Construction AI Leaderboard was created to measure how well LLMs handle real-world construction tasks. It also puts particular emphasis on performance in BIM, an area ONESTRUCTION focuses on. Source: huggingface.co

OpenAI's BioBasicsBench tests AI agents on everyday lab work

OpenAI has introduced BioBasicsBench, a benchmark of 100 tasks. It measures how well AI agents interpret raw experimental data and handle routine wet-lab science. Source: openai.com

Nine LLMs benchmarked on professional Parkinson's disease clinical tasks

A benchmark of nine large language models on 200 professional Parkinson's disease tasks found minimal differences between them in factual accuracy. The models diverged more in other aspects of performance. Source: bioengineer.org

Harness adapters bring safe RL to agentic development

Harness adapters can make local agentic development more efficient and accurate. In smoke training tests on held-out eval tasks, they made fewer tool calls. Source: huggingface.co

Same model, different harness: CUGA, Hermes, Deep Agents compared

The test ran CUGA, Hermes, and Deep Agents on AppWorld, a benchmark of multi-step tasks spanning 9 apps and 457 APIs, including email, payments, and file storage. The comparison shows how much the surrounding harness matters when the underlying model stays the same. Source: huggingface.co

StateBench tests how LLMs track evolving state in long texts

StateBench is a new benchmark for long-context state tracking. It measures how well language models keep tabs on entities whose properties change throughout a long document. Source: huggingface.co

LifeVal benchmark tests whether AI can handle everyday life tasks

LifeVal is a new benchmark that measures how well AI helps people get ordinary tasks done. The scenarios range from finding childcare to managing personal finances and dealing with government paperwork. Source: openai.com

Alphabet lost $500B in a week despite shipping its best model, says Mostaque

Alphabet shed $500 billion in a week even as it released its best model yet. Emad Mostaque argues search ads will peak in Q1 2027, and that the benchmark crown comes with an asterisk. Source: finance.biggo.com