AI Benchmark Trust | AI Field Notes #66

AnalysisA benchmark is only useful if the model has not already seen the answers. Cursor disclosed that Grok 4.5, the coding model it co-launched with xAI, had an earlier snapshot of Cursor's own codebase baked into its training data, which is exactly what CursorBench, built from real Cursor user sessions, is supposed to test blind. Cursor excluded the score from its public comparison and says the tainted data is gone for future models. On the independent Artificial Analysis index, which Grok could not train against, the model ranks fourth with a score of 54. As labs train on everything, the tests meant to rank them keep leaking into the training set, and the cleanest scores are the ones a vendor cannot control.

AnalysisHand a frontier model a lawyer's brief, a teacher's lesson plan, or a nurse's chart and it botches most of them. Snorkel's GDPval+ test, roughly 2,000 tasks written by working professionals across the economy, found the best performer, xAI's Grok 4.5, passing 29% of expert criteria, ahead of GPT-5.5 at 22% and Anthropic's Opus 4.8 at 21%. Even in its strongest domain, education, it cleared 58%. The leaderboard scores that sell these models measure narrow puzzles; graded on the actual deliverables people are paid to produce, all three fail more than they pass.

AnalysisAfter a year of telling its champions to buy domestic, Beijing is quietly cracking the door for Nvidia. Reports say China will let a short list, Alibaba, ByteDance, and DeepSeek, import H200 accelerators, but cap the total below 200,000 chips, less than half what the firms asked for. The restriction runs the other way from Washington's: the H200s may only train on public data, while Huawei's Ascend chips keep priority for inference. It is rationing dressed as a reversal, keeping just enough American hardware in play to stay competitive without loosening the push for home-grown silicon.

AnalysisEvery big AI operator is trying to stop renting all its compute from Nvidia, and Meta just set a date. A company memo says its custom chip, codenamed Iris and designed with Broadcom, enters production in September, part of a plan to reach 14 gigawatts of computing power by 2027, double this year's target. Iris is the first of four generations Meta wants to ship roughly every six months. The company expects to spend as much as $145 billion on AI infrastructure this year. Owning the silicon is how these firms plan to make that spend survive the next price war.

AnalysisThe company that makes the memory stacked next to Nvidia's GPUs just became the largest foreign name ever to list in the US. SK Hynix, the South Korean maker of high-bandwidth memory (HBM, the fast memory that feeds AI chips), priced its American shares at $149 and raised $26.5 billion, topping Alibaba's 2014 record, then rose 13% on the first day. The cash goes to new factories. HBM is the real bottleneck in AI hardware right now, booked out past 2027, so a war chest aimed at more capacity matters more to compute supply than another data-center headline.

AnalysisThe simplest way to run an AI model on your own laptop just raised real money. Ollama, the open-source tool that lets developers download and run large language models locally instead of calling a cloud API, closed a $65 million round led by Theory Ventures, bringing total funding to $88 million. It now counts 8.9 million monthly active developers and more than 67,000 community-built integrations. The appeal is plain: no per-token bill, no data leaving the machine, no rate limits. A tool that started as a weekend convenience is now infrastructure that every major lab and hardware vendor ships against.

AnalysisThe pitch that you should rent all your intelligence from a frontier lab now has a well-funded rival. Prime Intellect raised $130 million at a $1 billion valuation to sell the plumbing companies need to train and run their own models, spreading the work across scattered chips using FSDP2, an open-source piece of PyTorch. Radical Ventures led the round; the venture arms of Nvidia and Intel joined. The company claims 6,000 customers and more than $100 million in annual revenue. The bet is that plenty of teams would rather own a smaller model tuned to their work than pay per token for someone else's.

AnalysisMeta stopped licensing its image generation and built its own. Muse Image, the first image model from Meta's Superintelligence Labs, went live across the Meta AI app, Instagram Stories, and WhatsApp, with a Muse Video preview alongside it. It works more like an agent than a single pass, calling search and coding tools to check facts and refining its own output before returning it. The launch drew immediate pushback over whether users' own photos trained it. Putting a capable generator inside apps that 3 billion people already open is a distribution advantage no image startup can match, whatever the quality gap.

AnalysisMathematicians spent five days seeing whether AI can handle the tedious part of their job. At an Imperial College London workshop tied to Kevin Buzzard's project to render the proof of Fermat's Last Theorem into Lean (software that checks every step of a proof for errors), researchers aimed autoformalization tools at the groundwork, funded by the AI firm Logos Research. Fermat scribbled the claim in a margin in 1637, and no one proved it until 1994. The revealing part was where the tools broke: passages already in the shared math library formalized cleanly, while definition-heavy sections stumped them. Turning loose human intuition into steps a computer will accept is still the wall.

AnalysisOne state just put real teeth into AI safety rules while Washington debates. Illinois Governor JB Pritzker signed the Artificial Intelligence Safety Measures Act, aimed at developers with more than $500 million in annual revenue, which is to say the frontier labs. It requires public safety disclosures, first-in-the-nation annual third-party audits, critical-incident reporting to the state within 72 hours, and whistleblower protections for employees who raise alarms. The requirements start January 1, 2028, with fines of $1 million per violation and $3 million after that. It is a template other states can copy, and the enforcement date is the part worth marking.

AnalysisThe AI agent that runs multi-hour tasks no longer needs your laptop open. Anthropic extended Claude Cowork to web, iOS, and Android, so a job started at a desk keeps running in the cloud and can be steered from a phone, pausing to ask when it needs a decision. The usage data is the real story. In a sample of 1.2 million sessions across 600,000 organizations, software development was just 8.7% of the work; the biggest slice, 33.4%, was business-process grind like reconciling spreadsheets and stitching scattered updates into reports. Coding is a sliver here; the reconciling and reporting is the main event.