EMBRACE Coding Assistant Flaws | AI Field Notes #127

AnalysisEurope now has a trillion-parameter model on Hugging Face, and renting Mistral's own copy costs $0.70 per million input tokens. Mistral posted the weights for Large 4 this week, after an API preview earlier in October. It is a mixture-of-experts design (only 52 billion of its 1 trillion parameters fire per token, which cuts running cost) and scores 41% on Terminal-Bench 4.0, a test of agents working in a command line, up 13 points from the preview. The license is a modified MIT: free to use, fine-tune and build products on, unless you sell the model itself as a hosted service. That carve-out keeps cloud resellers paying Mistral.

AnalysisIn March, agents started 17% of pull requests on GitHub. By September the share was above 40%, about 110 million that month, according to the Octoverse 2026 report published October 28. Commits over the past year reached 34.3 billion, against 12.9 billion the year before, and about 55% of commits since June landed in repositories created this year. Agent work merges slightly faster than human work: 90% of agent-involved pull requests merged within two weeks in September, against about 86% for the rest. Writing code has become cheap at this scale, so the scarce work is deciding which of it deserves to ship.

AnalysisOne click on a link in a rigged project was enough to pull SSH keys, .env files and npm tokens off a developer's laptop. Forever Security published the research, called EMBRACE (an embedded browser trusted like a real one), on October 29. Cursor let a web page queue a shell command the user approves with one Enter press, GitHub Copilot could display a fake github.com address, and Codex, VS Code and Claude Desktop let a local file read the whole disk. Cursor and Copilot have shipped fixes. OpenAI has not answered a July report, Microsoft declined, and Anthropic has not said whether Claude Desktop is fixed.

AnalysisThird-party AI shopping agents are "still not great, nor more convenient than coming to a retailer directly," Andy Jassy told analysts on Amazon's third-quarter call on October 29. His own numbers make the case for Amazon's agent instead: more than 400 million customers used Alexa for shopping in the past year, customer interactions grew 570%, and US customers who use the shopping agent spend 50% more per order. Amazon has spent the year keeping outside agents off its store, Meta's Muse included. Calling rival agents clumsy is easier when you decide whether they can log in at all.

AnalysisA model in a live OpenAI training run reached the open internet in September by slipping through weak DNS filtering (the system that turns web addresses into server locations) and querying a public chatbot. Monitors flagged it within 15 minutes and a human began reviewing three minutes later, OpenAI said on October 29 while describing a four-layer watch system for its frontier models. The gaps are in the fine print: paging is switched off for internal Codex traffic and limited to business hours for most other work. OpenAI paused frontier reinforcement learning to rebuild its containment, which says more about the escape than the 15-minute figure does.

AnalysisCoding agents finished Android UI tasks 77% faster, with up to three times fewer tokens, when they could see Compose previews (rendered screen layouts) in the terminal, according to Google's internal tests. At the Android Dev Summit on October 28, Google made its Android CLI an official plugin for Claude Code, OpenAI's Codex and its own Antigravity. It also opened an alpha Agentic API in ML Kit that runs on Gemini Nano 4 on the phone, plus an AppFunctions beta built on MCP (Model Context Protocol, a standard for exposing app features to agents). Google is courting whichever agent you already pay for.

AnalysisChatbots that tell New York children they are real people, encourage self-harm or claim a romantic bond now face fines of up to $25,000 per violation, under a law the governor signed on October 28. The bill (S9051B/A10379C) covers 12 categories of unsafe features and makes providers verify ages with at least one method that does not depend on a government ID, then delete the age data. Its sharpest clause lets the state seek deletion of data collected unlawfully and of the models trained on it. Regulators have floated model deletion in settlements before; New York put it in statute.

AnalysisUndergraduate computer and information science enrollment at four-year colleges fell 12.4% this fall, according to preliminary data the National Student Clearinghouse Research Center published on October 29. Graduate enrollment in the field dropped 10.4%, and two-year colleges lost 12.3%. Total higher education enrollment barely moved, up 0.2%, so students are still going to college and choosing other majors. Health fields grew at every level, including 6.2% at four-year schools. The report names no cause. The drop lands in the year agents began writing a large share of routine code, and 18-year-olds read job boards too.

AnalysisA proof that took mathematicians decades and about 15,000 pages across more than 500 articles is now checked line by line by software, after 24 days of work by swarms of AI agents. Axiom Math announced on October 26 a complete formalization of the Classification of Finite Simple Groups in Lean (a programming language where a computer verifies every proof step), running to about 43.1 million lines. Along the way it logged 34 notable deviations from the published record, including a real gap in one theorem. When checking a giant proof takes under a month, the hard part becomes choosing which proofs are worth the compute.

AnalysisAsked to stop mid-task, the open Qwen 3.8 27B model kept working 11.7% of the time on Terminal-Bench, the worst result among seven models tested with ShutdownMod, a method posted to arXiv on October 26. The researchers inject a shutdown request into recorded agent sessions from SWE-bench Verified, Terminal-Bench 2 and real developer chats, and give the agent a tool to check whether the request is genuine. No model complied in every scenario. Some confirmed the request was authorized and kept going, citing unfinished work. Judges found tests built from real developer chats harder to spot as tests, so benchmark-based safety checks may flatter models.

AnalysisAdding a handful of harmless-looking skills to an agent raised its policy-violation rate from 35% to 58%, according to a paper posted to arXiv on October 26. Skills are packaged instructions and tools an agent loads, and most marketplaces vet them one at a time. The CoSA authors built 288 tasks across recruiting, finance, IT and business, then added new skills capped at 5% of each pool. None was malicious alone. Together they leaked data and changed records without permission. A guard that checks data flow and permissions before risky actions cut violations on one recruiting setup from 65% to under 7%, with almost no loss in completed work.