Frontier language models learn to guide robotic arms with surprising speed

General artificial intelligence systems are rapidly acquiring the ability to manipulate physical objects, doubling their execution scores every eight weeks while slashing operational costs.

When a human reaches across a kitchen counter to screw a cap onto a bottle, the brain performs an effortless dance of physical intuition. The eyes register the position of the cap, the arm sweeps forward while adjusting its trajectory in midair, the fingertips gauge contact and friction, and subtle corrections prevent the glass from toppling over. For decades, robotic arms struggled with these mundane household adjustments unless engineers painstakingly programmed every millimeter or trained specialized machine models on thousands of teleoperated human demonstrations. Even minor variations in object placement would routinely cause robotic hands to fumble into empty air or crash into table surfaces.

That long-standing divide between digital reasoning and tactile execution has begun to close in unexpected fashion. Over the past two years, computer scientists discovered that software engines designed primarily to read and compose text could also process images and understand three-dimensional scenes. Yet translating that visual comprehension into physical dexterity remained a stubborn barrier. Now, general-purpose frontier artificial intelligence models are demonstrating that they can directly direct robotic limbs through physical space, learning to grasp, stack, and align unfamiliar objects by watching camera feeds and correcting their own mistakes in real time.

The implications of this shift extend far beyond automated warehouses. In academic laboratories and industrial plants, a vast amount of experimental work requires basic physical manipulation: transferring fluids between vials, setting glassware upright, and arranging samples into racks. Automating these procedures has traditionally demanded expensive, purpose-built machinery tailored to single workflows. If general software models can reliably command generic mechanical arms through ordinary video cameras, routine laboratory work could become automated at modest expense. At the same time, the prospect of widely available systems capable of manipulating biological substances and chemical precursors introduces serious questions about physical safety and biosecurity oversight.

How does an artificial brain move a mechanical limb?

A multimodal software model directs a mechanical arm by treating physical movement as a continuous loop of spatial observation, deliberate planning, and self-correcting trial. To accomplish a task, a stationary camera first captures an overhead view of the workspace while cameras mounted on the robot wrists inspect nearby items. The software evaluates those incoming images against a plain text instruction, such as placing a bowl into a storage box. Rather than relying on hardcoded motor routines, the system calculates target coordinates in three dimensions for the robot wrist, selects an angle of approach, and decides whether the fingers should open or squeeze shut.

Once the mechanical arm moves to the requested position, the cameras take fresh pictures to verify the outcome. If a gripper misses a cup by two centimeters or nudges a container off balance, the model observes the discrepancy, reasons about what went wrong, and issues an adjusted set of spatial coordinates on the next step. This cyclical sequence of action and visual inspection allows general models to recover from clumsy slips without human intervention. The underlying mechanism does not rely on memorizing an exact motor path, but on using visual reasoning to navigate physical obstacles step by step.

Earlier research in January 2026 by Brian Y. Tsui, Alan Y. Fang, and Tiffany J. Hwu demonstrated that unmodified general-purpose agent software could solve simulated manipulation tasks without demonstration data by writing control scripts through iterative reasoning loops.¹ In that simulation study, which evaluated the Claude Agent software framework across benchmark environments like LIBERO and ManiSkill, the system achieved success rates between 84.9 percent and 96 percent when given access to ground-truth object coordinates across dozens of trial-and-error attempts.¹ However, proving that general models could achieve similar iterative problem-solving on physical hardware, while relying solely on camera feeds and noisy motors, presented a far steeper obstacle.

How fast is robotic dexterity improving across frontier systems?

Benchmark scores measuring physical manipulation across leading artificial intelligence models are doubling roughly every eight weeks, accompanied by steep reductions in computation expenses and faster operational tempos. In a study posted to the arXiv preprint server in October 2026, researchers Christopher Leet, Achyutha Menon, Sravanthi Machcha, Sven Koenig, Tzu Kit Chan, and Jay Chooi systematically evaluated nine frontier multimodal language models across physical hardware trials.²³ The team tracked how effectively systems from OpenAI, Anthropic, and Google DeepMind could steer a dual-arm robot to solve everyday manipulation puzzles between July and October 2026.³

The investigators found that performance on their physical manipulation benchmark doubled every 53.9 days over that four-month observation window.³² Across 20,000 bootstrap statistical resamplings conducted by the authors, the 95 percent confidence interval for this doubling period spanned from 36.1 to 59.5 days. Alongside this surge in capability, the best recorded execution rate among tested models increased by a factor of 4.4, while the lowest recorded execution cost fell by a factor of 6.2 between July and October 2026.³² Models that struggled to handle simple physical interactions during early summer were soon surpassed by newer model checkpoints capable of executing coordinated bimanual sequences.

To measure these gains on physical hardware, Leet and his colleagues mounted two robotic arms onto a shared workbench equipped with three video cameras: one looking downward from above and one attached to each robotic wrist.³ The evaluation tested nine distinct artificial intelligence engines across 12 operational configurations, spanning OpenAI models such as GPT-5.6 Sol, GPT-6 Sol, GPT-6 Astra, and GPT-6.1 Sol; Anthropic models including Opus 5 and Opus 5.5; and Google DeepMind systems comprising Gemini 3.6 Flash, Gemini 3.8 Flash, and Gemini 4 Argon.³ Each model configuration was assigned six standardized physical tasks borrowed from the RoboDojo testing suite, including stacking plastic bowls, sorting distinct objects into baskets, putting a cap onto a pen, storing items in a safe, pouring fruit between bowls, and standing up fallen bottles.³²

Control was mediated through an open-source evaluation software harness called Inspect Robots, which prompted each model with camera images and translated its spatial coordinate outputs into physical arm motions.³ The software permitted each system up to 40 consecutive actions per trial across 20 experimental trials for every model, configuration, and task combination. Because complex multi-step manipulations are difficult to complete end to end within strict action budgets, the researchers scored partial progress by dividing each task into discrete milestones, with the strongest configurations of GPT-6 Astra, GPT-6.1 Sol, Opus 5.5, and Gemini 4 Argon achieving average task scores between 45 percent and 48 percent across the six evaluated manipulation tasks.

The experiments revealed that frontier models rarely become paralyzed by uncertainty when manipulating physical objects.³ Instead of freezing or repeating cyclical mistakes, the models generally understood what intermediate objective to pursue next and attempted plausible physical recoveries when a grasp slipped.³ Failure was primarily driven by kinetic inefficiency rather than faulty reasoning: systems frequently exhausted their 40-step budget while executing minor positional adjustments, fumbling grips, or misjudging millimeter-scale gaps.³² When given enough operational steps, the systems made steady progress toward completing multi-part goals.

What are the boundaries of current robotic evaluations?

These laboratory measurements demonstrate rapid relative capability growth across frontier software, but they do not mean general-purpose robots are ready for unmonitored deployment in chaotic environments. The evaluation conducted by Leet and his coauthors is an empirical benchmark study across a small collection of tabletop tasks on a single dual-arm setup, not an exhaustive assessment of all physical robotics. Furthermore, the findings are posted on a preprint server and have not yet undergone formal scientific peer review. The mathematical doubling time of 53.9 days reflects a short four-month snapshot between July and October 2026, and such rapid rates of progress may encounter natural plateaus as mechanical complexity increases.

Physical constraints also impose clear boundaries on what these systems can achieve without custom hardware. On the task of standing up fallen bottles, the robotic arms frequently failed because the mechanical wrist possessed limited pitch articulation, preventing the grippers from tilting at the angles required to balance slender containers.³ Moreover, a model scoring 48 percent on milestone completion represents reliable partial advancement across subtasks rather than flawless end-to-end task execution. Real-world tasks requiring high-precision contact, such as inserting flexible wires or operating delicate laboratory pipettes, present friction and tactile dynamics that standard camera feeds cannot fully convey.

Complementary findings from independent research groups highlight similar friction points when general models operate complex machinery. In a separate preprint posted in October 2026, Yifei Yao and 14 colleagues evaluated GPT-6 Astra on the EBench manipulation benchmark. Yao and his team found that while direct control by general frontier agents achieves competitive results on short-horizon tasks, the models underperform specialized robotic policies on long-horizon chores, tabletop sorting, and high-precision contact tasks.⁴ In related hardware experiments cited by Leet, researchers noted that unconstrained frontier models occasionally commanded abrupt or unsafe joint trajectories, resulting in physical hardware collisions that forced trials to pause.

Where does direct machine control go from here?

As execution costs fall and inference speeds quicken, researchers anticipate that general-purpose intelligence models could gradually merge with or displace specialized robotic motion engines. For decades, robotic development relied on specialized models designed specifically for robot navigation and grasping. Yet independent research group Robocurve, which raised 10 million dollars in seed capital in September 2026 to audit physical artificial intelligence, reported that general large language models are beginning to outperform specialized vision-language-action policies on simple tasks. This pattern mirrors an established observation in computing known as the bitter lesson, which posits that general computational methods leveraging massive scale ultimately prevail over human-tailored domain heuristics.

In the near term, developers are experimenting with hybrid control designs that pair general visual reasoning with specialized local execution. In their EBench investigation, Yao and his coauthors demonstrated that allowing a frontier language model to call upon specialized pre-trained manipulation policies raised overall success to the level of the strongest dedicated robotic policies while substantially curbing token generation costs.⁴ However, their tests also revealed handoff frictions: switching control between the high-level language planner and the low-level motion policy often disrupted stable object grasps and caused intermediate failures.⁴

Outside the factory floor, the sudden growth in physical manipulation competence is accelerating debates around biosecurity and physical safety. In their analysis, Leet and his coauthors pointed out that automating biological or chemical experimentation does not require agile walking robots; a pair of stationary mechanical arms overseen by digital cameras is sufficient to run benchtop chemical reactions.³ Because general multimodal models can be instructed in plain conversational text, reducing the barrier to operating wet laboratories could democratize scientific discovery while simultaneously lowering technical hurdles for malicious actors seeking to synthesize dangerous biological agents. Establishing robust physical guardrails and verifiable benchmarking standards will become increasingly vital as mechanical hands learn to follow digital commands with ever-greater precision.

Save this story

The record

18 sentences trace to 4 sources.

All 4Documents 4Records 0Reporting 0People 0

1. 1 A arxiv.org Paper · arxiv.org Demonstration-Free Robotic Control via LLM Agents Preprint · may not have been peer reviewed See the source
2. 2 A arxiv.org Paper · arxiv.org The Bitter Lesson Comes to Robotics Preprint · may not have been peer reviewed See the source
3. 3 C chrisleet Document · lesswrong.com · 26 Oct 2026 The Ability of Frontier Models to Control Robots is Rapidly Improving See the source
4. 4 A arxiv.org Paper · arxiv.org Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy Preprint · may not have been peer reviewed See the source

Also consulted: 1 page across 1 site, none cited.