Robotic surgical models gain precision through specialized operating video

Training predictive vision models on thousands of hours of console footage establishes stronger visual foundations for robotic surgical assistance, according to a multi-task benchmarking study.

During a robot-assisted operation, a surgeon peers into a three-dimensional console to manipulate articulating instruments deep inside a patient. The camera captures an environment where tiny, critical events occur within fractions of a second, while the broader procedure unfolds across dozens of minutes. An automated system trying to assist in this setting must recognize both levels of action at once. It must distinguish delicate interactions like clipping an artery or retracting tissue from broader surgical phases, such as dissecting an organ from the liver bed.

Developing artificial intelligence for this operating environment has proven difficult because the visual demands of robotic surgery differ substantially from standard video analysis. General computer vision models are built to track people walking across rooms or vehicles moving along roads. Even medical vision systems have largely been trained on traditional laparoscopic procedures, where rigid instruments enter through straight ports with visual dynamics that do not match the multiaxis articulation and optical depth of modern surgical consoles. When standard vision models attempt to track complex robotic maneuvers, they frequently misidentify instruments or lose track of which procedural step is underway.

To build a system capable of understanding robotic surgery, researchers must follow a distinct causal chain. First, a video model must ingest thousands of hours of unlabeled surgical video to learn how surgical tools interact with living anatomy. Second, the model must learn to predict unseen visual patches in representational space, allowing it to build an internal map of anatomical motion and tool articulation. Third, this broad visual representation must be adapted to recognize specific surgical actions, such as identifying the active instrument, the verb describing its action, and the anatomical target being manipulated. Finally, the adapted system must be evaluated across both brief instrument motions and multi-minute procedural phases to verify that fine-grained spatial acuity and broad temporal context remain intact.

A research team led by Leonardo Borgioli and Milos Zefran at the University of Illinois Chicago, alongside Neil Getty and colleagues at Argonne National Laboratory and the Affiliated Hospital of Qingdao University, developed a model family called OmniRAS to address these challenges.¹² The researchers introduced specialized encoders designed to standardize foundation model training and evaluation directly in robot-assisted surgery.¹² Their findings, described in a preprint posted to arXiv, demonstrate that pretraining predictive visual models on thousands of hours of robotic footage produces substantial gains across a wide range of surgical understanding tasks.¹³

Why does robotic surgery present distinct visual challenges for artificial intelligence?

Robotic surgery presents distinct visual challenges because clinically meaningful actions occur simultaneously at widely differing spatial scales and temporal durations. A brief maneuver, such as clipping a cystic artery, lasts only a few seconds and requires precise spatial tracking of millimeter-wide instrument tips. Meanwhile, the surrounding phase of the operation, such as exposing the surgical field or dividing anatomical structures, extends across many minutes and demands sustained temporal memory. In an evaluation conducted by the research team on robotic cholecystectomy video, a general frontier multimodal model, Claude Opus 5, achieved a mean average precision of only 0.139 when attempting to predict tool, verb, and target triplets, whereas a domain-adapted surgical representation reached 0.426.¹²

Evaluating general commercial models on this task exposes practical boundaries. Borgioli said in response to questions from Primary that the comparison was designed primarily as an exploratory baseline, noting that obtaining the results cost approximately $350 in token usage and that such frontier architectures would not be a good fit for robotics operating at frequencies of at least once per second.^contributed Borgioli explained that the goal was to illustrate that even a capable general system struggles with fine-grained surgical understanding when prompted across 1,696 validation clips and 278 supported triplets.^contributed

Most prior computational benchmarks for surgery have focused on conventional laparoscopy rather than robotic systems. This historical imbalance means that models designed for robotic consoles have frequently been tested off-domain on laparoscopic footage, obscuring how well they actually interpret robotic procedures. Furthermore, previous studies often altered the downstream evaluation architecture alongside the underlying visual backbone, making it impossible to determine whether performance improvements arose from the learned visual representation or from the specialized readout heads attached to them.

To establish a consistent benchmark, the researchers released two densely annotated datasets covering robotic cholecystectomy, which is the surgical removal of the gallbladder. The first dataset, OmniRAS-PR, contains 51 complete procedures and approximately 11.1 hours of video annotated under an 11-class procedural phase ontology.²¹ This collection combines 41 private surgical recordings totaling approximately 7.3 hours with 10 publicly available procedures from SurgeNet totaling approximately 3.8 hours, while holding out two full public procedures to prevent clip-level data leakage.²

The second dataset, known as YT-Chole Triplets, re-annotates ten batches of surgical video totaling approximately 6.4 hours into multi-label triplet combinations of five tool categories, six action verbs, and 12 anatomical targets.¹ This format represents the first multi-label triplet annotation established specifically for robotic cholecystectomy video.¹² By evaluating tools, actions, and anatomical structures simultaneously, the benchmark tests whether a vision model can recognize multiple concurrent actions in a single frame.

How can predictive video models learn the flow of an operation?

Predictive video models learn the flow of an operation by reconstructing masked representations of future or hidden video frames without relying on manual labels. Rather than generating every raw pixel, which consumes vast computing resources on irrelevant background texture, joint-embedding predictive architectures project video clips into an abstract feature space and learn to predict the latent representations of missing spatio-temporal blocks. The researchers adapted this approach by continuing the pretraining of 1-billion-parameter and 2-billion-parameter V-JEPA-2.1 vision encoders on a specialized surgical video catalog.¹²

The pretraining catalog combined 19 sources totaling approximately 2,650 hours of surgical footage.¹² Robotic procedures performed on da Vinci surgical systems accounted for 51 percent of the catalog, amounting to 1,348 hours.¹² Mixed surgical video made up 34 percent, or 902 hours, while conventional laparoscopic video comprised the remaining 15 percent, or 388 hours.¹⁵ Pretraining was conducted at scale across up to 256 compute nodes with a global batch size of 6,144 samples, spanning three primary production runs of 9.22 million, 18.43 million, and 36.86 million video samples that required a cumulative 3,038 node-hours of high-performance computing.

To verify that the annotations themselves were reliable before measuring model performance, the authors conducted an inter-rater reliability study comparing independent surgeon annotations on a 10 percent sample of YT-Chole video spanning 39.5 minutes.² When evaluating the 11-class phase ontology with zero temporal boundary tolerance, the mean pairwise Cohen's kappa reached 0.664.¹ When allowing a temporal boundary tolerance of plus or minus four seconds to account for slight differences in transition timing between raters, the pairwise agreement rose to 0.807.¹ The researchers found that 82.1 percent of residual disagreements occurred within the five-phase Calot's triangle cluster, where sub-activities often occur concurrently rather than in strict sequential order.¹²

Borgioli told Primary that this concentration showed that the limitation was partly in the ontology, where concurrent actions create soft boundaries and temporal gating differences.^contributed A parallel reliability assessment of the action triplet ontology revealed that two-rater Krippendorff's alpha reached 0.882 for instruments, 0.653 for verbs, 0.624 for anatomical targets, and 0.655 for complete triplets under a half-second temporal tolerance.¹ These consistency checks confirmed that the underlying label vocabulary captures genuine surgical actions that clinical experts independently agree upon.

What did the multi-task evaluation reveal across surgical benchmarks?

The multi-task evaluation revealed that domain-adapted OmniRAS encoders consistently outperformed both raw generic visual models and existing specialized surgical backbones across diverse spatial and temporal benchmarks. Downstream testing encompassed 254 distinct evaluation runs across three random seeds, including 109 runs featuring partial backbone fine-tuning of the final four transformer blocks.¹² The tasks tested fine-grained action recognition, procedural step segmentation, phase classification, and localized bounding-box instrument detection.

Borgioli noted that fine-tuning four blocks was chosen as a controlled adaptation regime to provide enough flexibility while keeping most of the backbone fixed.^contributed He explained that deeper tuning can improve specific tasks, such as eight-block adaptation on YT-Chole, but substantially increases computing costs across multi-seed evaluations.

On the frame-wise action recognition benchmark SAR-RARP50, the fully fine-tuned OmniRAS model achieved an F1 score of 91.6 percent at a 10 percent overlap threshold, exceeding the 75.0 percent achieved by LemonFM, the 69.8 percent achieved by EndoViT, and the 68.1 percent recorded by SurgeNet-XL.¹² On the GraSP benchmark for surgical phase recognition, OmniRAS reached a mean average precision of 85.3 percent under full fine-tuning, compared with 76.7 percent for TAPIS, 75.4 percent for LemonFM, and 67.6 percent for SurgeNet-XL.¹² For GraSP surgical step recognition, fine-tuning the last four blocks of OmniRAS yielded 57.9 percent mean average precision, outperforming TAPIS at 52.0 percent, LemonFM at 48.6 percent, and SurgeNet-XL at 45.4 percent.¹²

Spatial localization tests showed similar improvements. On the SARAS-ESAD action detection benchmark, fine-tuning the final four blocks of OmniRAS alongside data augmentation produced a detection mean average precision of 23.2 percent, topping the previous challenge-best score of 19.3 percent and surpassing LemonFM at 15.9 percent.¹² On the newly released OmniRAS-PR phase recognition test, OmniRAS achieved a frame-level F1 score of 47.9 percent, compared with 36.6 percent for SurgeNet-XL and 25.6 percent for EndoViT.¹ On the YT-Chole Triplets task, OmniRAS scored 39.9 percent mean average precision under an eight-block fine-tuning regime, compared with 26.0 percent for SurgeNet-XL and 9.1 percent for EndoViT.¹

The authors also conducted a label-free test of predictive fidelity by evaluating how accurately the models predicted masked visual features in latent space without any downstream task head. OmniRAS predicted masked targets in its own representation space more accurately than the unadapted baseline model across 60 out of 60 paired unit-level comparisons, indicating that the continued pretraining process improved the model's fundamental ability to model surgical dynamics.¹² Borgioli emphasized that because each checkpoint possesses its own latent space, this measurement reflects stronger predictive fidelity rather than proof of a universally superior visual representation.^contributed

What are the main limitations of current surgical foundation models?

The reported experimental results represent benchmark evaluation scores across curated datasets rather than real-time clinical validations in active operating rooms. Although the pretraining corpus spans approximately 2,650 hours, the data collection remains concentrated primarily on cholecystectomy and prostatectomy procedures, meaning the representations may not generalize equally to cardiac, thoracic, or neurological robotic operations.² Furthermore, the pretraining campaign varied both training sample budgets and data catalog mixtures concurrently across compute configurations, meaning that compute scale and dataset composition effects cannot be fully disentangled into strict scaling laws.¹² The findings are documented in a research preprint posted to the arXiv server and have not yet undergone formal journal peer review.¹

Addressing the pretraining data mixture, Borgioli said that the pretraining catalog spans more than 30 distinct procedures, providing broad procedural diversity even though downstream evaluations focused on cholecystectomy and prostatectomy.^contributed He also clarified that pretraining alone does not drive the gains in isolation, stating that “the dominant gain comes from downstream adaptation” while continued pretraining provides a more effective predictive initialization.^contributed If unadapted baseline encoders consistently matched OmniRAS across tasks under identical adaptation budgets, Borgioli noted that domain-specific pretraining would offer little transferable value.^contributed

For clinical engineering teams and robotic platform developers, the findings offer a practical pathway toward standardized vision systems. Demonstrating that pretraining on domain-specific robotic video produces measurable gains across both spatial detection and temporal workflow tasks helps establish clear baselines for future surgical assistive tools. As robotic platforms capture growing volumes of procedure footage, foundation models trained on structured surgical video may eventually assist operating room teams with workflow tracking, post-operative video indexing, and safety monitoring.

Future work will center on expanding the pretraining corpus across a wider variety of surgical specialties and testing the representation on real-time video streams during live robotic procedures. Researchers also plan to investigate whether incorporating kinematic data from surgical robot arms alongside console video can further improve action segmentation and tool pose estimation.

This piece was prepared from the arXiv preprint and public records together with answers from Leonardo Borgioli to six questions from the Primary news team, completed September 2026.

References

This article is based on 10 sources, with 6 statements from 1 contributor, listed in the order they are cited.

1. 1 LB Leonardo Borgioli, Neil Getty, Wenli Xiu, Jessica Cassiani, Alvaro Ducas, Carlos Agustin Orda, Hira Waris, Fangfang Xia, Rick Stevens, Pier Cristoforo Giulianotti, Milos Zefran announcement · 1 Sep 2026 OmniRAS: Standardizing Foundation Model Training and Evaluation in Robot-Assisted Surgery See the source
2. 2 A arxiv.org OmniRAS: Standardizing Foundation Model Training and Evaluation in Robot-Assisted Surgery See the source
3. 3 A arxiv.org Image and Video Processing See the source
4. 4 LB Leonardo Borgioli, lead author added 10 Sep 2026 Contributor commentary — Leonardo Borgioli 6 statements added to this article
5. 5 P pol.polaimath.com OmniRAS: Standardizing Foundation Model Training and Evaluation in Robot-Assisted Surgery See the source
6. 6 A arxiv.org V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning See the source
7. 7 A arxiv.org Image and Video Processing See the source
8. 8 G github.com GitHub - keon/jepa: implementing minimal versions of joint-embedding predictive architecture (JEPA) See the source
9. 9 G github.com GitHub - SITL-Eng/CRCD See the source
10. 10 P physionet.org Table1.csv - Electroencephalogram and eye-gaze datasets for robot-assisted surgery performance evaluation Download the document
11. 11 P poissonlabs.ai Where V-JEPA 2.1 See the source

Article history

1. 6 statements 10 Sep 2026, 14:21
   What was added

   Borgioli noted that fine-tuning four blocks was chosen as a controlled adaptation regime to provide enough flexibility while keeping most of the backbone fixed.

   On the record as reference 4 go to the sentence ↑
   What was added

   Addressing the pretraining data mixture, Borgioli said that the pretraining catalog spans more than 30 distinct procedures, providing broad procedural diversity even though downstream evaluations focused on cholecystectomy and prostatectomy.

   On the record as reference 4 go to the sentence ↑
   What was added

   Borgioli emphasized that because each checkpoint possesses its own latent space, this measurement reflects stronger predictive fidelity rather than proof of a universally superior visual representation.

   On the record as reference 4 go to the sentence ↑
   What was added

   Borgioli told Primary that this concentration showed that the limitation was partly in the ontology, where concurrent actions create soft boundaries and temporal gating differences.

   On the record as reference 4 go to the sentence ↑
   What was added

   Borgioli said in response to questions from Primary that the comparison was designed primarily as an exploratory baseline, noting that obtaining the results cost approximately $350 in token usage and that such frontier architectures would not be a good fit for robotics operating at frequencies of at least once per second.

   On the record as reference 4 go to the sentence ↑
   What was added

   He also clarified that pretraining alone does not drive the gains in isolation, stating that “the dominant gain comes from downstream adaptation” while continued pretraining provides a more effective predictive initialization.

   On the record as reference 4 go to the sentence ↑
   LB Leonardo Borgioli, lead author · Contributor
2. Published 11 Sep 2026, 14:48
   Assembled by the Primary desk from 10 sources · 36 cited sentences