VGM-VS

Rethinking Visual Geometry Model for High-Precision Visual Servoing

Yimin Pan1, Sen Wang1,2,*, You Zhou1, Jianfeng Gao1, Pengbo Sun1,

Ahmed M. Naguib1, Zoltan-Csaba Marton1
1Agile Robots SE, 2Technical University of Munich
*Corresponding author
Paper arXiv
Overview of VGM-VS on the RAM insertion task

TL;DR. VGM-VS uses a pretrained visual geometry model to predict the relative pose between the current camera view and a reference image, and applies it iteratively as the pose increment of a closed-loop visual servoing controller. A short, autonomously recorded dataset makes the predictions metric, enabling submillimeter alignment on real-world assembly tasks.

Abstract

We present VGM-VS, a visual servoing method built on a pretrained feed-forward visual geometry model. Given the current view and a reference image captured at the target configuration, we estimate the relative camera pose with a visual geometry model and apply it iteratively as the pose increment of a closed-loop pose-based visual servoing (PBVS) scheme. The geometry-aware representation acquired from large-scale pretraining keeps this estimate reliable when the target is occluded, weakly textured, or covers only a small part of the image. However, the scale ambiguity inherent to these models leaves the predicted translation defined up to an unknown scale, while the pose increment must be metric for robot control. We close this gap with a scene-specific metric adaptation: the robot autonomously records image–pose pairs along a predefined motion starting from the target pose, and we fine-tune the camera head on these data, jointly learning the hand–eye transform and thus removing the need for a dedicated calibration process. We evaluate our method on three real-world assembly tasks with demanding tolerances: USB-C cable picking, cable insertion, and RAM insertion. Running in real time at 30Hz, VGM-VS converges to submillimeter terminal accuracy on the cable tasks, and reaches success rates of 90–100% when the target is moved during servoing. It converges in all trials under initial displacements of up to 30cm from the reference pose and with 50% of the target object occluded, outperforming the compared visual servoing baselines.

Method

Overview of the VGM-VS pipeline

(a) Data collection: the end-effector follows a conical spiral with its apex at the target pose, yielding synchronized image–pose pairs for the target workspace. (b) Fine-tuning: the VGGT-Ω aggregator is kept frozen, while the camera head and the hand–eye transform are trainable and optimized against the ground-truth relative pose. (c) Pose-based visual servoing loop: the predicted increment is composed with the current robot pose and sent to a Cartesian motion controller. When its translational and rotational magnitudes fall below the thresholds τt and τR, the predefined task motion is executed; otherwise, the loop continues with the new observation.

Demonstrations

Real-robot runs of VGM-VS on the three assembly tasks under random initial poses, target occlusion, and a moving target. In RAM insertion, the module is first grasped from a fixture with a predefined motion. In all methods, the final insertion or picking is a simple linear motion defined in the TCP frame, executed once the alignment process has converged.

RAM Insertion

USB-C Cable Picking

USB-C Cable Insertion

Quantitative Results

Task Eval. type Setting Method Conv. (%) ↑ et (mm) ↓ eR (°) ↓ Iter. ↓ Success (%) ↑
RAM insertion Init. pose pert. Small ORB-IBVS1003.450.8777.34–
ART-VS4632.976.47140.11–
VGM-VS w/o FT1006.61.0423–
VGM-VS1000.950.4526.64–
Large ORB-IBVS663.690.78926.33–
ART-VS2765.3415.11110.35–
VGM-VS w/o FT966.951.1129.75–
VGM-VS1001.150.5526.79–
Scene dist. 25% occ. ORB-IBVS405.431.151101–
ART-VS6530.126.2079–
VGM-VS w/o FT1005.551.2521–
VGM-VS1003.370.8717.65–
50% occ. ORB-IBVS157.321.661297–
ART-VS5529.825.8441.18–
VGM-VS w/o FT8514.82.8543.29–
VGM-VS1004.211.0719.4–
Dynamic ORB-IBVS55––195130
ART-VS20––39.510
VGM-VS w/o FT90––2760
VGM-VS100––20.295
Cable picking Init. pose pert. Large VGM-VS w/o FT1004.610.3236–
VGM-VS1000.580.2026.56–
Scene dist. Dynamic VGM-VS w/o FT90––17.3810
VGM-VS100––12.29100
Cable insertion Init. pose pert. Large VGM-VS w/o FT1001.980.3326.62–
VGM-VS1000.670.0833.06–
Scene dist. Dynamic VGM-VS w/o FT95––11.60
VGM-VS100––1590

Quantitative comparison on three real-world tasks. ↑/↓ indicate that higher/lower is better; the best result within each setting is shown in bold, and a dash denotes a metric that is not measured.

Servoing Behaviour on RAM Insertion

Predicted translation increment

(a) Predicted translation increment Δt (mm)

Predicted rotation increment

(b) Predicted rotation increment Δr (deg)

Translation error

(c) Translation error et (mm)

Camera linear velocity

(d) Camera linear velocity v (m/s)

Camera angular velocity

(e) Camera angular velocity ω (rad/s)

Rotation error

(f) Rotation error eR (deg)

RAM insertion under large initial pose perturbation. The x axis represents the number of inference steps (%). VGM-VS variants are shown in (a), (b); the keypoint-based baselines in (d), (e); all four methods in (c), (f). The increments of the fine-tuned model decay monotonically to zero, whereas VGM-VS w/o FT overshoots and changes sign before settling, and the velocities of the keypoint-based baselines keep fluctuating along the trajectory.

BibTeX

@misc{pan2026vgmvs,
  title={VGM-VS: Rethinking Visual Geometry Model for High-Precision Visual Servoing},
  author={Yimin Pan and Sen Wang and You Zhou and Jianfeng Gao and Pengbo Sun and Ahmed M. Naguib and Zoltan-Csaba Marton},
  year={2026},
  eprint={2609.28312},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2609.28312},
}