30997

UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization

Yaozhi Zheng, Yilei Jiang, Manyuan Zhang, Yuxuan Wan, Kaituo Feng, Tianshuo Peng, Bo Zhang, Xiangyu Yue
The Chinese University of Hong Kong
arXiv:2606.31732 [cs.CV], (30 Jun 2026)

@misc{zheng2026unicoder,

   title={UniCoder: Unified Visual-to-Code Generation via Symbolic Rewards and Reference-Guided Code Optimization},

   author={Yaozhi Zheng and Yilei Jiang and Manyuan Zhang and Yuxuan Wan and Kaituo Feng and Tianshuo Peng and Bo Zhang and Xiangyu Yue},

   year={2026},

   eprint={2606.31732},

   archivePrefix={arXiv},

   primaryClass={cs.CV},

   url={https://arxiv.org/abs/2606.31732}

}

Visual-to-Code generation, which transforms scientific plots, vector graphics, and webpages into executable scripts, demands a level of pixel-precise alignment that standard Multimodal Large Language Models (MLLMs) fail to achieve through Supervised Fine-Tuning (SFT) alone. While Reinforcement Learning (RL) offers a theoretical pathway to bridge this gap, its application is hindered by two fundamental obstacles: (1) Reward Coarseness, where semantic metrics like CLIP scores fail to penalize fine-grained element deviations, and (2) Exploration Stagnation, where the sparse, heterogeneous code search space prevents the policy from bootstrapping valid trajectories. To overcome these limitations, we introduce UniCoder, a unified RL framework that integrates two novel mechanisms. First, we propose Symbolic Attribute Alignment, which employs a lightweight auxiliary LLM to parse generated code into discrete visual attributes (e.g., hex colors, coordinate limits), enabling dense, element-wise reward computation. Second, to escape local optima, we devise Reference-Guided Code Optimization, a strategy that dynamically injects ground-truth trajectories into low-performing rollout groups, transforming blind exploration into guided policy improvement. Extensive experiments on ChartMimic, UniSVG, Design2Code and ScreenBench benchmarks demonstrate that our 8B-parameter model not only surpasses all open-source baselines but also achieves state-of-the-art performance comparable to proprietary models, establishing a new paradigm for generalized visual-to-code synthesis.
No votes yet.
Please wait...

You must be logged in to post a comment.

* * *

* * *

HGPU group © 2010-2026 hgpu.org

All rights belong to the respective authors

Contact us: