Paper

The working title is Grounding Executable Visual Semantics in Unified Multimodal Models.

Idea

Models that emit graphics code (TikZ, SVG, matplotlib, manim, slide DSLs) treat the program purely as text. The model never observes what its code renders to, so any signal about visual fidelity has to come through a text-only proxy.

Unified models remove this restriction. In Chameleon, Transfusion, and BAGEL, image and text tokens live in one transformer with shared global attention, so an interleaved <code>…</code><image>…</image> sequence is a supported training example. Also, the renderer is deterministic, so the supervision is exact and can be generated at scale.

Setup

  • Backbone. BAGEL 7B active / 14B total (MoT), continued from the released checkpoint. Hybrid loss: cross entropy on text, flow matching on images.
  • Data. TikZ is the primary domain because it renders deterministically and existing corpora (DaTikZ v2, arXiv sources) are large. SVG, matplotlib, manim, and python-pptx can be used to test cross-DSL transfer. Each program is rendered in a sandbox, deduplicated by AST hash and CLIP neighbors, and decontaminated against public evals.
  • Sequence mix. About 70% prompt-conditioned full programs (<prompt><code><image>), 20% render traces where the program is chunked and the partial render is interleaved after each chunk, and 10% prompt-free <code><image> / <image><code> pairs.
  • Baseline. The same checkpoint continued on the same sequences with the flow loss on noised image tokens zeroed. The model still sees the images as context but is never supervised to produce them. This separates “image-side supervision helps” from “more code helps”.
  • After mid-training. A short SFT pass, then GRPO with a renderer-as-verifier reward: a compile check, CLIP image–image similarity, and a term for the failures CLIP might underweigh (text fidelity, fine geometry).

Hypotheses

  1. Grounding. (code, image) mid-training improves code-to-image generation over the text-only baseline.
  2. Understanding. It also improves predicting what a program renders to.
  3. Transfer. Mid-training on TikZ alone transfers to held-out SVG and matplotlib.
  4. RL initialization. It makes the renderer-verified RL stage more sample efficient.
  5. Context unrolling. The model can condition on its own intermediate renders to write better final code than single-shot generation.