Discrete Visual Tokens Just Matched Continuous Ones on Multimodal Understanding — Rewriting a Core Assumption of Native Multimodal Models
A new arXiv paper shows quantization error is an unreliable proxy for semantic loss: with Factorized Vector Quantization, discrete visual token models become competitive with Qwen3-VL on OCR and understanding benchmarks.
A widely held assumption in multimodal AI is that converting images into discrete tokens — like words — inevitably loses information, so continuous visual features set an upper bound that discrete-representation models can only approach. A paper posted to arXiv on October 27 argues that this assumption was never properly tested, and that when you fix the right variable, discrete representations can match continuous ones outright.
The paper, “Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?” by Jialv Zou, Xiaotong Li, Zhengrong Yue and colleagues at Huazhong University of Science and Technology, Meituan LongCat, Shanghai Jiao Tong University and other institutions, introduces Factorized Vector Quantization (FVQ). Its central claim: the thing that determines whether discrete visual tokens preserve meaning is not reconstruction error but the effective bit budget of the discrete indices — and existing quantizers simply stop delivering useful information long before they should.
The experiment that breaks the assumption
The authors attack the conventional reasoning directly. Vision-language models fed discrete tokens never see reconstructed pixels — they see token indices, whose embeddings are learned from scratch. So the mean-squared error of reconstructing the original image feature should not tell you how much semantic information the indices carry. To prove it, they compared a single-codebook vector quantizer (VQ) against half of an eight-group residual VQ: RVQ-Odd keeps only codebook groups 1, 3, 5 and 7.
RVQ-Odd has 9.1× higher reconstruction error than VQ — yet scores 9.5 points higher on multimodal understanding (measured as an eight-benchmark average with OCRBench scaled down by 10, under a shared Qwen3-1.7B setting). RVQ-Even has reconstruction error close to VQ and also understands substantially better. In other words, more “loss” by the metric everyone uses corresponded to better downstream reasoning. Reconstruction error, the authors conclude, is not a reliable measure of semantic fidelity.
The right variable: bit budget, and why codebooks scale better than codebook size
In any digital system, even continuous features are finite-bit encodings. For discrete indices the per-patch bit budget is B = L · log₂C, where L is the number of codebooks and C the size of each codebook. The budget grows linearly with L but only logarithmically with C — so adding codebooks is a much more efficient scaling strategy than enlarging each codebook.
The catch is that existing multi-codebook quantizers saturate at high budgets. Later residual codebooks in residual VQ receive progressively smaller residuals; product quantization splits features into increasingly narrow channel groups. Extra nominal bits stop carrying new semantic information. The authors attribute this high-budget saturation to diminishing information per codebook, and build FVQ to fix exactly that.
How FVQ works
Factorized Vector Quantization follows three design principles: symmetric roles across codebooks, fixed-dimensional views derived from the full feature, and collective feature preservation before quantization. During tokenizer training, multiple dense projections produce complementary views of the same visual feature, each quantized in parallel by an independent codebook. A tight-frame synthesis transform combines the selected codewords for reconstruction, and the transform itself introduces no reconstruction error — all distortion comes from replacing projected views with nearest codewords. Crucially, adding a codebook adds a new fixed-dimensional view rather than shrinking later residuals or channel groups, so each additional codebook stays semantically informative.
The pipeline is clean: the tokenizer trains only to reconstruct frozen features from a pretrained vision model with no pixel-level supervision. In the subsequent VLM stage, the quantizer is frozen and only the discrete indices are mapped to newly initialized token embeddings. The released model pairs Qwen3-8B with an any-resolution tokenizer built from 16-codebook FVQ applied to the Qwen3 ViT, using standard 1D RoPE and omitting DeepStack connections to preserve a native next-token-prediction setup.
Results: closing the gap to Qwen3-VL
Trained on industrial-scale multimodal data (roughly 100B tokenizer tokens, then 10B alignment, 60B pre-training and 30B SFT tokens per the paper’s recipe), the 8B FVQ model is state of the art among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2% respectively on a nine-metric average, according to the arXiv abstract and the project’s GitHub repository.
Selected numbers from the repository’s results tables:
| Model | MMMU | MathVista | OCRBench | DocVQA |
|---|---|---|---|---|
| Qwen3-VL-8B (continuous) | 69.6 | 77.2 | 896 | 96.1 |
| Emu3-Chat-8B | 31.6 | 47.6* | 687 | 76.3 |
| UniAR-8B | 44.3 | 50.8* | 833 | 91.4 |
| FVQ-8B (discrete) | 64.3 | 77.4 | 867 | 95.4 |
* Re-evaluated by the authors under their own evaluation setting for fair comparison.
The authors describe this as the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL — including on fine-grained OCR, where FVQ-8B scores 867 on OCRBench and 58.7/59.5 (en/zh) on OCRBenchV2 against Qwen3-VL-8B’s 896 and 65.4/61.2. The gap has narrowed dramatically but not fully closed on every metric, which is worth keeping in mind when reading the “competitive” claim.
Why this matters for agent builders
If discrete visual tokens under a shared next-token-prediction objective can carry essentially the same semantics as continuous features, then language-native multimodal foundation models — one tokenizer, one objective, one autoregressive stack for text and vision — become a viable architecture rather than a compromise. That simplifies inference, unifies training recipes, and removes a structural argument that pushed teams toward hybrid continuous-discrete designs. The Hugging Face Daily Papers listing, where the paper was submitted on October 28, already shows strong community uptake with 41 upvotes.
Code and model weights are open-sourced under Apache 2.0: a standalone FVQ-G16 tokenizer and the FVQ-8B VLM are available on Hugging Face, and the GitHub repository includes training launchers for tokenizer, alignment, pre-training and SFT stages, plus quick-start inference.
A few caveats are in order. This is an arXiv preprint that has not yet been peer reviewed. The headline training recipe is described at a level (“industrial-scale multimodal data”) that will be hard to reproduce exactly, and the authors themselves note the released stage YAMLs are configurable examples rather than a replication of the full run. The comparison to Qwen3-VL uses the authors’ own eval harness for starred baseline numbers.
Even with those caveats, the core finding — that an entire research subfield was optimizing the wrong proxy — is the kind of result that reshapes how teams reason about vision tokenizers, and the released weights give practitioners a way to test the claim directly. For more on how model architecture choices ripple through agent systems, see our coverage of MIT’s multimodal model for materials science.
