Frontier Vision-Language Models: Architectural Evolution, Benchmarks, Applications, and Challenges

🌐 Language: English · 简体中文

A most Frontend Collection and survey of vision-language model papers, and models GitHub repository

VLM design has gone through four distinct architectural eras in just six years — and Era 3 has split into two parallel branches. Early models kept frozen vision and language towers, aligned contrastively (CLIP) or bridged by a learnable connector into a frozen LM (BLIP-2, Flamingo). The 2023–2025 generation made a pretrained LLM the trunk and treated vision as a bolt-on adapter (LLaVA, Qwen2.5-VL, GPT-4V). The 2025–2026 generation drops the bridge entirely and early-fuses all modalities into a single transformer — forking along the output axis — and in 2026 the trunk is becoming a world model that predicts and acts:

• Era 3a — Native Multimodal Input → Text Out. Image, video, and (sometimes) audio enter a single early-fused token stream, but generation is still autoregressive text. This is the design used by today's general-purpose flagships: Qwen3.5 / Qwen3.6, Gemma 4, Gemini 3, GPT-5.4, Phi-4-Reasoning-Vision, Claude Opus 4.6, Nemotron 3 Nano Omni.
• Era 3b — Omni-Modal Unified I/O. The same fused trunk plus dedicated image / video decoder (VAE / DiT / flow-matching) and/or audio codec decoder heads, so the model can also generate images, video, and speech — via autoregression or, increasingly, discrete diffusion / AR-Diffusion (LLaDA2.0-Uni, Mamoda2.5). This is the design used by unified models: BAGEL, Qwen3.5-Omni, InternVL-U, Emu3 / Emu3.5, Erin 5.0, DeepSeek-Janus-Pro, LLaDA2.0-Uni, Mamoda2.5. Generation-only specialists share the same decoder stack without the understanding half — Sora 2, Veo 3, Kling now generate video with synchronized audio, and they double as the substrate for Era 4 world models (DreamX-World builds on Wan, OmniDreams on Cosmos).
• Era 4 — World-Action Models (2026 →). The unified trunk adds action as a first-class modality and closes the loop with the environment: it predicts future observations, maintains persistent state and spatial memory, and emits actions — generator, perceiver, and policy in one network: Cosmos 3, Kairos, DreamX-World 1.0, OmniDreams (see §1.1 World Models).

Reading the diagram (left → right). Era 1 uses a two-tower design — aligned contrastively (CLIP: no generative decoder) or with a learnable cross-modal bridge (e.g. Q-Former) into a frozen LM — text-only output. Era 2 puts a pretrained LLM at the center; an MLP/Resampler projects visual tokens into the LLM's vocabulary, and the LLM does all the reasoning — still text-only output. Era 3a drops the bridge: image, video, audio, and text share a single tokenizer/embedding space and flow through one early-fused transformer — but the output is still autoregressive text. Era 3b keeps that fused trunk and adds decoder heads (image/video DiT, VAE, audio codec) so the model can natively output text, image, video, and/or speech — generation-only video models (Sora 2, Veo 3, Kling) reuse this decoder stack, now with synchronized audio. Era 4 adds an action token stream, persistent state, and a policy head, closing the observation → action → next-observation loop: the world-action model is generator, perceiver, and policy at once. Eras 3a, 3b, and 4 coexist; the choice is essentially "how much do you want the model to generate — and to act?"

Below we compile awesome papers and model and github repositories that

• State-of-the-Art VLMs Collection of newest to oldest VLMs (we'll keep updating new models and benchmarks).
• Evaluate VLM benchmarks and corresponding link to the works
• Post-training/Alignment Newest related work for VLM alignment including RL, sft.
• Applications applications of VLMs in embodied AI, robotics, etc.
• Contribute surveys, perspectives, and datasets on the above topics.

VLM Trends is the live companion to this repository. This README records what exists; VLM Trends tracks what changed today — new model releases, papers, benchmarks, and datasets pulled daily from arXiv, Hugging Face, GitHub, and Semantic Scholar, scored on a published rubric, and grouped by topic and model family. It also makes this survey browsable, and charts how each research direction is moving over time.

We track new VLMs, benchmarks, and post-training methods that haven't yet been folded into the main tables in dated mini-surveys:

Welcome to contribute and discuss!

🤩 Papers marked with a ⭐️ are contributed by the maintainers of this repository. If you find them useful, we would greatly appreciate it if you could give the repository a star or cite our paper.

• 📄 Paper Link/⛑️ Citation

• 1. 📚 SoTA VLMs
  • 1.1. 🌍 World Models
• 1. 🗂️ Dataset and Evaluation
  • 2.1. Large Scale Pre-Training & Post-Training Dataset
  • 2.2. Datasets and Evaluation for VLM
  • 2.3. Benchmark Datasets, Simulators and Generative Models for Embodied VLM

• 1. 🔥 Post-Training/Alignment/prompt engineering 🔥

  • 3.1. RL Alignment for VLM
  • 3.2. Regular finetuning (SFT)
  • 3.3. VLM Alignment Github
  • 3.4. Prompt Engineering
• 1. ⚒️ Applications
  • 4.1. Embodied VLM agents
  • 4.2. Generative Visual Media Applications
  • 4.3. Robotics and Embodied AI
    • 4.3.1. Manipulation
    • 4.3.2. Navigation
    • 4.3.3. Human-robot Interaction
    • 4.3.4. Autonomous Driving
  • 4.4. Human-Centered AI
    • 4.4.1. Web Agent
    • 4.4.2. Accessibility
    • 4.4.3. Medical and Healthcare
    • 4.4.4. Social Goodness
• 1. ⛑️ Challenges
  • 5.1. Hallucination
  • 5.2. Safety
  • 5.3. Fairness
  • 5.4. Alignment
    • 5.4.1. Multi-modality Alignment
      • 5.4.2. Commonsense and Physics Alignment
  • 5.5. Efficient Training and Fine-Tuning
  • 5.6. Scarce of High-quality Dataset

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zongxia and Wu, Xiyang and Du, Hongyang and Liu, Fuxiao and Nghiem, Huy and Shi, Guangyao},
    title     = {A Survey of State of the Art Large Vision Language Models: Benchmark Evaluations and Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {1587-1606}
}

World models moved from research demos to release-grade artifacts in mid-2026: unified trunks that serve as generator, perceiver, and policy at once. Foundation releases first, then memory/state analyses and benchmarks.

Benchmarks & analyses

See also the earlier world-model entries in §2.3 (UniSim, GAIA-1, LWM, Genesis, RoboGen) and the 2026-06-23 / 2026-06-27 progressive reports.