🌐 Language: English · 简体中文

A most Frontend Collection and survey of vision-language model papers, and models GitHub repository

VLM design has gone through three distinct architectural eras in just five years — and Era 3 has split into two parallel branches. Early models bridged a frozen vision encoder to a frozen language model with a learnable connector (CLIP, BLIP, Flamingo). The 2023–2025 generation made a pretrained LLM the trunk and treated vision as a bolt-on adapter (LLaVA, Qwen2.5-VL, GPT-4V). The latest 2025–2026 generation drops the bridge entirely and trains a single transformer from scratch on mixed-modality data — but it forks along the output axis:

• Era 3a — Native Multimodal Input → Text Out. Image, video, and (sometimes) audio enter a single early-fused token stream, but generation is still autoregressive text. This is the design used by today's general-purpose flagships: Qwen3.5 / Qwen3.6, Gemma 4, Gemini 3, GPT-5.4, Phi-4-Reasoning-Vision, Claude Opus 4.6.
• Era 3b — Omni-Modal Unified I/O. The same fused trunk plus dedicated image decoder (VAE / MMDiT / flow-matching) and/or audio codec decoder heads, so the model can also generate images and speech. This is the design used by unified models: BAGEL, Qwen3.5-Omni, InternVL-U, Emu3 / Emu3.5, Erin 5.0, DeepSeek-Janus-Pro.

Reading the diagram (left → right). Era 1 uses a two-tower design with a learnable cross-modal bridge (e.g. Q-Former) into a frozen LM — text-only output. Era 2 puts a pretrained LLM at the center; an MLP/Resampler projects visual tokens into the LLM's vocabulary, and the LLM does all the reasoning — still text-only output. Era 3a drops the bridge: image, video, audio, and text share a single tokenizer/embedding space and flow through one transformer trained from scratch — but the output is still autoregressive text. Era 3b keeps that fused trunk and adds decoder heads (image VAE/MMDiT, audio codec) so the model can natively output text, image, and/or speech. Era 3a and Era 3b coexist; the choice is essentially "how much do you want non-text generation?"

Below we compile awesome papers and model and github repositories that

• State-of-the-Art VLMs Collection of newest to oldest VLMs (we'll keep updating new models and benchmarks).
• Evaluate VLM benchmarks and corresponding link to the works
• Post-training/Alignment Newest related work for VLM alignment including RL, sft.
• Applications applications of VLMs in embodied AI, robotics, etc.
• Contribute surveys, perspectives, and datasets on the above topics.

We track new VLMs, benchmarks, and post-training methods that haven't yet been folded into the main tables in dated mini-surveys:

• 📰 2026-06-02 — latest: Mamoda2.5 (AR-Diffusion DiT-MoE, 95.9× faster editing), VLM3 (native 3D learners), AlphaGRPO (RL for unified-model generation), Stage-wise Preference Optimization, FastOCR / WindowQuant (KV-cache efficiency), Fast-dDrive / CLOVER / CoWorld-VLA (driving VLA), Lost in Fog (reasoning-as-safety-signal), LiteGUI (SFT-free GUI agents), Health-Conditioned VLA, POLAR, TOC-Bench / VGenST-Bench (video), HalluCXR (medical) — 16 new entries since May 16.
• 📰 2026-05-16 — LensVLM (Apple), Nemotron 3 Nano Omni (NVIDIA), LLaDA2.0-Uni, PLaMo 2.1-VL, S-GRPO / Faithful GRPO / GRPO-TTA / OpenSearch-VL, MindVLA-U1 (surpasses human driving), VLADriver-RAG, Green-VLA, Anticipation-VLA, VLA Foundry, LAMO, ScreenExplorer, VideoZeroBench, Video-Oasis, MedThinkVQA, data curation at 87× less compute — 34 new entries since April 28.
• 📰 2026-04-28 — Qwen3.6-27B & Qwen3.6-35B-A3B, Claude Mythos (gated), S1-VL, GLM-5V-Turbo, FreshPER / GMPO / ARPO / GRPO-VPS, QUOTA, Fast-dVLM, VLA-World, SpanVLA, VLA-Forget, R-VLM, UILoop, WebForge, WorldMark, Video-MME-v2, CrossMath, BabyVision, SlowBA — 30 new entries since April 13.
• 📰 2026-04-13 — LFM2.5-VL-450M, EXAONE 4.5, Gemma 4, Granite 4.0 3B Vision, InternVL-U, GLM-4.6V, Vero, MolmoWeb, UniDriveVLA, QAPruner, Firebolt-VL, CoME-VL, and more.
• 📰 2026-03-25 — GPT-5.4, Phi-4-Reasoning-Vision-15B, Gemini 3.0, Qwen3.5, Claude Opus 4.6, Molmo2, and more.

Welcome to contribute and discuss!

🤩 Papers marked with a ⭐️ are contributed by the maintainers of this repository. If you find them useful, we would greatly appreciate it if you could give the repository a star or cite our paper.

• 📄 Paper Link/⛑️ Citation

• 1. 📚 SoTA VLMs
• 1. 🗂️ Dataset and Evaluation
  • 2.1. Large Scale Pre-Training & Post-Training Dataset
  • 2.2. Datasets and Evaluation for VLM
  • 2.3. Benchmark Datasets, Simulators and Generative Models for Embodied VLM

• 1. 🔥 Post-Training/Alignment/prompt engineering 🔥

  • 3.1. RL Alignment for VLM
  • 3.2. Regular finetuning (SFT)
  • 3.3. VLM Alignment Github
  • 3.4. Prompt Engineering
• 1. ⚒️ Applications
  • 4.1. Embodied VLM agents
  • 4.2. Generative Visual Media Applications
  • 4.3. Robotics and Embodied AI
    • 4.3.1. Manipulation
    • 4.3.2. Navigation
    • 4.3.3. Human-robot Interaction
    • 4.3.4. Autonomous Driving
  • 4.4. Human-Centered AI
    • 4.4.1. Web Agent
    • 4.4.2. Accessibility
    • 4.4.3. Medical and Healthcare
    • 4.4.4. Social Goodness
• 1. ⛑️ Challenges
  • 5.1. Hallucination
  • 5.2. Safety
  • 5.3. Fairness
  • 5.4. Alignment
    • 5.4.1. Multi-modality Alignment
      • 5.4.2. Commonsense and Physics Alignment
  • 5.5. Efficient Training and Fine-Tuning
  • 5.6. Scarce of High-quality Dataset

@InProceedings{Li_2025_CVPR,
    author    = {Li, Zongxia and Wu, Xiyang and Du, Hongyang and Liu, Fuxiao and Nghiem, Huy and Shi, Guangyao},
    title     = {A Survey of State of the Art Large Vision Language Models: Benchmark Evaluations and Challenges},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2025},
    pages     = {1587-1606}
}