Tutorial: An overview of 3D Vision-Language Models

Márcus Lobo^*^†¹ Vitor Matias^*¹ Afonso Paiva¹ Jeová Farias²
Tiago Novello³ Moacir Ponti¹

ICMC-USP¹ Bowdoin College² IMPA³

*: Equal contribution †: Corresponding Author

SIBGRAPI 2026

TL;DR

An overview of 3D Vision-Language Models: from 3D representations and their embeddings, through contrastive alignment with pretrained image and text encoders, to 3D VLLMs and applications in Gaussian splatting, shape generation, and embodied AI — with a hands-on notebook.

Abstract

Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification and cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

What you will learn

The tutorial is given in two parts, with a hands-on session at the end of Part 1. See the schedule for the full program and Slides & Code for the material.

Preliminaries: why 3D and language, 3D representations (point clouds, meshes, multi-view, 3D Gaussian splatting), datasets, and how 3D VLMs differ from 3D VLLMs.
Input and encoders: (3D, image, text) triplets and the Point-BERT 3D encoder.
Geometric view of contrastive alignment: L2-normalization, the unit hypersphere, and cosine similarity against a frozen CLIP space.
Contrastive alignment for 3D VLMs: batch matching, the symmetric InfoNCE loss, and the ingredients that make it work in practice.
A taxonomy of 3D VLMs: five model families, from closed-set recognition to 3D VLLMs, with representative methods.
Hands-on: a CPU-only notebook that trains a 3D encoder into CLIP space end to end.
Current trends and emerging applications: zero-shot classification, cross-modal retrieval, spatial reasoning, and embodied AI.

3D VLLMs: the generative paradigm

A 3D VLM asks whether a 3D object or scene can be connected to language, and answers with a similarity score or a ranking. A 3D Vision-Large Language Model (3D VLLM) goes further: it uses the 3D information to understand a question, reason about it, and answer in language.

3D VLLM pipeline: a 3D input goes through an encoder and a projector/adapter into a large language model, which answers the prompt 'What is this object?' with 'This is a cow'

Pipeline: 3D input, encoder, projector, and a Large Language Model (LLM).
Behavior: generates a response conditioned on geometry and a text prompt.
Scale: from 1 billion to more than 13 billion parameters.
Output: natural language tokens.

A taxonomy of 3D VLMs

From fixed labels, to open vocabulary, to language generation: five families, distinguished by where language connects to the 3D pipeline.

Family	Output	Methods	Tasks
1. Closed-set 3D recognition	Class label (no language: a fixed set of classes)	MVCNN, PointNet, DGCNN, Point-BERT	Classification, part segmentation, detection
2. Projection-based adaptation	View–text similarity (render views, reuse frozen CLIP)	PointCLIP, PointCLIP V2, EPCL	Zero-shot and few-shot classification
3. Native joint embeddings	Shared 3D–image–text space (train a 3D encoder into CLIP space)	ULIP, ULIP-2, OpenShape, Uni3D, 3D-MRL	Zero-shot classification, text/image → 3D retrieval, open-vocabulary recognition
4. Language-aligned 3DGS	Language features in 3D (attach CLIP features to Gaussians)	UniGS, CLIP-GS, OpenGaussian, SceneSplat	Open-vocabulary segmentation, scene understanding, object grouping and editing, Gaussian retrieval
5. 3D VLLMs	Generated text (3D tokens fed to an LLM)	PointLLM, ShapeLLM, Multi-3DLLM, N3D-VLM	Captioning, 3D question answering, spatial reasoning, instruction following

From model families to tasks

One aligned space (or one LLM), many tasks. They differ in what they output:

Eight tasks enabled by 3D VLMs and 3D VLLMs: zero-shot classification, cross-modal retrieval, visual grounding, question answering, scene understanding, spatial reasoning, 3D generation, and embodied AI

Scores: zero-shot classification
Ranked 3D assets: cross-modal retrieval
Regions: grounding, scene understanding, spatial reasoning
Text: question answering, spatial reasoning
3D content: 3D generation
Actions: embodied AI

When & Where

Event: SIBGRAPI 2026 — Conference on Graphics, Patterns and Images (September 29 to October 2, 2026)
Date and time: Tuesday, September 29, 2026, 14:00–17:30
Room: D
Location: Federal University of Goiás (UFG), Goiânia, GO, Brazil
Session: tutorial #266 in the SIBGRAPI 2026 tutorials program
Format: half-day tutorial in two parts, with a hands-on notebook in Part 1 (see the schedule)

Prerequisites

Basic Python programming.
Familiarity with deep learning fundamentals (neural networks, embeddings, loss functions).
Basic PyTorch is helpful for the hands-on session, which runs on CPU — no GPU is required.
No prior experience with 3D data is assumed.

BibTeX

If you find our work useful, please consider citing our paper.

                        
@misc{lobo2026overview3dvisionlanguagemodels,
    title={An overview of 3D Vision-Language Models}, 
    author={Márcus Lobo and Vitor Matias and Afonso Paiva and Jeová Farias and Tiago Novello and Moacir Ponti},
    year={2026},
    eprint={2609.05583},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2609.05583}, 
}

Acknowledgments

This work was supported in part by the São Paulo Research Foundation (FAPESP), under grants #2024/09462-1 and #2026/01721-3, and by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) under fellowship grant #315158/2023-9.