The tutorial was presented at SIBGRAPI 2026 by Márcus Lobo and Vitor Matias, in two parts, with a hands-on session closing Part 1.
| Topic | Description |
|---|---|
| Part 1 — Foundations and contrastive alignment Slides | |
| Goals and motivation | Why 3D and language: applications in robotics, LiDAR, medical imaging and engineering; the move beyond closed-set 3D recognition; industry and research impact. |
| 3D representations and datasets | Point clouds, meshes, multi-view images and 3D Gaussian splatting; where the language comes from; ShapeNet, Objaverse / Objaverse-XL and Objaverse-LVIS. |
| 3D VLMs vs. 3D VLLMs | The alignment paradigm (similarity scores and rankings) against the generative paradigm (encoder, projector and LLM producing text); overview of the 3D VLM training pipeline. |
| Input and encoders | (3D, image, text) triplets; image, text and 3D encoders with projection layers; 3D encoder options and a closer look at Point-BERT and masked point modeling. |
| Geometric view of contrastive alignment | Frozen OpenCLIP image and text embeddings as the semantic reference space; L2-normalization, the unit hypersphere and cosine similarity before and after alignment. |
| Contrastive alignment for 3D VLMs | Alignment as batch matching, the similarity matrix, the symmetric InfoNCE loss and its PyTorch implementation; key ingredients: data scale, augmentation, batch size, hard negatives, temperature and compute. |
| A taxonomy of 3D VLMs | Five families, from fixed labels to open vocabulary to language generation. |
| Hands-on | Contrastive learning for 3D VLMs, end to end, in a notebook: point cloud → 3D encoder → joint embedding space → similarity matrix → InfoNCE loss → training loop. |
| Part 2 — Model families, tasks and applications Slides | |
| Recap | Problem statement, contrastive alignment, and 3D VLMs vs. 3D VLLMs. |
| Closed-set recognition and projection-based adaptation | Fixed classifiers (MVCNN, PointNet, DGCNN, Point-BERT) and their limits; rendering 3D into 2D views for frozen CLIP, with PointCLIP V2. |
| Native joint embeddings | Training a dedicated 3D encoder into a shared space with image and text: OpenShape, Uni3D and 3D-MRL. |
| Language-aligned 3D Gaussian splatting | Gaussians as the 3D modality: UniGS and related methods. |
| 3D Vision-Large Language Models | The encoder → projector → LLM recipe and two-stage training: PointLLM, ShapeLLM, PointAlign and N3D-VLM. |
| From model families to tasks | Zero-shot classification, cross-modal retrieval and spatial reasoning. |
| Embodied AI | Vision-language-action models and the role of 3D data: PointVLA and GR00T N1. |
| Discussion | |
| Q&A | Questions and discussion. |