Slides & Code | Tutorial: An overview of 3D Vision-Language Models

Slides

Slides — Part 1 (PDF) Slides — Part 2 (PDF) Paper (arXiv)

Part 1 — Foundations and contrastive alignment: motivation, 3D representations and datasets, 3D VLMs vs. 3D VLLMs, the Point-BERT encoder, the geometric view of contrastive alignment, the symmetric InfoNCE loss, and the hands-on notebook.
Part 2 — Model families, tasks and applications: the five-family taxonomy in depth (closed-set, projection-based, native joint embeddings, language-aligned 3D Gaussian splatting, 3D VLLMs), downstream tasks, and embodied AI.

Code

The hands-on notebook implements a 3D Vision-Language contrastive-learning pipeline end to end: point cloud → 3D encoder (Point-BERT) → normalization → joint embedding space with frozen OpenCLIP image and text embeddings → similarity matrix → symmetric InfoNCE loss → a training loop that updates only the 3D encoder.

Repository Notebook Open in Colab

Data: a self-contained subset of 110 ShapeNet objects across 54 categories, with precomputed OpenCLIP image and text embeddings, shipped in the repository.
Hardware: everything runs on CPU; no GPU is required.
Helpers: the handson_utils/ package provides data loading, the Point-BERT encoder, and visualization utilities.

Running it

Clone the repository and open the notebook. PyTorch is assumed to be installed; the command below installs the remaining dependencies.

git clone https://github.com/usmarcv/Tutorial-3DVLMs.git
cd Tutorial-3DVLMs
pip install torch_redstone einops plotly ipywidgets scikit-learn matplotlib nbformat
jupyter lab handson_3DVLM_contrastive.ipynb

On Google Colab, the notebook needs the repository files next to it. Run this in a cell before the notebook's own setup cell:

!git clone https://github.com/usmarcv/Tutorial-3DVLMs.git
%cd Tutorial-3DVLMs
!pip install -q torch_redstone