The Content–Geometry Tradeoff in World Models: When JEPA Learns Latent Maps

Mengye Ren1, Ying Wang1*, Chris Hoang1*, Yilun Kuang1, and Yann LeCun1,2

1New York University, 2AMI Labs
*Equal contribution

Technical report

TL;DR: Sliced variance regularization learns smooth latent maps for planning, and slow channels with a gated residual predictor keep the task-relevant content.

Abstract

A latent world model for planning needs two properties: latent distances that smoothly track progress toward a goal (geometry), and task-relevant factors that remain decodable from the latent representation (content). Common approaches prevent representation collapse with a stop-gradient (SG) target or a distributional regularizer, but we find that both often yield poor latent geometry. In this paper, we introduce sliced variance regularization (SVReg), which imposes a variance floor on latent projections along random directions. SVReg learns a smooth map of the agent's state. Its latent trajectories are smooth, and its latent distances track progress in navigation and reaching. These geometric benefits, however, can come with a tradeoff in content. SVReg can discard slowly changing, hard-to-predict factors, such as the pose of a manipulated object, that SG variants tend to preserve. To address this tradeoff, we propose Slow SVReg, which favors slowly changing factors with a slowness term and a gated residual predictor. It keeps these factors and improves planning. This simple recipe has the highest planning success in the hard settings of five visual control tasks, and it learns latent maps from first-person mazes and city walks.

Overview

(a) One first-person route through the Medium maze in each latent (jag: route jaggedness, 1 is smooth; succ: planning success on hard goals). (b) Held-out readout R² of the PushT agent and resting block, six seeds. SVReg learns a smooth map but can lose the block, and Slow SVReg keeps both.
(a) One first-person route through the Medium maze in each latent (jag: route jaggedness, 1 is smooth; succ: planning success on hard goals). (b) Held-out readout R² of the PushT agent and resting block, six seeds. SVReg learns a smooth map but can lose the block, and Slow SVReg keeps both.

A latent world model for planning needs two properties. Latent distances should track progress toward a goal (geometry), and task-relevant factors should stay decodable from the latent (content). Training without collapse usually relies on a stop-gradient (SG) target or on a variance regularizer trained without SG. We find that both choices often give poor geometry.

Sliced variance regularization (SVReg) puts a variance floor on the latent along random directions. It learns a smooth map of the agent's state, but it can drop slowly changing, hard-to-predict factors such as the pose of a manipulated object. Slow SVReg adds a slowness term and a gated residual predictor, which favor these factors. It keeps them, plans best in the hard settings of five visual control tasks, and learns latent maps from first-person mazes and city walks.

Method

(a) SVReg floors the standard deviation of the latent along random directions. (b) Slow channels change slowly between frames. (c) The gated residual predictor copies the current latent and learns the change.
(a) SVReg floors the standard deviation of the latent along random directions. (b) Slow channels change slowly between frames. (c) The gated residual predictor copies the current latent and learns the change.

SVReg puts a one-sided floor on the standard deviation of the latent along random unit directions $\mathbf{u}$:

$$\mathcal{R}_{\mathrm{SVReg}}(z) = \mathbb{E}_{\mathbf{u}}\!\left[\max\!\left(0,\,1-\mathrm{std}_t(\mathbf{z}_t \cdot \mathbf{u})\right)^2\right].$$

A small fixed set of slow channels $\mathbf{b}_t$ gets a variance-normalized slowness term, so persistent factors such as a resting cube keep room in the latent:

$$\mathcal{R}_{\mathrm{slow}}(\mathbf{b}) = \frac{\mathbb{E}_t\|\mathbf{b}_{t+1}-\mathbf{b}_t\|^2}{2\,\mathrm{tr}\,\mathrm{Var}(\mathbf{b})}.$$

The predictor copies the current latent and learns the change, scaled by a per-dimension gate $\mathbf{c}$ initialized at 0.1:

$$\hat{\mathbf{z}}_{t+1} = \mathbf{z}_t + \mathbf{c}\odot f(\mathbf{z}_{\le t},\mathbf{a}_t).$$

Latent maps

SVReg and Slow SVReg lay out the maze in the leading principal components of the latent, from top-down and first-person views alike.

Top two principal components of each latent on PointMaze, coloured by the agent's position.
Top two principal components of each latent on PointMaze, coloured by the agent's position.
Slow SVReg on egocentric PointMaze Large, a smooth tour through the maze: first-person view, top-down position and the top three principal components of the latent.

Content

On Cube, SVReg loses the cube height in three of six seeds, the ones whose random initial encoder reads the least height. From the same starts, SG and Slow SVReg acquire it.

Held-out cube-height readout during training on Cube. Dashed: SG and SVReg seeds 2, 3 and 5.
Held-out cube-height readout during training on Cube. Dashed: SG and SVReg seeds 2, 3 and 5.
Cube, one held-out episode with two lifts: the camera next to the top three principal components of SG, SIGReg-BN and Slow SVReg. Colour: cube height while grasped.

Planning

Slow SVReg plans best in the hard settings. Planning success (%), mean ± s.d. over six training seeds. Hard goals on the mazes, all goals on Cube. Bold: best; underline: second best.
Navigation and reachingObject manipulation
MethodU-mazeMediumReacherCubePushT
Stop-gradient
SG65.0±18.771.6±8.786.5±1.680.4±2.144.0±19.3
SG + straightening88.0±3.172.8±5.686.6±2.985.4±1.062.1±7.7
SG + SVReg95.8±1.162.7±6.296.3±1.677.2±1.376.9±4.3
No stop-gradient
VICReg94.3±2.377.2±2.393.1±1.769.6±5.855.0±3.6
SIGReg-BN82.2±13.778.2±3.895.8±1.387.4±0.860.4±2.6
SVReg (ours)96.4±1.381.3±2.398.7±0.976.9±8.875.0±8.6
Slow SVReg (ours)98.7±0.784.7±2.799.6±0.489.1±1.687.3±3.8

City walks

We train on first-person walks rendered from 3D Gaussian-splat reconstructions of real city blocks (Wanderland), with actions and no poses. The walked layout appears in the top two principal components of the Slow SVReg latent.

Wanderland scene 13Skv: held-out walks in the top two principal components of each latent, coloured by true position.
Wanderland scene 13Skv: held-out walks in the top two principal components of each latent, coloured by true position.
A smooth walk around the block: first-person view, top-down position and the top three principal components of the Slow SVReg latent.
The top three principal components of five latents, rotating. SVReg and Slow SVReg trace the walked loop as a thin curve, and the other objectives spread the frames into a cloud.

BibTeX

@misc{ren2026content,
  title  = {The Content--Geometry Tradeoff in World Models: When {JEPA} Learns Latent Maps},
  author = {Ren, Mengye and Wang, Ying and Hoang, Christopher and Kuang, Yilun and LeCun, Yann},
  year   = {2026}
}