Good Views Hidden in Plain Sight

A Closer Look at the Next-Token Baselines in “Proving and Improving the Platonic Representation Hypothesis with JEPAs”

In Proving and Improving the Platonic Representation Hypothesis with JEPAs [1], Randall Balestriero and Yann LeCun prove under idealised assumptions that JEPA encoders learn the same representation up to one rotation when trained on good views (distinct observations of the same scene, such as two captions of one image). Bad views corrupt a single observation, such as masking one caption twice; because those views share surface wording, the encoder retains wording alongside scene content. Empirically, a small BERT trained as a good-view JEPA on Flickr captions aligns with frozen DINOv2 much better than a bad-view JEPA or next-token prediction (NTP). The authors argue that NTP acts like a bad-view JEPA whose positives are a prefix and its continuation, and therefore aligns with vision mainly through scale.

I ran a few quick experiments with LLM coding agents to test that comparison. The paper’s paired-caption NTP baseline appears to underestimate what NTP learns from good views because its mean-pooled readout dilutes the cross-caption signal.

The paired baseline

The paper’s main NTP baseline trains on single captions, which matches the data given to the bad-view JEPA (both score ~.12–.13 against DINOv2 on Flickr) but not the good-view JEPA, which sees which captions share an image. Section 3.1 addresses this with a paired NTP control:

“Third, the gain comes from the objective and not from the information that two captions describe the same image: next-token prediction trained on two captions of the same image, [CLS] a [SEP] b [SEP], barely moves (mutual 10-NN .042 to .043, R@1 .021 to .023), i.e., the same positives help next-token prediction far less than they help as JEPA views, as per Proposition 4”

(Table 1 measures alignment against an image JEPA trained from scratch on Flickr. Against frozen DINOv2 in Table 2, R@1 moves from .118 to .137. All results below use frozen DINOv2.)

At evaluation time, the paper feeds a single caption [CLS] a [SEP] and mean-pools the final hidden states across all tokens. In earlier layers, tokens in b can attend back to all of a as well as [SEP], so cross-caption gradients reach the whole prefix. At the final layer, however, each state feeds only the output head at its own position: tokens inside a only see a prefix of a and predict the next token of a (or [SEP]), whereas [SEP] has seen all of a and is the only position trained to predict into the second caption b (specifically its first token). That makes h([SEP]) an interesting readout candidate.

Appendix B of the paper tests a last-token readout only on single-caption NTP, where it trails mean pooling (R@1 .061 vs .119). That is expected: there [SEP] is followed by ignored padding and receives no loss. In the paired model, [SEP] predicts the first token of b, yet wasn’t evaluated separately.

Using the authors’ training code unchanged (BERT-S, Flickr30k, 200 epochs, seed 0) and changing only the readout:

• R@1, 1k-pair rotation: 13.2% → 24.5% (good-view JEPA: 24.6%).
• R@1, 30k-pair rotation: 23.9% → 32.1% (good-view JEPA: 34.6%).
• Mutual 10-NN: .127 → .170 (good-view JEPA: .232).

On retrieval through an orthogonal map (the paper’s “direct test of equality up to rotation”), paired NTP read at h([SEP]) matches the good-view JEPA on the 1k-pair fit, while the JEPA retains a clear lead on mutual 10-NN. An independent GPT-written run stopped at epoch 139 shows the same jump (14.0% → 22.2% on 1k, 26.1% → 31.5% on 30k).

Reading h([SEP]) hurts both single-caption NTP (11.1% → 6.6%) and the JEPA (11.1% vs 24.9% for h([CLS])). It only helps when [SEP] precedes a second caption of the same image during training: in a 25-epoch pilot where caption a was paired with a caption from a different image, h([SEP]) dropped to chance (Appendix A1).

Alignment at h([SEP]) also keeps rising while the validation loss more than doubles (Figure 2). The paper’s validation metric (used for its early-stopped row, though main tables use final checkpoints) is next-token loss on single captions, which rises after ~20–30 epochs as the model memorises training phrasing. Because single captions have only ignored padding after [SEP], that validation loss never evaluates [SEP] or cross-caption prediction; early-stopping on it would pick epoch 20 (17.5% R@1 instead of 24.5%).

Predicting only the other caption

Paired NTP still spends roughly half its loss predicting caption a, whereas the JEPA only penalises disagreement between views. To isolate cross-caption prediction, I trained a conditional NTP variant using the same [CLS] a [SEP] b [SEP] input, initialization, and schedule, but masking the loss on a so only b is predicted (with gradients still flowing into a and both caption orders sampled).

Read at h([SEP]), conditional NTP reaches 27.8% / 38.3% R@1 (1k / 30k fit), ahead of the good-view JEPA’s 24.6% / 34.6%, and even mean pooling reaches 22.2% / 33.5%. Its mutual 10-NN (.185) remains below the JEPA’s (.232). Part of the gain over standard paired NTP may simply come from b’s tokens receiving roughly twice the weight in the loss.

Readout choice affects the JEPA as well: its native h([CLS]) readout (the projector input) beats mean pooling on Flickr R@1 (24.9% vs 16.7%), while the projector output itself scores 26.7% on the 1k fit (2.2 points above paired NTP’s h([SEP])) and 32.4% on the 30k fit (Appendix A1). On Flickr, the paper thus compares the JEPA at its trained summary token against paired NTP mean-pooled over tokens trained for within-caption prediction. (On WIT-1024 and COCO-5k, the paper instead mean-pools all models across layers following the PRH protocol [2].)

All numbers

Bold marks the better readout per model. Paper values are three-seed means ± half-range; ours are single seeds.

Without a fitted map

Because fitting an orthogonal map could flatter a weaker representation, I also checked three of the paper’s map-free text evaluations: caption→caption retrieval, STS-B sentence similarity, and the Table 6 “scene share” estimator (variance explained by the shared image rather than caption wording). At h([SEP]), paired NTP closes most of the gap to the JEPA on all three (caption retrieval 36.7% → 53.8% vs 60.6%), and conditional NTP slightly exceeds the JEPA on each. These three metrics are text-only; on mutual 10-NN against DINOv2, the JEPA still leads.

Single seeds, 200 epochs. Paired NTP: Claude-run with the authors’ code; conditional NTP and the JEPA: GPT-run. More tasks and readouts in Appendix A1.

At web scale: the 2M Recap point

To test beyond Flickr, I trained paired BERT-S NTP on the smallest web-scale point in the paper’s scaling plot: 2.24M Recap-DataComp images (one alt-text and two LLaVA recaptions per image), following the exact schedule of the Table 7 good-view JEPA control (20 × 1M samples, learning rate 10⁻⁴, gradient clip 1.0, one seed).

Mean pooling again shows little gain from pairing: .075 WIT mutual 10-NN (vs .070–.071 for single-caption NTP) and 23.2% Flickr R@1 (vs 22.3–24.6%). Reading h([SEP]) raises WIT mutual 10-NN to .093, close to the JEPA’s range (.095–.115, mean .107), but only lifts Flickr R@1 to 25.3% against the JEPA’s 33.9%, a gap substantially larger than the seed variation the paper reports.

Reading one token further by appending “the” (which starts nearly every Recap recaption) reaches .104 on WIT and 30.1% on Flickr, though this was the best of ten readouts scored on the same test sets, so treat it as a lead rather than a result (Appendix A2). Testing the analogous pre-declared heuristic on Flickr by appending “a” (the most common first word in Flickr) gave smaller, inconsistent changes: +2.1 points on 30k-pair retrieval and +0.016 on mutual 10-NN, but −0.2 on 1k-pair retrieval, with +0.9 on 30k-pair retrieval for the JEPA control as well (Appendix A1). One possible reason paired NTP lags further behind on Recap is that two of the three captions per image come from the same LLaVA captioner, so predicting b from a also rewards matching generator style. That is untested.

One seed. WIT: the paper’s PRH protocol (per-layer features, best layer pair; the JEPA’s .107 is mean-pooled). Flickr: final layer, rotation fitted on 30k pairs, frozen DINOv2. Full readout search, including generated continuations, in Appendix A2.

Readouts for frozen language models

Pretrained LLM baselines in the paper are also mean-pooled over the raw caption. Appending a short prompt and reading the final token’s hidden state (without generating tokens) substantially changes their scores. Prompted last-token embeddings are well known [3–6]; here they just show that the LLM baselines are also sensitive to the readout:

Caption: {A}
Alternative caption of the same scene:

Under the paper’s Flickr protocol (final layer, 30k-pair rotation), Qwen2.5-0.5B R@1 jumps from 24.1% to 35.4% (14.0% to 22.2% with a 1k-pair fit), passing the Flickr-trained good-view JEPA (34.6%). On Qwen2.5-1.5B, a continuation prompt reaches 42.9% (raw caption 24.5%, alternative prompt 35.3%). On WIT-1024, mutual 10-NN rises from .102 to .138 (0.5B), .108 to .155 (1.5B), and .153 to .170 (72B). The relative gain shrinks with scale, the continuation prompt hurts at 72B, and 72B remains below the paper’s 109M web-trained JEPA (.184).

Prompt wording itself may matter less than providing a post-caption summary position: “Continuation of this caption:” matches “Alternative caption of the same scene:” on 0.5B and beats it on 1.5B Flickr, whereas bare markers like literal “[SEP]” or Qwen’s end-of-text token do not help (Appendix A3).

Closest to this, Wang et al. raise frozen Qwen3’s alignment with DINOv2 on WIT with sensory prompts (“imagine what it would be like to see…”) [5]. They let the model generate a continuation and average its hidden states over the generated tokens and over all layers; in a single forward pass averaged over the input tokens, the prompt did not help. The prompted last-token state here needs no generation. Wang et al. also find that mean-pooling over generated tokens beats single-token readouts [7]. Here, greedily generating an alternative caption and pooling its states did not beat the single prompted last-token state (33.9% vs 35.4% Flickr R@1 on 0.5B; Appendix A3), but that comparison reads single layers only. Averaging over all layers, which Wang et al. find often beats any single layer [5], I haven’t tested yet.

Conclusion

On Flickr with 33M models, the paper’s paired control does not show that alignment gains come from the JEPA objective rather than the pairing information. Evaluated at [SEP], the paper’s own paired NTP checkpoint gains 11 R@1 points and matches the good-view JEPA on retrieval through a rotation, while conditional NTP surpasses it. Mean-pooling therefore understates what paired NTP learns, even though the JEPA still leads on mutual 10-NN (.232 vs .170–.185) and on Flickr retrieval at the 2.24M-image Recap point (33.9% vs 25.3%).

Several confounds also remain untested. In the paper, hyperparameters were tuned only for the good-view JEPA, while NTP “use[s] the same recipe and [was] not tuned”; each JEPA sample draws eight views from an image’s five captions whereas paired NTP sees two; and the JEPA encoder is bidirectional while NTP is causal. All runs here use a single seed, prompted LLM readouts were chosen after inspecting Flickr results, conditional NTP was not run at web scale, and the paper’s largest regime (109M models on 940M images) is not tested here. I also haven’t gone through the theory (whose extension to NTP the authors note is heuristic).

To reproduce the main Flickr result with the authors’ repository, train with train.py --modality text --objective ntp --caption-views and extract features with evaluate.py embed <checkpoint> <out> --last-token (where the last token of [CLS] a [SEP] is [SEP]).

Disclosure. LLMs from OpenAI, Anthropic and Google wrote and reviewed the code, and helped edit this post. Two independently written pipelines reproduce the paper’s baselines (ordinary NTP 11.1% vs reported 11.8%, paired NTP 13.2% vs 13.7%, good-view JEPA 24.9% vs 24.6%, Qwen2.5-0.5B WIT .102 vs .102) and agree on the h([SEP]) effect.

Additional results

References

1. R. Balestriero, Y. LeCun. Proving and Improving the Platonic Representation Hypothesis with JEPAs. arXiv:2610.39302, 2026. Code: galilai-group/prh-proof.
2. M. Huh, B. Cheung, T. Wang, P. Isola. The Platonic Representation Hypothesis. ICML 2024.
3. T. Jiang et al. Scaling Sentence Embeddings with Large Language Models (PromptEOL). arXiv:2307.16645, 2023.
4. J. Springer et al. Repetition Improves Language Model Embeddings (echo embeddings). arXiv:2402.15449, 2024.
5. S. L. Wang, P. Isola, B. Cheung. Words That Make Language Models Perceive. arXiv:2510.02425, 2025.
6. S. Zhuang, X. Ma, B. Koopman, J. Lin. PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval. arXiv:2404.18424, 2024.
7. S. L. Wang, P. Isola, B. Cheung. The Truth Lies Somewhere in the Middle (of the Generated Tokens). arXiv:2605.09969, 2026.

@article{bernwald2026goodviews,
  title  = "Good Views Hidden in Plain Sight",
  author = "Bernwald, Thomas",
  year   = "2026",
  month  = "November",
  url    = "https://thomasaas.github.io/good-views.html"
}