AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation

01 ABSTRACT

AC-JEPA is an SFT-compatible plug-in that learns predictable cross-modal structure in a joint embedding space by predicting masked intermediate states in both directions, while retaining token-level SFT supervision. On optical chemical structure recognition, AC-JEPA reaches 76.07% accuracy with clean data versus 67.40% for SFT alone; with the same pseudo-labeled data for self-training, it rises to 80.23% while SFT drops to 63.47%. The authors view SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.

02 KEY FINDINGS

1. AC-JEPA predicts masked intermediate states in both cross-modal directions; the joint-embedding objective targets cross-modal structure predictable from context, while the SFT path retains token-level supervision for the target sequence.
2. Self-training uses continuous latent targets derived from pseudo-labeled pairs, avoiding promotion of pseudo-tokens to hard categorical supervision and offering intrinsic tolerance to local data errors.
3. On the Simple set of the WildMol benchmark, AC-JEPA trained on clean pairs achieves 76.07% accuracy versus 67.40% for SFT; with the same pseudo-labeled data for self-training, AC-JEPA reaches 80.23% while SFT drops to 63.47%.