Language Guidance for Supervised Vision Training: An Empirical Study of Generalization

Summary

This empirical study tests whether pretrained language models can serve as lightweight auxiliary supervision for supervised vision training, without paired image-text data, prompt engineering, or contrastive objectives. The authors evaluate Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG) across six generalization regimes, including out-of-distribution, shortcut resistance, texture and shape bias, adversarial robustness, and continual learning. Explicit guidance helps in-distribution, low-data, and continual-learning retention, while implicit guidance helps more in shortcut-sensitive and strongly adversarial settings.