Loading…
Glossary term
Learning representations by pulling similar items together and pushing different items apart.
Contrastive learning trains a model to put matching pairs close together and mismatched pairs far apart. For example, a photo and its caption should map to nearby points, and a photo and an unrelated caption to distant ones. CLIP used this to connect images and text.
A contrastive loss for image-text pairs
import torch
import torch.nn.functional as F
logits = img_emb @ txt_emb.T / temperature # similarity of every pair
labels = torch.arange(len(logits)) # the matching pair is on the diagonal
loss = F.cross_entropy(logits, labels)It produces strong general-purpose representations without hand-made labels, and underpins image search and multimodal models.