clip model

CLIP (Contrastive Language-Image Pretraining) is a vision-language model trained to align images and text in a shared embedding space — enabling zero-shot image classification, image search, and serving as the vision backbone of modern generative AI.

How CLIP Works

• Training Data: 400M (image, text) pairs scraped from the internet.
• Architecture: Two encoders — ViT for images, Transformer for text.
• Objective: Contrastive loss — maximize similarity between correct (image, text) pairs, minimize for incorrect pairs.
• Result: Images and their descriptions have similar embeddings; unrelated images/texts have dissimilar embeddings.

Zero-Shot Classification

1. Encode candidate class labels as text: "a photo of a dog", "a photo of a cat".
2. Encode the query image.
3. Find the most similar text embedding → predicted class.
4. No task-specific training required — generalizes to arbitrary categories.

Why CLIP Revolutionized AI

• Zero-shot transfer: Competitive with supervised models on 30+ vision benchmarks without task-specific training.
• Universal features: CLIP embeddings work for retrieval, classification, generation conditioning.
• Stable Diffusion backbone: CLIP text encoder guides the denoising process in most image generation models.
• Semantic search: Enables image search by text description (used in Google Photos, Pinterest).

CLIP Variants

• OpenCLIP: Open-source CLIP trained on LAION-5B (5 billion pairs).
• SigLIP (Google): Sigmoid loss instead of softmax — better performance at smaller batch sizes.
• MetaCLIP: Meta's CLIP using curated data curation methodology.

CLIP is the foundation of modern vision-language AI — its shared embedding space enabled the entire ecosystem of multimodal models and controllable image generation.