SigLIP, Explained: Why a Simpler Loss Beat CLIP's

CLIP taught image and text encoders to agree with each other by making every pair in a batch compete against every other pair. SigLIP asks a much smaller question of each pair, one at a time: is this image-text pair a match, yes or no? That's the entire idea, and it's why SigLIP — first from Google DeepMind in 2023, now in its second generation — has quietly become the default vision encoder bolted into a large share of recent open vision-language models, PaliGemma among them.

The two models are otherwise the same architecture. Everything up to the last step — the encoders, the embeddings, the similarity score — is identical; only the loss stage, highlighted below, diverges:

The problem SigLIP was built to fix

CLIP (2021) trains two separate encoders — one for images, one for text — so that matching image-text pairs land close together in a shared embedding space and mismatched pairs land far apart. It does this with a contrastive softmax loss: for a batch of $N$ image-text pairs, you build the full $N \times N$ matrix of similarity scores, then normalize each row and each column with softmax, treating it as an $N$ -way classification problem — "which of these $N$ texts matches this image?"

That works, but the softmax normalization means every single pair's loss depends on every other pair in the batch at that step. You need the whole matrix materialized before you can compute anything, which costs memory that grows with batch size and creates an asymmetry between rows and columns that has to be handled with two passes (one softmax over images-to-texts, one over texts-to-images).

The fix: treat each pair independently

SigLIP — Sigmoid Loss for Language Image Pre-Training, Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, Google DeepMind, ICCV 2023 — replaces the softmax with a sigmoid loss. Every image-text pair, matching or not, is scored on its own as a binary classification problem: for pair $(j, k)$ , the logit is

$s_{j,k} = t \cdot (\mathbf{i}_j \cdot \mathbf{t}_k) + b$

where $\mathbf{i}_j$ and $\mathbf{t}_k$ are the (normalized) image and text embeddings, $t$ is a learned temperature, and $b$ is a learned bias. The loss is then just binary cross-entropy against a label of $+1$ for the true pair $(j = k)$ and $-1$ for every mismatched pair — no row, no column, no shared normalization term:

$\mathcal{L} = -\frac{1}{|B|}\sum_{j \in B}\sum_{k \in B} \log \frac{1}{1+e^{z_{j,k}(-t(\mathbf{i}_j \cdot \mathbf{t}_k)-b)}}, \quad z_{j,k} = \begin{cases} 1 & j = k \\ -1 & j \ne k \end{cases}$

Diagram comparing CLIP's softmax loss, which normalizes similarity scores across every row and column of the batch matrix, with SigLIP's sigmoid loss, which scores each image-text pair independently as a binary classification problem

The learned bias $b$ matters more than it looks: with $N-1$ negatives for every positive in a batch, the loss starts out dominated by negatives, so the authors initialize $b$ strongly negative to counteract that imbalance early in training.

Because no pair needs information from any other pair to compute its own loss, you never build the dense normalized matrix — each device can compute its local chunk of pairs and only needs to exchange the raw embeddings, not intermediate softmax statistics. That's a smaller, more parallelizable operation, and it's the whole reason for the efficiency gains below.

What that bought them

The paper's headline results, per the abstract and ICCV presentation:

Better at small batch sizes, and scales further at large ones. The sigmoid loss removes the softmax's dependence on very large batches to get a reliable signal, while also parallelizing more cheaply when you do scale up.
Batch size saturates around 32k, far below CLIP-style setups that pushed toward hundreds of thousands; gains from going all the way to a 1M batch size are marginal.
A SigLiT model — SigLIP's loss combined with Locked-image Tuning — hit 84.5% zero-shot ImageNet accuracy training on just four TPUv4 chips in two days.

That last number is the one that mattered practically: a result in that range no longer required a lab-scale training cluster.

The architecture itself is deliberately plain

Nothing about the encoders is novel — that's the point of the paper. SigLIP is a standard dual-encoder: a Vision Transformer for images, a Transformer text encoder for text, each producing a single pooled embedding, compared by dot product. All the design effort goes into the loss, not the towers.

The Hugging Face transformers implementation exposes exactly that split — SiglipVisionModel, SiglipTextModel, and a combined SiglipModel — plus SiglipForImageClassification for the classification head case, and support for FlashAttention/SDPA and int4 quantization via bitsandbytes. Two usage details the docs call out as easy to get wrong: always pass padding="max_length" to the tokenizer (SigLIP was trained without an [EOS]-based variable-length convention CLIP relies on), and for zero-shot classification use the literal prompt template "This is a photo of {label}.", which is what the model was calibrated against.

from PIL import Image
import requests
from transformers import AutoProcessor, AutoModel
import torch

model = AutoModel.from_pretrained("google/siglip-so400m-patch14-384")
processor = AutoProcessor.from_pretrained("google/siglip-so400m-patch14-384")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
texts = ["a photo of 2 cats", "a photo of 2 dogs"]

inputs = processor(text=texts, images=image, padding="max_length", return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.sigmoid(outputs.logits_per_image)  # independent probabilities, not a softmax distribution

Notice the last line: because the loss is a sigmoid, inference-time scores are independent probabilities per label rather than a distribution that has to sum to 1 across your candidate captions — a small but real difference from how you'd read CLIP logits.

The so400m in that checkpoint name is worth a footnote: it's a "shape-optimized" ViT (SoViT-400m) from a separate Google paper on compute-optimal vision transformer shapes, not something SigLIP itself introduced — SigLIP just adopted it as one of its released variants alongside plain ViT-B and ViT-L backbones.

SigLIP 2: the same idea, generalized

SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features (Tschannen et al., Google DeepMind, February 2025) keeps the sigmoid loss as the backbone objective and folds in several training-recipe additions on top: captioning-based pretraining, self-supervised losses (self-distillation and masked prediction, in the style of DINO/BEiT), and online data curation. It's trained on WebLI, roughly 10 billion images paired with 12 billion alt-texts across 109 languages, and ships in four sizes — B (86M), L (303M), So400m (400M), and a new g (1B) — plus a NaFlex variant that accepts native aspect ratios and multiple resolutions instead of forcing a fixed square crop. Per the paper, SigLIP 2 outperforms SigLIP 1 at every matched model scale on zero-shot classification, image-text retrieval, and as the frozen vision tower feeding downstream VLMs.

Why this design choice ended up mattering beyond the paper

The efficiency numbers are the headline, but the more durable effect is architectural: because SigLIP is cheap enough to pretrain well without a large-batch, multi-pod training setup, it became an easy, high-quality default to bolt on as the frozen (or lightly tuned) vision encoder in later vision-language models — PaliGemma being the clearest example, using SigLIP's So400m encoder directly. A loss function that removed a scaling constraint turned into a component other teams could reuse without re-deriving it — the usual way a good simplification actually propagates through a field.