When images speak louder than words: semantic image search with CLIP

Introduction

Ever wished your image search engine understood concepts instead of just rummaging for matching pixels and file names? Enter CLIP — no, not the hair accessory, but a brainy model from OpenAI that can handle both images and text like they’re best buds. Think of CLIP as a bilingual translator that speaks “visual” and “textual,” mapping them to the same secret language (embeddings), so you can run your searches with style.

In this blog post, we’ll build a semantic image search pipeline in Python. Buckle up, because your cat pictures will never look the same again.

1. Setup: Lock & Load Your CLIP

What is CLIP?

CLIP (Contrastive Language-Image Pre-training) is a model developed by OpenAI that connects images and text in the same embedding space. It was trained on large pairs of images and text, learning to associate a given image with the text that best describes it (and vice versa). Because both images and text end up in a shared vector space, you can easily measure how semantically close they are — making CLIP a perfect fit for building image search systems driven by meaning rather than just raw pixel values or filenames.

Installation

To get started, make sure you have the right Python packages installed. Run the following commands:

pip install torch torchvision torchaudio --upgrade
pip install sentence-transformers
pip install pillow

Loading the Model

Once installed, you can load a pre-trained CLIP model with sentence-transformers like so:

from sentence_transformers import SentenceTransformer
model_name = "clip-ViT-B-32"
model = SentenceTransformer(model_name)

If you have a GPU, consider moving the model to the GPU for faster processing:

import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = model.to(device)

With that, you have a powerful encoder ready to convert both text and images into meaningful embeddings. In the next steps, we’ll use these embeddings to build a semantic image search system.

Pro tip: If you have a GPU, you can use torch.cuda.is_available() to speed things up.

2. Building the Image Embeddings

What Are Image Embeddings?

In deep learning, embeddings are numerical representations of data — usually high-dimensional vectors. When working with images, an embedding is essentially a summary of the image’s content, learned by a neural network. Two images that are similar in content or context should (ideally) have embeddings that are closer together in this high-dimensional space, while very different images will have embeddings that are farther apart.

CLIP excels at generating image embeddings because it was trained on a large collection of (image, text) pairs. During training, CLIP learned to associate visual features in an image (like shapes, objects, and textures) with textual descriptions that best match it. This means the embeddings it produces are semantically rich — they capture more than just raw pixels, focusing instead on higher-level concepts like “cat,” “mountain,” or “red car.”

Generating Embeddings with CLIP

Below is a function that takes a folder of images and uses a CLIP model (via SentenceTransformer) to produce embeddings for each image. These embeddings are stored in a dictionary, keyed by the image filename.

import os
from PIL import Image
import torch

def build_image_embeddings(image_folder, model):
    """
    Loads images from a folder and computes embeddings using the provided model.
    Returns a dictionary mapping image filenames to their embedding tensors.
    """
    embeddings_dict = {}

    # Go through each file in the folder
    for filename in os.listdir(image_folder):
        # Check for common image extensions
        if filename.lower().endswith(('.jpg', '.jpeg', '.png', '.bmp')):
            file_path = os.path.join(image_folder, filename)
            image = Image.open(file_path).convert("RGB")

            # Convert the image to an embedding tensor
            embedding = model.encode(image, convert_to_tensor=True)
            embeddings_dict[filename] = embedding

    return embeddings_dict

# Example usage
image_folder_path = "path/to/your/images"
embeddings_dict = build_image_embeddings(image_folder_path, model)

1. We load each image from the specified folder.
2. The image is converted to RGB (in case it’s grayscale or has a different format).
3. We pass the image to our CLIP-based model, which produces a vector (commonly 512 dimensions for many CLIP variants).
4. We store this vector in a dictionary keyed by the image filename.

These embeddings will form the basis of our semantic image search. In the upcoming steps, we’ll use them to quickly find images relevant to a text query or to another image.3. Serving Up the Search

3.1 Text-to-Image

So you want to find all pictures of “cats wearing a pirate hat?” Just feed CLIP the text “cat wearing a pirate hat” and watch it return the best matches. It’s practically telepathy, minus the tinfoil hats.

from sentence_transformers import util

def search_by_text(query_text, embeddings_dict, model, top_k=5):
    query_emb = model.encode(query_text, convert_to_tensor=True)
    scores = {}
    for fname, emb in embeddings_dict.items():
        # Cosine similarity is like CLIP's BFF
        sim = util.cos_sim(query_emb, emb)[0][0].item()
        scores[fname] = sim
    # Sort by descending similarity
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]
# Example usage
text_query = "A cat wearing a pirate hat"
text_results = search_by_text(text_query, embeddings_dict, model, top_k=3)
for fname, score in text_results:
    print(f"{fname} = {score:.4f}")

These top matches are presumably the best co-captains for your next pirate-themed adventure. Arrr, indeed.

3.2 Image-to-Image

Got an image and want to find its doppelgängers (or its long-lost siblings)? With image-to-image search, you can play detective.

def search_by_image(query_image_path, embeddings_dict, model, top_k=5):
    image = Image.open(query_image_path).convert("RGB")
    query_emb = model.encode(image, convert_to_tensor=True)
    scores = {}
    for fname, emb in embeddings_dict.items():
        sim = util.cos_sim(query_emb, emb)[0][0].item()
        scores[fname] = sim
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]

# Example usage
query_image = "path/to/query.jpg"
image_results = search_by_image(query_image, embeddings_dict, model, top_k=3)
for fname, score in image_results:
    print(f"{fname} = {score:.4f}")

You can now hunt down similar cats — er, images — in your dataset, effectively organizing your image library like a champ.

4. Full Example

Here’s a single script that ties it all neatly in a bow, so you can copy-paste and act like a wizard among muggles:

import os
from sentence_transformers import SentenceTransformer, util
from PIL import Image
import torch

def build_image_embeddings(image_folder, model):
    embeddings_dict = {}
    for filename in os.listdir(image_folder):
        if filename.lower().endswith(('.jpg', '.jpeg', '.png', '.bmp')):
            path = os.path.join(image_folder, filename)
            image = Image.open(path).convert("RGB")
            embedding = model.encode(image, convert_to_tensor=True)
            embeddings_dict[filename] = embedding
    return embeddings_dict

def search_by_text(query_text, embeddings_dict, model, top_k=5):
    query_emb = model.encode(query_text, convert_to_tensor=True)
    scores = {}
    for fname, emb in embeddings_dict.items():
        sim = util.cos_sim(query_emb, emb)[0][0].item()
        scores[fname] = sim
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]

def search_by_image(query_image_path, embeddings_dict, model, top_k=5):
    image = Image.open(query_image_path).convert("RGB")
    query_emb = model.encode(image, convert_to_tensor=True)
    scores = {}
    for fname, emb in embeddings_dict.items():
        sim = util.cos_sim(query_emb, emb)[0][0].item()
        scores[fname] = sim
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)[:top_k]

if __name__ == "__main__":
    # Load CLIP
    model_name = "clip-ViT-B-32"
    model = SentenceTransformer(model_name)
    # (Optional) Put model on GPU if available:
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    model = model.to(device)
    image_folder_path = "path/to/your/images"
    embeddings_dict = build_image_embeddings(image_folder_path, model)
    # Example 1: Text Query
    text_query = "A scenic view of a mountain"
    text_results = search_by_text(text_query, embeddings_dict, model, top_k=3)
    print("Text-to-Image results:")
    for fname, score in text_results:
        print(f"{fname}: {score:.4f}")
    # Example 2: Image Query
    query_image = "path/to/query.jpg"
    image_results = search_by_image(query_image, embeddings_dict, model, top_k=3)
    print("\nImage-to-Image results:")
    for fname, score in image_results:
        print(f"{fname}: {score:.4f}")

5. Leveling Up

1. Vector Databases: If your image folder is bigger than your appetite at an all-you-can-eat buffet, consider hooking up a specialized vector store (like Faiss, Annoy, or Milvus) for faster searches.
2. Batch Embeddings: To avoid your CPU/GPU screaming for help, embed images in batches if you have thousands or millions of them.
3. Fine-Tuning: If you want your model to be extra good at, say, diagnosing cat memes, consider fine-tuning it on a domain-specific dataset. Yes, that’s actually a thing.
4. GPU Usage: Always remember to send your data and the model to the same device (cpu or cuda). Mismatching them is like wearing socks with sandals—someone’s bound to complain.

Conclusion

With this guide, you now have the building blocks for a semantic image search pipeline using CLIP. By converting both images and queries (text or other images) into embeddings, you can retrieve contextually relevant images rather than relying on exact matches of file names or low-level features.

Real-World Applications

• E-commerce: Shoppers can upload photos of items they like or describe them in natural language (“sleek black running shoes with white soles”), and the system can find visually similar products.
• Travel and Hospitality: Users can search photo galleries or travel destinations with phrases like “sunset on a tropical beach,” helping them discover locations that match their interests.
• Media Archives and Libraries: Large collections of images or videos can be quickly indexed and made searchable by content, saving time for journalists, researchers, and content creators.
• Art and Design: Museums, design studios, and photographers can use image-to-image search to find pieces similar in style, color, or theme. This can streamline the discovery of related artwork or design references.
• Corporate and Enterprise: Internal asset libraries at companies (e.g., marketing materials, branding images) become easier to search, increasing productivity.

By leveraging the CLIP model’s ability to understand both text and images, you can create a flexible, intelligent search experience for almost any image collection. Whether you’re building a small personal project or powering a large-scale platform, semantic search can significantly improve how users find and interact with visual content.