Reranking with Cohere and cross-encoders is surprisingly effective because it shifts the focus from retrieving any relevant document to retrieving the most relevant document, often by a significant margin.
Let’s see what that looks like in practice. Imagine you’ve got a user query like "What are the latest advancements in AI for drug discovery?" and you’ve retrieved a set of candidate documents using a simpler, faster method like BM25 or a dense retriever.
Here’s a snippet of raw output from a hypothetical retriever, showing a few documents it thought were relevant:
[
{
"id": "doc_101",
"text": "Deep learning models are revolutionizing drug discovery by predicting molecular properties and designing novel compounds. Recent breakthroughs in generative adversarial networks (GANs) show promise for de novo drug design.",
"score": 0.85
},
{
"id": "doc_205",
"text": "Artificial intelligence (AI) has been applied to various fields, including healthcare, finance, and autonomous driving. In healthcare, AI is used for diagnosis and personalized medicine.",
"score": 0.72
},
{
"id": "doc_312",
"text": "The pharmaceutical industry is investing heavily in research and development. New drug candidates are constantly being screened using high-throughput methods.",
"score": 0.65
},
{
"id": "doc_450",
"text": "Recent advancements in AI, particularly in natural language processing (NLP) and computer vision, are impacting many sectors. For instance, NLP is improving chatbots and translation services.",
"score": 0.60
}
]
The retriever picked doc_101 first, which is good. But doc_205 and doc_450 are quite generic. doc_312 is related to pharmaceuticals but doesn’t mention AI specifically. A simple reranker might just look at keywords. A cross-encoder, however, treats the query and each document together as a single input to a powerful transformer model.
Here’s how you might set up a Cohere reranker (or a similar cross-encoder model):
from cohere import Client
co = Client("YOUR_COHERE_API_KEY")
query = "What are the latest advancements in AI for drug discovery?"
documents = [
"Deep learning models are revolutionizing drug discovery by predicting molecular properties and designing novel compounds. Recent breakthroughs in generative adversarial networks (GANs) show promise for de novo drug design.",
"Artificial intelligence (AI) has been applied to various fields, including healthcare, finance, and autonomous driving. In healthcare, AI is used for diagnosis and personalized medicine.",
"The pharmaceutical industry is investing heavily in research and development. New drug candidates are constantly being screened using high-throughput methods.",
"Recent advancements in AI, particularly in natural language processing (NLP) and computer vision, are impacting many sectors. For instance, NLP is improving chatbots and translation services."
]
# Cohere's rerank endpoint expects a list of (document, relevance_score) tuples
# If you're using raw text, you can just pass the documents.
# For demonstration, let's assume we're not passing initial scores.
# If you have initial scores, you'd pass them as a second element in the tuple.
# Example: [(doc_text, initial_score), ...]
# Cohere's rerank endpoint often needs a `top_k` parameter to limit the number of documents to rerank.
# Let's say we want to rerank the top 50 documents from our initial retrieval.
# For this example, we only have 4, so we'll use 4.
reranked_results = co.rerank(query=query, documents=documents, top_k=4, model='rerank-english-v2.0')
# The output `reranked_results.results` is a list of dictionaries, each with 'document' and 'relevance'.
# Let's print them sorted by their new relevance score.
sorted_reranked_results = sorted(reranked_results.results, key=lambda x: x['relevance'], reverse=True)
for item in sorted_reranked_results:
print(f"Relevance: {item['relevance']:.4f}, Document: {item['document'][:100]}...") # Print first 100 chars
Running this would produce output more like:
Relevance: 0.9876, Document: Deep learning models are revolutionizing drug discovery by predicting molecular properties and designing novel compounds. Recent breakthroughs in...
Relevance: 0.7543, Document: Artificial intelligence (AI) has been applied to various fields, including healthcare, finance, and autonomous driving. In healthcare, AI is used for...
Relevance: 0.5567, Document: Recent advancements in AI, particularly in natural language processing (NLP) and computer vision, are impacting many sectors. For instance, NLP is improvin...
Relevance: 0.2345, Document: The pharmaceutical industry is investing heavily in research and development. New drug candidates are constantly being screened using high-throughput meth...
Notice how doc_101 maintained a very high score, doc_205 and doc_450 are now lower, and doc_312 is significantly demoted. The cross-encoder model, by processing the query and document simultaneously, can understand the nuanced relationship between "AI for drug discovery" and the specific details in each text. It’s not just about keyword overlap; it’s about semantic entailment and contextual relevance.
The problem this solves is the "recall vs. precision" trade-off in initial retrieval. Initial retrievers are fast and have high recall (they find most of the relevant documents), but their precision can be lower (they also return many irrelevant or weakly relevant documents). Rerankers significantly boost precision.
Internally, a cross-encoder is typically a transformer model (like BERT, RoBERTa, or Cohere’s proprietary models) that takes the concatenation of the query and a document as input. The model then outputs a single score representing the likelihood that the document is relevant to the query. This is fundamentally different from a bi-encoder, where the query and document are encoded independently, and relevance is calculated based on the similarity of their separate embeddings (e.g., cosine similarity). The sequential processing of query and document together allows for a much deeper, contextual understanding.
The exact levers you control are primarily the model parameter (e.g., rerank-english-v2.0, rerank-multilingual-v2.0) and top_k. top_k is crucial for performance; you typically only rerank the top N documents from your initial retrieval, where N might be 50, 100, or 500, depending on your latency budget and the number of documents you need to present to the user. The model selection depends on your language needs and desired performance characteristics.
What most people don’t realize is that the "relevance" score from a cross-encoder isn’t a probability in the traditional sense. It’s an arbitrary score derived from the model’s internal workings, often the output of a linear layer after the transformer’s final hidden state. While it’s highly indicative of relative relevance, directly interpreting it as a "percentage chance of being relevant" is a misapplication. The ordering it provides is the primary value.
The next concept you’ll likely run into is how to integrate this reranking step into a larger application pipeline, considering latency and cost.