Pinecone’s similarity search isn’t just about finding any match; it’s about finding the best matches, and often, the "best" is defined by a confidence score that you can, and should, filter on.
Let’s see this in action. Imagine you have a Pinecone index named my-index with vectors and associated metadata, including a text field. You want to find vectors similar to a query vector [0.1, 0.2, ..., 0.9] but only if their similarity score is above 0.8.
from pinecone import Pinecone, ServerlessSpec
import os
# Initialize Pinecone (replace with your API key and environment)
api_key = os.environ.get("PINECONE_API_KEY")
pc = Pinecone(api_key=api_key)
# Assume index 'my-index' already exists and has data
index_name = "my-index"
index = pc.Index(index_name)
# Example query vector
query_vector = [0.1] * 1536 # Assuming a 1536-dimensional vector
# Perform a similarity search with a score threshold
results = index.query(
vector=query_vector,
top_k=10,
filter={
"score": {"$gte": 0.8} # Filter for scores greater than or equal to 0.8
},
include_metadata=True,
include_values=False
)
# Process results
for match in results.matches:
print(f"ID: {match.id}, Score: {match.score}, Metadata: {match.metadata}")
In this example, the filter argument is where the magic happens. We’re using a $gte (greater than or equal to) operator to specify that we only want results where the similarity score is at least 0.8. This is crucial for applications where a certain level of confidence is required before acting on a retrieved item.
The core problem Pinecone’s scoring and filtering solves is distinguishing between genuinely relevant results and those that are merely coincidentally similar. In high-dimensional vector spaces, it’s easy to find many vectors that have some degree of overlap with a query vector. However, not all overlaps are meaningful. A low similarity score might indicate that while two vectors share a few dimensions with similar values, they don’t represent the same underlying concept or entity. By setting a threshold, you’re essentially saying, "I only care about matches that are highly likely to be relevant."
Internally, when you perform a query, Pinecone uses approximate nearest neighbor (ANN) algorithms. These algorithms are designed for speed and scalability, but they don’t guarantee finding the absolute nearest neighbors. They return a set of candidate vectors that are likely to be among the closest. Each candidate is then assigned a similarity score (often cosine similarity or dot product, depending on your index configuration). The filter parameter acts as a post-processing step on these ANN-candidate results. Before Pinecone returns the top_k results to you, it checks each candidate against your specified filter. If a candidate’s score (or any other metadata field you’re filtering on) doesn’t meet the criteria, it’s discarded, and Pinecone might fetch more candidates from the ANN search to still fulfill the top_k request if enough filtered results aren’t found.
The filter argument is incredibly flexible. You can filter not only on the similarity score but also on any metadata fields associated with your vectors. This allows for complex querying, such as finding similar items that also belong to a specific category ({"category": {"$eq": "electronics"}}) or were created after a certain date ({"created_at": {"$gt": 1678886400}}). You can combine multiple conditions using logical operators like $and and $or. For instance, to find items similar to your query vector with a score above 0.8 AND belonging to the 'electronics' category:
results = index.query(
vector=query_vector,
top_k=10,
filter={
"$and": [
{"score": {"$gte": 0.8}},
{"category": {"$eq": "electronics"}}
]
},
include_metadata=True
)
When you’re filtering on the score itself, it’s important to understand that Pinecone’s internal score calculation is based on the distance metric you chose when creating the index (e.g., cosine similarity, dot product, Euclidean distance). For cosine similarity, a score of 1.0 is a perfect match, and scores decrease as vectors become less similar. For dot product, the range depends on vector magnitudes. For Euclidean distance, a score of 0.0 is a perfect match, and higher scores mean less similarity. You need to align your $gte, $lte, $gt, $lt operators with the expected range and direction of your chosen metric. If you’re using cosine similarity, {"score": {"$gte": 0.8}} makes intuitive sense for high confidence.
A common misconception is that filtering on score is equivalent to asking the ANN search for exactly the top K results with scores above a certain value. In reality, the ANN search finds candidates, and then the filter is applied. If the filter is very strict, Pinecone might need to perform additional ANN searches to find enough qualifying candidates to return top_k results. This can sometimes impact latency if the filter is too aggressive or the index is very sparse in high-scoring matches.
The next step after mastering score thresholds is understanding how to combine metadata filtering with vector similarity search for even more precise results, enabling you to build highly contextual and relevant search experiences.