The most surprising thing about Pinecone is that it’s not a database in the traditional sense; it’s a vector similarity search engine designed to find conceptually similar items, not just exact matches.
Let’s see it in action. Imagine you have a collection of product descriptions, and you want to find items that are similar in meaning, even if they use different words.
import pinecone
import os
from dotenv import load_dotenv
load_dotenv() # Load environment variables from a .env file
# Initialize connection to Pinecone
pinecone.init(
api_key=os.environ.get("PINECONE_API_KEY"),
environment="us-west1-gcp" # Replace with your actual environment
)
# Define index name
index_name = "my-product-index"
# Check if the index exists, create it if not
if index_name not in pinecone.list_indexes():
pinecone.create_index(
name=index_name,
dimension=8, # This should match your embedding dimension
metric="cosine" # Or "euclidean", "dotproduct"
)
print(f"Index '{index_name}' created.")
else:
print(f"Index '{index_name}' already exists.")
# Connect to the index
index = pinecone.Index(index_name)
# --- Upserting data ---
# In a real scenario, you'd generate these embeddings using an ML model.
# For demonstration, we'll use dummy vectors.
vectors_to_upsert = [
("prod_1", [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8], {"description": "A red apple"}),
("prod_2", [0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2], {"description": "A green apple"}),
("prod_3", [0.15, 0.25, 0.35, 0.45, 0.55, 0.65, 0.75, 0.85], {"description": "A shiny apple"}),
("prod_4", [0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1], {"description": "A ripe banana"})
]
# Upsert the vectors. This adds or updates vectors in the index.
# The first element of the tuple is the unique ID, the second is the vector,
# and the third is optional metadata.
upsert_response = index.upsert(vectors=vectors_to_upsert)
print(f"Upserted {upsert_response.upserted_count} vectors.")
# Wait a moment for the index to update (especially important for new indexes)
import time
time.sleep(10) # Adjust as needed
# --- Querying data ---
# Let's say we want to find items similar to a "sweet red fruit".
# Again, this would typically be an embedding from an ML model.
query_vector = [0.12, 0.22, 0.32, 0.42, 0.52, 0.62, 0.72, 0.82]
# Query the index for the top 2 most similar items
query_response = index.query(
vector=query_vector,
top_k=2,
include_metadata=True # We want to see the descriptions
)
print("\nQuery Results:")
for match in query_response.matches:
print(f"ID: {match.id}, Score: {match.score:.4f}, Metadata: {match.metadata}")
This example shows the core loop: initialize Pinecone, connect to an index, upsert (add/update) vectors with their IDs and optional metadata, and then query using a vector to find the most similar items. The dimension parameter during index creation is crucial – it must match the dimensionality of the vectors you intend to store and query. The metric defines how similarity is calculated; cosine is common for text embeddings as it measures the angle between vectors, ignoring magnitude.
Pinecone handles the complex task of organizing these high-dimensional vectors so that similarity searches are incredibly fast, even with millions or billions of items. When you upsert, Pinecone builds an index structure (often a form of Approximate Nearest Neighbor – ANN – search) that allows it to quickly narrow down potential matches. When you query, it uses this structure to find the top_k most similar vectors to your query vector based on the chosen metric.
The metadata you attach to vectors is stored alongside them. This is powerful because your query might return vectors that are conceptually similar, but the metadata allows you to filter or select based on specific attributes (e.g., "find me similar red fruits"). You can also use metadata filtering directly within the query call to refine results.
What most people don’t realize is how sensitive the dimension parameter is. If your embedding model outputs vectors of dimension 768, your Pinecone index must be created with dimension=768. Mismatching this will lead to errors during upsert or query, or worse, incorrect similarity results because the vector spaces won’t align. It’s not a value you can easily change after index creation, so it’s a fundamental decision made at the outset.
The next step is often to explore more advanced querying capabilities like filtering by metadata during the search itself.