Embeddings Explained

Text: "The dog is cute"
Embedding: [0.2, -0.5, 0.8, ..., 0.1]  (e.g., 1536 dims)

Text: "A cute dog"
Embedding: [0.21, -0.48, 0.79, ..., 0.12]  (similar!)

Text: "The car is red"
Embedding: [-0.8, 0.2, 0.1, ..., -0.5]  (not similar)

Query: "How do I cook spaghetti?"

Document A: "Cooking pasta: 1. Boil water..."
Distance: 0.05 (close) ✓

Document B: "History of Rome"
Distance: 0.9 (far) ✗

Input: "The cat is black"
       "A black cat"
       "The car is blue"

Goal:
- First two get similar vectors
- Third gets different vector

from openai import OpenAI

client = OpenAI(api_key="sk-...")

response = client.embeddings.create(
    input="Hello, I'm text",
    model="text-embedding-3-small"
)

embedding = response.data[0].embedding

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('nomic-ai/nomic-embed-text-v1')
embedding = model.encode("Hello")

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")
embedding = model.encode("What is AI?")

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

doc_embeddings = [
    [0.2, 0.5, 0.1],
    [0.21, 0.48, 0.12],
    [-0.8, 0.2, -0.5],
]

query = "Cute dog"
query_embedding = [0.21, 0.49, 0.11]

similarities = cosine_similarity([query_embedding], doc_embeddings)[0]
# [0.99, 0.95, 0.1]

top_indices = np.argsort(similarities)[::-1][:2]
# [0, 1] → Most similar docs

import chromadb

client = chromadb.Client()
collection = client.create_collection("documents")

collection.add(
    ids=["doc_1", "doc_2"],
    embeddings=[[0.2, 0.5], [0.21, 0.48]],
    documents=["Text 1", "Text 2"]
)

results = collection.query(
    query_embeddings=[[0.21, 0.49]],
    n_results=2
)

text-embedding-3-small: 1536 dims
nomic-embed: 768 dims
DistilBERT: 384 dims

from sklearn.decomposition import PCA

pca = PCA(n_components=256)
reduced = pca.fit_transform(embeddings)

# Quality: ~90%, Memory: -83%

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-large")

embeddings = model.encode([
    "How do I cook pasta?",
    "Wie koche ich Pasta?",
    "¿Cómo cocino pasta?"
])

# Fine-tune text-embedding-3-small on your domain
from sentence_transformers import models, losses, SentenceTransformer
from sentence_transformers.util import sentences2labels

# Your training pairs (similar sentences)
train_examples = [
    ("How do I cook pasta?", "Cooking spaghetti: boil water, add salt..."),
    ("What's machine learning?", "ML is a type of AI that learns from data..."),
]

model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

# Train with contrastive loss
train_loss = losses.ContrastiveLoss(model)
model.fit(
    [(e[0], e[1]) for e in train_examples],
    epochs=1,
    warmup_steps=100,
    show_progress_bar=True
)

model.save('./my-domain-embeddings')

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

v1 = np.array([0.2, 0.5, 0.8])
v2 = np.array([0.19, 0.51, 0.79])
v3 = np.array([-0.5, 0.1, 0.2])

print(cosine_similarity([v1], [v2])[0][0])  # 0.9999 (nearly identical)
print(cosine_similarity([v1], [v3])[0][0])  # 0.34 (different)

from scipy.spatial.distance import euclidean

v1 = np.array([0.2, 0.5, 0.8])
v2 = np.array([0.19, 0.51, 0.79])

dist = euclidean(v1, v2)  # 0.014 (small = similar)

# Fastest for large-scale search
dot_product = np.dot(v1, v2)  # 0.85 (needs normalization first)

import pinecone

pinecone.init(api_key="...", environment="us-west1-gcp")

index = pinecone.Index("documents")

# Upsert vectors
index.upsert(vectors=[
    ("doc_1", [0.2, 0.5, 0.1, ...], {"title": "Doc 1"}),
    ("doc_2", [0.21, 0.48, 0.12, ...], {"title": "Doc 2"}),
])

# Query
results = index.query([0.21, 0.49, 0.11], top_k=2)
# Returns: [("doc_1", 0.99), ("doc_2", 0.95)]

import weaviate

client = weaviate.Client("http://localhost:8080")

# Define schema
client.schema.create_class({
    "class": "Document",
    "vectorizer": "text2vec-transformers",
    "properties": [{"name": "content", "dataType": ["text"]}]
})

# Add documents
client.data_object.create(
    data_object={"content": "The cat is on the mat"},
    class_name="Document"
)

# Query
result = client.query.get("Document", ["content"]).with_near_text({
    "concepts": ["cat mat"]
}).do()

from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct

client = QdrantClient("localhost", port=6333)

# Create collection
client.recreate_collection(
    collection_name="documents",
    vectors_config={"size": 1536, "distance": "Cosine"}
)

# Add vectors
client.upsert(
    collection_name="documents",
    points=[
        PointStruct(id=1, vector=[0.2, 0.5, ...], payload={"text": "Doc 1"}),
    ]
)

# Search
results = client.search(
    collection_name="documents",
    query_vector=[0.21, 0.49, ...],
    limit=5
)

from openai import OpenAI
import chromadb

# 1. Index documents
client = chromadb.Client()
collection = client.create_collection("documents")

docs = [
    "Paris is the capital of France",
    "Tokyo is the capital of Japan",
    "The Earth orbits the Sun"
]

collection.add(ids=[str(i) for i in range(len(docs))], documents=docs)

# 2. User question
question = "What's the capital of France?"

# 3. Retrieve relevant docs (embedding search)
results = collection.query(query_texts=[question], n_results=1)
retrieved_doc = results["documents"][0][0]

# 4. Generate answer with LLM
llm = OpenAI()
response = llm.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": f"Based on: {retrieved_doc}\n\nAnswer: {question}"
        }
    ]
)

print(response.choices[0].message.content)
# Output: "According to the context, Paris is the capital of France."

# Hybrid search with Weaviate
result = client.query.get("Document", ["content"]).with_where({
    "operator": "And",
    "operands": [
        {"path": ["content"], "operator": "ContainsAny", "valueText": ["pasta"]},  # Keyword
        {"path": ["embedding"], "operator": "WithinDistance", "valueDistance": 0.1}  # Semantic
    ]
}).do()

from sklearn.decomposition import PCA

# 1536 → 384 dims
pca = PCA(n_components=384)
compressed = pca.fit_transform(embeddings_1536)

# Quality loss: ~2-5%
# Storage gain: 75% reduction

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-large")

# All languages encoded to same space
embeddings = model.encode([
    "What is artificial intelligence?",        # English
    "What es inteligencia artificial?",         # Spanish
    "Qu'est-ce que l'IA?",                     # French
    "What ist künstliche Intelligenz?",         # German (sic)
])

# All close together in vector space
# Cross-language similarity search works!

Embedding cost:
- 10k docs × 200 tokens/doc = 2M tokens
- text-embedding-3-small: $0.02/1M = $0.04

Inference cost (monthly, 1000 queries):
- 1000 queries × 200 tokens = 200k tokens = $0.004

Total monthly: $0.004 (embedding cost amortized, negligible)

One-time indexing:
- 500k docs × 200 tokens = 100M tokens = $2.00

Monthly inference:
- 10k queries/day × 30 days × 200 tokens = 60M tokens = $1.20

Infrastructure (vector DB):
- Pinecone: ~$1000/month (managed)
- Self-hosted: $200/month (VPS)

Monthly total: $1200+ (with Pinecone)