Context Windows

Input Tokens:  [tok1, tok2, tok3, ..., tok_N]
               └─ max N = context window ─┘

Output Tokens: [response_tok1, response_tok2, ...]

Example:
Context: 4K (4096 tokens)
Input: 3500 tokens
Output: max 596 tokens (4096 - 3500 remaining)

Important distinction:

Llama 3 70B advertises 8K context
But with extrapolation:
  - 16K: unstable
  - 32K: hallucinations
→ Effective: 8K

Claude 3.5 advertises 200K context
Needle-in-Haystack test:
  - 200K: 95% accuracy
  - 150K: 98% accuracy
→ Effective: Really 200K (exceptional!)

Rule of thumb: Advertised ÷ 2-4 = safe to use
               (except Claude 3.5)

Experiment:
1. Fill with 100K tokens "irrelevant info"
2. Hide fact: "Chapter 3 written in 1823"
3. Ask: "When was Chapter 3 written?"

Results:

GPT-4 Turbo (128K):
- Fact at 5%: 100% accuracy
- Fact at 50%: 94% accuracy
- Fact at 95%: 87% accuracy
Problem: Middle of context weaker!

Claude 3.5 Sonnet (200K):
- All positions: 98-99% accuracy
Problem: Minimal!

Gemini 2.0 (1M):
- All positions: 95-98% (middle)
Problem: Larger window but not perfect

Large document (1M tokens)?

RAG approach:
1. Split into 10K chunks
2. Embed + vector search
3. Retrieve top-5 chunks (~50K tokens)
4. Put in prompt

Overhead: +100ms for search
Quality: +20-30%

Document (1000 pages)
  ├─ Chapter 1 → Summary
  ├─ Chapter 2 → Summary
  └─ ... (8 more)

Prompt: Summaries (50K) + Chapter 5 full text (50K) + Query
Result: Overview + detail

Iteratively:
1. Retrieve "AI 2020-2021" → process
2. Retrieve "AI 2022-2023" → process
3. Retrieve "AI 2024" → process

Multiple calls, each under limit
User gets progressive results

With Claude 3.5 (200K):
- System: 1K tokens
- 100 examples (few-shot): 150K tokens
- User query: 1K tokens
- Available for response: 48K tokens

Huge number of exemplars → model learns deeply
Quality 50%+ improvement possible!

Position Effectiveness:
First 10%:       98% (attention starts here)
Next 20%:        95%
Middle 40%:      85% (U-shaped!)
Next 20%:        95%
Last 10%:        98% (recency bias)

→ Optimal structure:
[Important info at start]
→ [Support details in middle]
→ [User question at end]

OpenAI GPT-4 Turbo:
Input: $0.01 per 1K tokens

Claude 3.5 Sonnet:
Input: $0.003 per 1K tokens

For 100K token input:
- GPT-4: $1.00
- Claude: $0.30

Claude 3.3× cheaper for long context!

# Document: 1M tokens, Context: 100K tokens

windows = []
overlap = 20000  # 20% overlap for continuity

for i in range(0, 1000000, 80000):  # 80K stride
    window = {
        'start': i,
        'end': min(i + 100000, 1000000),
        'text': document[i:min(i + 100000, 1000000)]
    }
    windows.append(window)

# Process each window
for window in windows:
    result = llm.process(window['text'])
    # Results have context from overlapping sections

# Query 1: Process document section (100K tokens)
response1 = client.chat.completions.create(
    messages=[{"role": "user", "content": document[:100000]}],
    cache_control={"type": "ephemeral"}  # Cache this!
)

# Query 2: Same document + different question
response2 = client.chat.completions.create(
    messages=[{"role": "user", "content": document[:100000] + "\n\nNEW QUESTION"}],
)

# Query 2 cost: 90% cheaper (reuses cached 100K tokens)

from langchain.vectorstores import Qdrant
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# 1. Index large document (~1M tokens)
document = load_document("large_book.pdf")
chunks = split_into_chunks(document, size=1000)  # ~1M chunks
embeddings = encode_all(chunks)

vector_db = Qdrant.from_embeddings(embeddings, chunks)

# 2. User asks question
question = "What's the main argument in chapter 5?"

# 3. Retrieve relevant chunks
relevant = vector_db.similarity_search(question, k=5)  # Top-5 chunks
context = "\n".join([c.page_content for c in relevant])  # ~5K tokens

# 4. Use LLM with just relevant context
qa = RetrievalQA.from_chain_type(llm=OpenAI(), retriever=vector_db)
answer = qa.run(question)

# Document: 500K tokens
# Task: Summarize with specific focus

# Step 1: Extract facts (fast)
facts = llm.extract_facts(document)
# Output: "CEO: John Smith", "Founded: 2010", "Revenue: $1B"

# Step 2: Use facts to guide analysis (within context)
analysis = llm.analyze(document[:200000], facts=facts)
# Facts act as retrieval hints, document provides detail

# WRONG: Question first
prompt = f"Question: {question}\n\nDocument: {document}"

# RIGHT: Question last
prompt = f"Document: {document}\n\nQuestion: {question}"

# Claude 3.5: 98% accuracy everywhere
# GPT-4: 87% accuracy in middle
# → Use Claude for long contexts

# Retrieve only relevant chunks
# No "middle" concept
# Perfect accuracy

# DON'T: GPT-4 with 200K tokens (slow, expensive)
response = gpt4_client.completions.create(
    model="gpt-4-turbo",
    messages=[{"role": "user", "content": huge_document}]
)
# Latency: 30+ seconds
# Cost: $2.00

# DO: Claude 3.5 with 200K (faster, cheaper)
response = claude_client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1000,
    messages=[{"role": "user", "content": huge_document}]
)
# Latency: 8 seconds
# Cost: $0.30

contract_text = load_contract("agreement.pdf")
response = claude.messages.create(
    max_tokens=2000,
    messages=[{
        "role": "user",
        "content": f"Analyze risks in this contract:\n\n{contract_text}"
    }]
)

kb = VectorDB(articles)  # Index all articles

query = "How do I reset my password?"
relevant = kb.search(query, k=3)  # Get top-3 articles

response = llm.answer(query, context=relevant)

papers = load_papers(10)

for paper in papers:
    chunks = split_paper(paper, size=100000)

    summary_tasks = []
    for chunk in chunks:
        task = asyncio.create_task(
            llm.summarize(chunk)
        )
        summary_tasks.append(task)

    summaries = await asyncio.gather(*summary_tasks)
    final = llm.synthesize(summaries)