Fresh

Token Economics & Log Probabilities

Log probabilities reveal what an LLM "believes" before it generates a response. By examining the probability distribution over the next token at each step, you can measure entity alignment, discover content gaps, audit competitive positioning, and expand keyword strategies -- all without relying on the model's generated text.

---

Log Probabilities Explained

When an LLM generates text, it does not pick words at random. At each step, it calculates a probability for every token in its vocabulary. The token with the highest probability is (usually) selected. Log probabilities are the natural logarithm of these probabilities.

Log Probability	Actual Probability	Interpretation
-0.01	99.0%	Model is near-certain
-0.10	90.5%	Very confident
-0.50	60.7%	Moderately confident
-1.00	36.8%	Uncertain, considering alternatives
-2.00	13.5%	Low confidence
-5.00	0.7%	Very unlikely

Key principle: A log probability closer to 0 means the model strongly associates that token with the context. A log probability far from 0 means the model is uncertain or the association is weak.

---

Measurement Methodology

How to Access Log Probabilities

1. OpenAI API: Set logprobs=true and optionally top_logprobs=5 in your API call to get the top 5 token probabilities at each generation step.

2. Prompt design for measurement:

   • Use completion-style prompts, not chat-style
   • Structure prompts to force the model to "fill in the blank"
   • Example: "The leading company in [industry] for [service] is" -- then examine which company name tokens have the highest log probabilities

3. Batch measurement: Run the same prompt template across multiple entities/topics to build a comparative dataset.

Reproducibility

Set temperature to 0 for measurement. This ensures the model always selects the highest-probability token, making results deterministic and comparable across runs. See Temperature Zero Trick for more on this technique.

---

4 SEO Applications

Application 1: Entity Alignment

Question answered: How strongly does the LLM associate your brand with your target topics?

Method:

1. Create prompts like: "The best [service] provider in [city] is ___"
2. Examine log probabilities for your brand name vs. competitors
3. Map the probability distribution across your target topic space

What high alignment looks like:

• Your brand name appears in top-5 tokens with log probability > -1.0
• The association is consistent across related prompts
• Competitor brands do not dominate the probability space

What low alignment looks like:

• Your brand name does not appear in top-20 tokens
• Generic terms ("many companies," "various providers") have higher probability
• A competitor consistently occupies the top position

Actionable insight: Low entity alignment means the model was not trained on enough content connecting your brand to the topic. The fix is creating more branded content that explicitly connects your entity to your target topics across multiple high-authority platforms.

Application 2: Query Expansion

Question answered: What related terms and topics does the model associate with your target keyword?

Method:

1. Prompt: "Topics closely related to [keyword] include ___"
2. Examine top-5 tokens at each generation step
3. Build a tree of associated terms by chaining completions

What this reveals:

• The semantic neighborhood of your keyword as the model understands it
• Related topics you may not have considered for content planning
• The "expected" content structure for your topic (what the model predicts should be covered)

Actionable insight: If the model predicts a sub-topic as highly probable but your content does not cover it, that is a content gap. If the model predicts a sub-topic that your competitors cover but you do not, that is a competitive gap.

Application 3: Content Auditing

Question answered: Does your content match what the model expects for this topic?

Method:

1. For each section of your content, create a prompt that would lead to that section's key claim
2. Measure whether the model's highest-probability completion aligns with your content
3. Flag sections where the model's expectation diverges significantly from what you wrote

What divergence means:

• Your content says X, model expects Y: Either your content has a unique angle (good if backed by data) or it misaligns with consensus (bad if factually wrong)
• Model expects a section you do not have: Content gap -- the model has been trained to expect this sub-topic as part of comprehensive coverage
• Your content covers something the model does not expect: Either ahead-of-the-curve content or off-topic drift

Application 4: Competitive Analysis

Question answered: Which competitors does the model associate most strongly with your target topics?

Method:

1. Run entity alignment prompts across your full competitor set
2. Create a matrix: topics (rows) x brands (columns) x log probability (values)
3. Identify which competitor "owns" each topic in the model's associations

What this reveals:

• Which competitors have the strongest training-data presence for your target topics
• Topics where no single competitor dominates (opportunity)
• Topics where a competitor's association is so strong you need a different angle

---

Variability Factors

Log probability measurements are not perfectly stable. Several factors introduce variability:

Factor	Impact	Mitigation
Temperature setting	High variability if not set to 0	Always use temperature=0 for measurement
Model version	Different weights produce different probabilities	Document which model version you used
Prompt phrasing	Different wordings produce different distributions	Test 3-5 prompt variants and average
Context window	Prior tokens in the prompt affect subsequent probabilities	Standardize prompt structure
Token boundary effects	Some words split differently depending on context	Check tokenization of target terms before measuring
API caching	Some providers cache responses, reducing variability	Use unique prompt prefixes

---

Key Limitations

What Log Probabilities Cannot Tell You

1. They do not reflect real-time search rankings. Log probabilities are based on training data, not live search results. A brand that trained heavily but lost market share will still show high probability.

2. They are not user intent. High probability for a brand name does not mean users prefer that brand -- it means the model saw that brand more often in training data for that context.

3. Measurement is point-in-time. Log probabilities change when the model is updated. Measurements are valid for the model version they were taken on.

4. Not all tokens are equal. A brand name that tokenizes as a single token is easier to measure than one that splits into 3 sub-word tokens. Multi-token brand names require more sophisticated measurement (joint probability across tokens).

5. Correlation with citations is imperfect. High entity alignment in log probabilities correlates with but does not guarantee citation in AI search. The retrieval and reranking pipeline (documented in other articles) adds additional filtering.

When to Use This

Log probabilities are a diagnostic tool, not an optimization target. Use them to understand where you stand, identify gaps, and measure the impact of your content strategy over time. Do not optimize for log probabilities directly -- optimize for the things that improve them (comprehensive coverage, high-authority mentions, entity clarity).