Technical Whitepaper
GenRank: A Methodology for Measuring Entity Visibility in Large Language Model
Abstract
This document presents the GenRank methodology—a systematic framework for quantifying entity visibility within large language model (LLM) recommendation outputs. As AI-driven discovery increasingly influences consumer behavior and market dynamics, understanding how LLMs prioritize and recommend entities becomes critical for researchers, marketers, and policymakers. GenRank employs a multi-model polling approach combined with market-weighted scoring to produce normalized visibility indices. This methodology document details our data collection protocols, mathematical scoring framework, model weight allocation procedures, and known limitations.
1. Introduction
1.1 Background
The proliferation of large language models (LLMs) as primary information discovery tools has fundamentally altered how consumers identify and evaluate products, services, and brands. Unlike traditional search engines that display ranked results based on explicit relevance signals, LLMs generate synthesized recommendations that reflect patterns learned during training and reinforcement learning processes.1
This shift presents both opportunities and challenges. Entities that achieve favorable positioning in LLM outputs may benefit from increased visibility and consideration, while those absent from recommendations face potential market invisibility—a phenomenon we term AI-mediated discovery bias.2
1.2 Research Objectives
GenRank addresses the following research objectives:
- Establish a reproducible framework for measuring entity visibility across multiple LLM platforms
- Develop a market-weighted scoring system that reflects real-world AI usage patterns
- Create transparent, open-access datasets for academic and commercial research
- Enable longitudinal analysis of recommendation patterns and temporal trends
1.3 Definitions
- Entity
- A distinct brand, product, service, organization, or concept that can be identified and tracked across LLM outputs.
- Visibility Score
- A normalized metric (0–100) representing an entity's prominence in LLM recommendations within a specific category.
- Market Weight
- A coefficient assigned to each LLM reflecting its estimated share of global AI assistant usage.
2. Data Collection
2.1 Query Design
Queries are designed to elicit ranked recommendation lists from LLMs. Each query undergoes validation against the following criteria:
- Specificity — Queries must target a defined category or use case
- Neutrality — Queries must not contain leading language or brand mentions
- Reproducibility — Queries must consistently elicit list-format responses
- Temporal stability — Queries should remain relevant across update cycles
2.2 Polling Parameters
Approved queries are submitted to each active LLM via official API endpoints. We deliberately do not override decoding parameters: a poll is a sample of what the model actually tells a user, and pinning temperature would measure something no user receives. Variance is acknowledged rather than suppressed: in a standard cycle each query-model pair is polled once, and the pipeline supports repeated sampling — in a sampled cycle each pair is polled multiple times and the resulting spread is published as a confidence interval (§4.5) rather than hidden.
Web search.Polling is executed with the model's native web search enabled. Each polled model is chosen specifically because it supports real-time web search, so recommendations reflect current information rather than training-cutoff knowledge. This constrains the active model set (see §4.2) and is versioned in the Changelog (§6).
2.3 Entity Resolution
Each response is parsed into the items it names, and each item is mapped to a canonical entity. Entities are resolved within a language: the same brand named in English and in Korean is ranked separately in each. From v3.3 the process is:
- What counts as an answer — each item is classified by its role in the response. An answer is recommended or named as an answer to the question. A source is only cited as evidence or pointed to as a place to look (a review site, a study, a search platform). A generic item is not a specific named thing, such as a product type, an ingredient, a feature or a piece of advice, in a question that asks for named ones. Only answers are ranked. When a question asks for types (e.g. horse breeds) or for platforms and sources, those are its answers. Sources and generic items are recorded with each response, alongside the web citations the model returned, but are not ranked.
- Lexical normalization — case folding, punctuation and whitespace normalization.
- Alias mapping — a maintained database of verified aliases. An alias that would route a name to a different real-world thing (a parent brand, a bare ticker, a generic word) is rejected.
- Match judgement — a mention is matched to an existing entity automatically only when the match is unambiguous. Uncertain matches are judged by a language model against the question and the answer text. A match is uncertain when the name is new to the category, matched only through an alias, previously rejected, ambiguous between candidates, or possibly a duplicate. Every decision is recorded, and a rejected match is never re-applied. Distinct items named in the same answer are never merged. When in doubt, the mention is credited to a new entity: a wrong merge is worse than a duplicate.
- Audit and correction— a cycle's resolution can be audited in full, and every correction is published with its cause (§6).
3. Scoring Methodology
3.1 Theoretical Foundation
The GenRank scoring system is grounded in information retrieval theory, specifically adapting the logarithmic discounting principles used in normalized discounted cumulative gain (nDCG) metrics.3 We employ logarithmic decay to model the diminishing marginal value of lower-ranked positions while ensuring all mentions contribute non-zero utility.
3.2 Mathematical Formulation
Definition 3.1 — Score Function
For an entity e at rank position r from model m with weight wm:
S(e, r, m) = [1 / (1 + log10(r))] × wm × 100
The final GenRank score aggregates across all queries Q and models M:
G(e) = Σq∈Q Σm∈M S(e, rq,m, m)
3.3 Score Decay
The logarithmic decay function produces the following position-relative values:
| Position | Raw Score | Relative Value | Decay |
|---|---|---|---|
| 1 | 1.000 | 100.0% | — |
| 2 | 0.769 | 76.9% | −23.1% |
| 3 | 0.677 | 67.7% | −9.2% |
| 5 | 0.588 | 58.8% | −8.9% |
| 10 | 0.500 | 50.0% | −8.8% |
| 20 | 0.435 | 43.5% | −6.5% |
3.4 Example Calculation
Consider entity "Notion" receiving the following rankings for the query "What are the best productivity applications?":
| Model | Weight | Rank | Raw | Weighted |
|---|---|---|---|---|
| GPT-5.6 Luna | 0.41 | 1 | 1.000 | 41.00 |
| Claude Sonnet 5 | 0.30 | 2 | 0.769 | 23.07 |
| Gemini 3.7 Flash | 0.29 | 3 | 0.677 | 19.63 |
| Total (single query) | 83.70 | |||
4. Model Weight Allocation
4.1 Methodology
Model weights are derived from a composite index incorporating three primary data sources, each addressing different aspects of real-world AI influence:
- Consumer market share (40%) — Monthly active users and web traffic data from Statcounter, SimilarWeb, and company disclosures
- Enterprise adoption (35%) — Deployment metrics from industry reports and API revenue estimates5
- Developer API usage (25%) — Token throughput from aggregated gateways6
4.1.1 Weight Derivation (October 2026)
Each component below is the provider's share normalized within the three-provider active set, from the most recent public figures at recalibration time. Values marked ~ are estimates where a source publishes partial breakdowns. Providers whose composite falls below a 3% materiality threshold are excluded from the active set entirely (see the note below the sources).
| Provider | Consumer (40%) | Enterprise (35%) | Developer (25%) | Composite |
|---|---|---|---|---|
| OpenAI | 59.2% | 30.7% | ~25.2% | 0.41 |
| Anthropic | 10.1% | 45.5% | ~41.4% | 0.30 |
| 30.7% | 23.9% | ~33.5% | 0.29 |
Sources: consumer — Similarweb web-visit share of major generative-AI chatbots, May–August 2026 (ChatGPT 53.9%, Gemini 27.9%, Claude 9.2%, Grok 2.4%, Perplexity 1.3% of all tracked assistants, renormalized to the active set). Enterprise — enterprise LLM API usage share, mid-2026 industry reports (Anthropic 40%, OpenAI 27%, Google 21% of total enterprise usage; xAI and Perplexity estimated). Developer — model-author token share on public aggregated gateways (OpenRouter), June 2026; shares for providers primarily consumed via direct APIs are estimates and this component is capped at 25% accordingly. Composite = 0.40·consumer + 0.35·enterprise + 0.25·developer, rounded to two decimals.
Materiality threshold. A provider whose composite falls below 3% is excluded: at that weight its influence on published scores is within rounding noise, while its polling cost is not. Perplexity (composite 0.01 within the prior five-provider set) and xAI (0.03 — and roughly a fifth of per-cycle polling cost for under 3% of score influence) were retired from the active set in October 2026. Their historical poll data is preserved, and both are re-evaluated at each recalibration.
4.2 Current Allocation (Q4 2026)
The following table presents active model weights as of October 2026, derived in §4.1.1. The active set is restricted to one web-search-capable model per major provider so recommendations reflect current information rather than training-cutoff knowledge. Weights are recalibrated periodically; every change is recorded in the Changelog (§6).
| Provider | Model | Weight |
|---|---|---|
| OpenAI | GPT-5.6 Luna | 0.41 |
| Anthropic | Claude Sonnet 5 | 0.30 |
| Gemini 3.7 Flash | 0.29 | |
| Total (n=3 active models) | 1.00 | |
4.3 Provider Distribution
Aggregate weights by provider:
4.5 Reliability & Disclosure
4.5.1 Sampling and confidence intervals
A language model does not return the same answer twice. Across the 994 questions polled in both the July and August 2026 cycles, an average of 5.12 of the top 10 entities persisted (51.9%), and with a single sample there is no way to separate genuine change from sampling noise. Per-question figures are therefore labeled directional, and the robust signals are the aggregated ones (§4.5.2).
From v3.0 the pipeline supports repeated sampling of each question–model pair. In a standard monthly cycle each pair is polled once, and scores publish without an interval — an absent interval is reported as absent, never as a zero-width claim of certainty. In a sampled cycle (n = 3; first run September 2026), model contributions are averaged before weighting, so scores remain comparable across all cycles, and the spread across samples yields a published confidence interval:
Where published, these intervals are wide — at the level of a single question, wide enough that per-question figures remain directional. They were always this wide; unsampled cycles simply cannot report them. Category rankings aggregate many questions, models and samples, and are correspondingly tighter.
4.5.2 Inclusion rate is the primary metric
Position within a model's answer is the fragile signal; whether a model names an entity at all is the robust one. GenRank therefore reports inclusion rate — the share of all (model × sample) trials mentioning the entity — as its primary measure, with position retained as a secondary quality signal.
4.5.3 What we measure, stated plainly
- Collection channel: provider APIs, not consumer chat interfaces. These surfaces differ substantially, and we do not claim to measure what a consumer sees in a chat window.
- Model tier: the small/fast tier of each provider, named with versions in §4.2 — not the flagship models.
- Grounding: native web search enabled, capped at 3 results per call.
- Decoding parameters: provider defaults, deliberately unset by us (§2.2).
- Version stamp: every score records the methodology version that produced it, so figures are never silently compared across revisions.
4.6 Independence
GenRank does not sell, and will not sell, any service to an entity it ranks that is intended to change that entity's rank. No fee depends on the level or direction of any score, and correction of a mismeasured entity is always free — never a paid service.
This follows the separation that credit rating, benchmark and audit regulation all treat as load-bearing: publishing data and diagnostics is legitimate; selling advice on how to move the measured attributes is not. The full policy, including our material-change and corrections procedures, is published separately. Methodology changes and corrections are logged in §6.
5. Limitations
Users should consider the following methodological limitations when interpreting results:
5.1 Temporal Variability
LLM outputs may vary over time due to model updates, fine-tuning, and reinforcement learning from human feedback. GenRank captures point-in-time snapshots and should not be interpreted as static ground truth.
5.2 Stochastic Sampling
Despite controlled parameters, LLM outputs exhibit inherent randomness. Identical queries may produce different rankings across executions. Statistical aggregation mitigates but does not eliminate this variance.
5.3 Market Weight Estimation
Model weights are derived from publicly available market data which may not fully reflect actual usage patterns. Enterprise API usage is estimated from secondary sources and may contain measurement error.
5.4 Query Design Bias
The selection and phrasing of queries may influence recommendation outcomes. Query design inherently reflects researcher assumptions about relevant use cases and natural language patterns.
5.5 Entity Resolution Errors
Automated entity resolution may introduce errors through incorrect alias mappings or failure to distinguish between similarly-named entities. A full audit of the October 2026 cycle found and corrected such errors in 2,299 of its 2,784 questions (§6, Corrections). Earlier cycles have not been re-audited and likely contain errors of the same kinds.
6. Changelog & Corrections
The methodology is version-controlled. Each revision below is immutable and dated; results should always be interpreted against the methodology version in effect at poll time. Versions follow semantic numbering (major.minor): a major bump denotes a change that breaks longitudinal comparability (e.g., scoring formula), a minor bump denotes the model set, cadence, weights, or entity resolution changing. Changes are announced before they take effect, with their estimated magnitude. Every score and ranking snapshot records the version that produced it.
- Only answers are ranked. GenRank ranks the named things AI models recommend: brands, products, organizations, people, places and works. Generic terms (product types, ingredients, features, advice) are no longer ranked when a question asks for named things. Websites, publications and platforms that an answer only cites or points to are no longer ranked as answers. Both are still recorded with each response (§2.3).
- Entity matching. Uncertain matches are decided by a language-model judge against the question and the answer text, and every decision is recorded. Rejected matches are never re-applied, and distinct items named in the same answer are never merged (§2.3).
- Citations stored. The web sources each model cites are now stored with every response.
- Question corpus. 20 questions whose answers named only generic terms, never a specific entity, were retired. Each was replaced by a brand-focused question in the same category. Two questions, each asked in English and Korean, were added for the platforms people use to find daycares and pet groomers. Retired questions keep their pages and history, but their scores no longer count toward any ranking.
- Estimated magnitude. Measured on the October 2026 cycle, where these rules were applied together with the error corrections below:
- category top-10s kept 8.5 of their 10 entities on average;
- per-question top-10s kept 83% of their entities;
- 13.7% of score rows were removed;
- total score mass fell 11.5%.
- Timing. The pipeline changes are deployed now, so the 24 questions added on 6 October are first polled under v3.3, on 7 October. Every other question moves to v3.3 with the 1 November cycle.
- Rationale: a ranking must contain only answers to the question asked. A source an answer cites, or a generic term it uses, is not a recommendation.
- Provider weights re-derived from published market data via the §4.1 composite and the full derivation published (§4.1.1): OpenAI 0.30 → 0.41, Anthropic 0.25 → 0.30, Google 0.20 → 0.29.
- Perplexity and xAI retired from the active set: their derived composites (0.01 and 0.03) fall below the 3% materiality threshold (§4.1.1) — weights at which polling cost is incurred without measurable influence on published scores. Historical Sonar and Grok poll data is preserved.
- Per-model score contributions now stored with every score, enabling published weight-sensitivity analysis: rankings can be recomputed under any weight set.
- Rationale: weights must be reproducible from cited sources, not asserted. The previous allocation materially over-weighted xAI and Perplexity relative to every published usage measure.
- Standard cycles poll each question–model pair once; sampled cycles (n = 3) run on demand and publish confidence intervals (§4.5).
- Model set refreshed to the current consumer tier per provider: GPT-5.6 Luna (replacing GPT-5.4 mini), Claude Sonnet 5 (replacing Claude Haiku 4.5), Gemini 3.7 Flash (replacing Gemini 3.1 Flash Lite). Grok 4.3 and Sonar unchanged.
- Provider weights unchanged (OpenAI 0.30, Anthropic 0.25, Google 0.20, xAI 0.15, Perplexity 0.10); retired models keep their historical poll data.
- Repeated sampling introduced: a cycle may poll each question–model pair n times; the September 2026 cycle was the first sampled cycle (n = 3 for part of the corpus).
- Confidence intervals published where sampling depth allows; absent intervals are reported as absent, never as zero-width (§4.5).
- Inclusion rate adopted as the primary metric; position within an answer retained as a secondary quality signal.
- Model contributions averaged before weighting, keeping scores comparable with all earlier cycles.
- Gemini 3 Flash replaced by Gemini 3.1 Flash Lite(provider's successor model); weight unchanged at 0.20.
- Active model set reduced to five web-search-capable models, one per major provider (OpenAI, Anthropic, Google, xAI, Perplexity). Models without native real-time web search were retired.
- Web search enabled at poll time so recommendations reflect up-to-date information rather than training-cutoff knowledge.
- Polling cadence changed from weekly to monthly (1st of each month, UTC).
- Provider weights reallocated: OpenAI 0.30, Anthropic 0.25, Google 0.20, xAI 0.15, Perplexity 0.10.
- Rationale: data recency, representative provider coverage, and cost efficiency. Historical poll data from retired models is preserved for longitudinal comparison.
- Scoring framework formalized using nDCG-style logarithmic decay (Definition 3.1).
- Market-weighted active set of 16 models across nine providers; weights recalibrated quarterly.
- Initial public methodology: multi-model polling with market-weighted aggregation; weekly cadence.
Corrections
Substantiated errors in published numbers are corrected, and each correction is logged here with its cause. Correcting a mismeasured entity is always free.
- Review. All 76,418 question–entity scores from the 1 October 2026 cycle were audited. Each was checked against its question, the entity record and the answer text by two language-model reviewers. One checked that the mention was credited to the right entity; the other looked for duplicates, cited sources and generic terms. Every flagged error was independently verified before it was corrected, and merges required two independent confirmations.
- Errors found and corrected.
- Mentions credited to the wrong real-world entity, such as a homonym, a sibling product or a parent brand, often through a wrong alias. For example, the stablecoin issuer Circle had been credited to the community platform Circle.so. 5,722 scores were re-attributed, 1,135 missing entities were created, and 1,622 wrong aliases were removed.
- 1,736 duplicate entity records were merged (e.g. “Costco” and “Costco Wholesale”).
- 10,598 scores were removed, mostly cited websites (e.g. rtings.com) and generic terms that had been ranked as answers.
- Effect. 2,299 of 2,784 questions changed. Category top-10s kept 8.5 of their 10 entities on average, and per-question top-10s kept 83%. Total score mass fell 11.5%. The October history snapshots were rebuilt to match.
- Cause. The pipeline had five flaws:
- it consulted its list of rejected matches in only one fallback path;
- it extracted citation links as entities;
- it grouped distinct products named in one answer;
- it accepted aliases naming a parent brand, a bare ticker or a generic word;
- it extracted generic terms as entities.
- Scope. Cycles before October 2026 were not re-audited. Their published history is unchanged and likely contains errors of the same kinds.
7. References
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.
- Shah, C., & Bender, E. M. (2024). Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web? ACM Transactions on the Web.
- Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.
- Wilkinson, M. D., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.
- Menlo Ventures. (2025). 2025 Mid-Year LLM Market Update: Foundation Model Landscape + Economics.
- OpenRouter. (2025). LLM Rankings: Model Usage Statistics.
- Statcounter. (2025). AI Chatbot Market Share Worldwide.