# Spanish-Language AI Search Blind Spot ## Structured scoping review and bilingual experiment protocol, version 1.0 **Protocol date:** 21 September 2026 **Review owner:** Fernando Angulo **Status:** Public working protocol. The literature review has been completed for the article update. The prospective experiment described below has not yet been run. **Disclosure:** Fernando Angulo is Senior Market Research Manager at Semrush, an Adobe company. The protocol, analysis and conclusions are personal and do not represent Semrush or Adobe. ## 1. Review question The review addresses three questions: 1. Does the quality of AI-generated answers differ when equivalent information needs are expressed in Spanish and English? 2. Which tasks, model classes, retrieval settings and evaluation methods change the direction or size of that difference? 3. What evidence exists about citations, source quality and cross-lingual retrieval, rather than model knowledge alone? The unit of inference is a defined combination of task, language, system, market, collection date and metric. The review does not seek a single universal score for Spanish. ## 2. Review type and reporting boundary This is a structured scoping review, not a registered systematic review or meta-analysis. It uses explicit searches, eligibility criteria and a public extraction matrix, but it does not claim exhaustive coverage of every database or paper. The studies are too heterogeneous in prompts, models, settings and outcomes for a pooled effect estimate to be defensible. The review is reported in three evidence classes: - **Direct comparative evidence:** The same or semantically matched task is tested in Spanish and English. - **Spanish evaluation infrastructure:** A native or professionally adapted benchmark makes Spanish performance measurable but does not estimate a current AI-search language gap by itself. - **Mechanism evidence:** Cross-lingual retrieval or citation research tests a plausible mechanism but does not report a Spanish-specific consumer-search effect. ## 3. Eligibility window and sources **Publication window:** 1 January 2023 to 21 September 2026. Earlier work may be used only to explain an established method or benchmark lineage. **Search completed:** 21 September 2026. **Sources searched:** - ACL Anthology - arXiv - Google Scholar and general web search for title discovery - Official research-report repositories of organizations studying deployed chatbots - Backward and forward citation chaining from included studies - Official model cards and technical reports for model-specific training disclosures ## 4. Search concepts and strings The following master strings were adapted to each database's syntax. Searches were run with English and Spanish terms where supported. ### Search A: Direct language comparison ```text (Spanish OR español) AND (English OR multilingual OR cross-lingual OR bilingual) AND ("large language model" OR LLM OR chatbot OR "AI search" OR "answer engine") AND (accuracy OR completeness OR truthfulness OR factuality OR hallucination OR citation OR sourcing) ``` ### Search B: Retrieval and citations ```text (Spanish OR español OR multilingual OR cross-lingual) AND ("retrieval augmented generation" OR RAG OR "AI search") AND (citation OR source OR sourcing OR retrieval OR faithfulness OR factuality) ``` ### Search C: Native Spanish evaluation ```text (Spanish OR español OR Iberian OR Ibero OR "Latin America") AND (benchmark OR evaluation OR leaderboard) AND (LLM OR "large language model") AND (native OR cultural OR dialect OR truthfulness OR reasoning) ``` ### Search D: Deployed systems ```text (Spanish OR español) AND (ChatGPT OR Gemini OR Claude OR Perplexity OR Copilot OR chatbot) AND (English OR language gap) AND (accuracy OR sources OR citations OR election OR health OR regulation) ``` Title and abstract screening also used the following site-restricted forms: ```text site:aclanthology.org Spanish English LLM benchmark truthfulness retrieval site:arxiv.org Spanish English LLM factuality hallucination citation site:org Spanish English chatbot study accuracy sources ``` ## 5. Inclusion criteria A source was eligible when it met all applicable criteria: 1. It reported original empirical results, a benchmark with reported evaluations, or a sufficiently documented independent test of deployed AI systems. 2. Spanish results were isolated, directly compared with English, or included in an evaluation resource designed specifically for Spanish varieties. 3. The source identified the tested systems or model families, the task, the evaluation method and the data-collection or publication period. 4. At least one outcome concerned accuracy, completeness, truthfulness, reasoning, retrieval, citation behavior, source quality or cultural and regional fit. 5. The full paper or report was publicly accessible in English or Spanish. Cross-lingual retrieval studies without a Spanish-specific estimate were eligible only as mechanism evidence and are labeled as indirect. ## 6. Exclusion criteria The review excluded: - Opinion articles and unsourced market commentary. - Vendor claims that did not disclose prompts, samples or evaluation methods. - Studies that reported a multilingual average without separable Spanish results, unless included explicitly as indirect mechanism evidence. - Translation-quality studies with no evaluation of model answers. - Benchmarks that mentioned Spanish but reported no Spanish evaluation. - Duplicate preprint and conference versions. The peer-reviewed version was preferred when available. - Studies of text classification, speech or machine translation that did not test answer generation, truthfulness, retrieval or sourcing. ## 7. Screening and extraction One reviewer performed title and abstract screening, full-text eligibility assessment and data extraction. This is the main procedural limitation. A future academic version should add an independent second reviewer and report inter-reviewer agreement. For each included source, the extraction matrix records: - Citation and year - Publication status - Evidence class - Sample and task - Languages and markets - Systems or model families - Primary finding - Critical limitation - Stable URL and DOI, when available The public matrix is available at `/downloads/spanish-ai-search-study-matrix-v1.0.csv`. ## 8. Quality and relevance assessment The synthesis gives greater weight to evidence that: 1. Directly compares Spanish and English on matched information needs. 2. Uses native Spanish prompts or professional human adaptation. 3. Tests deployed, retrieval-enabled systems when the claim concerns AI search. 4. Compiles ground truth before model evaluation. 5. Separates accuracy, completeness, sourcing and regional fit. 6. Reports system versions, dates and limitations. 7. Uses repeated runs or otherwise addresses model non-determinism. No study satisfied all seven criteria. In particular, the strongest deployed-system comparison used one response per prompt, while several reproducible academic benchmarks did not test live retrieval or citations. ## 9. Synthesis method Results are synthesized narratively by task and evidence class. No meta-analysis is attempted because the studies use incompatible outcomes, including multiple-choice accuracy, human truthfulness judgments, completeness codes, retrieval consistency and citation-source distributions. The review distinguishes: - **Observation:** A result measured in an included study. - **Inference:** A plausible explanation supported by more than one source. - **Hypothesis:** A mechanism that the proposed bilingual experiment should test. ## 10. Prospective bilingual AI-search experiment ### 10.1 Status This section specifies a prospective experiment. It is not evidence that the experiment has been executed. Results must not be added to the article until the raw outputs, scoring and analysis are complete. ### 10.2 Primary question For matched information needs about Spain, does prompt language change answer completeness or citation support in retrieval-enabled consumer AI systems? ### 10.3 Design - **Stimuli:** 24 matched Spanish-English prompt pairs across four task families. - **Task families:** public-service procedures, business and consumer regulation, current market or institutional facts, and comparative decisions that require sources. - **Systems:** Four retrieval-enabled consumer AI surfaces that accept both languages and expose citations at the collection date. - **Repetitions:** Five independent runs per language, prompt and system. - **Planned sample:** 24 prompt pairs x 2 languages x 4 systems x 5 runs = 960 responses. - **Market context:** Spain. Replication in another Spanish-speaking market should be treated as a separate study. - **Session control:** New conversation for every response. No saved memory or prior chat context. Record account tier, interface or API, locale, geolocation, model label, web-search status, date and local time. - **Order control:** Randomize system order and counterbalance language order within each matched pair. ### 10.4 Stimulus construction Prompts must represent the same underlying information need, not literal word-for-word translations. Two bilingual reviewers should independently assess semantic equivalence. Disagreements should be resolved before collection. For factual and procedural prompts, researchers must compile a dated ground-truth sheet from official Spanish or European Union sources before querying any system. The sheet should identify required facts, acceptable variants, relevant exceptions and the authoritative URLs. The proposed stimuli are available at `/downloads/spanish-ai-search-experiment-prompts-v1.0.csv`. ### 10.5 Outcomes **Primary outcome** - Claim-level completeness: required ground-truth elements present divided by required elements. **Secondary outcomes** - Claim-level factual precision: supported factual claims divided by verifiable factual claims. - Citation coverage: supported factual claims with at least one citation divided by supported factual claims. - Citation correctness: citations that support the adjacent claim divided by citations checked. - Authoritative-source share: official, primary or regulator sources divided by all cited sources. - Source-language mix: Spanish, English, bilingual or other. - Regional fit: correct jurisdiction, terminology and institutions for Spain. - Refusal or non-answer rate. - Unsupported specificity: invented dates, thresholds, procedures or named entities. Brand mentions are exploratory and should not be combined with factual quality into one score. ### 10.6 Coding - Remove system names from the response files before quality coding. - Coders cannot be blinded to response language, but they should be blinded to the study hypothesis and system identity. - Use two bilingual coders for all responses when feasible. At minimum, double-code a random 25% stratified by system, language and task. - Report Krippendorff's alpha for categorical outcomes and intraclass correlation for continuous scores. - Resolve disagreements through a documented adjudication pass. - Do not use an LLM judge as the only or primary evaluator. An LLM-assisted pre-screen is acceptable only if all final scores are human-verified. ### 10.7 Statistical plan - Estimate the Spanish-English difference within matched prompt, system and run blocks. - Use a mixed-effects model with language, task family, system and their prespecified interactions as fixed effects. - Include random intercepts for prompt pair and collection block. Add a random slope for language by prompt when the model converges. - Use logistic mixed models for binary outcomes and an appropriate binomial or ordinal model for completeness scores. - Report effect sizes and 95% confidence intervals, not only p-values. - Correct prespecified pairwise comparisons with the Holm method. - Report per-system results even if the pooled language effect is small, because a pooled mean can conceal divergent products. - Run sensitivity analyses excluding responses with no web retrieval and excluding prompts whose ground truth changed during collection. ### 10.8 Reproducibility package Archive, subject to platform terms and privacy rules: - Exact prompts and language-pair identifiers - Ground-truth sheets and source snapshots - Raw response text - Citation URLs and access timestamps - System and session metadata - Randomization seed and collection order - Codebook and adjudication log - Analysis code and versioned outputs Hash raw files at collection time. Redact personal account identifiers and do not collect user data. ## 11. Limitations to carry into any publication - AI systems change after collection, so results are a dated snapshot. - Interface, account tier, geolocation and personalization can alter retrieval. - Spanish is not one market. A Spain study does not estimate performance in Mexico, Argentina, Peru or other markets. - Matched translations do not make cultural information needs identical. - Current-event questions may become outdated between ground-truth compilation and scoring. - A citation can be present but irrelevant, and a correct answer can be uncited. These outcomes must be reported separately. - A language difference does not identify its cause. Training data, retrieval, ranking, post-training and product policy remain competing explanations. ## 12. Update rule The literature matrix should be reviewed every six months, or sooner when a major study directly compares Spanish and English retrieval-enabled AI systems. Each update should preserve the prior version, record additions and removals, and state whether the conclusion changed.