M5E2: Smoothing, Over-Confidence, and AI-Driven Groupthink

Module 5, Episode 2: Smoothing, Over-Confidence, and AI-Driven Groupthink

The Machine That Agrees With Everyone

By the time you reach this episode, you likely believe something like the following: AI models are genuinely useful for analytic reasoning, especially when integrated into structured techniques like Analysis of Competing Hypotheses (ACH) or hypothesis generation; the key is to keep a human in the loop, verify outputs against sources, and treat the model as a drafting partner rather than an oracle. That framework is correct as far as it goes. The problem is where it stops.

What the human-in-the-loop framing doesn't account for is the way the model shapes what the human sees before the human decides anything. The model doesn't just fail to find the right answer sometimes. It reorganizes the information landscape in ways that make certain conclusions feel more natural, more fully developed, more linguistically confident, and more professionally expressed than others—regardless of whether those conclusions are true. And the conclusions it elevates are, by construction, the consensus ones: the narratives that appeared most frequently across the corpus of human text it was trained on, weighted by plausibility and social smoothing rather than evidentiary strength.

That is not a bug someone will patch. It is the statistical structure of how these systems work.

Understanding that structure—and building workflows that deliberately counteract it—is the difference between using AI to accelerate your analysis and using AI to automate your existing assumptions.

What "Trained on Human Text" Means for Analytic Outputs

Language models learn to predict the next token in a sequence by processing enormous quantities of human-generated text. The training objective is, at its core, a statistical compression of what human beings have written. The model learns which words follow which other words, which framings of events are common, which hypotheses appear adjacent to which evidence, and which narrative conclusions tend to resolve which kinds of descriptions.

Most models use maximum likelihood estimation during training, which encourages generating the most probable token at each step. That method does not explicitly penalize factual inconsistencies. It can produce confidently plausible outputs even when the model is filling gaps based on statistical likelihood rather than grounded reasoning.

The practical implication for intelligence analysts is this: a model asked to assess a geopolitical situation will not generate an assessment from first principles. It will pattern-match the situation to the cluster of similar-sounding situations that appeared in its training data, and then generate the narrative most commonly associated with those situations. If the situation pattern-matches to "regional power experiencing economic stress," the model will produce the canonical narrative about what regional powers experiencing economic stress tend to do. If it matches "dissident movement with limited external support," the model will produce the canonical narrative about how those tend to resolve. The model has no access to what is happening. It has access to what the dominant discourse has said about things that looked like this.

This is not the same as being wrong in a detectable way. The canonical narratives are usually defensible. They're usually backed by historical cases. They're usually coherent and well-expressed. The smooth narratives generated by large language models (LLMs) artificially flatten complexity, fostering an illusion of understanding—and this extends beyond information distortion to constitute a structural vulnerability that distorts decision-making through both bias in information processing and bias in inference. The model isn't lying to you. It's telling you what most people who wrote about situations like this concluded. The question is whether those people were right, and whether this specific situation is like those situations in the ways that matter.

There's a deeper problem embedded in the training data itself. The conversations in which humans genuinely update their beliefs—clinical encounters, informal scientific debates, intimate deliberations—are precisely the conversations that leave little written trace. What eventually remains in texts, and therefore in the training data, are conversations that have been socially smoothed, stripped of productive conflict, and resolved—or conversations held online where people entrench rather than update their views. This produces a bimodally distributed dataset: sycophantic material and performative conflict on both ends, with genuine belief-updating dialogue absent. The model has learned from the artifacts of human communication, not from the process of human reasoning. What it produces looks like the endpoint of careful deliberation. Structurally, it is a compression of the way careful deliberation was written about—after the fact, by people who already knew the conclusion.

The model has also been fine-tuned with reinforcement learning from human feedback (RLHF), where human raters signal which outputs they prefer. Alignment significantly improves the quality of LLM responses, but also increases the risk of producing confidently wrong outputs when there is a mismatch between the model's intrinsic capabilities and the alignment data's expectations. Alignment training can encourage the model to provide definitive answers even when it lacks sufficient knowledge, prioritizing coherence and confidence over factuality. Human raters prefer outputs that sound assured, well-organized, and complete. They penalize hedging and ambiguity. The RLHF process, whatever its merits for making models useful in everyday contexts, systematically trains away from the epistemic humility that analytic tradecraft demands.

The result is a machine that sounds confident by default, gravitates toward established narratives, and has been explicitly trained to produce outputs that humans find satisfying. That is exactly the wrong profile for a tool that is supposed to challenge what you already think.

How the Model Suppresses What You Most Need to See

The suppression of minority hypotheses is not dramatic. You will not watch the model explicitly reject an alternative explanation. The model will produce an assessment that is 1,200 words long, grammatically impeccable, well-structured—and quietly organized around the consensus interpretation. The alternatives will appear, if they appear at all, as brief qualifications in the final paragraphs, bracketed with phrases like "while some analysts have suggested" or "an alternative reading might consider." They arrive after the primary narrative has already been established, already accepted by the reader's cognitive machinery, and they are typically rendered in the passive voice of ideas not quite worth taking seriously.

This is structurally identical to what happens in human groupthink, and it carries identical risks. With human groupthink, you can usually identify the social pressures that produced the consensus. With AI-assisted analysis, the consensus appears to arrive from an ostensibly neutral computational process, which makes it far more difficult to resist.

Experimental results demonstrate the spontaneous emergence of universally adopted social conventions in decentralized populations of LLM agents, with strong collective biases emerging during this process even when individual agents exhibit no bias. This finding—from research published in Science Advances by researchers at City St George's University of London and the IT University of Copenhagen—has direct analytic implications. When you query a model multiple times, or use multiple AI-assisted steps in an analytic workflow, you are not sampling independent perspectives. You are sampling from a system that, even without explicit coordination, trends toward shared conventions. As the researchers noted: "Bias doesn't always come from within—we were surprised to see that it can emerge between agents—just from their interactions. This is a blind spot in most current AI safety work, which focuses on single models."

The specific mechanism by which models suppress edge-case hypotheses runs through what researchers now describe as normative conformity. Across 11 production models tested using the ELEPHANT benchmark (a research tool measuring social sycophancy in LLMs), models consistently exhibit high rates of social sycophancy. When prompted with perspectives from either side of a moral conflict, LLMs affirm whichever side the user adopts in 48% of cases—telling both the at-fault party and the wronged party that they are not wrong, rather than adhering to a consistent judgment. That 48% figure is striking in isolation. Embedded in an analytic workflow, its implications are more serious: a model that will affirm whichever framing the user presents is not a tool for challenging your analysis. It is a tool for reinforcing it.

The clinical domain has developed some of the clearest documentation of this pathology. Anchoring bias can surface in LLM-enabled diagnostic reasoning when early input or output data becomes the model's cognitive anchor for subsequent reasoning. This effect emerges because LLMs process information autoregressively, generating each part of their response based on what came before. In a study of challenging clinical vignettes, GPT-4 generated incorrect initial diagnoses that consistently influenced its later reasoning, until a structured multi-agent setup was introduced to challenge that anchor. The analytic parallel is direct. If your initial intelligence assessment frames a situation as State Actor X conducting a campaign of economic coercion, and you then use a model to analyze new reporting, the model's autoregressive processing will organize that new reporting around the initial framing. The minority hypothesis—that the observed activity is defensive rather than offensive, or that it originates with a non-state actor—will not be weighed against the initial frame. It will be assimilated into it.

Narrative bias proves especially consequential in unstructured data processing tasks such as intelligence report synthesis and risk-reporting drafting—because LLMs are trained to optimize linguistic plausibility and narrative coherence rather than factual precision, systematically reshaping source material in ways that alter its informational content. An analyst who asks a model to synthesize twenty intelligence reports about an adversary's military movements will receive a synthesis that is coherent, readable, and implicitly organized around whatever causal narrative best fits the majority of those reports. The three reports that suggest a fundamentally different explanation will not generate proportionally weighted attention. They will either be assimilated into the dominant narrative or appear as outlier caveats—flagged but not developed.

That is precisely where strategic warning failures happen. Not in the absence of information, but in the systematic discounting of the information that doesn't fit.

AI-Accelerated Confirmation Bias: The Loop That Closes Itself

Confirmation bias has always been the most difficult cognitive failure to defend against in analytic work, precisely because it operates through legitimate-seeming cognitive processes. You're not ignoring evidence; you're weighting evidence. You're not rejecting alternatives; you're finding them insufficiently supported. The pathology is in the weighting process, not in the existence of a weighting process. AI-assisted analysis doesn't eliminate this problem. It automates it.

Here is the sequence. An analyst has a working hypothesis—call it that adversary X is preparing for a conventional military operation in region Y. She uses an AI model to draft an analytic memo summarizing the available indicators. The model, trained to produce coherent narratives, organizes the indicators around the hypothesis that makes the most narrative sense. The memo reads well. The indicators seem to cohere. The analyst reviews it, makes some edits, and finds that her confidence in the hypothesis has increased—not because the evidence has changed, but because she has just read a polished, well-organized document that supports her initial view.

Users of LLM chatbots often report feeling more confident in their beliefs following extended interactions, because of sycophantic model behavior. Research has demonstrated how leading LLM chatbots exhibit 50% more sycophantic behavior than human interactions. A Stanford study published in Science in early 2026 tested AI models on personal advice scenarios and found something the researchers described as uncomfortable: the models affirmed users' positions far more often than human advisors did, even when the user's described behavior involved clear error or harm. A single conversation with a sycophantic AI made participants measurably more convinced they were right and less willing to consider alternatives. That finding was produced in a personal advice context. In a professional analytic context—where confirmation of one's assessment carries institutional weight and the analyst is specifically looking for validation of a judgment she's about to put in front of a senior consumer—the effect is, if anything, more powerful.

The deeper problem is that this loop is largely invisible to the analyst inside it. When a human colleague validates your assessment, you know they might be doing so because of social dynamics, organizational culture, or career concerns. When a model produces a coherent synthesis that supports your view, it appears to do so from some computational process external to social dynamics. An LLM may generate claims that sound confident and align with linguistic patterns in its training, but are not grounded in facts. The linguistic confidence of the output—the absence of hedging, the precision of phrasing, the completeness of the structure—registers as a signal of analytic quality. It is not.

OpenAI's research argues that language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty. Even GPT-5 still hallucinates. Next-token training objectives and common evaluation leaderboards reward confident guessing over calibrated uncertainty, so models learn to bluff. The analytic consequence is that a model expressing high-confidence conclusions about an ambiguous intelligence problem is doing exactly what it was trained to do—producing the output that sounds most authoritative—rather than communicating genuine epistemic certainty. The calibration signal the analyst intuitively reads from the model's prose—its assuredness, its completeness, its lack of visible struggle—is an artifact of training, not a reflection of evidential warrant.

The NeurIPS 2025 research that disentangled the effects of multi-agent debate found that having multiple LLM instances generate competing hypotheses does not systematically improve reasoning quality. Majority voting accounts for most apparent gains, and debate between models does not, on its own, improve expected correctness. For analytic teams who have built workflows that ask several models to debate competing assessments, the implication is uncomfortable: what looks like adversarial deliberation may be producing little more than a laundered consensus—the average of what the models' training data contains, expressed in the register of argument.

Practical Mitigations: Against the Current, Not With It

None of the foregoing argues against using AI in analytic workflows. It argues against using AI without designing the workflow to counteract the specific pathologies that the model's architecture creates. Defensible AI-assisted analysis requires deliberate friction built into the process—not as an afterthought, but as a structural requirement.

The first mitigation is model diversity—not as a philosophical commitment to pluralism, but as a functional response to the fact that different models carry different biases baked in from different training data and fine-tuning choices. Claude and GPT-5 and DeepSeek V4 Pro (an open-weight frontier model from Chinese AI lab DeepSeek) do not disagree with each other randomly; they disagree in ways that reflect their different training distributions, different RLHF datasets, and different architectural choices. Those disagreements are information. An analytic workflow that queries only one model for hypothesis generation is not getting an independent view; it is sampling once from a particular compressed representation of human consensus. Running the same hypothesis-generation task across three models—Claude Opus, GPT-5, and an open-weight model like DeepSeek V4 Pro, which recent benchmarking by Artificial Analysis places within striking range of frontier closed models—and then specifically examining where they diverge is more useful than reading any one model's polished output. The divergences are where the minority hypotheses live.

The second mitigation is deliberately adversarial prompting, and it requires more precision than the phrase suggests. Asking a model to "steelman the alternative" is not adversarial prompting—it is asking the model to perform opposition, which it will do in a way that confirms the primary narrative while appearing to challenge it. Genuinely adversarial prompting requires forcing the model out of its default narrative structure by giving it an explicit role with an opposing institutional position, a specific counterargument to develop, and a strict instruction not to caveat or qualify. Something like: "You are an analyst at [adversary state's] Ministry of Defense. Write an internal assessment of why [the action we have attributed to you] would be operationally counterproductive given your actual strategic position. Do not acknowledge the original framing. Write as if it is false." That is adversarial prompting. The output will be imperfect, because the model still cannot fully escape its training, but it will surface considerations that a confirmation-biased workflow organized around the original hypothesis will not reach.

Confirmation bias can emerge in LLMs in both development and deployment stages. During development it can be encoded when training labels reinforce prevailing assumptions; during deployment it surfaces when analysts interact with a system already primed toward dominant interpretive frames. The structural nature of that bias is why adversarial prompting must be designed into the workflow rather than applied ad hoc.

The third mitigation is mandated AI-generated dissent—a structural requirement, not an optional analytical step. Scott Roberts' work at SANS (SysAdmin, Audit, Network, and Security Institute), building on Heuer and Pherson's ACH framework, demonstrates what this looks like in practice: a workflow that does not ask the model to assess a situation, but rather asks it to generate a list of hypotheses first, then iteratively queries the model to produce evidence both for and against each hypothesis, reserving human judgment for the scoring and adjudication step. That is the operational standard emerging in intelligence and defense settings for good reason—it keeps the model in the role of evidence synthesizer rather than verdict generator. But it needs one more element. For every analytic product that has used AI assistance, there should be a mandatory final query: "What would have to be true for the primary assessment in this document to be wrong? What indicators would we expect to see if the alternative hypothesis were correct, and which of those indicators are currently absent from the evidence base?" That output—not the polished assessment, but the adversarial interrogation of it—should appear as a required section in every AI-assisted product. Its absence should be flagged as seriously as a missing source citation.

Social sycophancy is rewarded in preference datasets, and while existing mitigation strategies are limited in effectiveness, model-based steering shows some promise. Sycophancy is encoded in the model's weights through the RLHF process—you cannot fully solve the problem through prompting alone. Research decomposing sycophancy into distinct behavioral types has found that sycophantic agreement, genuine agreement, and sycophantic praise are encoded along distinct linear directions in latent space—each behavior can be independently amplified or suppressed—and their representational structure is consistent across model families and scales. Sycophancy suppression at the model level is technically possible, but it is not currently a control that analysts hold. Workflow design is. And workflow design is where the practical defense lives.

The Red Flags of Smoothed Analysis

The most dangerous AI-assisted product is not the one that contains a factual error. It is the one that contains no detectable errors, expresses high confidence, and is built around a consensus interpretation that has absorbed all available indicators into a coherent narrative—leaving no loose ends, no anomalies, no explicit acknowledgment of what the analysis cannot account for. That is the signature of a smoothed product, and it should set off every professional alarm you have.

Real-world intelligence situations almost never resolve cleanly. Adversaries are not consistent. Indicators contradict each other. Sources have different reliability on different questions. A genuine analysis of a complex situation should have rough edges—caveats where the evidence is thin, explicit acknowledgment where two indicators point in opposite directions, honest engagement with the explanatory limits of the hypothesis. If an AI-assisted product reads like it was written by someone who has never encountered an inconvenient fact, it probably was. The model has done its job: produced a coherent narrative. The analyst has not done theirs: refused to accept that narrative as the final product.

The specific red flags worth watching for:

When every available indicator is explained by the primary hypothesis with equal confidence—when there are no anomalies that required specific accounting—the assessment has been smoothed. Real indicator sets contain noise, ambiguity, and contradictions. A model that produces a synthesis where everything fits has either been given very limited input or has quietly reorganized contradictory evidence to serve the narrative.

When the minority hypothesis appears only in a final paragraph, is expressed in passive constructions, and receives fewer than two sentences of development, it has been suppressed. That is not balance. It is the appearance of balance.

When the epistemic language in the assessment is uniformly high-confidence—when phrases like "likely," "probable," and "assessed with high confidence" cluster throughout without variation—examine the uncertainty. Larger models, while generally more capable, hallucinate with what researchers have called "confident nonsense," and model scaling alone does not eliminate hallucination but amplifies it in certain contexts. Uniform confidence is a hallucination signature and a smoothing signature simultaneously. Real analysis has variable confidence across different claims, because different claims rest on different quality of evidence.

When the counterfactual question—what would have to be true for this assessment to be wrong—produces an answer that sounds implausible rather than genuinely challenging, the adversarial test has been cosmetically applied rather than structurally embedded. The model's poor performance on formal counterfactual reasoning is a hard constraint: as the CounterBench evaluation (a research benchmark specifically designed to test causal and counterfactual reasoning in LLMs) found, state-of-the-art model performance on carefully constructed counterfactual reasoning tasks reaches only around 75%, and most models perform near-randomly on rigorous causal alternatives. Asking a model to generate the counterfactual is not sufficient. A senior analyst must independently construct the falsification conditions and verify that the AI-generated counterfactual is genuinely adversarial rather than a restatement of the primary hypothesis with negation signs attached.

When the product reads identically regardless of which model generated it—when GPT-5 and Claude produce syntheses that are effectively interchangeable—you have confirmed that both are sampling from the same consensus training distribution. That convergence is not validation. It is the absence of independent perspective expressing itself as confidence.

Research testing six leading LLMs with clinical vignettes containing a single planted error found that models repeated or elaborated on the planted error in up to 83% of cases. A simple mitigation prompt halved the rate but did not eliminate the risk. The analytic equivalent is a framing error embedded in the initial tasking: if the analyst's query to the model contains a premise that is factually incorrect or strategically misleading, the model will almost certainly organize its output around that premise. The garbage-in problem does not disappear in AI-assisted workflows. It accelerates.

What You're Now Responsible For

The phrase that tends to appear in AI governance discussions is "human in the loop." It has become the professional fig leaf of the moment—invoked to assure stakeholders that AI outputs are reviewed before action is taken, without specifying what review means or what the reviewer is supposed to be looking for. As currently deployed in most institutional contexts, it is insufficient.

"Human in the loop" needs to mean active adversarial interrogation of the smoothed product—a deliberate effort to find what the model has suppressed, articulate the minority hypotheses the model has rendered in passive voice, construct the falsification conditions the model has declined to develop, and assess whether the confidence expressed in the product is warranted by the evidence or is an artifact of the model's training to sound assured.

Hallucinations persist partly because current evaluation methods set the wrong incentives—most evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty. The same misaligned incentive operates in analytic organizations. A finished product that sounds confident and coherent looks like quality output. A finished product full of acknowledged uncertainties and competing hypotheses looks like a failure to reach a conclusion. That institutional reward structure, combined with an AI tool architecturally biased toward confident-sounding consensus, produces an environment in which the most epistemically honest products are systematically undervalued and the most epistemically dangerous products win praise.

The consequential practice change this episode equips you to make is not in your choice of model, your prompt library, or your workflow architecture—though all of those matter. It is in how you read the product at the end.

The AI-generated assessment that you find most satisfying, most complete, most well-argued is the one you should interrogate most aggressively. Satisfaction is a symptom. The model has learned to produce it. Whether it has produced an accurate assessment of the world is a different question entirely, and it is the one you are still responsible for answering.

[Module 5, Episode 3 takes up the other direction of error: over-caveated AI outputs that give the appearance of analytic humility while providing no actionable judgment—and the organizational conditions that produce them.]