Mitigating Hallucinations in Large Language Models via Contrastive Decoding and Entropy Minimization

Abstract

The proliferation of Large Language Models has revolutionized natural language processing, yet the persistent issue of hallucination—where models generate plausible but factually incorrect information—remains a critical impediment to their deployment in high-stakes environments. This paper introduces a novel decoding framework that synergizes Contrastive Decoding with Entropy Minimization to enhance factual reliability without sacrificing linguistic fluency. While traditional decoding strategies such as nucleus sampling rely heavily on probability truncation, they often fail to distinguish between high probability generic responses and contextually accurate specifics. Our approach leverages a dual-model system, pitting an expert model against an amateur model to penalize tokens that are statistically probable but conceptually vacuous, while simultaneously employing an entropy-based regularization term to suppress generation during states of high uncertainty. Through extensive experimentation on standard factual benchmarks, we demonstrate that this composite method significantly reduces hallucination rates compared to state-of-the-art baselines. The results suggest that addressing the stochastic flaws of auto-regressive generation requires not only better training data but also more rigorous, inference-time intervention mechanisms that account for the semantic entropy of the output distribution.