The Surprising Secrets Hidden in the Entropy of a Mixture
This article delves into the relationship between the entropy of a mixture of probability density functions and its interpolation factor. The author reveals that entropy, as a function of probabilities, is concave, and this concavity is directly tied to the mutual information between the two distributions. By introducing a Bernoulli variable and the concept of conditional entropy, the article elegantly explains how mutual information quantifies the change in the expected surprisal of a prediction given knowledge of the mixture factor. Furthermore, it introduces a novel concept, 'proclivity', connecting it to KL divergence and cross-entropy. The article also discusses Jensen-Shannon divergence and the Neyman χ² divergence appearing in higher-order Taylor expansions. Ultimately, it concludes that the entropy function of the mixture completely describes the distribution of likelihood ratios between the two probability distributions, offering a fresh perspective on understanding the relationship between probability distributions.