entropy of the sum of binomial distributions

Question

Suppose that one has $X_1 \sim Bin(n,p)$ and $X_2 \sim Bin(n,1-p)$ and that $Z$ is distributed s.t:

$$ P(Z = k) = .5 P(X_1=k) + .5P(X_2=k) $$

How do we compute the entropy? For a binomial distributed variable i have seen this proof:

Entropy of a binomial distribution

let $A = p^k (1-p)^{n-k} + (1-p)^k (1-p)^{n-k}$ then we see that:

\begin{align} H(Z) &= -.5 \sum {n \choose k} \Big[A\Big] \log \Big[.5{n \choose k}A\Big] \\&= 1 - .5\sum {n \choose k} \Big[A\Big] \log \Big[{n \choose k}\Big] - .5\sum {n \choose k} \Big[A\Big] \log \Big[A\Big] \end{align}

Using de-Moivre-Laplace theorem, i get:

\begin{align*} 1+\log_2(\sqrt{2\pi}\sigma) + \int_{-\infty}^{\infty} \Big[e^{{\frac{(x-\mu_1)^2}{2\sigma^2})}}+e^{{\frac{(x-\mu_2)^2}{2\sigma^2})}}\Big] \log_2(e^{{\frac{(x-\mu_1)^2}{2\sigma^2})}}+e^{{\frac{(x-\mu_2)^2}{2\sigma^2})}}) \end{align*}

where $\mu_1 = np, \mu2 = n(1-p)$ and $\sigma^2 = np(1-p)$ I tried to get the integral on the right hand side using Mathematica but that failed. Any suggestions on how to get the exact or approximation of this integral? What i tried is this:

\begin{align*} R &= \int_{-\infty}^{\infty}\frac{1}{2\sigma\sqrt{2 \pi}}\Big[e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\Big]\log_2\left(e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\right) \\ &= \int_{-\infty}^{\infty}\frac{1}{2\sigma\sqrt{2 \pi}}\Big[e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\Big]\Big[\log_2\left(e^{\frac{(x-\mu_{1})^2}{2\sigma^2}}\right)+\log_2\left(1+e^{\frac{-(2p-1)(n-2x)}{2(p-1)p}}\right)\Big] \\& \approx \frac{1}{2\sigma\sqrt{2 \pi}}\int_{-\infty}^{\infty}\Big[e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\Big]\log_2\left(\left(e^{\frac{(x-\mu_1)^2}{2\sigma^2}}\right)\right) + \frac{1}{\sigma\sqrt{2 \pi}} \int_\frac{n}{2}^{\infty}\left(e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\right) e^{\frac{-(2p-1)(n-2x)}{2(p-1)p}} \\&= \frac{1}{4}\log_2(e)\left(1 + (\sigma^2 + (n-2\mu_1)^2)\right) + \frac{1}{\sigma\sqrt{2 \pi}} \int_\frac{n}{2}^{\infty}\left(e^{-\frac{(x-\mu_1)^2}{2\sigma^2}}+e^{-\frac{(x-\mu_2)^2}{2\sigma^2}}\right) e^{\frac{-(2p-1)(n-2x)}{2(p-1)p}} \end{align*}

The approximation

$\log_2(1+e^{\frac{-(2p-1)(n-2x)}{2(p-1)p}}) \approx e^{\frac{-(2p-1)(n-2x)}{2(p-1)p}} \text{if } x \in [\frac{n}{2},\infty)$ and since the function is symetrical around $\frac{n}{2}$ i can just multiply the integration by 2. The right hand side gives me an ugly term so i was wondering if an approximation there could work?

Are there more theorems that i should consider looking at?

Answer 1

For small $n$, you just compute it numerically.

If you need an approximation for large $n$, (and $p$ not too close to $0$ or $1$) you can rely on the approximation for a single Binomial distribution from the linked question:

$$H(B;n,p)=\frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big) + O \left( \frac{1}{n} \right) \tag{1}$$

You can combine this with the known expression for the entropy of a mixture: Let $Z$ be a rv chosen from one of two rvs $X,Y$ with probabilities $\alpha,1-\alpha$; then, applying the entropy chain rule to $H(Z,A)$ (where $A$ is a variable that indicates which of the two rvs was chose) we get

$$H(Z)=H(A)+H(Z\mid A) - H(A\mid Z)=h(\alpha)+\alpha H(X) + (1-\alpha)H(Y)- H(A\mid Z) \tag{2}$$

where $h(\cdot)$ is the binary entropy function. In our case, $\alpha=\frac12$ and $H(X)=H(Y)=H(B;n,p)$, hence

$$H(Z)=1 + H(B;n,p) - H(A \mid Z) \tag{3}$$

Because $0\le H(A \mid Z)\le1$,

$$H(B;n,p)\le H(Z)\le 1 + H(B;n,p) $$ Assuming $0<p<1$ and $n\to \infty$, then $$ H(Z) = \frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big) + O(1)\\ \approx \frac1 2 \log_2 \big( 2\pi e\, np(1-p) \big)$$