1

There is an information source on the information source alphabet $A = \{a, b, c\}$ represented by the state transition diagram below markov chain

The $i$-th output from this information source is represented by random variable $X_i$. It is known that the user is now in state $s_1$. In this state, let $H (X_i \mid s_1)$ denote the entropy when observing the next symbol $X_i$, find the value of $H (X_i \mid s_1)$, entropy of this information source, Calculate $H (X_i\mid X_{i-1}) $ and $H (X_i)$ respectively. Assume $i$ is quite large.

How can I find $H(X_i|s1)?$ I know that $H(X_i\mid s_1) = -\sum_{i} p\left(x_i, s_1\right)\cdot\log_b\!\left(p\left(x_i|s_1\right)\right) = -\sum_{i} p\left(x_i, s_1\right)\cdot\log_b\!\left(\frac{p\left(x_i, s_1\right)}{p\left(s_1\right)}\right)$ but I don't know $p(s_1).$

$A=\begin{pmatrix}0.25 & 0.75 & 0\\0.5 & 0 & 0.5 \\0 & 0.7 & 0.3 \end{pmatrix}.$

from matrix I can know that $p(s_1|s_1)=0.25$, etc

but what is the probability of $s_1$? and how can I calculate $H (X_i|X_{i-1})$?　is this stationary distribution too?

Bernard
  • 175,478
vedss
  • 343

1 Answers1

0

First, notice that $H(X_i|s_1)$ is a succint way of writing $H(X_i|X_{i-1}=s_1)$. Then, recall that the entropy of a random variable conditioned on a particular value of another variable (different from the plain conditional entropy, if you are confused by this read here and here) is computed the same as the plain entropy, only replacing plain probabilities with conditional probabilities:

$$H(X_i \mid X_{i-1}=s_1)= - \sum_{X_i} P(X_i\mid X_{i-1}=s_1) \log P(X_i\mid X_{i-1}=s_1) \tag1$$

Because the values of $P(X_i\mid X_{i-1}=s_1)$ are given by the first row of $A$, you don't need any extra data to compute this.

But you indeed need the stationary distribution to compute $H(X_i \mid X_{i-1})$:

$$ H(X_i \mid X_{i-1})= \sum_k P(X_{i-1}=s_k) \, H(X_i \mid X_{i-1}=s_k) \tag2$$

leonbloy
  • 63,430