2

Relation of Entropy and SNR : Based on this question and answer, I had another question that struck me and I am curious to know, if somebody can shed some light, on the following situation: $y= desired_{signal} + noise$ is received by the receiver and the $error = y - \hat{y}$ where $\hat{y}$ is simulated at the receiver by using guessed parameters with the knowledge of the kind of model used at the transmitter.

For increasing SNR (attenuating noise) would Entropy of error decrease? In the link provided, the entropy of the transmitted signal decreases with increasing SNR since the uncertainty decreases when noise is getting attenuated. But what about the entropy of error? Should error entropy increase with increasing SNR? I am confused about this.

EDIT: The way I calculated entropy of error: Let the system model be AR(2): $y(t)= a1y(t-1) + b1y(t-2) + noise$. At the receiver end, I have $z(t) = y(t) - (a2y(t-1) + b2y(t-2))$ where $(a2,b2)$ are close guesses to ($a1,b1$). I calculated 2 entropies H1 for y(t) and H2 for z(t).

z(t) will be close to zero when a2,b2 will be equal to a1,b1 in which case entropy of z(t) is found to be minimum among all different z(t) for a particular SNR value of noise. So, for a particular noise value, I have 10 pairs of (a2,b2) and for each I get 10 values of H2(z(t)). I chose the minimum among all H2 for a particular SNR level. But, for different noise levels, as I increase the SNR at the transmitter end, the minimum entropy, H2(z(t)), increases with increasing SNR. I found that H1 decreases with increasing SNR but the reverse happens for H2. Is this trend correct?

Community
  • 1
SKM
  • 611
  • 12
  • 27

1 Answers1

1

According to the link you provided:

$H(error) = H(y-\hat{y}) = H(Var[y]+Var[\hat{y}])$. While $Var[y]$ decreases with increasing of SNR, $Var[\hat{y}]$ stays the same (because it's not dependent of the noise, it depends on the signal alphabet only). So we may guess $H(error)$ decreases with increasing of SNR.

Serj
  • 1,085
  • 7
  • 11
  • I used an AR model and tried to compute the entropy of the $errors = y -y_{desired}$ and it was increasing with increasing SNR. Also why is $Var[\hat{y}]$ independent of SNR?Could you please explain? – SKM Aug 13 '14 at 15:16
  • How do you compute entropy for your case? Actually I've used variance of the process in my work, so entropy is not native for me. $Var[\hat{y}]$ is independent of SNR because it doesn't contain any noise. $\hat{y}$ is either decisions or desired signal, this signal can get values from a given distribution (in the case of random process) or signal alphabet (in the case of modulation). So its variance is a known statistic and it's independent of SNR. – Serj Aug 14 '14 at 02:22
  • I have updated the question, please have a look. I have explained the way I calculated entropy. Could you please let me know what is wrong – SKM Aug 14 '14 at 17:42
  • I probably do not understand the way you input noise to your system. Why do you adjust SNR at the transmitter end? This parameter refers to the receiver one. If you increase SNR, $noise$ component in the equations above will be weaker, so $Var[y]$ will decrease. Is it true for your model? Then there is no $noise$ for $\hat{y}$ so it isn't affected, is it? Further you can estimate $Var[z(t)]$ for different SNR scenario, if it decreases with SNR increasing, entropy is calculated probably incorrect. – Serj Aug 15 '14 at 02:41
  • I adjust noise in the transmitter to see the effect on the entropy with respect to noise. When SNR is very high, then it means there is no noise and $\hat{y}$ is not affected i.e. it is almost equal to the original signal.Can you please explain why Var[z(t)] should decrease with SNR & how it indicates that entropy is incorrect?You mentioned I n your answer that $Var[\hat{y}]$ which I think is the same as $Var[z(t)]$ is independent of noise?As noise is included in the observation, won't noise be present in the error terms after all errors are caused due to noise? – SKM Aug 15 '14 at 16:57
  • Let $d$ be a desired signal, so as you've written $y=d+n$, where $n$ is noise component. It's a signal at the input of the receiver. Now you try to compute error signal and analize it, right? Having observations of $y$ you make assumptions of its true values: $\hat{y}$. If your model is right $\hat{y} \to d$ with possible errors due to noise only. Now error, $e=y-\hat{y} \to y-d=n$. So $Var[\hat{y}]$ and $Var[z(t)]$ is not the same. And furthermore $Var[\hat{y}]$ is independent of SNR because $\hat{y}$ doesn't contain $n$, while $z(t)$ does. – Serj Aug 16 '14 at 04:43
  • Ok,let me clarify 2 things. In my Question I mentioned about $z(t) = y(t) - \hat{y(t)}$, from your comment this appears to be $e(t)$. I am calculating $Entropy[y(t)]$and Entropy(e(t)) i.e. $Entropy(z(t))$ and not $Entropy[\hat{y(t)}]$. – SKM Aug 19 '14 at 15:58