Examples of Independent and uncorrelated data in real-life, and ways to measure/detect them

Question

We always hear about this vector of data VS this other vector of data being independent from each other, or uncorrelated, etc, and while it is easy to come across the math regarding those two concepts, I want to tie them into examples from real-life, and also find ways to measure this relationship.

From this stand point, I am looking for examples of two signals that are of the following combinations: (I will start with some):

• Two signals that are independent AND (necessarily) uncorrelated:

  • The noise from a car engine (call it $v_1[n]$) and your voice ($v_2[n]$) as you are talking.
  • A recording of humidity every day ($v_1[n]$) and the dow-jones index ($v_2[n]$).

Q1) How would you measure/prove that they are independent with those two vectors in hand? We know that independence means that the product of their pdfs equals their joint pdf, and thats great, but with those two vectors in hand, how does one prove their independence?

• Two signals that are NOT independent, but still uncorrelated:

Q2) I cant think of any examples here ... what would some examples be? I know we can measure correlatedness by taking the cross-correlation of two such vectors, but how would we prove that they are also NOT independent?

• Two signals that are correlated:
  • A vector measuring an opera singer's voice in the main hall, $v_1[n]$, while someone records her voice from somewhere inside the building, say in the rehearsal room ($v_2[n]$).
  • If you continuously measured your heart rate in your car, ($v_1[n]$), and also measured the intensity of blue lights impinging on your rear windshield ($v_2[n]$)... I am guessing those would be very correlated... :-)

Q3) Related to q2, but in the case of measuring cross-correlation from this empirical stand point, is it enough to look at the dot product of those vectors (since that is the value at the peak of their cross-correlation)? Why would we care about other values in the cross-corr function?

Thanks again, the more examples given the better for building of the intuition!

Answer 1

A few elements... (I know that this is not exhaustive, a more complete answer should probably mention moments)

Q1

To check whether two distributions are independent, you need to measure how similar their joint distribution $p(x,y)$ is to the product of their marginal distribution $p(x) \times p(y)$. To this purpose, you can use any distance between distributions. If you use the Kullback-Leibler divergence to compare those distributions, you will consider the quantity:

$\int_x \int_y p(x, y) \log \frac{p(x, y)}{p(x) p(y)} dx dy$

And you will have recognized... the Mutual Information! The lower it is, the more independent the variables are.

More practically, to compute this quantity from your observations, you can either estimate the densities $p(x)$, $p(y)$, $p(x, y)$ from your data using a Kernel density estimator and do a numerical integration on a fine grid ; or just quantify your data into $N$ bins and use the expression of the Mutual Information for discrete distributions.

Q2

From the Wikipedia page on statistical independence and correlation:

At the exception of the last example, these 2D distributions $p(x, y)$ have uncorrelated (diagonal covariance matrix), but not independent, marginal distributions $p(x)$ and $p(y)$.

Q3

There are indeed situations in which you might look at all the values of the cross-correlation functions. They arise, for example, in audio signal processing. Consider two microphones capturing the same source, but distant from a few meters. The cross-correlation of the two signals will have a strong-peak at the lag corresponding to the distance between microphones divided by the speed of sound. If you just look at the cross-correlation at lag 0, you won't see that one signal is a time-shifted version of the other one!

Answer 2

Inferring whether two signals are independent is very hard to do (given finite observations) without any prior knowledge/assumptions.

Two random variable $X$ and $Y$ are independent if the value of $X$ doesn't give any information about the value of $Y$ (i.e. doesn't affect our prior probability distribution for $Y$). This is equivalent to any nonlinear transformation of $X$ and $Y$ being uncorrelated i.e. $$\text{cov}(f_1(X),f_2(Y)) = E(f_1(X),f_2(Y)) = 0$$ for any non-linear $f_1$ and $f_2$ assuming wlog both variables have zero mean. The difference between independence and uncorrelatedness is that $X$ and $Y$ are uncorrelated if the above holds, only for $f_1(x) = f_2(x) = x$, the identity function.

If we assume joint Gaussianity, then all joint moments greater than order 2 are equal to zero and in this case uncorrelated implies independent. If we have no prior assumptions then estimation of the joint moments $E(X^iY^j)$ will give us information on 'how dependent' they are upon one another.

We can generalise this to signals $X(t)$ and $Y(t)$ by considering the cross-spectra $$S_{X,Y}(f),S_{X^2,Y}(f),S_{X,Y^2}(f) \dots$$ across all frequencies $f$.

Example:

After reading 'pichenettes' comment I was inspired to use his idea as an example. Consider the signals $$X(t) = \sin(2 \pi ft)$$ $$Y(t) = \sin( 2 \pi ftk)$$ for $k \in \mathbb{Z}$ and $k \neq 1$. Clearly there is no linear transform sending $X(t)$ to $Y(t)$ as they oscillate at different frequencies. However, it is well known that we can write $\sin(kx)$ as a function in $\sin(x)$ and therefore, $$Y(t) = f (X (t))$$ for some polynomial $f$.

Hence despite being uncorrelated signals, $X(t)$ and $Y(t)$ are not independent.