Why does over-modelling an adaptive AR NLMS filter fix sharp spikes?

Question

I just simulated an auto-regressive second-order model fueled by white noise and estimated the parameters with normalized least-mean-square filters of orders 1-4.

As the first-order filter under-models the system, of course the estimations are weird. The second-order filter finds good estimates, although it has a couple of sharp jumps. This is to be expected from the nature of NLMS filters.

What confuses me is the third- and fourth-order filters. They seem to eliminate the sharp jumps, as seen in the figure below. I can't see what they would add, as the second-order filter is enough to model the system. The redundant parameters hover around $0$ anyway.

Could someone explain this phenomenon for me, qualitatively? What causes it, and is it desirable?

I used step size $\mu=0.01$, $10^4$ samples, and the AR model $x(t)=e(t)-0.9x(t-1)-0.2x(t-2)$ where $e(t)$ is white noise with variance 1.

enter image description here

The MATLAB code, for reference:

% ar_nlms.m
function th=ar_nlms(y,order,mu)
N=length(y);
th=zeros(order,N); % estimated parameters
for t=na+1:N
    phi = -y( t-1:-1:t-na, : );
    residue = phi*( y(t)-phi'*th(:,t-1) );
    th(:,t) = th(:,t-1) + (mu/(phi'*phi+eps)) * residue;
end

% main.m
y = filter( [1], [1 0.9 0.2], randn(1,10000) )';
plot( ar_nlms( y, 2, 0.01 )' );

I don't quite understand what you're plotting there. What kind of filter is it you're simulating with the NLMS? — Obviously, the more parameters you have, the better you will be able to fit to an arbitrary filter; even if parameters "hover around 0" that doesn't mean they don't do anything. — leftaroundabout, Dec 05 '11 at 23:01
@left: I'm simulating an AR(2) model with constant parameters, which means that the NLMS(2) should be able to describe the system completely. Obviously the extra parameters do something, as they manage to reduce the spikes, but I'm wondering why -- the system is over-modelled, which usually just means that the confidence interval for the estimated parameters increases. — Andreas, Dec 06 '11 at 07:36
@left: Sorry, I missed your first sentence. I'm plotting the estimated values of the AR parameters of an adaptive NLMS filter over time. I.e. $a_n$ from the estimated model $x(t)=e(t)-a_1x(t-1)-a_2x(t-2)-...-a_nx(t-n)$ for $n\in\left{1,2,3,4\right}$ — Andreas, Dec 07 '11 at 17:59
Isn't NLMS a MA model while you are trying to approximate an AR model? — Memming, Mar 11 '12 at 15:47
@Memming: The NLMS is trying to invert the AR model, so an MA model is the right thing to do here. — Peter K., Feb 14 '13 at 01:49

score 2 · Accepted Answer · answered Feb 14 '13 at 02:15

What appears to be happening is, as you start over-modeling, the error signal becomes less and less white.

I modified your code to return the error signal (part of the residue term).

This plot shows the off-zero-lag coefficients of the xcorr of the error for order = 2 (blue), 3 (red), and 4 (green). As you can see, the close-to-but-not-zero lag terms are getting larger in magnitude.

If we look at the FFT (spectrum) of the xcorr of the error, then we see that the lower frequency terms (that cause the large drifts) are getting smaller (the error is containing more high frequencies).

So it seems the effect of over-modeling in this case is to high-pass filter the error, which (for this example) is beneficial.

enter image description here

function [th,err]=ar_nlms(y,order,mu)
eps = 0.000000001;
N=length(y);
th=zeros(order,N); // estimated parameters
err = zeros(1,N);
for t=order+1:N
    phi = -y( t-1:-1:t-order, : );
    err(t) = y(t)-phi'*th(:,t-1);
    residue = phi*( err(t) );
    th(:,t) = th(:,t-1) + (mu/(phi'*phi+eps)) * residue;
    size(residue)
end

Why does over-modelling an adaptive AR NLMS filter fix sharp spikes?

1 Answers1