Time Series processing using fft

Question

I have a set of real data (timestamp and value) with an unstable step between samples (5sec, 30sec etc.). The data is the % of fillage of a vehicle's tank through time.

Due to the harsh volatility of the data, I need to perform fft to transfer my time series to the frequency domain, select a cutoff point to remove all the noise and then transfer back to the time domain.

Is my thought process correct?

I want to use matlab but I cannot figure out what I have to do with my data. The first 10 rows (I have a document with 10ths of thousands of data) are as follows:

I have produced some dummy timestamp data with the same values and with time step at 0.01sec. After writing the code in matlab I get the following:

My code is:

load input.txt;
plot(input);
figure;
Fs = 1/0.01;
Ts = 1/Fs;
dt = 0:Ts:5-Ts;
x = input(:,1);
y = input(:,2);
nfft = length(y);
nfft2 = 2*nextpow2(nfft);
ff = fft(y,nfft2);
fff = ff(1:nfft2/2);ff
plot(abs(fff));

If you want some data don't hesitate to contact me. Thank you in advance!

Due to the harsh volatility of the data, I need to perform fft to transfer my time series to the frequency domain, select a cutoff point to remove all the noise and then transfer back to the time domain. You want to do the right thing, but the way you're approaching it is not good: FFT'ing data and then cutting off stuff leads to bad artifacts: https://dsp.stackexchange.com/questions/6220/why-is-it-a-bad-idea-to-filter-by-zeroing-out-fft-bins — Marcus Müller, Apr 07 '21 at 09:20
https://dsp.stackexchange.com/questions/6220/why-is-it-a-bad-idea-to-filter-by-zeroing-out-fft-bins — Marcus Müller, Apr 07 '21 at 09:20
My idea was to use a low pass filter on the fft data and then inverse it. — Dimitris Boukosis, Apr 07 '21 at 09:24
you'd need to use a window, not a filter on the frequency domain data. — Marcus Müller, Apr 07 '21 at 09:24

score 1 · Answer 1 · answered Apr 07 '21 at 13:43

1

Is my thought process correct?

No. Frequency domain filtering is difficult and even if you get it right, I sincerely doubt that low-pass filtering will solve your problem

Your data is very noisy. I looks like there are two noise sources: one with a "quantization" step of about 20 plus some smaller noise overlaid. Hard to tell without further analysis.

You haven't told us yet, what you actually want to get out of the analysis, so it's hard to give advise. Low-pass filtering will smear out your "refuel" transients a lot. I'm guessing, your best shot would be to build a parametric model and do a piece-wise least squares error fit, but again, you need to tell us what you actually want to achieve.

answered Apr 07 '21 at 13:43

Hilmar

44,604
1
32
63

Basically we have some data from a floater inside the vehicles tank which gives the % fillage of the tank. We need to denoise the data you see in order to have more precise data to create some reports. – Dimitris Boukosis Apr 07 '21 at 14:09
Sorry, "create some reports" is not specific enough to give you meaningful guidance. Denoising is hard, especially at the noise amount that your data has and ideally is tailored to the specific requirements of your application. – Hilmar Apr 07 '21 at 15:18

score 0 · Answer 2 · answered Apr 07 '21 at 09:31

Due to the harsh volatility of the data, I need to perform fft to transfer my time series to the frequency domain, select a cutoff point to remove all the noise and then transfer back to the time domain.

Is my thought process correct?

You want to do the right thing, but the way you're approaching it is not good: FFT'ing data and then cutting off stuff leads to bad artifacts: Why is it a bad idea to filter by zeroing out FFT bins?

You just want to filter (and you can do that comforatbly in time domain, 10s of thousands of samples is really not much data at all). To be able to filter, however, it's usually a good idea interpolate your data to a new, uniformly sampled time base.

That, however, is something that I routinely indeed do via FFT and zeroing out points; basically:

Your time step resolution seems to be 1s. So, first allocate an array large enough for your observation period in seconds, plus at least 2·(1/cutoff frequency) as "buffer" at the end or the beginning; zero it completely.
Fill in the values at the times you actually observed in that array
Now, do the FFT of that whole array (no, it doesn't have to be a power of 2 in length) (Seeing your data is probably real-valued, you want a real-to-complex FFT)
The "abrupt" changes between lots of zeros and samples inbetween lead to high-frequency components; suppressing them by point-wise multiplication with a window vector of the same length and appropriate shape will indeed suppress these. That window should be real-valued and symmetric (symmetry means it's symmetric to the 0 frequency, which, depending on how your FFT is shifted or not, its peak is at the zeroth bin and the last bin).
do the inverse FFT to go back to time domain
cut off the padding you added in step 1.

Ok let's see If I get this. First, I will keep my data as is and create a "dummy" column representing time in seconds (with step of 1 second). Where I do have a value I will replace the zero value with my value and then run the FFT. `load input.txt;
z1 = input(:,1);

z2 = input(:,2);

N=length(z2);

y=fft(z2,N);

plot(y);

e=y(1:floor(N/2)+1);

Fs=1;

f=linspace(0,Fs/2,floor(N/2)+1);

plot(f,abs(e));`

Should I use this code? — Dimitris Boukosis, Apr 07 '21 at 09:50
I'm not a great octave/matlab coder, but I don't see where your code calculates any necessary length for the data you have in your input, and where it generates the zero data? — Marcus Müller, Apr 07 '21 at 10:30
I generated the data via c# code so it reads from input. The length is N = length (z2). I just ran the code and it produces the following result: https://imgur.com/a/fShyD5k How should I proceed? — Dimitris Boukosis, Apr 07 '21 at 10:43
Again, I can't read your code, and I don't know what's in your variables. I don't know what this plot shows, and whether you've implemented 1. – 6. of my description above, doesn't look like it (I see no appropriate window function being created). In short: I'm really not the right person to debug your code! — Marcus Müller, Apr 07 '21 at 11:33

Time Series processing using fft

2 Answers2