14

One of my weekend projects has brought me into the deep waters of signal processing. As with all my code projects that require some heavy-duty math, I'm more than happy to tinker my way to a solution despite a lack of theoretical grounding, but in this case I have none, and would love some advice on my problem, namely: I'm trying to figure out exactly when the live audience laughs during a TV show.

I spent quite a bit of time reading up on machine learning approaches for detecting laughter, but realized that's more to do with detecting individual laughter. Two hundred people laughing at once will have much different acoustical properties, and my intuition is that they should be distinguishable through much cruder techniques than a neural network. I may be completely wrong, though! Would appreciate thoughts on the matter.

Here's what I've tried so far: I chopped up a five minute excerpt from a recent episode of Saturday Night Live into two second clips. I then labeled these "laughs" or "no-laughs". Using Librosa's MFCC feature extractor, I then ran a K-Means clustering on the data, and got good results -- the two clusters mapped very neatly to my labels. But when I tried to iterate through the longer file the predictions didn't hold water.

What I'm going to try now: I'm going to be more precise about creating these laughter clips. Rather than do a blind split and sort, I'm going to manually extract them, so that no dialogue is polluting the signal. Then I'll split them into quarter second clips, calculate the MFCC's of these, and use them to train a SVM.

My questions at this point:

  1. Is any of this making sense?

  2. Can statistics help here? I've been scrolling around in Audacity's spectrogram view mode and I can see pretty clearly where laughs occur. In a log power spectrogram, speech has a very distinctive, "furrowed" appearance. In contrast, laughter covers a broad spectrum of frequency quite evenly, almost like a normal distribution. It's even possible to visually distinguish applause from laughter by the more limited set of frequencies represented in applause. That makes me think of standard deviations. I see there's something called the Kolmogorov–Smirnov test, might that be helpful here? Log-power spectrogram (You can see the laugh in the above image as a wall of orange hitting at 45% of the way in.)

  3. The linear spectrogram seems to show that the laughter is more energetic in lower frequencies and fades out towards the higher frequencies -- does this mean it qualifies as pink noise? If so, could that be a foothold on the problem? Spectrogram

I apologize if I misused any jargon, I've been on Wikipedia quite a bit for this one and wouldn't be surprised if I got some jumbled.

jojeck
  • 11,107
  • 6
  • 38
  • 74
Erik
  • 141
  • 2
  • 1
    I agree on the "won't need a neural network to get a solid detector for laugh tracks". I also agree on you throwing Kolmogorov at the signal, considering that if you're right, and the laugh track is the (weighted) sum of iid laughs, you should be getting normal features of some kind. Maybe you'd still like to increase the frequency resolution. By the way, for someone who's "not into" DSP, your jargon is quite fine :) sadly, I'm not an audio guy, so I'm probably not very helpful. – Marcus Müller Oct 22 '17 at 21:09
  • I am happy to help. Do you have any data for training? One of the most important things is the data labelling. Garbage in - garbage out. One of the simplest and most effective approaches would be to train a bag-of-frames GMM and evaluate the likelihood. – jojeck Nov 14 '17 at 15:03
  • You might want to check first, if you can separate laughter from silence by checking the power/amplitude of your signal at given times. Giving you the possibility to threshold moments where "something" is happening. Then you could try to look at the distribution of the frequency. For example, speech might have some distinctive sharp peaks (don't care where exactly these peaks are, just that they exist), while laughter is uniform as you said. Tracking this might yield a value to decide if it is laughter. (You need the volume information, to make sure you dont have just uniform silence) – user6522399 Jan 19 '18 at 11:04

2 Answers2

1

Based on your observation, that spectrum of the signal is sufficiently distinguishable, you can use this as a feature to classify laughter from speech.

There are many ways you can look at the problem.

Approach #1

In once case, you can just look at vector of the MFCC. and apply this to any classifier. Since you have many co-efficient in frequency domain, you might want to look at Cascade Classifiers structure with boosting algorithms such as Adaboost based on this, you can compare between speech class vs. laugh class.

Approach #2

You realise that you speech is essentially a time varying signal. So one of the effective ways to do it is to look at time variation of the signal itself. For this, you can split signals in batches of samples, and look at the spectrum for that time. Now, you may realise that laugh might have more repetitive pattern for a stipulated duration where as speech inherently posses more information and hence the spectrum variation would be rather bigger. You can apply this to HMM type of model to see if you consistently remain within the same state for some frequency spectrum or you continuously keep changing. Here, even if occasionally the spectrum of speech resembles to that of laughter it will be more time changing.

Approach #3

Force to apply LPC/CELP type of coding on the signal and observe the residue. CELP Coding makes a very accurate model of speech production.

From the reference here: THEORY OF CELP CODING

The redundancies in the speech signal are almost removed after the short term prediction and long term prediction of the speech signal and the residual has very little correlation left in it. Then an excitation is searched which synthesizes the speech and the codebook index and gain are searched from the fixed codebook. The optimum codebook index selection criterion is based on MMSE between the locally synthesized speech and the original speech signal.

To put it simply, after all the speech which is predicted from analyser is removed - what is left is the residue which is transmitted to recreate exact waveform.

How does that help with your problem? Basically, if you apply CELP coding, speech in the signal is mostly removed, what remains is residue. In case of laughter a majority of the signal might be retained because CELP will fail to predict such a signal with vocal tract modelling, where as individual speech will have very little residue. You can also analyse this residue back in frequency domain, to see if it is laughter or speech.

Dipan Mehta
  • 5,587
  • 2
  • 30
  • 53
0

Most speech recognizers use not only the MFCC coefficients but also the first and 2nd derivatives of the MFCC levels. I’m guessing that the onsets would be very useful in this case and help you distinguish a laugh versus other sounds.

Bob
  • 548
  • 2
  • 6