System and method for tracking sound pitch across an audio signal using harmonic envelope

• US 9,473,866 B2
• Filed: 11/25/2013
• Issued: 10/18/2016
• Est. Priority Date: 08/08/2011
• Status: Active Grant

• Current Assignee: Friday Harbor LLC
• Original Assignee: Knuedge, Inc.
• Inventors: Bradley, David C., Gateau, Rodney, Goldin, Daniel S., Hilton, Robert N., Fisher, Nicholas K.
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): THOMAS-HOMESCU, ANNE L
• Application Number:
  US14/089,729
• Publication Number:
  US 20140086420A1
• Time in Patent Office: 1,058 Days
• Field of Search: 704/200, 704/200.1, 704/201, 704203-207, 704216-223, 704500-504
• US Class Current: 1/1
• CPC Class Codes: G10L 2025/906 Pitch tracking
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04R 29/00 Monitoring arrangements; Testing arrangements for hearing aids H04R25/30; detection of loudspeaker connection H04R5/04; sound-field adaptation dependent on speaker detection H04S7/308

Abstract

A system and method may be configured to analyze audio information derived from an audio signal. The system and method may track sound pitch across the audio signal. The tracking of pitch across the audio signal may take into account change in pitch by determining at individual time sample windows in the signal duration an estimated pitch and a representation of harmonic envelope at the estimated pitch. The estimated pitch and the representation of harmonic envelope may then be implemented to determine an estimated pitch for another time sample window in the signal duration with an enhanced accuracy and/or precision.

Claims

1. A system configured to track pitch in an audio signal, the system comprising:

an electronic storage storing computer program modules; and

one or more processors configured to execute the computer program modules, the computer program modules being configured to;

receive the audio signal obtained from a user input device;

obtain a first transformation of the audio signal in a first time period, wherein the first transformation represents the audio signal as a function of frequency in the first time period;

obtain a first pitch corresponding to a first sound in the first time period of the audio signal;

determine a first envelope vector of the first time period from the first transformation in a multi-dimensional space, wherein each dimension of the multi-dimensional space corresponds to one of a plurality of harmonics of a pitch and the first envelope vector of the first time period is defined by a first set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the first pitch in the first transformation;

obtain a second transformation of the audio signal in a second time period, wherein the second time period is different from the first time period and the second transformation represents the audio signal as a function of frequency in the second time period;

obtain a second pitch corresponding to a second sound in the second time period of the audio signal;

determine a second envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the second envelope vector of the second time period is defined by a second set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the second pitch in the second transformation;

determine a first correlation between the first envelop vector of the first time period and the second envelope vector of the second time period;

obtain a third pitch corresponding to a third sound in the second time period of the audio signal;

determine a third envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the third envelope vector of the second time period is defined by a third set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the third pitch in the second transformation;

determine a second correlation between the first envelop vector of the first time period and the third envelope vector of the second time period; and

determine, using the first correlation and the second correlation, that the first sound in the first time period of the audio signal and the second sound in the second time period of the audio signal are portions of a same harmonic sound.

2. The system of claim 1, wherein the first and second time periods of the audio signal correspond to a first and a second time sample windows of the audio signal.

3. The system of claim 2, wherein the second time sample window is adjacent to the first window of time before or after the first time sample window.

4. The system of claim 2, wherein the second time sample window overlaps with the first time sample window.

5. The system of claim 2, the computer program modules are further configured to identify a primary time sample window as the first time sample window.

6. The system of claim 1, wherein the first transformation of the audio signal in the first time period comprises an intensity coefficient related to an intensity of the audio signal as a function of frequency and fractional chirp rate.

7. The system of claim 6, wherein to obtain the first and second pitches comprises to search for a maximum across a plurality of frequencies for one common fractional chirp rate common to both the first transformation and second transformation respectively.

8. The system of claim 1, wherein the computer program modules are further configured to obtain a fractional chirp rate associated with the first sound, wherein to obtain the second pitch comprises incrementing the first pitch by an amount that corresponds to the obtained fractional chirp rate associated with the first sound and a time difference between the first and second time periods of the audio signal.

9. A method for tracking pitch in an audio signal, the method comprising:

receiving the audio signal obtained from a user input device;

obtaining a first transformation of the audio signal in a first time period, wherein the first transformation represents the audio signal as a function of frequency in the first time period;

obtaining a first pitch corresponding to a first sound in the first time period of the audio signal;

determining a first envelope vector of the first time period from the first transformation in a multi-dimensional space, wherein each dimension of the multi-dimensional space corresponds to one of a plurality of harmonics of a pitch and the first envelope vector of the first time period is defined by a first set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the first pitch in the first transformation;

obtaining a second transformation of the audio signal in a second time period, wherein the second time period is different from the first time period and the second transformation represents the audio signal as a function of frequency in the second time period;

obtaining a second pitch corresponding to a second sound in the second time period of the audio signal;

determining a second envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the second envelope vector of the second time period is defined by a second set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the second pitch in the second transformation;

determining a first correlation between the first envelop vector of the first time period and the second envelope vector of the second time period;

obtaining a third pitch corresponding to a third sound in the second time period of the audio signal;

determining a third envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the third envelope vector of the second time period is defined by a third set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the third pitch in the second transformation;

determining a second correlation between the first envelop vector of the first time period and the third envelope vector of the second time period; and

determining, using the first correlation and the second correlation, that the first sound in the first time period of the audio signal and the second sound in the second time period of the audio signal are portions of a same harmonic sound.

10. The method of claim 9, wherein the first and second time periods of the audio signal correspond to a first and a second time sample windows of the audio signal.

11. The method of claim 10, wherein the second time sample window is adjacent to the first window of time before or after the first time sample window.

12. The method of claim 10, wherein the second time sample window overlaps with the first time sample window.

13. The method of claim 10, further comprising identifying a primary time sample window as the first time sample window.

14. The method of claim 9, wherein the first transformation of the audio signal in the first time period comprises an intensity coefficient related to an intensity of the audio signal as a function of frequency and fractional chirp rate.

15. The method of claim 14, wherein obtaining the first and second pitches comprises searching for a maximum across a plurality of frequencies for one common fractional chirp rate for the first transformation and second transformation respectively.

16. The method of claim 9, further comprising obtaining a fractional chirp rate associated with the first sound, wherein obtaining the second pitch comprises incrementing the first pitch by an amount that corresponds to the obtained fractional chirp rate associated with the first sound and a time difference between the first and second time periods of the audio signal.

17. A non-transitory computer readable storage medium having data stored therein representing computer program modules executable by a computer, the computer program modules including instructions to track pitch in an audio signal, the storage medium comprising:

instructions for receiving the audio signal obtained from a user input device;

instructions for obtaining a first transformation of the audio signal in a first time period, wherein the first transformation represents the first portion of the audio signal as a function of frequency in the first time period;

instructions for obtaining a first pitch corresponding to a first sound in the first time period of the audio signal;

instructions for determining a first envelope vector of the first time period from the first transformation in a multi-dimensional space, wherein each dimension of the multi-dimensional space corresponds to one of a plurality of harmonics of a pitch and the first envelope vector of the first time period is defined by a first set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the first pitch in the first transformation;

instructions for obtaining a second transformation of the audio signal in a second time period, wherein the second time period is different from the first time period and the second transformation represents the second portion of the audio signal as a function of frequency in the second time period;

instructions for obtaining a second pitch corresponding to a second sound in the second time period of the audio signal;

instructions for determining a second envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the second envelope vector of the second time period is defined by a second set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the second pitch in the second transformation;

instructions for determining a first correlation between the first envelop vector of the first time period and the second envelope vector of the second time period;

instructions for obtaining a third pitch corresponding to a third sound in the second time period of the audio signal;

instructions for determining a third envelope vector of the second time period from the second transformation in the multi-dimensional space, wherein the third envelope vector of the second time period is defined by a third set of coordinates corresponding to intensity coefficients at a plurality of harmonics of the third pitch in the second transformation;

instructions for determining a second correlation between the first envelop vector of the first time period and the third envelope vector of the second time period; and

instructions for determining, using the first correlation and the second correlation, that the first sound in the first time period of the audio signal and the second sound in the second time period of the audio signal are portions of a same harmonic sound.

18. The non-transitory computer readable storage medium of claim 17, wherein the first and second time periods of the audio signal correspond to a first and a second time sample windows of the audio signal.

19. The non-transitory computer readable storage medium of claim 18, wherein the second time sample window is adjacent to the first window of time before or after the first time sample window.

20. The non-transitory computer readable storage medium of claim 18, wherein the second time sample window overlaps with the first time sample window.

21. The non-transitory computer readable storage medium of claim 18, further comprising instructions for identifying a primary time sample window as the first time sample window.

22. The non-transitory computer readable storage medium of claim 17, wherein the first transformation of the audio signal in the first time period comprises an intensity coefficient related to an intensity of the audio signal as a function of frequency and fractional chirp rate.

23. The non-transitory computer readable storage medium of claim 22, wherein instructions for obtaining the first and second pitches further comprises instructions for searching for a maximum across a plurality of frequencies for one common fractional chirp rate for the first transformation and second transformation respectively.

24. The non-transitory computer readable storage medium of claim 17, further comprising instructions for obtaining a fractional chirp rate associated with the first sound, wherein the instructions for obtaining the second pitch comprises instructions for incrementing the first pitch by an amount that corresponds to the obtained fractional chirp rate associated with the first sound and a time difference between the first and second time periods of the audio signal.