Systems and methods for concurrent signal recognition

• US 9,047,867 B2
• Filed: 02/21/2011
• Issued: 06/02/2015
• Est. Priority Date: 02/21/2011
• Status: Expired due to Fees

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Smaragdis, Paris
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Sharma, Neeraj
• Application Number:
  US13/031,353
• Publication Number:
  US 20130132082A1
• Time in Patent Office: 1,562 Days
• Field of Search: 704/240, 704/238, 704/243, 704/256, 704/234, 704/235, 715/767, 706/20
• US Class Current: 1/1
• CPC Class Codes: G10L 15/142 Hidden Markov Models [HMMs]
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence

Abstract

Methods and systems for recognition of concurrent, superimposed, or otherwise overlapping signals are described. A Markov Selection Model is introduced that, together with probabilistic decomposition methods, enable recognition of simultaneously emitted signals from various sources. For example, a signal mixture may include overlapping speech from different persons. In some instances, recognition may be performed without the need to separate signals or sources. As such, some of the techniques described herein may be useful in automatic transcription, noise reduction, teaching, electronic games, audio search and retrieval, medical and scientific applications, etc.

Claims

1. A method implemented by one or more computer systems, the method comprising:

receiving a mixed audio signal, the mixed audio signal including;

one or more portions including audio signals emitted from respective ones of a plurality of sources; and

at least one portion having audio signals concurrently emitted from the plurality of sources;

deriving a plurality of parameters for each of the audio signals within the mixed audio signal, the plurality of parameters derived from one of the mixed audio signal or training data and including;

initial state probabilities representing probabilities of beginning a Markov chain at each state in the Markov chain;

a transition matrix representing a set of transition probabilities between pair of states in the Markov chain; and

a set of state output distributions representing probabilities of generating observations from each of the states in the Markov chain;

generating, from the parameters and independent of using a Gaussian function, a plurality of models that each contain one or more state dictionaries containing two or more spectral vectors, such that each of the plurality of sources is represented by one or more of the plurality of models;

combining a plurality of the spectral vectors from the plurality of models into a set of spectral vectors representing the mixed audio signal;

calculating mixture weights for each spectral vector in the set of spectral vectors;

calculating a likelihood that each one of the plurality of models emitted one or more audio signals in the mixed audio signal based at least in part on the set of spectral vectors representing the mixed audio signal; and

selecting one or more models from the plurality of models with the highest calculated likelihood.

2. The method of claim 1, wherein deriving the plurality of parameters for each of the plurality of sources is performed without preprocessing the mixed audio signal by a separation algorithm.

3. The method of claim 1, wherein a number of dictionaries for each of the plurality of models is determined dependent, at least in part, on a number of phonemes within a corresponding said audio signal.

8. The method of claim 1, further comprising identifying one or more sources based on the selected one or more models.

13. The method of claim 1, wherein the training data is derived from an offline training method performed prior to receiving the mixed audio signal.

4. A non-transitory computer-readable storage medium storing program instructions that, when executed by one or more computer systems, the method cause the one or more computer systems to perform operations comprising:

receiving a mixed audio signal, the mixed audio signal including one or more portions of audio signals emitted from respective ones of a plurality of sources and at least one portion having audio signals concurrently emitted from the plurality of sources;

deriving a plurality of parameters for each of the audio signals within the mixed audio signal, the plurality of parameters derived from one of the mixed audio signal or training data and including;

initial state probabilities representing the probabilities of beginning a Markov chain at each state in the Markov chain;

a transition matrix representing the set of all transition probabilities between every pair of states in the Markov chain; and

a set of state output distributions representing the probability of generating observations from each of the states in the Markov chain;

generating, from the parameters and independent of using a Gaussian function, a plurality of models that each contain one or more state dictionaries containing two or more spectral vectors, such that each of the plurality of sources is represented by one or more of the plurality of models;

combining a plurality of the spectral vectors from the plurality of models into a set of spectral vectors representing the mixed audio signal;

calculating mixture weights for each spectral vector in the set of spectral vectors;

calculating a likelihood that each one of the plurality of models emitted a portion of one or more audio signals in the mixed audio signal based at least in part on the calculated mixture weights representing the mixed audio signal; and

selecting one or more models from the plurality of models with the highest calculated likelihood.

5. The non-transitory computer-readable storage medium of claim 4, wherein the combining of the plurality of spectral vectors into the set of spectral vectors representing the mixed audio signal is performed without preprocessing the mixed audio signal by a separation algorithm.

9. The non-transitory computer-readable storage medium of claim 4, further comprising identifying one or more sources based on the selected one or more models.

10. The non-transitory computer-readable storage medium of claim 4, wherein a number of dictionaries for each of the plurality of models is determined dependent, at least in part, on a number of phonemes within a corresponding audio signal.

14. The non-transitory computer-readable storage medium of claim 4, wherein the training data is derived from an offline training method performed prior to receiving the mixed audio signal.

6. A device, comprising:

at least one processor; and

a memory coupled to the at least one processor storing program instructions executable by the at least one processor to perform operations including;

receiving a mixed audio signal, the mixed audio signal including one or more portions of audio signals emitted from respective ones of a plurality of sources and at least one portion having audio signals concurrently emitted from the plurality of sources;

deriving a plurality of parameters for each of the audio signals within the mixed audio signal, the plurality of parameters derived from one of the mixed audio signal or training data and including;

initial state probabilities representing the probabilities of beginning a Markov chain at each state in the Markov chain;

a transition matrix representing the set of all transition probabilities between every pair of states in the Markov chain; and

a set of state output distributions representing the probability of generating observations from each of the states in the Markov chain;

generating, from the parameters and independent of using a Gaussian function, a plurality of models that each contain one or more state dictionaries containing two or more spectral vectors, such that each of the plurality of sources is represented by one or more of the plurality of models;

combining a plurality of the spectral vectors from the plurality of models into a set of spectral vectors representing the mixed audio signal;

calculating mixture weights for each spectral vector in the set of spectral vectors;

calculating a likelihood that each one of the plurality of models emitted a portion of one or more audio signals in the mixed audio signal based at least in part on the calculated mixture weights representing the mixed audio signal; and

selecting one or more models from the plurality of models with the highest calculated likelihood.

7. The device of claim 6, wherein the combining of the plurality of spectral vectors into the set of spectral vectors representing the mixed audio signal is performed without preprocessing the mixed audio signal by a separation algorithm.

11. The device of claim 6, further comprising identifying one or more sources based on the selected one or more models.

12. The device of claim 6, wherein a number of dictionaries for each of the plurality of models is determined dependent, at least in part, on a number of phonemes within a corresponding audio signal.

15. The device of claim 6, wherein the training data is derived from an offline training method performed prior to receiving the mixed audio signal.