Learning algorithm to detect human presence in indoor environments from acoustic signals

• US 10,068,587 B2
• Filed: 06/21/2015
• Issued: 09/04/2018
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Rajeev Conrad Nongpiur
• Original Assignee: Rajeev Conrad Nongpiur
• Inventors: Nongpiur, Rajeev Conrad
• Primary Examiner(s): Sarpong, Akwasi M
• Application Number:
  US14/745,461
• Publication Number:
  US 20150380013A1
• Time in Patent Office: 1,171 Days
• Field of Search: 704233, 704231, 704E15, 708300, 34053913
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/15 the extracted parameters being formant information
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

A system is described that constantly learns the sound characteristics of an indoor environment to detect the presence or absence of humans within that environment. A detection model is constructed and a decision feedback approach is used to constantly learn and update the statistics of the detection features and sound events that are unique to the environment in question. The learning process may not only rely on acoustic signal, but may also make use of signals derived from other sensors such as range sensor, motion sensors, pressure sensors, and video sensors.

Claims

1. A method for detecting the presence of one or more individual human occupants in an indoor environment comprising:

providing a database of a plurality of models for determining presence of one or more individual human occupants based upon acoustic features, with each model of the plurality of models associated with a respective individual of a plurality of individuals;

detecting acoustic events in an audio signal obtained from the indoor environment;

determining human presence of a first individual human in the indoor environment by;

comparing the acoustic events with each of the plurality of models;

computing a likelihood score for each of the plurality of models based at least on the comparing the acoustic events with each of the plurality of models;

identifying a first model of the plurality of models having a greatest likelihood score, the first model associated with the first individual human; and

determining that the first individual human is present in the indoor environment based at least in part on the first model associated with the first individual human being identified as having a greatest likelihood score; and

refining at least one of the models of the plurality of models based at least in part on the determining that the first individual human is present in the indoor environment, so that the models over time become more accurate and more specific to characteristics of the acoustic features of each of the respective individual human occupants and the particular indoor environment, wherein refining the models includes;

starting with generic models to detect, from acoustic features, the presence and absence of each of the individual human occupants in the indoor environment; and

during high-detection probability of presence of a particular individual human occupant, modifying a human-presence model for that particular individual human occupant so that relevant acoustic features specific to the particular individual human occupant and the indoor environment are emphasized, thereby improving the accuracy of the human-presence model for that particular individual human occupant over time.

2. The method of claim 1 further Comprising:

automatically modifying respective human presence or absence models corresponding to one or more individual human occupants, without human intervention or supervision, during high detection probability of individual human presence or absence in the indoor environment.

3. The method of claim 1, further comprising:

determining the likelihood scores for each of the plurality of models based at least on the comparing the acoustic events with each of the plurality of models, wherein the acoustic events include;

indoor speech;

footsteps;

household sounds; and

at least one of electronic audio or music.

4. The method of claim 1, wherein detecting the acoustic events in the audio signal includes:

extracting directions of the audio signals;

extracting magnitude of the audio signals;

extracting frequency spectrogram of the audio signals;

extracting autocorrelation coefficients of the audio signals; and

extracting time information of the audio signals.

5. The method of claim 1, wherein determining presence of individual human occupants in the indoor environment includes the computing a likelihood score for each of the plurality of models further comprising:

computing likelihood scores for presence and absence of individual human occupants associated with the respective models of the plurality of models; and

wherein the determining that the first individual human is present in the indoor environment is further based at least in part on determining a ratio of presence-score to absence-score corresponding to the first individual human is greater than a prescribed threshold.

6. The method of claim 5, wherein computing the likelihood score for presence or absence of each occupant in the indoor environment includes:

computing, using the respective models, detection probabilities of the detected acoustic events during presence or absence of a respective individual human occupant; and

computing from a human-presence model or a human-absence model the likelihood score for the presence or absence of the respective individual human occupant, from the detection probabilities of the acoustic events.

7. The method of claim 6, wherein computing the detection probabilities of acoustic events during presence or absence of each individual human occupant includes:

computing the acoustic features corresponding to the acoustic events from microphone array signals obtained from a microphone array in the indoor environment; and

computing the probability of acoustic features during presence and absence of each occupant by taking into account the prior probability distributions of acoustic features during determined presence and absence of each respective individual human occupant in the indoor environment.

8. The method of claim 1, wherein the refining at least one of the models of the plurality of models based at least in part on the determining that the first individual human is present in the indoor environment includes:

initializing a generic model for human presence or absence;

computing a probabilistic score for presence or absence of each individual human occupant from a joint probability of the acoustic events; and

updating a prior probability distribution of input features for a human-presence model or a human-absence model for each individual human occupant.

9. The method for claim 8, wherein updating the prior probability distribution of the input features for the human-presence model or the human-absence model of each individual human occupant includes:

confirmation of human presence or absence from one or more sensors that is different than a sensor that generated the audio signal; and

compute a score of presence or absence of each occupant from the human-presence model or human-absence model.

10. The method of claim 9 further comprising:

when presence or absence of an individual human occupant is confirmed or the score of presence and absence for an occupant exceeds a prescribed threshold;

updating joint probability of acoustic events and human presence or absence for the individual human occupant; and

updating a prior distribution of input features for the human-presence model or human absence model for the individual human occupant.

11. The method of claim 10, wherein the input features with updated prior distributions include:

direction of sound sources;

magnitude of the sound signal;

frequency spectrum;

autocorrelation coefficients;

time-of-day,

day-of-week, and

month-of-year.

12. The method of claim 10, wherein the acoustic events with updated joint probability include:

indoor human speech;

human footsteps; and

human-made household sounds.

13. The method of claim 1 wherein providing a database of models for determining presence of individual human occupants includes:

providing a system for identifying each of the individual human occupants in the indoor environment;

providing a database of discriminating features of individual human occupants within the indoor environment; and

updating the database of discriminating features for new individual human occupants within the indoor environment.

14. The method of claim 13, wherein the system for identifying each of the individual human occupants in the indoor environment comprises:

a microphone array for receiving the audio signal;

a vocal-tract length estimator that takes in the audio signal from the microphone array;

a pitch estimator that takes in the audio signal from the microphone array; and

a human individual detector coupled to the pitch and vocal-tract-length estimators and configured to identify one or more individual human speaker present within the indoor environment.

15. The method of claim 14, wherein the vocal-tract length estimator further includes:

a detector for schwa-like vowels that takes in the audio signal from the microphone array.

16. The method of claim 15, further comprising:

when schwa-like vowels are detected;

determining the formant peaks of vowels

determining the direction-of-arrival of the vowels;

determining the number of speakers from the direction-of-arrival of the vowels; and

storing the formant peaks for classification;

wherein if multiple speakers are present, configuring the formant peaks of the speakers to be mutually exclusive between the speakers.

17. The method of claim 14, wherein the pitch estimator further includes:

a detector for vowels of speech, and wherein;

one or more vowels are detected in speech in the audio signal;

extracting pitch features of the one or more vowels;

determining a direction-of-arrival of the one or more vowels;

determining a number of individual human speakers present in the indoor environment based at least in part on the determined direction-of-arrival and further based at least in part the one or more vowels; and

storing the pitch features for classification.

18. The method of claim 17, wherein storing the pitch features for classification includes:

in response to determining that multiple individual human speakers are present in the indoor environment, configuring the pitch features from the individual human speakers to be mutually exclusive between the individual human speakers.

19. The method of claim 13, further comprising:

determining pitch and vocal-tract-length features from the audio signal;

storing the pitch and vocal-tract-length features in a database;

performing semi-supervised classification on the stored pitch and vocal-tract-length features in a database;

wherein;

if the pitch and vocal-tract-length features come from multiple speakers;

configuring the pitch and vocal-tract-length features to be mutually exclusive between the individual human speakers during the semi-supervised classification.