Determining audio event based on location information

• US 10,134,422 B2
• Filed: 12/01/2015
• Issued: 11/20/2018
• Est. Priority Date: 12/01/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hwang, Kyu Woong, Cho, Yongwoo, Cho, Jun-Cheol, Moon, Sunkuk
• Primary Examiner(s): Pham, Thierry L
• Application Number:
  US14/956,212
• Publication Number:
  US 20170154638A1
• Time in Patent Office: 1,085 Days
• Field of Search: 704273, 704274, 704275, 704278
• CPC Class Codes: G01S 3/803 using amplitude comparison of signals derived from receiving transducers or transducer systems having differently-oriented directivity characteristics
• CPC Class Codes: G01S 3/808 using transducers spaced apart and measuring phase or time difference between signals therefrom, i.e. path-difference systems
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 5/20 Position of source determined by a plurality of spaced direction-finders
• CPC Class Codes: G01S 5/22 Position of source determined by co-ordinating a plurality of position lines defined by path-difference measurements G01S5/28 takes precedence
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/1672 using sonic detecting means, e.g. a microphone operating in the audio frequency range
• CPC Class Codes: G08B 13/19695 Arrangements wherein non-video detectors start video recording or forwarding but do not generate an alarm themselves
• CPC Class Codes: G08B 3/10 using electric transmission; using electromagnetic transmission
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 21/028 using properties of sound source
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/72 for transmitting results of analysis

Abstract

A method of determining, by an electronic device, an audio event is disclosed. The method may include receiving an input sound from a sound source by a plurality of sound sensors. The method may also extracting, by a processor, at least one sound feature from the received input sound, determining, by the processor, location information of the sound source based on the input sound received by the sound sensors, determining, by the processor, the audio event indicative of the input sound based on the at least one sound feature and the location information, and transmitting, by a communication unit, a notification of the audio event to an external electronic device.

Claims

1. A method of determining, by an electronic device, one or more audio events, the method comprising:

receiving, by a plurality of sound sensors, an input sound from a sound source;

extracting, by a processor, at least one sound feature from the input sound;

determining, by the processor, location information of the sound source based on the input sound received by the plurality of sound sensors;

identifying, by the processor, at least one object associated with the location information of the sound source based on environmental information, wherein the environmental information includes location data of the at least one object, wherein the location data of the at least one object indicates a position of the at least one object relative to one or more sound sensors of the plurality of sound sensors, and wherein the at least one object is distinct from the plurality of sound sensors;

determining, by the processor, an audio event indicative of the input sound based on the location information of the sound source, the at least one object associated with the location information of the sound source, and the at least one sound feature; and

transmitting, by a communication unit, a notification of the audio event to an external electronic device.

2. The method of claim 1, wherein the location information of the sound source indicates a position of the sound source relative to the one or more sound sensors of the plurality of sound sensors, wherein the environmental information further includes room type identification information, historical data of previously determined audio events near the position indicated by the location information, or a combination thereof, and wherein the position of the at least one object indicated by the location data is located near the position of the sound source indicated by the location information of the sound source.

3. The method of claim 1, wherein determining the audio event comprises:

modifying a detection model associated with the audio event based on the at least one object associated with the location information of the sound source; and

determining the audio event based on the modified detection model.

4. The method of claim 1, wherein at least one location near the plurality of sound sensors is associated with one or more types of audio events, and wherein transmitting the notification of the audio event comprises transmitting the notification of the audio event responsive to determining that the audio event corresponds to a particular type of audio event of the one or more types of audio events associated with a position indicated by the location information.

5. The method of claim 1, further comprising:

adjusting a direction of an image sensor towards the sound source based on the location information; and

capturing, by the image sensor, at least one image, wherein the at least one object is identified further based on the at least one image.

6. The method of claim 1, wherein the location information of the sound source includes at least one of a direction of the sound source or a position of the sound source relative to the one or more sound sensors of the plurality of sound sensors.

7. The method of claim 1, wherein at least one sound sensor of the plurality of sound sensors is located in a room, wherein the at least one object is located in the room, and wherein the position of the at least one object indicated by the location data further indicates a particular position in the room where the at least one object is located.

8. The method of claim 1, wherein the plurality of sound sensors are located in a plurality of rooms, wherein the environmental information further includes room identification information of each room of the plurality of rooms, and wherein the room identification information for each of the rooms includes room location information and a type of the room.

21. The method of claim 1, wherein the audio event comprises a speech sound, a screaming sound, a crying sound, a door knocking sound, a glass breaking sound, a gunshot sound, a footstep sound, a falling sound, or a television sound.

22. The method of claim 1, further comprising determining, by the processor, a second audio event indicative of a second input sound based on a second sound feature, second location information of a second sound source of the second input sound, and the environmental information, wherein the second input sound corresponds to a gunshot sound produced by a television, and wherein determining the second audio event includes:

determining that a position of the second sound source corresponds to a position associated with the television based on the second location information and the environmental information;

determining that the second audio event corresponds to a sound produced by the television; and

refraining from transmitting a notification based on determining that the second audio event corresponds to the sound produced by the television.

23. The method of claim 1, wherein determining the audio event is performed by a device that comprises a mobile device.

27. The method of claim 1, wherein the at least one sound feature corresponds to one or more Mel-frequency cepstral coefficients (MFCC), and wherein the audio event corresponds to a type of sound or a class of sound.

28. The method of claim 1, wherein determining the audio event comprises:

selecting a particular sound type from a plurality of sound types based on a classification model, the particular sound type corresponding to a sound type of the input sound; and

determining, based on the location data, that the particular sound type is associated with a position indicated by the location information of the sound source.

30. The method of claim 1, wherein determining the audio event comprises:

modifying a classification model based on the at least one object associated with the location information of the sound source; and

calculating detection probabilities of the one or more sound features for the audio events based on the modified classification model.

9. An electronic device comprising:

a plurality of sound sensors configured to receive an input sound from a sound source;

a feature extractor configured to extract at least one sound feature from the input sound;

a location determination unit configured to determine location information of the sound source based on the input sound received by the plurality of sound sensors; and

an audio event determination unit configured to;

identify at least one object associated with the location information of the sound source based on environmental information, wherein the environmental information includes location data of the at least one object, wherein the location data of the at least one object indicates a position of the at least one object relative to one or more sound sensors of the plurality of sound sensors, and wherein the at least one object is distinct from the plurality of sound sensors; and

determine an audio event indicative of the input sound based on the location information of the sound source, the at least one object associated with the location information of the sound source, and the at least one sound feature.

10. The electronic device of claim 9, wherein the audio event determination unit is configured to:

modify a detection model associated with the audio event based on the location information and the environmental information; and

determine the audio event based further on the modified detection model.

11. The electronic device of claim 9, further comprising a notification generating unit configured to generate a notification of the audio event responsive to determining that the audio event is an audio event type or audio event class associated with a position indicated by the location information.

12. The electronic device of claim 9, wherein the plurality of sound sensors are located in at least one room, and wherein the location data of the at least one object is associated with the at least one room.

24. The electronic device of claim 9, further comprising:

an antenna; and

a transmitter coupled to the antenna and configured to transmit a notification of the audio event to an external electronic device via the antenna.

25. The electronic device of claim 24, wherein the plurality of sound sensors, the feature extractor, the location determination unit, the audio event determination unit, the transmitter, and the antenna are integrated into a mobile device.

29. The electronic device of claim 9, wherein the audio event determination unit is configured to:

identify that the audio event is associated with the at least one sound feature based on a classification model; and

verify that the identified audio event is indicative of the input sound based on the at least one object associated with the location information of the sound source.

13. An electronic device comprising:

means for receiving an input sound from a sound source;

means for extracting at least one sound feature from the input sound;

means for determining location information of the sound source based on the input sound received by the means for receiving an input sound;

means for identifying at least one object associated with the location information of the sound source based on environmental information, wherein the environmental information includes location data of the at least one object, wherein the location data of the at least one object indicates a position of the at least one object relative to the means for receiving, and wherein the at least one object is distinct from the means for receiving; and

means for determining an audio event indicative of the input sound based on the location information of the sound source, the at least one object associated with the location information of the sound source, and the at least one sound feature.

14. The electronic device of claim 13, wherein the environmental information further includes historical data indicating prior occurrences of the audio event at a position indicated by the location information.

15. The electronic device of claim 13, wherein the means for determining the audio event is configured to:

determine a detection probability of the audio event based on the at least one sound feature;

modify the detection probability based on the location information and the environmental information; and

determine the audio event based on the modified detection probability.

16. The electronic device of claim 13, further comprising means for generating a notification of the audio event responsive to determining that a score associated with the audio event is greater than or equal to a threshold.

26. The electronic device of claim 13, wherein the means for receiving, the means for extracting, the means for determining the location information, the means for determining the audio event, and the means for determining whether the audio event is indicative of the input sound are integrated into a mobile device.

17. A non-transitory computer-readable storage medium comprising instructions causing at least one processor of an electronic device to perform operations comprising:

receiving, via a plurality of sound sensors, an input sound from a sound source;

extracting at least one sound feature from the input sound;

determining location information of the sound source based on the input sound received by the plurality of sound sensors;

identifying at least one object associated with the location information of the sound source based on environmental information, wherein the environmental information includes location data of the at least one object, wherein the location data of the at least one object indicates a position of the at least one object relative to one or more sound sensors of the plurality of sound sensors, and wherein the at least one object is distinct from the plurality of sound sensors;

determining an audio event indicative of the input sound based on the location information of the sound source, the at least one object associated with the location information of the sound source, and the at least one sound feature; and

transmitting a notification of the audio event to an external electronic device.

18. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise comparing a number of occurrences of particular previously determined audio events near a position indicated by the location information and having the same type of sound as the audio event to a threshold, and wherein the notification is transmitted based on the number being less than the threshold.

19. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise storing, at a memory of the electronic device, the audio event, the location information of the sound source, and a timestamp indicative of a time the input sound is received by the plurality of sound sensors.

20. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise:

updating an audio event log responsive to determining the audio event and prior to determining whether the audio event is indicative of the input sound;

capturing, via an image sensor, one or more images of a portion of a room near a position indicated by the location information, wherein the plurality of sounds sensors are located in the room; and

determining the environmental information based on the one or more images.