Sound event detection and localization system using power analysis

• US 7,567,676 B2
• Filed: 05/02/2003
• Issued: 07/28/2009
• Est. Priority Date: 05/03/2002
• Status: Active Grant

• Current Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Original Assignee: Harman International Industries Incorporated (Samsung Electronics Co. Ltd.)
• Inventors: Griesinger, David H.
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): SUTHERS, DOUGLAS JOHN
• Application Number:
  US10/428,405
• Publication Number:
  US 20040005064A1
• Time in Patent Office: 2,279 Days
• Field of Search: 381/92, 381/56, 381/58, 381/122, 381/91, 381/111, 381/59, 381 17- 23, 381/77, 381 80- 81
• US Class Current: 381/56
• CPC Class Codes: G01S 3/8083 determining direction of source
• CPC Class Codes: G01V 1/001 Acoustic presence detection

Abstract

The beginning detection, accommodation and frequency bias properties of the human hearing mechanism have been modeled to create systems that can detect directional transients (“sound events”) in a sound field and localize them. These systems break down a sound field into sound events and non-sound events and separately localize the sound events and non-sound events. Sound events are generally identified according to the frequency bias and beginning detection properties. Once detected, the sound events are generally localized according to differential steering angles (steering angles to which the steady-state signals have been accommodated) or ordinary steering angles, both of which reflect the direction of a sound event indicated during the rise-time of the sound event. When no sound events are detected, non-sound events are localized according to a steering angle that does not reflect rapid motion.

Claims

1. A method for detecting and localizing a sound event in a sound in multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the multiple audio channels comprising at least one channel pair, the channel pair comprising a first audio channel and a second audio channel, the method comprising:

detecting the sound event;

comparing power, using at least one hardware processor, in the first audio channel with power in the second audio channel to produce a power differential steering angle of the power of the sound in the channel pair;

when the sound event is detected, selecting, using the at least one hardware processor, which of the multiple audio channels to produce the sound event based on the power differential steering angle; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

58. The method of claim 1, where the at least one channel pair comprises a first channel pair;

where the multiple audio channels comprise the first channel pair and at least one additional channel pair; and

where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether to select only the first channel pair to produce the sound event or whether to select the first channel pair and the at least one additional channel pair to produce the sound event.

59. The method of claim 58, where the first channel pair comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

60. The method of claim 59, where the sound is recorded;

where the input channel pair comprises a left input channel and a right input channel;

where the first voltage comprises a recorded left input channel voltage;

where the second voltage comprises a recorded right input channel voltage; and

where the power differential steering angle comprise a left/right power differential steering angle.

61. The method of claim 1, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises selecting two channels or more than two channels.

62. The method of claim 61, where, when two channels are selected, producing the sound event comprises producing the sound event in stereo;

and

where, when more than two channels are selected, producing the sound event comprises producing the sound event in surround.

63. The method of claim 1, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises analyzing a direction of the power differential steering angle.

64. The method of claim 63, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether the direction of the power differential steering angle comprises a predetermined direction for at least a predetermined time.

65. The method of claim 64, where the predetermined direction comprises 0°

to about −

45°

indicating a rear direction of the power differential steering angle.

66. The method of claim 64, where an input channel pair comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

2. A method for detecting and localizing a sound event in a sound in multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the method comprising:

detecting the sound event;

when the sound event is detected, analyzing, using at least one hardware processor, power in the multiple audio channels to determine a power differential steering angle of the power in the sound in the multiple audio channels;

selecting, using the at least one hardware processor, which of the multiple audio channels to produce the sound event based on the power differential steering angle; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event, thereby localizing the sound event.

3. The method of claim 2, where determining the power differential steering angle of the power in the sound in the multiple audio channels includes isolating the power differential steering angle of the power of the sound indicated during a predetermined rise time.

4. The method of claim 3, where the power differential steering angle of the power of the sound in the multiple audio channels indicated during the predetermined rise time is used for a predetermined duration.

67. The method of claim 2, where the multiple audio channels comprise a first channel pair and at least one additional channel pair;

and

where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether to select only the first channel pair to produce the sound event or whether to select the first channel pair and the at least one additional channel pair to produce the sound event.

68. The method of claim 67, where the first channel pair comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

69. The method of claim 68, where the sound is recorded;

where the input channel pair comprises a left input channel and a right input channel;

where the first voltage comprises a recorded left input channel voltage;

where the second voltage comprises a recorded right input channel voltage; and

where the power differential steering angle comprise a left/right power differential steering angle.

70. The method of claim 2, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises selecting two channels or more than two channels.

71. The method of claim 70, where, when two channels are selected, producing the sound event comprises producing the sound event in stereo;

and

where, when more than two channels are selected, producing the sound event comprises producing the sound event in surround.

72. The method of claim 2, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises analyzing a direction of the power differential steering angle.

73. The method of claim 72, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether the direction of the power differential steering angle comprises a predetermined direction for at least a predetermined time.

74. The method of claim 73, where the predetermined direction comprises 0°

to about −

45°

indicating a rear direction of the power differential steering angle.

75. The method of claim 73, where the multiple audio channels comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

5. A method of detecting and localizing a sound event in a sound in multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the method comprising:

detecting a plurality of sound event types, each sound event type distinguished from background noise based on at least one characteristic of the sound event type;

when one of the plurality of sound event types is detected, analyzing, using at least one hardware processor, power in the multiple audio channels to determine a power differential steering angle of the power in the sound in the multiple audio channels;

selecting, using the at least one hardware processor, which of the multiple audio channels to produce the sound event based on the determined power differential steering angle and based on the detected sound event type; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event, thereby localizing the sound event.

6. The method of claim 5, where the one of the plurality of sound event types that is detected defines a detected sound event type and determining the power differential steering angle of the power in the sound includes isolating the power differential steering angle of the power of the sound indicated during a predetermined rise time of the detected sound event type.

7. The method of claim 5, where the one of the plurality of sound event types that is detected defines a detected sound event type and determining the power differential steering angle of the power in the sound includes isolating the power differential steering angle of the power of the sound for a predetermined duration of the detected sound event type.

8. A method of detecting and localizing a sound event in a sound produced by a plurality of input channel pairs each comprising multiple audio channels in order to determine how to produce the sound event in the multiple audio channels, the multiple audio channels comprising a left-right channel pair and at least one additional channel pair, the method comprising:

detecting a sound event in the plurality of input channel pairs;

when the sound event is detected, analyzing, using at least one hardware processor, power in the left-right channel pair to determine a left-right power differential angle;

analyzing, using the at least one hardware processor, power in the at least one additional channel pair to determine a second power differential angle;

analyzing, using the at least one hardware processor, the left-right power differential angle and the second power differential angle; and

producing the sound event in the multiple audio channels based on analyzing the left-right power differential angle and the second power differential angle.

9. The method of claim 8, further comprising, when the sound event is detected, determining whether at least one of the left-right power differential angle or the second power differential angle is accurate.

10. The method of claim 9, where the sound includes a background signal and determining whether at least one of the left-right power differential angle or the second power differential angle is accurate includes determining whether the background signal has dropped from that of a previous sound event.

11. The method of claim 10, where determining if the background signal has dropped includes determining if the background signal has dropped by at least about 3 dB.

12. The method of claim 9, where determining whether at least one of the left-right power differential angle or the second power differential angle is accurate includes determining whether a sum of absolute values of the left-right power differential angle and the second power differential angle is at most a predetermined value.

13. The method of claim 9, where determining whether at least one of the left-right power differential angle or the second power differential angle is accurate includes increasing an accuracy of the power differential steering angle of the power of the sound by determining a number of instances that the power differential steering angle of the power of the sound is not accurate during a predetermined period.

14. The method of claim 13, where detecting the sound event includes determining a threshold and increasing the accuracy of the power differential steering angle of the power of the sound further includes adjusting the threshold.

15. The method of claim 9, further comprising, if at least one of the left-right power differential angle or the second power differential angle is not accurate, determining and selecting a steering angle of the sound for the plurality of the input channel pairs, thereby localizing the sound event.

16. The method of claim 15, where determining the steering angle of the sound for the plurality of input channel pairs includes isolating the steering angle of the sound indicated during a predetermined rise time.

17. A method of detecting and localizing a sound event in a sound produced by a plurality of input channel pairs each comprising multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the method comprising:

detecting a plurality of sound event types; and

when one of the plurality of sound event types is detected, analyzing, using at least one hardware processor, power in one or more input channel pairs to determine a power differential steering angle of the power in the sound in the plurality of input channel pairs;

selecting, using the at least one hardware processor, which of the multiple audio channels to produce the sound event based on the determined power differential steering angle and based on the detected sound event type; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

18. The method of claim 17, further comprising, when one of the plurality of sound event types is detected, determining whether the power differential steering angle of the power of the sound is accurate.

19. The method of claim 18, further comprising, if the power differential steering angle of the power of the sound is accurate, determining the power differential steering angle of the power of the sound for the plurality of input channel pairs localizes the sound event.

20. The method of claim 19, where the one of the plurality of sound event types that is detected defines a detected sound event type, and where the power differential steering angle of the power of the sound is determined during a predetermined rise time of the detected sound event type.

21. The method of claim 20, where the power differential steering angle of the power of the sound indicated during the predetermined rise time of the detected sound event type is used for a predetermined duration of the detected sound event type.

22. A computer-readable medium comprising computer-executable instructions for detecting and localizing a sound event in a sound in multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the multiple audio channels comprising at least one channel pair, the channel pair comprising a first audio channel and a second audio channel, the computer-executable instructions comprising logic for performing the steps of:

detecting the sound event;

analyzing power in the first audio channel and in the second audio channel to produce a power differential steering angle of the power of the sound in the multiple audio channels;

when the sound event is detected, selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

23. An apparatus for detecting and localizing a sound event in a sound in multiple audio channels in order to determine in which of the multiple audio channels to produce the sound event, the multiple audio channels comprising at least one channel pair, the channel pair comprising a first audio channel and a second audio channel, the apparatus comprising a processor configured to perform the following:

detecting the sound event;

analyzing power in the first audio channel and in the second audio channel to produce a power differential steering angle of the power of the sound in the multiple audio channels;

when the sound event is detected, selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle; and

producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

24. A sound event detector, localizer, and sound producer for detecting and localizing a sound event in a sound in multiple audio channels and producing the sound event in the multiple audio channels, the multiple audio channels comprising at least one channel pair, the channel pair comprising a first audio channel and a second audio channel, comprising:

a sound event detector in communication with the sound and producing a first signal indicating when a sound event is detected;

a sound event localization circuit in communication with the sound event detector and including a differential steering angle circuit which, when the first signal indicates that the sound event is detected, produces a second signal based on analysis of power in the first audio channel with power in the second audio channel, the second signal indicating a power differential steering angle of a power of the sound; and

a sound producer in communication with the sound event localization circuit for selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle and producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

25. The detector, localizer, and sound producer of claim 24, where the detector and localizer is implemented by computer logic according to computer-executed instructions stored in a computer-readable medium.

26. The detector, localizer, and sound producer of claim 24, where the detector and localizer is implemented by computer logic according to computer-executed instructions embodied in a computer-readable electromagnetic signal.

27. The detector, localizer, and sound producer of claim 24, where the sound event localization circuit further include a sample-and-hold circuit in communication with the differential steering angle circuit from which the sample-and-hold circuit receives the second signal.

28. The detector, localizer, and sound producer of claim 27, where the sample-and-hold circuit is further in communication with the sound event detector from which the sample-and-hold circuit receives the first signal.

29. The detector, localizer, and sound producer of claim 28, where, when the first signal indicates that the sound event is detected, the sample-and-hold circuit isolates the second signal for a predetermined rise time.

30. The detector, localizer, and sound producer of claim 29, where the sound event includes a sound event type and the predetermined rise time is that of the sound event type.

31. The detector, localizer, and sound producer of claim 30, where the sound event type is a syllable.

32. The detector, localizer, and sound producer of claim 31, where the predetermined rise time of the syllable is about 20 ms to about 30 ms.

33. The detector, localizer, and sound producer of claim 29, where the sample-and-hold circuit holds the second signal for a predetermined duration of the sound event type.

34. The detector, localizer, and sound producer of claim 33, where the sound event type is an impulsive sound.

35. The detector, localizer, and sound producer of claim 34, where the predetermined duration of the impulsive sound is about 50 ms.

76. The detector, localizer, and sound producer of claim 24, where the at least one channel pair comprises a first channel pair;

where the multiple audio channels comprise the first channel pair and at least one additional channel pair; and

where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether to select only the first channel pair to produce the sound event or whether to select the first channel pair and the at least one additional channel pair to produce the sound event.

77. The detector, localizer, and sound producer of claim 76, where the first channel pair comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

78. The detector, localizer, and sound producer of claim 77, where the sound is recorded;

where the input channel pair comprises a left input channel and a right input channel;

where the first voltage comprises a recorded left input channel voltage;

where the second voltage comprises a recorded right input channel voltage; and

where the power differential steering angle comprise a left/right power differential steering angle.

79. The detector, localizer, and sound producer of claim 24, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises selecting two channels or more than two channels.

80. The detector, localizer, and sound producer of claim 79, where, when two channels are selected, the sound producer produces the sound event in stereo;

and

where, when more than two channels are selected, the sound producer produces the sound event in surround.

81. The detector, localizer, and sound producer of claim 24, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises analyzing a direction of the power differential steering angle.

82. The detector, localizer, and sound producer of claim 81, where selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle comprises determining whether the direction of the power differential steering angle comprises a predetermined direction for at least a predetermined time.

83. The detector, localizer, and sound producer of claim 82, where the predetermined direction comprises 0°

to about −

45°

indicating a rear direction of the power differential steering angle.

84. The detector, localizer, and sound producer of claim 82, where an input channel pair comprises a first input channel and a second input channel;

where a first voltage is associated with the first input channel and a second voltage is associated with the second input channel;

where the power in the first audio channel comprises a squaring of the first voltage for the first input channel; and

where the power in the first audio channel comprises a squaring of the second voltage for the first input channel.

36. A sound event detector, localizer, and sound producer for detecting and localizing a sound event in a sound in multiple audio channels and producing the sound event in the multiple audio channels, the multiple audio channels comprising a plurality of channel pairs, comprising:

a sound event detector in communication with the sound and producing a plurality of first signals indicating when a plurality of sound event types is detected;

a sound event localization circuit in communication with the sound event detector and including a differential steering angle circuit which, when any of the plurality of first signals indicate that one of the plurality of sound events is detected, produces a second signal based on analysis of power in the plurality of audio channels, the second signal indicating a power differential steering angle of a power of the sound; and

a sound producer in communication with the sound event localization circuit for selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle and producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

37. The detector, localizer, and sound producer of claim 36, where the detector and localizer is implemented by computer logic according to computer-executed instructions stored in a computer-readable medium.

38. The detector, localizer, and sound producer of claim 36, where the sound event localization circuit further includes a plurality of sample-and-hold circuits in communication with the differential steering angle circuit from which the sample-and-hold circuits receive the second signal.

39. The detector, localizer, and sound producer of claim 38, where each of the sample-and-hold circuits are further in communication with one of the first signals.

40. The detector, localizer, and sound producer of claim 39, where, when the one of the first signals with which one of the sample-and-hold circuits is in communication indicates that one of the sound event types is detected, the one of the sample-and-hold circuits isolates the second signal during a predetermined rise time.

41. The detector, localizer, and sound producer of claim 40, where, the one of the sound event types that is detected defines a detected sound event type and the predetermined rise time is that of the detected sound event type.

42. A sound event detector, localizer, and sound producer for detecting, localizing, and producing a sound event in multiple audio channels, the multiple audio channels comprising a plurality of channel pairs, comprising:

a sound event detector in communication with the plurality of input channel pairs and producing a first signal indicating when the sound event is detected in the plurality of input channel pairs;

a sound event localization circuit in communication with the sound event detector and including a plurality of differential steering angle circuits, which, when the first signal indicates that the sound event is detected, produces a plurality of second signals based on analysis of power in each of the plurality of channel pairs, the plurality of second signals indicating a power differential steering angle of a power of the sound in the plurality of channel pairs; and

a sound producer in communication with the sound event localization circuit for selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle and producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

43. The detector, localizer, and sound producer of claim 42, where the sound event localization circuit further includes a plurality of sample-and-hold circuits in communication with the plurality of differential steering angle circuits from which the sample-and-hold circuits receive the second signals.

44. The detector, localizer, and sound producer of claim 43, where the sample-and-hold circuits are further in communication with the sound event detector from which the sample-and-hold circuits receive the first signal.

45. The detector, localizer, and sound producer of claim 44, where, when the first signal indicates that the sound event is detected, the sample-and-hold circuits isolate the second signals during a predetermined rise time.

46. The detector, localizer, and sound producer of claim 44, where the sound event localizer further includes a verification circuit, which is in communication with the differential steering angle circuits and the sample-and-hold circuits and communicates to the sample-and-hold circuits a verification signal indicating whether the plurality of second signals is accurate.

47. The detector, localizer, and sound producer of claim 46, where, when the first signal indicates that the sound event is detected and if the verification signal indicates that the second signals are accurate, the sample-and-hold circuits isolate the second signals for a predetermined rise time.

48. A sound event detector, localizer, and sound producer for detecting, localizing, and producing a sound event in multiple audio channels, the multiple audio channels comprising a plurality of channel pairs, comprising:

a sound event detector in communication with the plurality of input channel pairs and producing a plurality of first signals indicating when a plurality of sound event types is detected in the plurality of input channel pairs; and

a sound event localization circuit in communication with the sound event detector and including a plurality of differential steering angle circuits, which, when any of the first signals indicate that one of the plurality of sound event types are detected, produces a plurality of second signals based on analysis of power in each of the plurality of channel pairs, the plurality of second signals indicating a power differential steering angle of a power of the sound in the plurality of channel pairs; and

a sound producer in communication with the sound event localization circuit for selecting which of the multiple audio channels to produce the sound event based on the power differential steering angle and producing the sound event in the multiple audio channels based on selecting which of the multiple audio channels to produce the sound event.

49. The detector, localizer, and sound producer of claim 48, where the sound event localization circuit further includes a plurality of sample-and-hold circuits in communication with the plurality of differential steering angle circuits from which the sample-and-hold circuits receive the second signals.

50. The detector, localizer, and sound producer of claim 49, where each of the sample and hold circuits are further in communication with one of the first signals.

51. The detector, localizer, and sound producer of claim 50, where, when the one of the first signals with which the any of the sample-and-hold circuits are in communication indicates that one of the sound event types is detected, the sample-and-hold circuits in communication with the one of the first signals isolate the second signals during a predetermined rise time.

52. The detector, localizer, and sound producer of claim 50, where the one of the sound event types that is detected defines a detected sound event type and the predetermined rise time is that of the detected sound event type.

53. The detector, localizer, and sound producer of claim 50, where the sound event localizer further includes a verification circuit, which is in communication with the differential steering angle circuits and the sample-and-hold circuits and communicates to the sample-and-hold circuits a verification signal indicating whether the plurality of second signals is accurate.

54. The detector, localizer, and sound producer of claim 51, where, when the one of the first signals with which any of the sample-and-hold circuits are in communication indicates that one of the sound event types is detected and if the verification signal indicates that the second signals are accurate, the sample-and-hold circuits in communication with the one of the first signals isolate the second signals during a predetermined rise time.

55. The detector, localizer, and sound producer of claim 52, where the sound event localizer further includes a plurality of ordinary steering angle circuits that is in communication with the plurality of sample-and-hold circuits and produces a plurality of third signals indicating a steering angle of the sound by the input channel pairs.

56. The detector, localizer, and sound producer of claim 55, where, when the one of the first signals with which any of the sample-and-hold circuits are in communication indicates that one of the sound event types is detected and if the verification signal indicates that the second signals are not accurate, the sample-and-hold circuits in communication with the one of the first signals isolate the third signals during a predetermined rise time.

57. A sound event detector and localizer for detecting and localizing a sound event in a sound in different audio channels in order to determine in which of the different audio channels to produce the sound event, comprising:

means for producing a signal indicating when a sound event is detected;

means for localizing the sound event in communication with the means for producing the signal and including a means for comparing power in the different audio channels to determine a power differential steering angle of the power of the sound when the means for producing indicates that the sound event is detected, thus localizing the sound event;

means for selecting which of the different audio channels to produce the sound event based on the power differential steering angle; and

means for producing the sound event in the different audio channels based on selecting which of the different audio channels to produce the sound event.