Apparatus and method performing audio-video sensor fusion for object localization, tracking, and separation

• US 20060075422A1
• Filed: 11/30/2004
• Published: 04/06/2006
• Est. Priority Date: 09/30/2004
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choi, Changkyu, Kong, Donggeon, Lee, Hyoung-Ki, Yoon, Sang Min
• Granted Patent: US 7,536,029 B2
• Time in Patent Office: Days
• US Class Current: 725/18
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems

Abstract

An apparatus for tracking and identifying objects includes an audio likelihood module which determines corresponding audio likelihoods for each of a plurality of sounds received from corresponding different directions, each audio likelihood indicating a likelihood a sound is an object to be tracked; a video likelihood module which receives a video and determines video likelihoods for each of a plurality of images disposed in corresponding different directions in the video, each video likelihood indicating a likelihood that the image is an object to be tracked; and an identification and tracking module which determines correspondences between the audio likelihoods and the video likelihoods, if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifies and tracks a corresponding one of the objects using each determined pair of audio and video likelihoods.

Claims

1. An apparatus for tracking and identifying objects using received sounds and video, comprising:

an audio likelihood module which determines corresponding audio likelihoods for each of a plurality of the sounds received from corresponding different directions, each audio likelihood indicating a likelihood the sound is an object to be tracked;

a video likelihood module which determines video likelihoods for each of a plurality of images disposed in corresponding different directions in the video, each video likelihood indicating a likelihood that the image in the video is an object to be tracked; and

an identification and tracking module which;

determines correspondences between the audio likelihoods and the video likelihoods,

if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifies and tracks a corresponding one of the objects using each determined pair of audio and video likelihoods, and

if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifies a source of the sound or image as not being an object to tracked.

2. The apparatus of claim 1, wherein, when the identification and tracking module determines a correspondence between multiple pairs of audio and video likelihoods, the identification and tracking module identifies and individually tracks objects corresponding to each of the pairs.

3. The apparatus of claim 2, wherein the identification and tracking module identifies and tracks a location of each determined pair.

4. The apparatus of claim 1, wherein, for each image in the received video, the video likelihood module compares the image against a pre-selected image profile in order to determine the video likelihood for the image.

5. The apparatus of claim 1, wherein the pre-selected image profile comprises a color of an object to be tracked, and the video likelihood module compares a color of portions of the image in order to identify features indicative of an object to be tracked.

6. The apparatus of claim 1, wherein the pre-selected image profile comprises a shape of an object to be tracked, and the video likelihood module detects an outer edge of each image and compares the outer edge of each image against the shape to identify features indicative of an object to be tracked.

7. The apparatus of claim 6, wherein the pre-selected image profile further comprises poses for the object to be tracked, and the video likelihood module further compares each outer edge against each of the poses to identify features indicative of the object to be tracked.

8. The apparatus of claim 7, wherein the pre-selected image profile comprises a color of an object to be tracked, and the video likelihood module compares a color of portions of the image in order to identify features indicative of an object to be tracked.

9. The apparatus of claim 8, wherein the video likelihood module uses the identified images to normalize each outer edge in order to be closer to a size of the poses and the shape in order to identify features indicative of an object to be tracked.

10. The apparatus of claim 9, wherein the video likelihood identifies an outer edge as not being an object to be tracked if the outer edge does not correspond to the shape and the poses.

11. The apparatus of claim 9, wherein the video likelihood identifies an outer edge as not being an object to be tracked if the outer edge does not include the color.

12. The apparatus of claim 1, wherein:

a first one of the objects is disposed in a first direction,

a second one of the objects is disposed in a second direction, and

based on the correspondences between the audio and video likelihoods, the identification and tracking module identifies the first object as being in the first direction and the second object as being in the second direction.

13. The apparatus of claim 12, wherein the identification and tracking module tracks the first object as the first object moves relative to the second object.

14. The apparatus of claim 13, wherein:

the video likelihood module receives the video included the images from a camera, and

the identification and tracking module tracks and identifies the first object as the first object moves relative to the second object such that the first object crosses the second object from a perspective of the camera.

15. The apparatus of claim 1, further comprising a beam-former which, for each identified object, determines a location of the identified object, and separates from the received sounds audio corresponding to a location of each identified object so as to output audio channels corresponding uniquely to each of the identified objects.

16. The apparatus of claim 15, wherein:

the apparatus receives the sounds using a microphone array outputting a first number of received audio channels,

each received audio channel includes an element of the sounds,

the beam-former outputs a second number of the audio channels other than the first number, and

the second number corresponds to the number of identified objects.

17. The apparatus of claim 16, further comprising a recording apparatus which records each beam formed audio channel for each identified object as separate audio tracks associated with each object.

18. The apparatus of claim 15, wherein:

each output channel includes audible periods in which speech is detected and silent periods between corresponding audible periods in which speech is not detected, and

the apparatus further comprises a speech interval detector which detects, for each output channel, a start and stop time for each audible period.

19. The apparatus of claim 18, wherein the speech interval detector further:

detects a proximity between adjacent audible periods,

if the proximity is less than a predetermined amount, determines that the adjacent audible periods are one continuous audible period and connects the adjacent audible periods to form the continuous audible period, and

if the proximity is more than the predetermined amount, determines that the adjacent audible periods are separated by the silent period and does not connect the adjacent audible periods.

20. The apparatus of claim 18, wherein the speech interval detector further:

detects a length of each audible period,

if the length is less than a predetermined amount, determines that the audible period is a silent period and erases the audible period, and

if the length is more than the predetermined amount, determines that the audible period is not a silent period and does not erase the audible period.

21. The apparatus of claim 18, wherein the speech interval detector further:

for each audible period, outputs the detected speech, and

for each silent period, deletes the sound from the audio channel.

22. The apparatus of claim 15, further comprising a post processor which, for each of plural audio channels received from the beam-former, detects audio portions related to cross channel interference caused by the remaining audio channels and removes the cross channel interference.

23. The apparatus of claim 1, further comprising a controller which controls a robotic element according to the identified object.

24. The apparatus of claim 23, wherein the robotic element comprises at least one motor used to move the apparatus according to the identified object.

25. The apparatus of claim 23, wherein the robotic element comprises at least one motor used to remotely move an element connected to the apparatus through an interface according to the identified object.

26. The apparatus of claim 1, further comprising an omnidirectional camera which outputs a 360°

panoramic view image to the video likelihood module.

27. The apparatus of claim 1, further comprising at least one limited field of view camera which outputs an image to the video likelihood module which has a field of view that is less than 360°

.

28. The apparatus of claim 1, wherein:

the audio likelihood module further detects, for each received sound, an audio direction from which a corresponding sound is received,

the video likelihood module further detects, for each image, a video direction from which the image is observed, and

the identification and tracking module further determines the correspondences based upon a correspondence between the audio directions and the video directions.

29. The apparatus of claim 1, wherein the video received by the video likelihood module is an infrared video received from a pyrosensor.

30. A method of tracking and identifying objects using at least one computer receiving audio and video data, the method comprising:

for each of a plurality of sounds received from corresponding different directions, determining in the at least one computer corresponding audio likelihoods, each audio likelihood indicating a likelihood the sound is an object to be tracked;

for each of a plurality of images disposed in corresponding different directions in a video, determining in the at least one computer video likelihoods, each video likelihood indicating a likelihood that the image in the video is an object to be tracked;

if a correspondence is determined to exist between one of the audio likelihoods and one of the video likelihoods, identifying and tracking in the at least one computer a corresponding one of the objects using each determined pair of audio and video likelihoods, and

if a correspondence does not exist between a corresponding one of the audio likelihoods and a corresponding one of the video likelihoods, identifying in the at least one computer a source of the sound or image as not being an object to tracked.

31. The method of claim 30, further comprising determining correspondences between multiple pairs of audio and video likelihoods, wherein the identifying and tracking comprises individually tracking the objects corresponding to each of the determined pairs.

32. The method of claim 33, further comprising, for each determined pair, identifying a location of each determined pair.

33. The method of claim 30, wherein the determining the video likelihoods comprises comparing each of the images against a pre-selected image profile in order to determine the video likelihoods for the corresponding images.

34. The method of claim 30, wherein the pre-selected image profile comprises a color of an object to be tracked, and the determining the video likelihoods comprises comparing a color of portions of the image in order to identify features indicative of an object to be tracked.

35. The method of claim 30, wherein the pre-selected image profile comprises a shape of an object to be tracked, and the determining the video likelihoods comprises detecting an outer edge of each image and comparing the outer edge of each image against the shape to identify features indicative of an object to be tracked.

36. The method of claim 35, wherein the pre-selected image profile further comprises poses for the object to be tracked, and the determining the video likelihoods comprises comparing each outer edge against each of the poses to identify features indicative of the object to be tracked.

37. The method of claim 36. wherein the pre-selected image profile comprises a color of an object to be tracked, and the determining the video likelihoods comprises comparing a color of portions of the image in order to identify features indicative of an object to be tracked.

38. The method of claim 37, wherein the determining the video likelihoods comprises using the identified images to normalize each outer edge in order to be closer to a size of the poses and the shape in order to identify features indicative of an object to be tracked.

39. The method of claim 38, wherein the determining the video likelihoods comprises determining that an outer edge of the image does not correspond to an object to be tracked if the outer edge does not correspond to the shape and the poses.

40. The method of claim 38, wherein the determining the video likelihoods comprises determining that an outer edge of the image does not correspond to an object to be tracked if the outer edge does not include the color.

41. The method of claim 30, wherein:

a first one of the objects is disposed in a first direction,

a second one of the objects is disposed in a second direction, and

the method further comprises;

determining that a correspondence exists between a first pair of the audio likelihoods and one of the video likelihoods and that another correspondence exists between a second pair of the one of the audio likelihoods and one of the video likelihoods, and

based on the correspondences between first and second determined pairs of the audio and video likelihoods, identifying the first object as being in the first direction using the first pair and the second object as being in the second direction using the second pair.

42. The method of claim 41, wherein method further comprising tracking the first object as the first object moves relative to the second object.

43. The method of claim 42, wherein:

the at least one computer receives the video included the images from a camera, and

the tracking and identifying comprises tracking and identifying the first object as the first object moves relative to the second object such that the first object crosses the second object from a perspective of the camera.

44. The method of claim 30, further comprising a performing beam-forming by, for each identified object, determining a location of the identified object, and separating from the received sounds audio corresponding to a location of each identified object so as to output audio channels corresponding uniquely to each of the identified objects.

45. The method of claim 44, wherein:

the at least one computer receives the sounds using a microphone array outputting a first number of received audio channels,

each received audio channel includes an element of the sounds,

the beam-forming comprising outputting a second number of the audio channels other than the first number, and

the second number corresponds to the number of identified objects.

46. The method of claim 45, further comprising recording each of the beam formed audio channels for each identified object as separate audio tracks associated with each object.

47. The method of claim 44, wherein:

each output channel includes audible periods in which speech is detected and silent periods between corresponding audible periods in which speech is not detected, and

the method further comprises detecting a speech interval by, for each output channel, a start and stop time for each audible period.

48. The method of claim 47, wherein the detecting the speech interval further comprises:

detecting a proximity between adjacent audible periods,

if the proximity is less than a predetermined amount, determining that the adjacent audible periods are one continuous audible period and connecting the adjacent audible periods to form the continuous audible period, and

if the proximity is more than the predetermined amount, determining that the adjacent audible periods are separated by the silent period and does not connect the adjacent audible periods.

49. The method of claim 47, wherein the detecting the speech interval further comprises:

detecting a length of each audible period,

if the length is less than a predetermined amount, determining that the audible period is a silent period and erasing the audible period, and

if the length is more than the predetermined amount, determining that the audible period is not a silent period and not erasing the audible period.

50. The method of claim 47, wherein the detecting the speech interval further comprises:

for each audible period, outputting the detected speech, and

for each silent period, deleting the sound from the audio channel.

51. The method of claim 44, further comprising a post processing the beam formed audio channels by, for each of plural beam formed audio channels, detecting audio portions related to cross channel interference caused by the remaining audio channels and removing the cross channel interference.

52. The method of claim 30, further comprising controlling a robotic element according to the identified object.

53. The method of claim 52, wherein the robotic element comprises at least one motor, and the method further comprises controlling the motor to move an apparatus according to the identified object.

54. The method of claim 53, wherein the robotic element comprises at least one motor used to remotely move an element connected to the at least one computer through an interface according to the identified object.

55. The method of claim 30, wherein:

the determining the audio likelihood further comprises detecting, for each received sound, an audio direction from which a corresponding sound is received,

the determining the video likelihood further comprises detecting, for each image, a video direction from which the image is observed, and

the method further comprises determining the correspondences based upon a correspondence between the audio directions and the video directions.

56. A computer readable medium encoded with processing instructions for performing the method of claim 30 using the at least one computer.

57. A computer readable medium encoded with processing instructions for performing the method of claim 37 using the at least one computer.

58. A computer readable medium encoded with processing instructions for performing the method of claim 55 using the at least one computer.