Method and apparatus for tracking moving objects using combined video and audio information in video conferencing and other applications

• US 6,850,265 B1
• Filed: 04/13/2000
• Issued: 02/01/2005
• Est. Priority Date: 04/13/2000
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Lee, Mi-Suen, Strubbe, Hugo J.
• Primary Examiner(s): Eng, George
• Application Number:
  US09/548,734
• Time in Patent Office: 1,755 Days
• Field of Search: 348 1401- 1403, 348/14.05, 348 1407- 1409, 348/14.16, 348142-143, 348169-170, 348/211.99, 348/211.1, 3482118-21112, 348/208.14, 348/211.14
• US Class Current: 348/14.05
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A video processing system tracks a moving person or other object of interest using a combined audio-video tracking system. The audio-video tracking system comprises an audio locator, a video locator, and a set of rules for determining the manner in which settings of a camera are adjusted based on outputs of the audio locator and video locator. The set of rules may be configured such that only the audio locator output is used to adjust the camera settings if the audio locator and video locator outputs are not sufficiently close and a confidence indicator generated by the audio locator is above a specified threshold. For example, in such a situation, the audio locator output alone may be used to direct the camera to a new speaker in a video conference. If the audio locator and video locator outputs are sufficiently close, the system determines if a confidence indicator generated by the video locator is above a specified level, and if so, the video locator output may be used to adjust the camera settings. For example, the camera may be zoomed in such that the face of a video conference participant is centered in and occupies a designated portion of a video frame generated by the camera.

Claims

1. A method for tracking an object of interest in a video processing system, the method comprising the steps of:

generating for particular ones of successive plural measurement intervals an audio locator output from an audio input derived from detecting sound from an object, and a video locator output from a video input derived partly from a camera detecting movement of an object, each indicative of a location of the object of interest;

applying a set of confidence level rules to each of the audio locator output and video locator output to determine which one of the audio locator output and the video locator output has a higher confidence level, whereby the one having the highest confidence level will be utilized independently from the other to adjust a setting of the camera during each one of said successive plural measurement intervals, but if in a measurement interval the confidence levels are equivalent, the video locator output is used if above an established threshold, otherwise the audio locator output is utilized; and

adjusting the camera setting utilizing only the selected one of the audio locator output and the video locator output in accordance with the applied set of confidence level rules.

2. The method of claim 1 wherein the object of interest comprises a moving person.

3. The method of claim 1 wherein the camera is a pan-tilt-zoom (PTZ) camera having adjustable pan, tilt and zoom settings.

4. The method of claim 1 wherein the set of rules includes determining if the audio locator and video locator outputs are sufficiently close for the given measurement interval, and utilizing only the audio locator output to adjust the camera setting if the audio and video locator outputs are not within a specified range of one another for the given measurement interval.

5. The method of claim 4 wherein the set of rules further includes utilizing the video locator output to adjust the camera setting only if the audio and video locator outputs are within a specified range of one another for the given measurement interval.

6. The method of claim 1 wherein the set of rules includes determining based on the audio locator output if the object of interest corresponds to a new speaker in a multiple-participant system, and if a new speaker is detected, directing the camera to zoom out by a predetermined amount and to turn in a direction of the new speaker.

7. The method of claim 1 wherein the set of rules includes determining based on the audio locator output if the object of interest corresponds to a same speaker in a multiple-participant system, and if the same speaker is detected, utilizing the video locator output to adjust the camera setting so as place the same speaker at a designated position within one or more video frames generated by the camera.

8. The method of claim 7 wherein the set of rules further includes adjusting a zoom setting of the camera until a head of the identified same speaker occupies a designated portion of a given one of the one or more video frames generated by the camera.

9. The method of claim 1 wherein the set of rules specifies that the camera is zoomed out by a predetermined amount after a detected period of continued silence exceeds a first amount of time.

10. The method of claim 9 wherein the set of rules further specifies that the camera is zoomed out by an additional amount if the detected period of continued silence exceeds a second amount of time greater than the first amount of time.

11. The method of claim 1 wherein the set of rules further includes determining if a confidence indicator associated with the video locator output is above a specified video locator threshold for the given measurement interval, and utilizing the video locator output to adjust the camera setting only if the video locator confidence indicator is above the video locator threshold for the given measurement interval.

12. An apparatus for tracking an object of interest in a video processing system, the apparatus comprising:

a camera; and

a processor coupled to the camera and operative (i) to process an audio locator output from an audio input signal, and a video locator output from a video input signal derived partly from movement of the object, each indicative of a location of the object of interest for particular ones of given measurement intervals of a plurality of successive measurement intervals; and

(ii) to apply a set of confidence level rules to each of the audio locator output and the video locator output to determine which one of the audio locator output and the video locator output has a higher confidence level, whereby the one having the highest confidence level will be utilized independently of the other to adjust a setting of the camera based on the given measurement interval.

13. The apparatus of claim 12 wherein the object of interest comprises a moving person.

14. The apparatus of claim 12 wherein the camera is a pan-tilt-zoom (PTZ) camera having adjustable pan, tilt and zoom settings.

15. The apparatus of claim 12 wherein the set of rules includes determining if the audio locator and video locator outputs are sufficiently close for the given measurement interval, and utilizing only the audio locator output to adjust the camera setting if the audio and video locator outputs are not within a specified range of one another for the given measurement interval.

16. The apparatus of claim 15 wherein the set of rules further includes utilizing the video locator output to adjust the camera setting only if the audio and video locator outputs are within a specified range of one another for the given measurement interval.

17. The apparatus of claim 12 wherein the set of rules includes determining based on the audio locator output if the object of interest corresponds to a new speaker in a multiple-participant system, and if a new speaker is detected, directing the camera to zoom out by a predetermined amount and to turn in a direction of the new speaker.

18. The apparatus of claim 12 wherein the set of rules includes determining based on the audio locator output if the object of interest corresponds to a same speaker in a multiple-participant system, and if the same speaker is detected, utilizing the video locator output to adjust the camera setting so as place the same speaker at a designated position within one or more video frames generated by the camera.

19. The apparatus of claim 18 wherein the set of rules further includes adjusting a zoom setting of the camera until a head of the identified same speaker occupies a designated portion of a given one of the one or more video frames.

20. The apparatus of claim 12 wherein the set of rules specifies that the camera is zoomed out by a predetermined amount after a detected period of continued silence exceeds a first amount of time.

21. The apparatus of claim 20 wherein the set of rules further specifies that the camera is zoomed out by an additional amount if the detected period of continued silence exceeds a second amount of time greater than the first amount of time.

22. The apparatus of claim 12 wherein the set of rules further includes determining if a confidence indicator associated with the video locator output is above a specified video locator threshold for the given measurement interval, and utilizing the video locator output to adjust the camera setting only if the video locator confidence indicator is above the video locator threshold for the given measurement interval.

23. An article of manufacture comprising a storage medium for storing one or more programs for tracking an object of interest in a video processing system, wherein the one or more programs when executed by a processor implement the steps of:

generating for particular ones of given measurement intervals of a plurality of successive measurement intervals, an audio locator output from an audio input, and a video locator output from a video input derived partly from detection of movement of the object, each indicative of a location of the object of interest;

applying a set of confidence level rules to each of the audio locator output and the video locator output to determine which one of the audio locator output and the video locator output has a higher confidence level, whereby the one having the highest confidence level will be utilized independently of the other to adjust a setting of the camera based on the given measurement interval; and

adjusting the camera setting utilizing only the selected one of the audio locator output and the video locator output in accordance with the applied set of confidence level rules.