Videoconferencing system having adjunct camera for auto-framing and tracking

• US 8,842,161 B2
• Filed: 08/20/2012
• Issued: 09/23/2014
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (Plantronics, Inc.)
• Inventors: Feng, Jinwei, Liu, Yibo, Chu, Peter L., Wang, Xiangdong
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US13/589,380
• Publication Number:
  US 20140049595A1
• Time in Patent Office: 764 Days
• Field of Search: 348/14.01, 348/14.08, 348/14.12, 348/14.16
• US Class Current: 348/14.12
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A videoconference apparatus and method coordinates a stationary view obtained with a stationary camera to an adjustable view obtained with an adjustable camera. The stationary camera can be a web camera, while the adjustable camera can be a pan-tilt-zoom camera. As the stationary camera obtains video, faces of participants are detected, and a boundary in the view is determined to contain the detected faces. Absence and presences of motion associated with the detected face is used to verify whether a face is reliable. To then capture and output video of the participants for the videoconference, the view of the adjustable camera is adjusted to a framed view based on the determined boundary. In the end, active video captured in the framed view with the adjustable camera can be sent to a far-end for the videoconference.

Claims

1. A videoconferencing method, comprising:

capturing first video in a first view of an environment with a first camera;

detecting one or more faces of participants in the first video;

verifying at least one of the one or more detected faces as being reliable;

determining a boundary in the first view of the environment containing the at least one verified face;

adjusting a second view of a second camera to a framed view of the environment based on the determined boundary; and

capturing second video in the framed view with the second camera.

2. The method of claim 1, wherein the steps of capturing, detecting, verifying, determining, and adjusting are performed repeatedly for frames of the first video.

3. The method of claim 1, wherein capturing the first video in the first view of the environment with the first camera comprises operating a web camera as the first camera.

4. The method of claim 1, wherein the act of verifying comprises analyzing an area of the first video associated with each of the one or more detected faces for motion and indicating the at least one of the one or more detected faces as being reliable based on the analysis.

5. The method of claim 4, wherein the act of analyzing comprises, for a subject face of the one or more detected faces:

detecting a presence of motion in an on-face region of the area associated with the subject face; and

indicating the subject face as reliable based on the detected motion.

6. The method of claim 4, wherein the act of analyzing comprises, for a subject face of the one or more detected faces:

detecting an absence of motion in an on-face region of the area associated with the subject face;

determining that the subject face has been indicated as previously reliable; and

indicating the subject face as currently reliable based on the previous indication.

7. The method of claim 1, wherein detecting the one or more faces of participants in the first video comprises determining that a subject face of the one or more detected faces has been previously detected in the first video and has not been currently detected in the first video.

8. The method of claim 7, wherein the act of verifying comprises, for the subject face:

detecting an absence of motion in a region surrounding the subject face; and

indicating the subject face as being reliable based on the absence of detected motion.

9. The method of claim 8, wherein the act of verifying comprises disregarding the subject face as reliable by detecting a presence of motion in a region surrounding the subject face.

10. The method of claim 1, wherein detecting the one or more faces of the participants in the first video comprises detecting for a presence of motion in the first video.

11. The method of claim 10, wherein detecting for the presence of motion in the first video comprises:

detecting for any motion pixels indicative of motion in the first video by comparing first and second frames of the first video;

determining, for each block of multiple pixels in the first video, whether the detected motion pixels in the each block exceeds a threshold; and

indicating each block determined to exceed the threshold as having the presence of motion.

12. The method of claim 11, wherein detecting for any motion pixels indicative of motion in the first video comprises analyzing a blue component of the pixels in the first video.

13. The method of claim 1, wherein determining the boundary in the first view of the environment containing the at least one verified face comprises:

starting with an initial boundary in the first view;

determining that the initial boundary contains less than all of the at least one verified face in the first view; and

determining a subsequent boundary by adjusting the initial boundary to contain all of the at least one verified face in the first view.

14. The method of claim 1, wherein detecting the one or more faces of participants in the first video and determining the boundary in the first view of the environment containing the at least one verified face comprises repeatedly:

storing one or more current face locations of the one or more faces currently detected in the first view;

detecting one or more new face locations of any one or more verified faces newly detected in the first view;

detecting a difference between the one or more current face locations and the one or more new face locations; and

adjusting the boundary in the first view based on the detected difference.

15. The method of claim 1, wherein adjusting the second view of the second camera to the framed view of the environment based on the determined boundary comprises controlling one or more of a pan, a tilt, and a zoom of the second camera.

16. The method of claim 1, further comprising:

capturing audio of the environment; and

using the captured audio in adjusting the second view of the second camera.

17. The method of claim 16, wherein using the captured audio in adjusting the second view of the second camera comprises:

determining from the captured audio one or more bearing angles of one or more source locations in the environment;

coordinating the one or more bearing angles of the one or more source locations with one or more face locations of the one or more detected faces in the environment; and

using the coordinated one or more bearing angles in adjusting the second view of the second camera.

18. A program storage device having program instructions stored thereon for causing a programmable control device to perform a method according to claim 1.

19. A videoconferencing apparatus, comprising:

a first camera capturing first video in a first view of an environment;

a second camera capturing second video in a second view of the environment; and

processing circuitry operatively coupled to the first and second cameras, the processing circuitry configured to;

detect one or more faces of participants in the first video,

verify at least one of the one or more detected faces as being reliable,

determine a boundary in the first view of the environment containing the at least one verified face, and

adjust the second view of the second camera to a framed view of the environment based on the determined boundary.

20. The apparatus of claim 19, wherein the first camera comprises a web camera, and wherein the second camera comprises a pan-tilt-zoom camera.

21. The apparatus of claim 20, wherein a video device comprises the first camera, the second camera, and the processing circuitry associated therewith.

22. The apparatus of claim 21, further comprising a videoconferencing unit separate from the video device and being operatively coupled thereto.

23. The apparatus of claim 21, wherein to verify at least one of the one or more detected faces as being reliable, the processing circuitry is configured to analyze an area of the first video associated with each of the one or more detected faces for motion.

24. The apparatus of claim 22, further comprising at least two microphones capturing audio in the environment, and wherein the processing circuitry is programmed to:

determine from the captured audio one or more bearing angles of one or more source locations in the environment;

coordinate the one or more bearing angles of the one or more source locations with one or more face locations of the one or more detected faces in the environment; and

use the coordinated one or more bearing angles in adjusting the second view of the second camera.