Automatic Camera Framing for Videoconferencing

• US 20110285809A1
• Filed: 05/18/2010
• Published: 11/24/2011
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: GALLMEIER, Jonathan, FENG, Jinwei, SCHAEFER, Stephen P., DUNLAP, Wayne, NIMRI, Alain, CHU, Peter
• Granted Patent: US 8,248,448 B2
• Time in Patent Office: Days
• US Class Current: 348/14.16
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A videoconferencing apparatus automatically tracks speakers in a room and dynamically switches between a controlled, people-view camera and a fixed, room-view camera. When no one is speaking, the apparatus shows the room view to the far-end. When there is a dominant speaker in the room, the apparatus directs the people-view camera at the dominant speaker and switches from the room-view camera to the people-view camera. When there is a new speaker in the room, the apparatus switches to the room-view camera first, directs the people-view camera at the new speaker, and then switches to the people-view camera directed at the new speaker. When there are two near-end speakers engaged in a conversation, the apparatus tracks and zooms-in the people-view camera so that both speakers are in view.

Claims

1. An automated videoconferencing method, comprising:

segmenting a wide view of an environment into first tight views;

directing a first camera on an endpoint to frame each of the first tight views;

determining relevance of each of the first tight views by analyzing the first video captured with the first camera in each of the first tight views;

determining a framed view defined by the first tight views determined relevant;

directing a second camera of the endpoint to frame the framed view; and

outputting second video for the videoconference, the second video captured with the second camera in the framed view.

2. The method of claim 1, wherein the first and second cameras are a same camera.

3. The method of claim 1, wherein the first camera is a different camera from the second camera.

4. The method of claim 3, wherein the first camera comprises a steerable pan-tilt-zoom camera, and wherein the second camera comprises an electronic pan-tilt-zoom camera.

5. The method of claim 3, further comprising:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

directing the first camera in a second tight view at the source location; and

switching output for the videoconference from the second video to third video, the third video captured by the first camera in the second tight view of the source location.

6. The method of claim 1, wherein segmenting the wide view of the environment into the first tight views comprises:

directing the first camera in a plurality of directions at the environment;

capturing video images with the first camera directed in each of the directions;

combining the video images into the wide view; and

dividing the wide view into the first tight views.

7. The method of claim 1, wherein directing the first or second camera comprises adjusting one or more of a pan, a tilt, and a zoom.

8. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting motion in the first video captured with the first camera in each of the first tight views.

9. The method of claim 8, wherein detecting motion comprises:

determining a first luminance value for a first frame of the first video;

determining a second luminance value for a second frame of the first video;

determining whether a difference between the first and second luminance values indicates motion between the frames.

10. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting tone indicative of human skin in the first video captured with the first camera in each of the first tight views.

11. The method of claim 10, wherein determining tone comprises:

determining a chrominance value within at least a portion of a frame of the first video; and

determining whether the chrominance value falls within a range of human skin tone.

12. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting a human face in the first video captured with the first camera in each of the first tight views.

13. The method of claim 12, wherein detecting a human face comprises:

determining the first tight views having tone indicative of human skin; and

finding the determined first tight views having a feature indicative of a human face.

14. The method of claim 1, wherein determining a framed view defined by the first tight views determined relevant comprises restricting the wide view at least by a left-most, a right-most, and a top-most ones of the first tight views determined relevant.

15. The method of claim 1, further comprising periodically segmenting a wide view, capturing video, determining relevance, determining a framed view, and directing the second camera.

16. The method of claim 1, further comprising:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

correlating the source location to one or more of the first tight views determined relevant.

17. A program storage device having program instructions stored thereon for causing a programmable control device to perform a method according to claim 1.

18. A videoconferencing apparatus, comprising:

at least one camera being controllable to pan, tilt, and zoom; and

a processing unit operatively coupled to the at least one camera, the control unit being operable to;

segment a wide view of an environment captured by the at least one camera into a plurality of first tight views;

direct the at least one camera to capture first video in each of the first tight views;

analyze the first video in each of the first tight views for relevance;

restrict the wide view to a framed view defined by the first tight views analyzed as relevant; and

direct the at least one camera to capture second video in the framed view.

19. The apparatus of claim 18, wherein the at least one camera comprises a first camera directed to capture the first tight views and comprises a second camera directed to capture the framed view.

20. The apparatus of claim 19, wherein the first camera comprises a steerable pan-tilt-zoom camera, and wherein the second camera comprises an electronic pan-tilt-zoom camera.

21. The apparatus of claim 19, wherein the processing unit is operable to:

capture the second video with the second camera in the framed view of the environment; and

output the second video for the videoconference.

22. The apparatus of claim 21, further comprising a plurality of microphones, wherein the processing unit is operatively coupled to the microphones and is operable to:

detect audio indicative of speech in the environment with the microphones;

determine a source location of the detected audio in the environment;

direct the first camera in a second tight view at the source location; and

switch output for the videoconference from the second video to third video, the third video captured with the first camera in the second tight view of the source location.