Automatic camera framing for videoconferencing

• US 8,248,448 B2
• Filed: 05/18/2010
• Issued: 08/21/2012
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Feng, Jinwei, Chu, Peter, Dunlap, Wayne, Gallmeier, Jonathan, Nimri, Alain, Schaefer, Stephen P.
• Primary Examiner(s): Nguyen, Joseph J
• Assistant Examiner(s): El-Zoobi, Maria
• Application Number:
  US12/782,155
• Publication Number:
  US 20110285809A1
• Time in Patent Office: 826 Days
• Field of Search: 348 1401- 1408
• US Class Current: 348/14.01
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

A videoconferencing apparatus automatically tracks speakers in a room and dynamically switches between a controlled, people-view camera and a fixed, room-view camera. When no one is speaking, the apparatus shows the room view to the far-end. When there is a dominant speaker in the room, the apparatus directs the people-view camera at the dominant speaker and switches from the room-view camera to the people-view camera. When there is a new speaker in the room, the apparatus switches to the room-view camera first, directs the people-view camera at the new speaker, and then switches to the people-view camera directed at the new speaker. When there are two near-end speakers engaged in a conversation, the apparatus tracks and zooms-in the people-view camera so that both speakers are in view.

Claims

1. An automated videoconferencing method, comprising:

segmenting a wide view of an environment into first tight views;

directing a first camera of an endpoint to frame each of the first tight views

determining relevance of each of the first tight views by analyzing the first video

captured with the first camera in each of the first tight views;

composing a framed view defined by the first tight views determined relevant;

directing a second camera of the endpoint to frame the framed view; and

outputting second video for the videoconference, the second video captured with the second camera in the framed view.

2. The method of claim 1, wherein the first and second cameras are a same camera.

3. The method of claim 1, wherein the first camera is a different camera from the second camera.

4. The method of claim 3, wherein the first camera comprises a steerable pan-tilt-zoom camera, and wherein the second camera comprises an electronic pan-tilt-zoom camera.

5. The method of claim 3, further comprising:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

directing the first camera in a second tight view at the source location; and

switching output for the videoconference from the second video to third video, the third video captured by the first camera in the second tight view of the source location.

6. The method of claim 1, wherein segmenting the wide view of the environment into the first tight views comprises:

directing the first camera in a plurality of directions at the environment;

capturing video images with the first camera directed in each of the directions;

combining the video images into the wide view; and

dividing the wide view into the first tight views.

7. The method of claim 1, wherein directing the first or second camera comprises adjusting one or more of a pan, a tilt, and a zoom.

8. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting motion in the first video captured with the first camera in each of the first tight views.

9. The method of claim 8, wherein detecting motion comprises:

determining a first luminance value for a first frame of the first video;

determining a second luminance value for a second frame of the first video;

determining whether a difference between the first and second luminance values indicates motion between the frames.

10. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting tone indicative of human skin in the first video captured with the first camera in each of the first tight views.

11. The method of claim 10, wherein determining tone comprises:

determining a chrominance value within at least a portion of a frame of the first video; and

determining whether the chrominance value falls within a range of human skin tone.

12. The method of claim 1, wherein determining relevance of each of the first tight views comprises detecting a human face in the first video captured with the first camera in each of the first tight views.

13. The method of claim 12, wherein detecting a human face comprises:

determining the first tight views having tone indicative of human skin; and

finding the determined first tight views having a feature indicative of a human face.

14. The method of claim 1, wherein determining a framed view defined by the first tight views determined relevant comprises restricting the wide view at least by a left-most, a right-most, and a top-most ones of the first tight views determined relevant.

15. The method of claim 1, further comprising periodically segmenting a wide view, capturing video, determining relevance, determining a framed view, and directing the second camera.

16. The method of claim 1, further comprising:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

correlating the source location to one or more of the first tight views determined relevant.

25. The method of claim 1, wherein the acts of segmenting the wide view, directing the first camera, determining relevance, determining the framed view, and directing the second camera are initially performed at a start of the videoconference.

26. The method of claim 1, wherein the acts of segmenting the wide view, directing the first camera, determining relevance, determining the framed view, and directing the second camera are performed at a time during the videoconference.

17. A non-transitory program storage device having program instructions stored thereon for causing a programmable control device to perform an automated videoconferencing method, the method comprising:

segmenting a wide view of an environment into first tight views;

directing a first camera of an endpoint to frame each of the first tight views;

determining relevance of each of the first tight views by analyzing the first video captured with the first camera in each of the first tight views;

composing a framed view defined by the first tight views determined relevant;

directing a second camera of the endpoint to frame the framed view; and

outputting second video for the videoconference, the second video captured with the second camera in the framed view.

27. The program storage device of claim 17, wherein the first and second cameras are a same camera.

28. The program storage device of claim 17, wherein the first camera is a different camera from the second camera.

29. The program storage device of claim 28, wherein the first camera comprises a steerable pan-tilt-zoom camera, and wherein the second camera comprises an electronic pan-tilt-zoom camera.

30. The program storage device of claim 28, wherein the method further comprises:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

directing the first camera in a second tight view at the source location; and

switching output for the videoconference from the second video to third video, the third video captured by the first camera in the second tight view of the source location.

31. The program storage device of claim 17, wherein segmenting the wide view of the environment into the first tight views comprises:

directing the first camera in a plurality of directions at the environment;

capturing video images with the first camera directed in each of the directions;

combining the video images into the wide view; and

dividing the wide view into the first tight views.

32. The program storage device of claim 17, wherein directing the first or second camera comprises adjusting one or more of a pan, a tilt, and a zoom.

33. The program storage device of claim 17, wherein determining relevance of each of the first tight views comprises detecting motion in the first video captured with the first camera in each of the first tight views.

34. The program storage device of claim 33, wherein detecting motion comprises:

determining a first luminance value for a first frame of the first video;

determining a second luminance value for a second frame of the first video;

determining whether a difference between the first and second luminance values indicates motion between the frames.

35. The program storage device of claim 17, wherein determining relevance of each of the first tight views comprises detecting tone indicative of human skin in the first video captured with the first camera in each of the first tight views.

36. The program storage device of claim 35, wherein determining tone comprises:

determining a chrominance value within at least a portion of a frame of the first video; and

determining whether the chrominance value falls within a range of human skin tone.

37. The program storage device of claim 17, wherein determining relevance of each of the first tight views comprises detecting a human face in the first video captured with the first camera in each of the first tight views.

38. The program storage device of claim 37, wherein detecting a human face comprises:

determining the first tight views having tone indicative of human skin; and

finding the determined first tight views having a feature indicative of a human face.

39. The program storage device of claim 17, wherein determining a framed view defined by the first tight views determined relevant comprises restricting the wide view at least by a left-most, a right-most, and a top-most ones of the first tight views determined relevant.

40. The program storage device of claim 17, wherein the method further comprises periodically segmenting a wide view, capturing video, determining relevance, determining a framed view, and directing the second camera.

41. The program storage device of claim 17, wherein the method further comprises:

detecting audio indicative of speech in the environment;

determining a source location of the detected audio in the environment;

correlating the source location to one or more of the first tight views determined relevant.

42. The program storage device of claim 17, wherein the acts of segmenting the wide view, directing the first camera, determining relevance, determining the framed view, and directing the second camera are initially performed at a start of the videoconference.

43. The program storage device of claim 17, wherein the acts of segmenting the wide view, directing the first camera, determining relevance, determining the framed view, and directing the second camera are performed at a time during the videoconference.

18. A videoconferencing apparatus, comprising:

at least one camera being controllable to pan, tilt, and zoom; and

a processing unit operatively coupled to the at least one camera, the control unit

being operable to;

segment a wide view of an environment captured by the at least one camera into a plurality of first tight views;

direct the at least one camera to capture first video in each of the first tight views;

analyze the first video in each of the first tight views for relevance;

restrict the wide view to a framed view defined by the first tight views analyzed as relevant; and

direct the at least one camera to capture second video in the framed view.

19. The apparatus of claim 18, wherein the at least one camera comprises a first camera directed to capture the first tight views and comprises a second camera directed to capture the framed view.

20. The apparatus of claim 19, wherein the first camera comprises a steerable pan-tilt-zoom camera, and wherein the second camera comprises an electronic pan-tilt-zoom camera.

21. The apparatus of claim 19, wherein the processing unit is operable to:

capture the second video with the second camera in the framed view of the environment; and

output the second video for the videoconference.

22. The apparatus of claim 21, further comprising a plurality of microphones, wherein the processing unit is operatively coupled to the microphones and is operable to:

detect audio indicative of speech in the environment with the microphones;

determine a source location of the detected audio in the environment;

direct the first camera in a second tight view at the source location; and

switch output for the videoconference from the second video to third video, the third video captured with the first camera in the second tight view of the source location.

23. The apparatus of claim 18, wherein the processing unit is configured to segment the wide view, direct the first camera, determine relevance, determine the framed view, and direct the second camera initially at a start of the videoconference.

24. The apparatus of claim 18, wherein the processing unit is configured to segment the wide view, direct the first camera, determine relevance, determine the framed view, and direct the second camera at a time during the videoconference.