Videoconferencing endpoint having multiple voice-tracking cameras

• US 9,392,221 B2
• Filed: 03/05/2013
• Issued: 07/12/2016
• Est. Priority Date: 05/18/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Polycom Incorporated (HP Inc.)
• Inventors: Feng, Jinwei, Chu, Peter, Dunlap, Wayne, Gallmeier, Jonathan
• Primary Examiner(s): Woo, Stella L
• Application Number:
  US13/786,260
• Publication Number:
  US 20130271559A1
• Time in Patent Office: 1,225 Days
• Field of Search: 348/14.08, 348/14.09
• US Class Current: 1/1
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A videoconferencing apparatus automatically tracks speakers in a room and dynamically switches between a controlled, people-view camera and a fixed, room-view camera. When no one is speaking, the apparatus shows the room view to the far-end. When there is a dominant speaker in the room, the apparatus directs the people-view camera at the dominant speaker and switches from the room-view camera to the people-view camera. When there is a new speaker in the room, the apparatus switches to the room-view camera first, directs the people-view camera at the new speaker, and then switches to the people-view camera directed at the new speaker. When there are two near-end speakers engaged in a conversation, the apparatus tracks and zooms-in the people-view camera so that both speakers are in view.

Claims

1. An automated videoconferencing method for an environment having a plurality of participants, the method comprising:

capturing video for a videoconference from one vantage point of the environment with at least one camera;

capturing audio in the environment with microphones co-located with the at least one camera;

determining at least one wide view of the participants in the captured video from the one vantage point of the environment;

determining one or more narrow views of one or more of the participants in the captured video from the one vantage point of the environment, the one or more narrow views contained in the at least one wide view; and

selectively outputting the wide or narrow views of the participants in the captured video for the videoconference based in part on the captured audio of the participants during the videoconference,

wherein to determine and selectively output the wide or narrow views, the method comprises;

determining from the captured audio that at least two locations of the one or more of the participants have audio indicative of speech and are engaged in one type of audio exchange in the environment relative to the one vantage point based on a designation of an active speaker in the environment having alternated between the same at least two locations within a time frame,

determining a selected view from the one or more narrow views of the captured video framing the at least two determined locations from the one vantage point of the environment, and

outputting the selected narrow view of the captured video framing the at least two determined locations from the one vantage point for the videoconference.

2. The method of claim 1, wherein to determine and selectively output the wide or narrow views, the method comprises:

initially outputting by default the at least one wide view of the captured video for the videoconference from the one vantage point of the environment;

determining the selected view as one of the one or more narrow views of the captured video framing the one or more determined locations from the one vantage point of the environment, and

subsequently outputting the selected narrow view of the captured video for the videoconference instead of the at least one wide view.

3. The method of claim 1, wherein capturing the video for the videoconference from the one vantage point of the environment with the at least one camera comprises capturing the video for the videoconference by sharing the one vantage point of the environment with at least two cameras co-located on an endpoint.

4. The method of claim 3, wherein outputting the at least one wide view of the captured video for the videoconference from the one vantage point of the environment comprises outputting the video captured with a first of the at least two cameras by directing the first camera in the at least one wide view of the environment from the one vantage point.

5. The method of claim 4, wherein to determine and selectively output the wide or narrow views, the method comprises:

directing a second of the at least two cameras co-located on the endpoint in the selected narrow view at the one or more determined locations while outputting the at least one wide view captured with the first camera for the videoconference, and

subsequently switching output for the videoconference from the at least one wide view to the selected narrow view captured with the second camera.

6. The method of claim 5, wherein outputting the selected narrow view of the captured video framing the at least two determined locations from the one vantage point for the videoconference comprises initially outputting the at least one wide view of the first camera from the one vantage point for the videoconference instead of outputting the selected narrow view of the second camera from the one vantage point.

7. The method of claim 6, further comprising:

directing the second camera in the selected narrow view at the at least two locations detected in the audio exchange; and

switching output for the videoconference from the at least one wide view to the selected narrow view captured with the second camera.

8. The method of claim 1, wherein determining the one or more narrow views of the one or more of the participants in the captured video comprises framing the one or more narrow views inside the wide view.

9. The method of claim 1, wherein determining the one or more narrow views of the one or more of the participants in the captured video comprises one or more of:

detecting motion in the captured video of the at least one camera;

detecting tone indicative of human skin in the captured video of the at least one camera; and

detecting a human face in the captured video of the at least one camera.

10. The method of claim 1, wherein determining from the captured audio that the one or more locations of the one or more of the participants have audio indicative of speech in the environment relative to the one vantage point comprises determining at least two orthogonal bearing angles by analyzing the audio captured with the microphones.

11. The method of claim 1, wherein determining from the captured audio that the one or more locations of the one or more of the participants have audio indicative of speech in the environment relative to the one vantage point comprises:

obtaining samples of the captured audio;

calculating coordinates for each of the samples; and

calculating a transiency value for each sample indicative of the transiency of the captured audio in the sample.

12. The method of claim 11, wherein determining from the captured audio that the one or more locations of the one or more of the participants have audio indicative of speech in the environment relative to the one vantage point comprises:

clustering the calculated coordinates for the samples; and

determining a coordinate for the one or more location based on the transiency values and the clustering of the samples.

13. The method of claim 11, wherein calculating the transiency value for each sample comprises:

filtering each of the samples with a high-pass filter;

correlating each of the filtered samples to a match filter; and

deriving a scalar value based on the correlation.

14. The method of claim 1, wherein to determine and selectively output the wide or narrow views, the method comprises:

detecting an absence of audio indicative of speech in the environment; and

outputting the at least one wide view of the captured video for the videoconference instead of the one or more narrow views.

15. The method of claim 1, wherein to determine and selectively output the wide or narrow views, the method comprises:

detecting another type of audio exchange between at least one of the locations in the environment and a far-end endpoint;

subsequently detecting an absence of audio indicative of speech in the environment; and

maintaining output of the selected view as one of the narrow views of the at least one location even when determining the absence of audio indicative of speech in the environment.

16. A non-transitory program storage device having program instructions stored thereon for causing a programmable control device to perform a method according to claim 1.

17. A videoconferencing apparatus for an environment having a plurality of participants, the apparatus comprising:

at least one camera capturing video for a videoconference from one vantage point of the environment;

a plurality of microphones co-located with the at least one camera and capturing audio;

a network interface communicatively coupling to a network; and

a processing unit operatively coupled to the network interface, the at least one camera, and the microphones, the processing unit programmed to;

determine at least one wide view of the participants in the captured video from the one vantage point of the environment,

determine one or more narrow views of one or more of the participants in the captured video from the one vantage point of the environment, the one or more narrow views contained in the at least one wide view, and

selectively output the wide or narrow views of the participants in the captured video for the videoconference based in part on the captured audio during the videoconference,

wherein to determine and selectively output the wide or narrow views, the processing unit is programmed to;

determine from the captured audio that at least two locations of the one or more of the participants have audio indicative of speech and are engaged in one type of audio exchange in the environment relative to the one vantage point based on a designation of an active speaker in the environment having alternated between the same at least two locations within a time frame,

determine a selected view from the one or more narrow views of the captured video framing the at least two determined locations from the one vantage point of the environment, and

output the selected narrow view of the captured video framing the at least two determined locations from the one vantage point for the videoconference.

18. The apparatus of claim 17, wherein the at least one camera comprises an electronic pan-tilt-zoom camera.

19. The apparatus of claim 18, wherein the at least one camera comprises a second camera co-located on the apparatus with the electronic pan-tilt-zoom camera.

20. The apparatus of claim 19, wherein the second camera comprises a steerable pan-tilt-zoom camera.

21. The apparatus of claim 17, wherein the at least one camera comprises at least two cameras co-located on the apparatus, the at least two cameras capturing the video for the videoconference and sharing the one vantage point of the environment.

22. The apparatus of claim 21, wherein the at least two cameras comprise a housing integrally housing the at least two cameras thereon and integrally housing the microphones thereon.

23. The apparatus of claim 21, wherein the at least two cameras comprise a base removably housing one or both of the at least two cameras thereon and integrally housing the microphones thereon.