Systems and methods for selecting videoconferencing endpoints for display in a composite video image

• US 8,514,265 B2
• Filed: 10/02/2008
• Issued: 08/20/2013
• Est. Priority Date: 10/02/2008
• Status: Active Grant

• Current Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Original Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Inventors: Goyal, Ashish, Kulkarni, Hrishikesh Gopal, Rao, Shantha Kumari Harohalli Sathyanarayana
• Primary Examiner(s): Brandt, Christopher M
• Application Number:
  US12/244,436
• Publication Number:
  US 20100085419A1
• Time in Patent Office: 1,783 Days
• Field of Search: 348/14.08, 348/14.09, 709/204, 709/205, 715/751, 715/753, 715/755, 715/756, 715/759
• US Class Current: 348/14.09
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/152 Multipoint control units therefor
• CPC Class Codes: H04N 7/173 with two-way working, e.g. subscriber sending a programme selection signal

Abstract

In some embodiments, a videoconferencing endpoint may be an MCU (Multipoint Control Unit) or may include embedded MCU functionality. In various embodiments, the endpoint may thus conduct a videoconference by receiving/compositing video and audio from multiple videoconference endpoints. The endpoint may select a subset of endpoints and form a composite video image from the subset of the videoconference endpoints to send to the other videoconference endpoints. In some embodiments, the subset of endpoints that are selected for compositing into the composite video image may be selected according to criteria such as the last N talking participants. In some embodiments, the master endpoint may request the non-talker endpoints to stop sending video to help conserve the resources on the master endpoint. In some embodiments, the master endpoint may ignore video from endpoints that are not being displayed.

Claims

1. A method, comprising:

utilizing a processor of a videoconferencing system to perform operations including;

receiving an audio stream from each of N remote endpoints that are participating in a videoconference, wherein N is greater than two, wherein the processor has access to M video decoders, wherein M is greater than one and smaller than N;

analyzing the audio streams from the N remote endpoints to maintain a list of the M most recently voice-active endpoints among the N remote endpoints;

receiving M video streams from the M most recently voice-active endpoints respectively;

directing the decoding of the M video streams using respectively the M video decoders to generate M component images respectively;

generating a composite image including at least the M component images;

transmitting the composite image to one or more of the N remote endpoints;

updating the list of M most recently voice-active endpoints to remove a least recently voice-active endpoint and to add a new endpoint in response to detecting current voice activity in the audio stream corresponding to the new endpoint;

signaling the new endpoint to start transmitting a video stream; and

reallocating a first of the M video decoders to the video stream transmitted from the new endpoint.

2. The method of claim 1, wherein M is selected by a user.

3. The method of claim 1, wherein said detecting current voice activity in the audio stream corresponding to the new endpoint includes detecting that a signal strength measure exceeds a threshold.

4. The method of claim 1 further comprising:

signaling said least recently voice-active endpoint to stop video stream transmission.

5. The method of claim 1 further comprising:

analyzing the audio streams corresponding to the M most recently voice-active endpoints to determine an endpoint having a dominant talker, and allocating a larger portion of the composite image to the component image belonging to that endpoint than to others of the M endpoints.

6. The method of claim 1, wherein the processor is included in a master endpoint, wherein the master endpoint receives a local video signal from a local video camera, wherein said generating the composite image includes incorporating a local image derived from the local video signal in the composite image.

7. The method of claim 1 further comprising:

generating a composite audio signal including a mixture of audio information from the N audio streams; and

transmitting the composite audio signal to said one or more of the N remote endpoints.

8. A system for performing multi-way videoconferencing, the system comprising:

a memory that stores program instructions;

a processor configured to execute the program instructions, wherein the program instructions, if executed, cause the processor to;

receive an audio stream from each of N remote endpoints that are participating in a videoconference, wherein N is greater than two, wherein the processor has access to M video decoders, wherein M is greater than one and smaller than N;

analyze the audio streams from the N remote endpoints to maintain a list of the M most recently voice-active endpoints among the N remote endpoints;

receive M video streams from the M most recently voice-active endpoints respectively;

direct the decoding of the M video streams respectively in the M video decoders to generate M component images respectively;

generate a composite image including at least the M component images;

transmit the composite image to one or more of the N remote endpoints;

update the list of M most recently voice-active endpoints to remove a least recently voice-active endpoint and to add a new endpoint in response to detecting current voice activity in the audio stream corresponding to the new endpoint;

signal the new endpoint to start transmitting a video stream; and

reallocate a first of the M video decoders to the video stream transmitted from the new endpoint.

9. The system of claim 8, wherein the integer M is selected by a user.

10. The system of claim 8, wherein said detecting current voice activity in the audio stream corresponding to the new endpoint includes detecting that a signal strength measure exceeds a threshold.

11. The system of claim 8, wherein the program instructions, if executed, further cause the processor to:

signal said least recently voice-active endpoint to stop video stream transmission.

12. The system of claim 8, wherein the program instructions, if executed, further cause the processor to:

analyze the audio streams corresponding to the M most recently voice-active endpoints to determine an endpoint having a dominant talker, and

allocate a larger portion of the composite image to the component image belonging to that endpoint than to others of the M endpoints.

13. The system of claim 8, further comprising:

an input port configured for coupling to a local video camera, wherein the local video camera is configured to generate a local video signal, wherein said generating the composite image includes incorporating a local image derived from the local video signal in the composite image.

14. The system of claim 8, wherein the program instructions, if executed, further cause the processor to:

generate a composite audio signal including a mixture of audio information from the N audio streams; and

transmit the composite audio signal to said one or more of the N remote endpoints.

15. The system of claim 8, wherein the M video streams are received via the Internet.

16. The system of claim 8 further comprising an interface for coupling to a local speakerphone.

17. The system of claim 8 further comprising a set of video compressors configured to compress a video signal containing the composite image.

18. A videoconferencing system operable to perform multi-way videoconferencing, the videoconferencing system comprising:

an audio input;

a video input;

a set of M decoders coupled to the video input;

a memory that stores program instructions;

a processor configured to execute the program instructions, wherein the program instructions, if executed, cause the processor to;

receive an audio stream from each of N remote endpoints via the audio input, wherein the N remote endpoints are participants in a videoconference, wherein N is greater than two, wherein the processor has access to the M video decoders, wherein M is greater than one and smaller than N;

analyze the audio streams from the N remote endpoints to maintain a list of the M most recently voice-active endpoints among the N remote endpoints;

receive M video streams, via the video input, from the M most recently voice-active endpoints respectively;

direct the decoding of the M video streams respectively in the M video decoders to generate M component images respectively;

generate a composite image including at least the M component images;

transmit the composite image to one or more of the N remote endpoints,

update the list of M most recently voice-active endpoints to remove a least recently voice-active endpoint and to add a new endpoint in response to detecting current voice activity in the audio stream corresponding to the new endpoint;

signal the new endpoint to start transmitting a video stream; and

reallocate a first of the M video decoders to the video stream transmitted from the new endpoint.

19. The videoconferencing system of claim 18, wherein said detecting current voice activity in the audio stream corresponding to the new endpoint includes detecting that a signal strength measure exceeds a threshold.

20. The videoconferencing system of claim 18, wherein the audio streams and the M video streams are received via the Internet.

21. The videoconferencing system of claim 18 further comprising an interface for coupling to a speakerphone.