Video processing in a multi-participant video conference

• US 7,817,180 B2
• Filed: 04/28/2005
• Issued: 10/19/2010
• Est. Priority Date: 04/28/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Normile, Jim, Jeong, Hyeonkuk, Abuan, Joe
• Primary Examiner(s): Woo; Stella L
• Application Number:
  US11/118,553
• Publication Number:
  US 20060244812A1
• Time in Patent Office: 2,000 Days
• Field of Search: 348/14.08, 348/14.09, 370/260, 370/263, 370/270, 379/93.21
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant'"'"'s image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved.

Claims

1. A method of distributing video images by a central distributor in a video conference, said conference having a plurality of participants, the method comprising:

at one particular video-conference participant'"'"'s video-conference device that houses a central focus distributor for distributing video images to a plurality of non-focus video conference devices of a plurality of other participants during the video conference;

receiving directly video images from each of the other plurality of participants through the non-focus video conference device of each of the other participants;

generating composite images from (i) the received video images and (ii) video images captured locally with the device that houses the central focus distributor; and

transmitting said composite images directly to each of the plurality of the other participants'"'"' non-focus video-conference devices.

2. The method of claim 1, wherein the device housing the central focus distributor comprises a data storage, the method further comprising placing each received video image in said data storage, wherein each received video image is not interleaved with any other received video image.

3. The method of claim 1 further comprising:

creating a separate composite image for each particular participant; and

for each particular participant, removing the video image received from the particular participant from a particular composite image for the particular participant before transmitting the particular composite image to the particular participant.

4. The method of claim 3 further comprising placing an indicator in place of each particular participant'"'"'s removed image in the particular participant'"'"'s separate composite image.

5. The method of claim 3 further comprising supplying the particular composite image to the particular participant, wherein a participant image is (a) captured locally at the particular participant device and (b) rendered on a screen of the particular participant along with the particular composite image.

6. The method of claim 1, wherein a particular composite image comprises a plurality of pixel sets, each pixel set comprising an identifier that identifies a particular video image associated with the pixel set, wherein the identifiers of the pixel sets are used to identify the video images included within the transmitted composite image.

7. The method of claim 1 further comprising resizing a particular received video image before generating the composite image.

8. The method of claim 1 further comprising:

saving said received video images before generating said composite images;

comparing a particular received video image of a particular participant with a previously saved video image of the particular participant; and

utilizing the received video image in generating the composite image when the received video image and the previously saved received video image are not identical.

9. The method of claim 8 further comprising foregoing inclusion of a video image of the particular participant in generating the composite image when the received video image and the previously saved video image are identical.

10. The method of claim 1 further comprising including video images of the central distributor in the generated composite images.

11. The method of claim 1 further comprising receiving, at the device of said particular participant, the locally captured video images of the particular participant.

12. The method of claim 1, wherein receiving directly a video image from a non-focus video conference device at the focus video-conference device comprises receiving a video image that does not traverse any other non-focus video conference device.

13. The method of claim 1, wherein transmitting directly a composite image from the focus video-conference device to a non-focus video conference device comprises transmitting the composite image to the non-focus video conference device without the composite image traversing any other non-focus video conference device.

14. The method of claim 1, wherein the locally captured video images are received from a camera attached to the device housing the central focus distributor.

20. The method of claim 1, wherein the video conference device that serves as the central focus distributor is a computer.

21. The method of claim 20, wherein the plurality of non-focus video conference devices are also computers.

15. A central distributor module for a multi-participant video conference, said central distributor module comprising:

a) a set of decoders for decoding video images received from a plurality of participants of the video conference, through the participants'"'"' respective computers;

b) a compositing engine for generating composite video images from the decoded video images; and

c) at least one encoder for encoding said composite video images,

wherein the central distributor module is stored on a computer readable medium housed within a first particular participant'"'"'s computer and is active on only the first particular participant'"'"'s computer during the video conference.

16. The central distributor module of claim 15 further comprising a set of intermediate buffers for storing a set of decoded video images before the compositing engine generates a composite video image based on the set of decoded video images.

17. The central distributor module of claim 15 further comprising a frame rate controller for adjusting a rate at which the compositing engine generates composite video images.

18. The central distributor module of claim 15 further comprising a redundancy remover to remove a second particular participant'"'"'s image from a composite video image that is to be sent to the second particular participant.

19. The central distributor module of claim 15 further comprising a resizer for receiving a decoded video image and resizing said decoded video image before the compositing engine generates an encoded composite image from the decoded video image.

22. For receiving and transmitting video content in a multi-participant video conference, a system comprising:

a) a plurality of non-focus point modules, each non-focus point module stored on a computer readable medium housed within a different participant device; and

b) a focus point module stored on a computer readable medium housed within a participant device different from the participant devices with the non-focus point modules, the focus point module for directly receiving video images from and directly transmitting composite video images to said plurality of non-focus point modules, wherein each composite video image is a composite of a plurality of video images received (i) from at least one non-focus point module and (ii) locally at the focus point module, wherein the directly receiving and directly transmitting requires the focus point module and the non-focus point module to communicate to each other without using another non-focus point module.

23. The system of claim 22, wherein the focus point module is further for eliminating a particular participant'"'"'s image from a composite video image that is sent to the particular participant.

24. The system of claim 22, wherein a directly received video image from a non-focus point module at the focus point module and a directly transmitted composite image from the focus point module to a non-focus point module do not traverse any other non-focus point module.

25. A computer readable medium housed within a computing device, the computer readable medium storing a computer program for distributing video content in a multi-participant video conference, the computer program for execution by at least one processor, the computer program comprising sets of instructions for:

at the device, receiving, through a network, a separate stream of video images from each of a plurality of participants;

at the device, locally capturing video images from another participant that is using the device to participate in the conference;

with the device, generating composite images from the received and captured video images; and

from the device, transmitting, through the network, separate streams of said composite images to each of the plurality of participants.

26. The computer readable medium of claim 25, wherein the device includes a data storage, the computer program further comprising a set of instructions for placing each received video image in said data storage, wherein each received video image is not interleaved with any other received video image.

27. The computer readable medium of claim 25, wherein the computer program further comprises a set of instructions for:

creating a separate composite image for each particular participant; and

removing each particular participant'"'"'s received video image from a particular composite image for the particular participant before transmitting the separate composite image to the particular participant.

28. The computer readable medium of claim 25, wherein the computer program further comprises a set of instructions for resizing a particular received video image before generating the composite image.

29. A method of providing a participant module for a video conference having a plurality of participants at a plurality of different devices, wherein a central distributor module at one of the different participant devices is connected to all other participant devices and the other participant devices are connected only to the central distributor module, said method comprising:

providing a decoder for;

receiving, from the central distributor module, an encoded composite image comprising a plurality of independent sub-images of at least two different participants, wherein image information for one sub-image does not rely on image information for any other sub-image; and

decoding the encoded composite image in order to extract each independent sub-image as a separate image; and

providing a local image controller for displaying the extracted sub-images.

30. The method of claim 29 further comprising providing a perspective adjuster for performing a transformation on a particular sub-image prior to displaying the particular sub-image, wherein the transformation is at least one of a skew and a resize operation.

31. The method of claim 29, wherein the provided participant module is not a central distributor of video content.

32. The method of claim 29, wherein the local image controller is further for displaying a placeholder for a particular participant when the sub-image for the particular participant is not available.

33. A method of providing a central distributor module for distributing video content from a first participant device for a video conference having a plurality of participants, the method comprising:

providing a set of decoders for receiving and decoding video images, at the first participant device, from a set of other participant devices of the video conference, wherein said set of other participant devices are not directly connected to each other;

providing a compositing engine for generating a plurality of different composite video images from the decoded video images at the participant device; and

providing an encoder for encoding said composite video images at the participant device.

34. The method of claim 33 further comprising providing a local image capture module for capturing video images at the participant device, wherein the compositing engine generates the plurality of different composite video images from (i) the decoded video images and (ii) the locally captured video images.