Video encoding in a video conference

• US 7,692,682 B2
• Filed: 04/28/2005
• Issued: 04/06/2010
• Est. Priority Date: 04/28/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Jeong, Hyeonkuk, Pun, Thomas, Wu, Hsi Jung
• Primary Examiner(s): Woo; Stella L
• Application Number:
  US11/118,615
• Publication Number:
  US 20060244819A1
• Time in Patent Office: 1,804 Days
• Field of Search: 348/14.08, 348/14.09, 348/14.01, 348/14.12, 348/14.14, 348/14.15
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Some embodiments provide an architecture for establishing multi-participant video conferences. This architecture has a central distributor that receives video images from two or more participants. From the received images, the central distributor generates composite images that the central distributor transmits back to the participants. Each composite image includes a set of sub images, where each sub image belongs to one participant. In some embodiments, the central distributor saves network bandwidth by removing each particular participant'"'"'s image from the composite image that the central distributor sends to the particular participant. In some embodiments, images received from each participant are arranged in the composite in a non-interleaved manner. For instance, in some embodiments, the composite image includes at most one sub-image for each participant, and no two sub-images are interleaved.

Claims

1. A method of encoding an image during a video conference with multiple participants, the method comprising:

generating a composite image comprising at least two sub-images for at least two participants; and

constraint encoding each sub-image separately, wherein said constraint encoding comprises encoding each sub-image based on a set of constraints that allow the encoding of the sub-image to be independent of the encoding of any other sub-image.

2. The method of claim 1,

wherein the encoding comprises performing a motion estimation process for an inter-block encoding operations,

wherein the set of constraints includes a constraint on a location of a search window used during the motion estimation process.

3. The method of claim 2, wherein performing the motion estimation process comprises performing a search within a target sub-image, wherein the location constraint requires the search window to be located within the target sub-image.

4. The method of claim 3 further comprising:

identifying a search window that partially falls outside the target sub-image; and

cropping the identified search window so that it entirely falls within the target sub-image.

5. The method of claim 3 further comprising:

identifying a search window that at least partially falls outside the target sub-image; and

moving the identified search window so that it entirely falls within the target sub-image.

6. The method of claim 1, wherein the encoding comprises examining blocks that are aligned with sub-pixel locations,

wherein the set of constraints includes a constraint on a region within which the blocks aligned with the sub-pixel locations are examined.

7. The method of claim 6, wherein said examination comprises performing interpolation operations for sets of image values of sets of pixels.

8. The method of claim 7, wherein said interpolation operations are performed on sets of horizontal pixels and sets of vertical pixels.

9. The method of claim 8, wherein a particular vertical set of pixels includes only pixels within a particular sub-image.

10. The method of claim 8, wherein a particular horizontal set of pixels does not include sub-pixel values that are only generated through vertical interpolation operations that span more than one sub-image.

11. The method of claim 7 further comprising establishing a set of boundary regions between the sub-images, wherein dimensions of the boundary regions are determined by a number of pixels in the sets of pixels used for said interpolation operations.

12. The method of claim 11, wherein said interpolation operations are performed on sets of horizontal pixels, wherein a particular horizontal set of pixels includes pixels within a boundary region of one particular sub-image.

13. The method of claim 11 further comprising foregoing examining blocks aligned with a plurality of sub-pixel locations that are within a particular boundary region.

14. The method of claim 13, wherein the blocks that are not examined within the boundary regions are a plurality of blocks aligned with a plurality of half pixel locations.

15. The method of claim 14 further comprising examining a plurality of blocks aligned with a plurality of half pixel locations that are within the boundary regions and are located between two integer pixel locations within the same boundary region.

16. The method of claim 13, wherein the blocks that are not examined within the boundary regions are a plurality of blocks aligned with quarter pixel locations.

17. The method of claim 16 further comprising examining a plurality of blocks aligned with a plurality of quarter-pixel locations that are within the boundary regions and are interpolated utilizing integer pixel locations and half-pixel locations within two integer locations in the same boundary region.

18. The method of claim 1, wherein the constraint encoding comprises:

performing intra prediction for a set of pixels in a particular sub-image; and

excluding from the intra prediction sets of pixels that fall outside of the particular sub-image.

19. The method of claim 1 further comprising:

performing a loop-filter operation during the encoding of a particular sub-image; and

excluding pixels that are not in the particular sub-image from the loop-filter operation.

20. The method of claim 1, wherein said generating and encoding are performed at a computing device of a first participant that serves as a central distributor of video content during the video conference.

21. A method of distributing video content in a video conference between at least two participants, the method comprising:

detecting a condition requiring refreshing of a first participant'"'"'s image data through intra-block encoded data;

intra-block encoding a portion of a first image of the first participant;

transmitting the intra-block encoded portion of the first image to a second participant;

intra-block encoding a first portion of a second image of the first participant; and

transmitting the intra-block encoded portion of the second image to the second participant,

wherein the intra-block encoded portions of the first and second images at least partially refresh the first participant'"'"'s image data.

22. The method of claim 21 further comprising:

inter-block encoding a second portion of the second image; and

transmitting the inter-block encoding of the second portion to the second participant, wherein the second portion corresponds to the portion of the first image that was intra-block encoded.

23. The method of claim 21, wherein the video conference is a peer-to-peer conference.

24. The method of claim 21, wherein the first and second images are first and second portions of an I frame.

25. The method of claim 21, wherein said condition is a transmission error from the first participant to the second participant.

26. The method of claim 21, wherein the intra-block encoded portions of the first and second images fully refresh the first participant'"'"'s image data.

27. The method of claim 21 further comprising:

intra-block encoding a portion of at least a third image of the first participant; and

transmitting the intra-block encoded portion of the third image to the second participant, wherein the intra-block encoded portions of the first, second, and third images at least partially refresh the first participant'"'"'s image data.

28. The method of claim 27, wherein the intra-block encoded portions of the first, second, and third images completely refresh the first participant'"'"'s image data.

29. A method of distributing video content in a video conference between at least two participants, the method comprising:

storing a long term reference frame at a first participant;

notifying a second participant that the first participant has dropped video data transmitted by the second participant; and

receiving, from the second participant, inter-encoded data, wherein the inter-encoded data is defined by reference to the long term reference frame.

30. The method of claim 29, wherein the video conference is a peer-to-peer conference.

31. The method of claim 29, wherein the long term reference frame is also stored at the second participant.

32. The method of claim 29, wherein the long term reference frame is one of an I frame and a P frame.

33. A method of distributing video content in a video conference having a plurality of participants, the method comprising:

at a computing device of a first participant that serves as a central distributor of video content during the video conference;

receiving at least one inter-encoded video image from a second participant;

detecting a condition that an expected inter-encoded video image has not been received from a third participant;

requesting an intra-encoded video image from the third participant;

receiving the requested intra-encoded video image; and

transmitting a composite video image comprising the received intra-encoded image of the third participant to at least said second participant.

34. The method of claim 33, wherein said expected inter-encoded video image is a P frame, wherein said requested intra-encoded video image is an I frame.

35. The method of claim 33, wherein the transmitted composite video image further comprises inter-encoded video images from at least one other participant from the plurality of participants.

36. A computer readable medium storing a computer program for encoding an image during a video conference with multiple participants, the computer program comprising sets of instructions for:

generating a composite image comprising at least two sub-images for at least two participants; and

constraint encoding each sub-image separately, wherein said constraint encoding comprises encoding each sub-image based on a set of constraints that allow the encoding of the sub-image to be independent of the encoding of any other sub-image.

37. A computer readable medium storing a computer program for distributing video in a video conference between at least two participants, the computer program comprising sets of instructions for:

detecting a condition requiring refreshing of a first participant'"'"'s image data through intra-block encoded data;

intra-block encoding a portion of a first image of the first participant;

transmitting the intra-block encoded portion of the first image to a second participant;

intra-block encoding a first portion of a second image of the first participant; and

transmitting the intra-block encoded portion of the second image to the second participant,

wherein the intra-block encoded portions of the first and second images at least partially refresh the first participant'"'"'s image data.

38. A computer readable medium storing a computer program for distributing video in a video conference between at least two participants, the computer program comprising sets of instructions for:

storing a long-term reference frame at a first participant;

notifying a second participant that the first participant has dropped video data transmitted by the second participant; and

receiving, from the second participant, inter-encoded data, wherein the inter-encoded data is defined by reference to the long term reference frame.

39. A computer readable medium storing a computer program for distributing video content in a video conference having a plurality of participants, the computer program comprising sets of instructions for:

at a computing device of a first participant that serves as a central distributor of video content during the video conference;

receiving at least one inter-encoded video image from a second participant;

detecting a condition that an expected inter-encoded video image has not been received from a third participant;

requesting an intra-encoded video image from the third participant;

receiving the requested intra-encoded video image; and

transmitting a composite video image comprising the received intra-encoded image of the third participant to at least said second participant.