Video decoder which processes multiple video streams

• US 8,633,962 B2
• Filed: 06/19/2008
• Issued: 01/21/2014
• Est. Priority Date: 06/22/2007
• Status: Active Grant

• Current Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Original Assignee: Lifesize, Inc. (Enghouse Systems Limited)
• Inventors: King, Keith C., Mock, Wayne E.
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): ANDERSON II, JAMES M
• Application Number:
  US12/142,377
• Publication Number:
  US 20080316298A1
• Time in Patent Office: 2,042 Days
• Field of Search: None
• US Class Current: 348/14.08
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

In some embodiments, a spatially multiplexed output decoder may spatially multiplex video packets received in a time multiplexed video stream. A video stream with video packets from two or more sources may be received along with metadata. In some embodiments, a decoder may organize the video packets into respective buffers (e.g., each buffer including video packets for a respective video image). The spatially multiplexed output decoder may spatially multiplex the video images (which are made up of data from the respective video packets) into a video frame to be outputted (e.g., to a separate buffer). The video images in the video frame may then be demultiplexed in other parts of the system (e.g., in a virtual decoder) using information provided about the video frame (e.g., in metadata formed with the video frame).

Claims

1. A method, comprising:

receiving, by a decoder of a first video conferencing endpoint, a video stream comprising time multiplexed video packets from two or more video conferencing endpoints of a plurality of video conferencing endpoints participating in a video conference with the first video conferencing endpoint;

sorting, by the decoder, the video packets into buffers according to the video packets respective video source conferencing endpoint;

spatially multiplexing, by the decoder, two or more video images into a first composite video frame, wherein the two or more video images are formed using the video packets from the respective buffers;

passing the first composite video frame to a virtual decoder of the first video conferencing endpoint;

finding, by the virtual decoder, boundaries of the two or more video images within the first composite video frame;

separating, by the virtual decoder, the two or more video images into separate video images;

compositing, by the first videoconferencing endpoint, a second composite video frame comprising a plurality of video images, wherein the plurality of video images includes at least one of the separate video images;

transmitting to a second video conferencing endpoint of the plurality of video conferencing endpoints the second composite video frame; and

transmitting to the second video conferencing endpoint information regarding which video images of the plurality of video images of the second composite video frame are to be included in a third composite video frame, wherein the third composite video frame is to be generated by the second video conferencing endpoint.

2. The method of claim 1, further comprising writing out the first composite video frame to a single buffer, wherein said passing comprises passing the first composite video frame from the single buffer.

3. The method of claim 1, further comprising receiving metadata with the video stream, wherein the received metadata includes identification information and is used to sort the video packets into the respective buffers.

4. The method of claim 1, further comprising displaying the new video layout by the first video conferencing endpoint.

6. The method of claim 1, further comprising:

generating, by the decoder, metadata for the first composite video frame, wherein the metadata for the first composite video frame comprises coordinate information identifying locations of the two or more video images within the first composite video frame; and

passing the metadata to the virtual decoder, wherein the virtual decoder uses the coordinate information in the finding the boundaries of the two or more video images within the first composite video frame.

7. The method of claim 6, wherein the metadata for the first composite video frame further comprises identifying information for a video conferencing endpoint corresponding to the coordinate information of each respective video image.

8. The method of claim 1, wherein the information regarding which video images of the plurality of video images of the second composite video frame are to be included in a third composite video frame identifies at least one video conferencing endpoint of the plurality of video conferencing endpoints, wherein video images originated by the identified at least one video conferencing endpoint are to be included in the third composite video frame.

9. The method of claim 1, wherein the information regarding which video images of the plurality of video images of the second composite video frame are to be included in a third composite video frame also identifies one or more video conferencing endpoints of the plurality of video conferencing endpoints to which the third composite video frame is to be transmitted by the second video conferencing endpoint.

10. The method of claim 1, further comprising:

transmitting to the second video conferencing endpoint a local video frame, wherein the local video frame is originated by an image capture device of the first video conferencing endpoint.

5. A method, comprising:

receiving, by a decoder of a first video conferencing endpoint, a video stream comprising time multiplexed video packets from two or more video conferencing endpoints of a plurality of video conferencing endpoints participating in a video conference with the first video conferencing endpoint;

receiving, by the decoder, metadata with the video stream;

wherein the metadata includes identification information for the video packets in the video stream;

sorting, by the decoder, the video packets into buffers according to the video packets respective video conferencing endpoint, wherein the metadata is used to sort the video packets into the respective buffers;

spatially multiplexing, by the decoder, video images into a first composite video frame, wherein the video images are formed using the video packets from the respective buffers;

passing the first composite video frame to a virtual decoder of the first video conferencing endpoint;

finding, by the virtual decoder, boundaries of the video images within the first composite video frame;

separating, by the virtual decoder, the video images of the first composite video frame into separate video images;

receiving from a second video conferencing endpoint of the plurality of video conferencing endpoints a set of instructions specifying which of the separate video images are to be included in a second composite video frame;

compositing, by the first video conferencing endpoint, the second composite video frame comprising the specified separate video images; and

transmitting the second composite video frame to a third video conferencing endpoint of the plurality of video conferencing endpoints.

11. The method of claim 5, further comprising:

generating, by the decoder, metadata for the first composite video frame, wherein the metadata for the first composite video frame comprises coordinate information identifying locations of the video images within the first composite video frame; and

passing the metadata to the virtual decoder, wherein the virtual decoder uses the coordinate information in the finding the boundaries of the video images within the first composite video frame.

12. The method of claim 11, wherein the metadata for the first composite video frame further comprises identifying information for a video conferencing endpoint corresponding to the coordinate information of each respective video image.

13. The method of claim 5, wherein the set of instructions specifies which of the separate video images are to be included in the second composite video frame by identifying at least one video conferencing endpoint of the plurality of video conferencing endpoints, wherein each of the separate video images originated by the identified at least one video conferencing endpoint are to be included in the second composite video frame.

14. The method of claim 5, wherein the set of instructions further identifies one or more video conferencing endpoints of the plurality of video conferencing endpoints to which the second composite video frame is to be transmitted by the first video conferencing endpoint, wherein the second video conferencing endpoint is one of the identified one or more video conferencing endpoints, the method further comprising:

transmitting the second composite video frame to a third video conferencing endpoint of the plurality of video conferencing endpoints.

15. The method of claim 5, further comprising:

transmitting to the second video conferencing endpoint a local video frame, wherein the local video frame is originated by an image capture device of the first video conferencing endpoint.

21. The video conferencing endpoint of claim 13, wherein each respective set of instructions further identifies one or more video conferencing endpoints of the plurality of remote video conferencing endpoints to which the composite video frames transmitted by the respective endpoint are to be transmitted.

16. A video conferencing endpoint operable to perform multi-way video conferencing, the video conferencing endpoint comprising:

a communication device configured to;

receive a video stream comprising time multiplexed video packets from two or more source video conferencing endpoints of a plurality of remote video conferencing endpoints participating in a video conference with the video conferencing endpoint;

transmit to a first video conferencing endpoint of the plurality of remote video conferencing endpoints a second composite video frame; and

transmit to each video conferencing endpoint of the plurality of remote video conferencing endpoints a respective set of instructions;

a decoder coupled to the communication device, and configured to;

sort the video packets into buffers according to the video packets respective video conferencing endpoint; and

spatially multiplex two or more video images into a first composite video frame, wherein the two or more video images are formed using the video packets from the respective buffers; and

a virtual decoder coupled to the decoder, and configured to;

receive the first composite video frame from the decoder; and

separate the two or more video images into separate video images;

a compositor coupled to the virtual decoder and the communication device, and configured to;

composite the second composite video frame comprising a plurality of video images, wherein the plurality of video images includes at least one of the separate video images; and

a processor coupled to the communication device, and configured to;

determine the respective sets of instructions, wherein, for each respective endpoint of the plurality of remote video conferencing endpoints, the respective set of instructions specifies at least one endpoint of the plurality of remote video conferencing endpoints, wherein frames originated by the at least one endpoint are to be included in composite frames transmitted by the respective endpoint.

17. The video conferencing endpoint of claim 16, wherein the communication device is further configured to receive metadata with the video stream, wherein the received metadata includes identification information and is used by the decoder to sort the video packets into the respective buffers.

18. The video conferencing endpoint of claim 16, wherein the virtual decoder is implemented as virtual decoder program instructions, wherein the processor is further configured to execute the virtual decoder program instructions.

19. The video conferencing endpoint of claim 16,

wherein the decoder is further configured to generate metadata for the first composite video frame, wherein the metadata for the first composite video frame comprises coordinate information identifying locations of the two or more video images within the first composite video frame;

and

wherein the virtual decoder is further configured to receive the metadata from the decoder, and use the coordinate information in the separating the two or more video images.

20. The video conferencing endpoint of claim 19, wherein the metadata for the first composite video frame further comprises identifying information for a video conferencing endpoint corresponding to the coordinate information of each respective video image.

22. The video conferencing endpoint of claim 16, further comprising:

a video image capture device;

wherein the communication device is further configured to transmit to the first video conferencing endpoint a local video frame, wherein the local video frame is originated by the video image capture device.