Method and system for multimedia communication control

• US 6,300,973 B1
• Filed: 01/13/2000
• Issued: 10/09/2001
• Est. Priority Date: 01/13/2000
• Status: Expired due to Term

• Current Assignee: Polycom Israel Limited (HP Inc.)
• Original Assignee: Polycom Israel Limited (HP Inc.)
• Inventors: Eshkoli, Noam, Elbaz, Moshe, Yona, Ilan, Feder, Meir, Eisenberg, Aviv
• Primary Examiner(s): Chan, Wing F.
• Application Number:
  US09/506,861
• Time in Patent Office: 635 Days
• Field of Search: 348/14-7-, 379/202
• US Class Current: 348/14.09
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/134 characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/184 the unit being bits, e.g. of the compressed video stream
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A multipoint control unit (MCU) for facilitating communication between a plurality of endpoints. Each endpoint sends a compressed video signal and receives a compressed video signal. The MCU has a plurality of video input modules and a video output module. Each of the video input modules receives a video signal from one of the endpoints and generally decodes the data into a primary data stream and a secondary data stream. The video output module includes a rate control unit and a generalized encoder that receive the primary and secondary data streams for generally encoding to a compressed output stream for transmission to an endpoint.

Claims

1. A multipoint control unit for facilitating communication between a plurality of endpoints, each respective endpoint sending a compressed video output signal and receiving a compressed video input signal, comprising:

a plurality of video input modules, each video input module receiving a respective video output signal from a respective endpoint, each video input module comprising;

a generalized decoder for reading the respective video output signal and for generating a respective primary data stream comprising video information and a respective secondary data stream comprising side information; and

a video output module comprising;

a rate control unit operable to read each of the respective secondary data streams, pre-process the respective secondary data stream, and control a generalized encoder; and

the generalized encoder, in communication with the rate control unit and operable to receive each of the respective primary data streams from each respective video input module, and

encode the respective primary data stream into a compressed video output stream for transmission to an endpoint.

2. A multipoint control unit for facilitating communication between a plurality of endpoints, each of said plurality of endpoints including a video screen and a video camera, each of said plurality of endpoints being operative to send a compressed video input signal to said multipoint control unit and receive a compressed video output signal from said multipoint control unit, the multipoint control unit comprising:

at least one video input module, for receiving a compressed video input signal from at least one endpoint of said plurality of endpoints, the video input module comprising;

a generalized decoder operative to decode the compressed video input signal and generate a primary video data stream, the generalized decoder comprising;

a data processing unit operative to process said compressed video input signal and said primary video data stream to generate a secondary data stream, said secondary data stream being associated with said primary video stream; and

at least one video output module operative to receive at least one of said primary video data stream and said secondary data stream, the output module comprising;

a rate control unit; and

a generalized encoder, in communication with said rate control unit and operative to receive said primary data stream from said at least one input module and encode said primary data stream into a compressed video output stream for transmission to at least one endpoint of said plurality of endpoints;

means to route said primary data stream from at least one input module to at lest one output module; and

means to route said secondary data stream from at least one input module to the at lest one output module;

whereby the use of said secondary data stream by the output module improves the speed of encoding and the quality of the compressed video output signal.

3. The multipoint control unit of claim 2, wherein the association between the secondary data stream and the primary video data stream is that the secondary data stream includes side information.

4. The multipoint control unit of claim 2, wherein the compressed video input signal includes at least one type of information selected from a group consisting of:

frame type, resolution, motion vectors, filter indication, DCT coefficients and quantizer values.

5. The multipoint control unit of claim 3, wherein said side information includes at least one type of information selected from a group consisting of:

frame type, resolution, motion vectors, filter usage indication, quantizer identifications, coded/uncoded decisions, the amount of information within each macroblock, image segmentation indication, scene cut off indication, camera zoom identification, camera pan identification, camera movements identification and statistical information.

6. The multipoint control unit of claim 2, wherein said rate control unit comprises:

means to read said secondary data stream;

means to process said secondary data stream; and

means to control a generalized encoder based upon said processed secondary data stream.

7. The multipoint control unit of claim 6, wherein said rate control unit comprises:

means to read feedback data from a generalized encoder;

means to process said secondary data stream with said feedback data; and

means to control said generalized encoder based upon said processed secondary data stream and said feedback.

8. The multipoint control unit of claim 2, wherein said means to route said primary data stream includes a common interface selected from a group consisting of:

shared memory, ATM bus, TDM bus, switching and direct connection.

9. The multipoint control unit of claim 2, wherein said means to route the secondary data stream includes a common interface selected from a group consisting of:

shared memory, ATM bus, TDM bus, switching and direct connection.

10. The multipoint control unit of claim 2, wherein said primary data stream includes information in the DCT domain.

11. The multipoint control unit of claim 2, wherein said primary data stream includes information in spatial domain.

12. The multipoint control unit of claim 2, wherein said video output module receives at least one of said primary video data streams and it'"'"'s associated said secondary data stream and control information from an external device.

13. The multipoint control unit of claim 12, wherein said rate control unit of said video output module comprises:

means to read said secondary data stream;

means to read said control information;

means to process said secondary data stream;

means to process the control information; and

means to control said generalized encoder based upon the processed secondary data stream and the processed control information.

14. The multipoint control unit of claim 13, wherein said rate control unit of said video output module comprises:

means to read feedback data from a generalized encoder; and

means to process said secondary data stream with said control information and said feedback; and

means to control said generalized encoder based upon the results.

15. The multipoint control unit of claim 12, wherein the control information includes at least one type of information selected from a group consisting of:

region of interest indication, screen layout requirements, user quality preferences and special effects.

16. The multipoint control unit of claim 12 wherein the control information is bidirectional information.

17. The multipoint control unit of claim 2 wherein said video output module receives said primary video data stream and said secondary data stream, and said rate control unit of said video output module comprises:

means to read said secondary data stream;

means to read data on the actual amount of bits used after variable length coding;

means to process the respective secondary data stream with the variable length coding information; and

means to control said generalized encoder based on the said processed information, whereby the use of said variable length coding information and the secondary data stream by generalized encoder improves the speed of encoding and the quality of the compressed video output signal by improving the output bits allocation.

18. The multipoint control unit of claim 17 wherein said video output module receives said primary video data stream and said secondary data stream, and said rate control unit of said video output module comprises:

means to read feedback data from a generalized encoder;

means to process the respective secondary data stream with the variable length coding information and said feedback data; and

means to control said generalized encoder based on the said processed information.

19. A multipoint control unit for facilitating communication between a plurality of endpoints, each endpoint including a video screen and a video camera, each endpoint being operative to send a compressed video input signal to the multipoint control unit and receive a compressed video output signal from said multipoint control unit, said multipoint control unit comprising:

at least one video fat port, each video fat port receiving at least one compressed video input signal from at least one endpoint and sending a compressed video output signal to at least one endpoint of said plurality of endpoints, each video fat port comprising;

at least one video input module, each video input module receiving a compressed video input signal from one endpoint of said plurality of endpoints, each video input module comprising;

a generalized decoder for decoding the compressed video input signal from said endpoint for generating a primary video data stream; and

a video output module, the output module receiving at least one of said primary video data streams, the output module comprising;

a rate control unit;

a generalized encoder, in communication with said rate control unit and operative to receive each of said primary data stream from at least one of said input modules and encode said primary data streams into a compressed video output stream for transmission to at least one endpoint of said plurality of endpoints; and

means to route said primary data stream from at least one input module to the output module;

whereby the use of said fat port enables sharing video streams of a conference in the compressed domain on the BackPlane this increases the number of participants in a conference and eliminates fragmentation compared to a case where the video sharing is done in the spatial domain on an open video bus.

20. A multipoint control unit for facilitating communication between a plurality of endpoints, each endpoint including a video screen and a video camera, each endpoint being operative to send a compressed video input signal to the multipoint control unit and receive a compressed video output signal from said multipoint control unit, the multipoint control unit comprising:

at least one video fat port, each video fat port receiving at least one compressed video input signal from at least one endpoint and sending a compressed video output signal to at least one endpoint of said plurality of endpoints, each video fat port comprising;

at least one video input module, each video input module receiving a compressed video input signal from one endpoint of said plurality of endpoints, each video input module comprising;

a generalized decoder for decoding the compressed video input signal from said endpoint for generating a primary video data stream, the generalized decoder comprising;

a data processing unit for processing said compressed video input signal and said primary video data stream for generating a secondary data stream associated with said primary video data stream comprising side information; and

a video output module, the video output module receiving at least one of said primary video data streams and said secondary data streams, the output module comprising;

a rate control unit; and

a generalized encoder, in communication with said rate control unit and operative to receive each of said primary data stream from at least one of said input modules and encode said primary data streams into a compressed video output stream for transmission to at least one endpoint of said plurality of endpoints; and

means to route said primary data stream from at least one input module to the output module; and

means to route said secondary data stream from at least one input module to the output module,

whereby the use of said fat port enables sharing video streams of a conference in the compressed domain on the BackPlane this increases the number of participants in a conference and eliminates fragmentation in compare to a case where the video sharing is done in the spatial domain on an open video bus.

21. The multipoint control unit of claim 20, wherein the association between the secondary data stream and the primary video data stream is that the secondary data stream includes side information.

22. The multipoint control unit of claim 20, wherein the compressed video input signal includes at least one type of information selected from a group consisting of:

frame type, resolution, motion vectors, filter indication, DCT coefficients and quantizer values.

23. The multipoint control unit of claim 21, wherein said side information includes at least one type of information selected from a group consisting of:

frame type, resolution, motion vectors, filter usage indication, quantizer identifications, coded/uncoded decisions, the amount of information within each macroblock, image segmentation indication, scene cut of indication, camera zoom identification, camera pan identification, camera movements identification and statistical information.

24. The multipoint control unit of claim 20, wherein said rate control unit comprises:

means to read each of the respective secondary data streams;

means to process the respective secondary data streams; and

means to control said generalized encoder based upon said processed secondary data stream.

25. The multipoint control unit of claim 20, wherein said means to route said primary data stream includes a common interface selected from a group consisting of:

shared memory, ATM bus, TDM bus, switching and direct connection.

26. The multipoint control unit of claim 20, wherein said means to route said secondary data stream includes a common interface selected from a group consisting of:

shared memory, ATM bus, TDM bus, switching and direct connection.

27. The multipoint control unit of claim 20, wherein said primary data stream includes information in a DCT domain.

28. The multipoint control unit of claim 20, wherein said primary data stream includes information in spatial domain.

29. The multipoint control unit of claim 20, wherein said video output module receives said at least one primary video data stream and said secondary data stream and control information from an external device.

30. The multipoint control unit of claim 29, wherein said rate control unit of said video output module comprises:

means to read the respective secondary data stream;

means to read said control information;

means to process the respective secondary data stream;

means to process the control information; and

a means to control the generalized encoder based upon the processed information.

31. The multipoint control unit of claim 29, wherein said rate control unit of said video output module comprises:

means to read feedback data from a generalized encoder;

means to process said secondary data stream with said control information and said feedback; and

means to control said generalized encoder based upon the results.

32. The multipoint control unit of claim 29, wherein the control information includes at least one type of information selected from the group consisting of:

region of interest indication, screen layout requirements, user quality preferences and special effects.

33. The multipoint control unit of claim 29, wherein the control information is bidirectional information.

34. The multipoint control unit of claim 20, wherein said video output module receives said primary video data stream and said secondary data stream, and said rate control unit of said video output module comprises:

means to read said respective secondary data stream;

means to read data on the actual amount of bits used after variable length coding;

means to process the respective secondary data stream with the variable length coding information; and

means to control a generalized encoder based on the said processed information,

whereby process of said variable length coding information and said secondary data stream the generalized encoder improves the speed of encoding ant the quality of the compressed video output signal by improving the output bits allocation.

35. The multipoint control unit of claim 34 wherein said video output module receives said primary video data stream and said secondary data stream, and said rate control unit of said video output module comprises:

means to read feedback data from a generalized encoder;

means to process the respective secondary data stream with the variable length coding information and said feedback data; and

means to control said generalized encoder based on the said processed information.

36. A method of performing operations on a compressed video stream, the method comprising the steps of:

reading encoding parameters embedded within the compressed input video stream;

processing the compressed input video stream into two data stream, a primary data stream and a secondary data stream;

routing said primary data stream and secondary data stream to at least one output unit; and

encoding said primary data stream by using the information associated with said secondary data stream,

whereby using said secondary data stream improves the speed of encoding and the quality of the compressed video output signal.

37. The method of claim 36, wherein the step of reading encoding parameters further comprises the step of reading from the compressed video stream at least one type of parameter selected from a group of parameters consisting of:

DCT coefficients, frame type, resolution, motion vectors, filter indication, quantizer values.

38. The method of claim 36, wherein the step of reading encoding parameters further comprises the step of reading quantizer values from the compressed video stream.

39. The method of claim 36, wherein the step of reading encoding parameters further comprises the step of reading statistical information from the compressed video stream.

40. The method of claim 36, wherein the step of pseudo-encoding further comprises the step of using the DCT coefficients to assist in the pseudo-encoding.

41. The method of claim 36, wherein the step of processing said compressed video stream further comprises the step of analyzing at least one type of indications selected from a group of indication consisting of:

coded/uncoded decision, the amount of information within each macroblock, image segmentation, scene cut off, camera zoom identification, camera pan identification, camera movements identification and statistical information.

42. The method of claim 36, wherein the step of encoding further comprises the step of using at least one parameter from a group of parameters, associated with said secondary data stream, consisting of:

DCT coefficients, frame type, resolution, motion vectors, filter indication, quantizer values coded/uncoded decision, the amount of information within each macroblock, image segmentation indications, scene cut off indications, camera zoom identification, camera pan identification, camera movements identification, and statistical information.

43. An apparatus embodying the method of claim 36.

44. A system for performing operations on a compressed video stream, the system comprising:

at least one generalized decoder for decoding the compressed video stream into a primary data stream and for reading and analyzing encoding parameters embedded within the compressed video stream and for creating a secondary data stream; and

at least one editor that receives the primary streams and generates modified pseudo-decoded video stream; and

a generalized encoder for encoding said modified decoded video stream into a second compressed output video stream using the encoding parameters from the generalized decoders.

45. The system of claim 44, wherein the editor is further operable to scale said primary data stream.

46. The system of claim 44, wherein the editor is further operable to receive a second primary data stream.

47. The system of claim 44, wherein the editor is further operable to scale the secondary primary data stream.

48. The system of claim 44, wherein the editor is further operable to composite the first primary data stream and the second primary data stream.

49. The system of claim 44, further comprising a router for sending the primary data stream to the editor.

50. The system of claim 44, further comprising a router for sending the secondary data stream to the output unit.