Method and apparatus for reconstruction of low frame rate video conferencing data

• US 6,704,357 B1
• Filed: 09/28/1999
• Issued: 03/09/2004
• Est. Priority Date: 09/28/1999
• Status: Expired due to Term

• Current Assignee: Open Invention Network LLC
• Original Assignee: 3Com Corporation (HP Inc.)
• Inventors: Ketcham, Carl
• Primary Examiner(s): Kelley, Chris
• Assistant Examiner(s): VO, TUNG T
• Application Number:
  US09/408,974
• Time in Patent Office: 1,624 Days
• Field of Search: H04/B.166, H04/N.514, 375/240.01, 375/240.02, 375/240.03, 375/240.12, 375/240.13, 375/240.15, 375/240.16, 375/240.26, 375/240.27, 375/240.28, 375/240.29, 375/240.24, 348/411.1, 348/412.1, 348/413.1, 348/414.1, 348/416.1, 348/699, 348/700, 348/701
• US Class Current: 375/240.01
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 7/148 Interfacing a video terminal to a particular transmission medium, e.g. ISDN

Abstract

A method and apparatus for improving the playback quality of audio/video streams encoded for transmission using low frame rate motion compensated coding. When a transmitting system sends an audio/video stream encoded using low frame rate motion compensated waveform, a receiving system may decode and then re-encode the stream to perform a more exhaustive search for new motion vectors that were not detected by the transmitting system. The method and system further provides improved motion compensated interpolation processes that the receiving system may use in conjunction with, or separately from re-encoding of an audio/video stream to produce a better quality playback of the stream. The method and system is particularly useful for video conferencing sessions transmitted at low rates using H.263 standard, but may be useful in other scenarios as well.

Claims

1. A method of reconstructing a video stream using a computer, the video stream including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream, the method comprising:

receiving the first frame and a plurality of vectors describing changes from the first frame to the second frame on the computer;

decoding the first frame and the plurality of vectors to produce a reconstructed second frame, the reconstructed second frame corresponding to the second frame;

determining whether a number of vectors in the plurality of vectors exceeds a predetermined number using the computer;

responsive to the determining, encoding the first frame and the reconstructed second frame to generate a second plurality of vectors using the computer; and

using the second plurality of vectors to generate at least one intermediate frame between the second frame and the third frame.

2. The method of claim 1, wherein the video stream includes a format, a region size, and an application, the format describing a size for each frame of the video stream, the region size describing a region size covered by vectors in the plurality of vectors, and the application specifying a type for the video stream, the method further comprising setting the predetermined number using at least one of the format, the region size, and the application prior to the determining.

3. The method of claim 2, wherein the type for the video stream comprises a video conference and wherein the predetermined number comprises 80% of a total number of regions for the format.

4. The method of claim 1, wherein the using the second plurality of vectors to generate at least one intermediate frame between the second frame and the third frame comprises:

receiving a third plurality of vectors describing changes from the second frame to the third frame on the computer;

decoding the third plurality of motion vectors to produce a reconstructed third frame using the third plurality of vectors, the reconstructed third frame corresponding to the third frame;

encoding the reconstructed second frame and the reconstructed third frame to generate a fourth plurality of vectors; and

using the fourth plurality of vectors, the at least one intermediate frame, and the reconstructed third frame to generate a new frame, the new frame for display in place of the reconstructed third frame.

5. A method of motion compensated interpolation on a video conferencing station for a video stream accessible to the video conferencing station as a plurality of encoded frames, the plurality of encoded frames including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream, the method comprising:

decoding the first frame and the second frame to produce a reconstructed second frame, the reconstructed second frame corresponding to the second frame;

encoding the first frame and the reconstructed second frame using the video conferencing station to produce a plurality of vectors corresponding to the reconstructed second frame, the plurality of vectors describing changes from the first frame to the second frame;

determining a plurality of motion vectors corresponding to the reconstructed second frame from the corresponding plurality of vectors; and

using the plurality of motion vectors to add at least one intermediate frame between the reconstructed second frame and the third frame, the at least one intermediate frame generated by a process of motion compensated interpolation.

6. The method of claim 5, wherein the plurality of encoded frames are accessed through receipt by the video conferencing station of a transmission of the video stream from a remote video conferencing station.

7. The method of claim 5, wherein the plurality of encoded frames comprises a plurality of Moving Pictures Expert Group level 4 (MPEG-4) format frames.

8. The method of claim 5, wherein the plurality of encoded frames comprises a plurality of H.263 format frames.

9. A method of making an intermediate frame in a video stream, the video stream including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream, the method comprising;

generating a plurality of motion vectors by a video conferencing station receiving the video stream;

using the plurality of motion vectors generated by the video conferencing station receiving the video stream to make a first intermediate frame between the second frame and the third frame,

fitting a curve according to the plurality of motion vectors generated by the video conferencing station receiving the video stream for the second frame and a second plurality of motion vectors generated by the video conferencing station receiving the video stream for the third frame; and

using the curve to generate a second intermediate frame.

10. The method of claim 9, wherein the using further comprises:

encoding the first frame and the second frame to generate a plurality of vectors; and

determining the plurality of motion vectors from the plurality of vectors.

11. A computer data signal embodied in a carrier wave comprising:

a computer program for improved motion compensated interpolation, the computer program comprising;

a first set of instructions for receiving an encoded video stream comprised of a plurality of encoded frames, the plurality of encoded frames including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream;

a second set of instructions for decoding the first frame and the second frame to create a reconstructed second frame, the reconstructed second frame corresponding to the second frame;

a third set of instructions for encoding the first frame and the reconstructed second frame with a motion compensated waveform encoding to produce a plurality of motion vectors corresponding to the reconstructed second frame; and

a fourth set of instructions for using the plurality of motion vectors corresponding to the reconstructed second frame to perfonn motion compensated interpolation between the second frame and the third frame.

12. The computer program of claim 11, further comprising a fifth set of instructions for sequencing the first set of instructions, the second set of instructions, the third set of instructions, and the fourth set of instructions to operate in order on the video stream as each of the plurality of encoded frames in the video stream in received and processed by the previous set of instructions.

13. The computer program of claim 11, further comprising a fifth set of instructions for motion compensated waveform encoding and decoding.

14. The computer program of claim 11, wherein the motion compensated waveform encoding is an H.263 encoding.

15. A method of displaying a video stream using a computer coupled to a display, the video stream including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream, the method comprising:

receiving the first frame and a plurality of vectors describing changes from the first frame to the second frame on the computer;

displaying the first frame on the display;

decoding the first frame and the plurality of vectors to produce a reconstructed second frame using the computer, the reconstructed second frame corresponding to the second frame;

displaying the reconstructed second frame on the display;

generating at least one intermediate frame between the second frame and the third frame using the computer and the plurality of vectors;

displaying the at least one intermediate frame;

receiving a second plurality of vectors describing changes from the second frame to the third frame on the computer;

using the second plurality of vectors, the at least one intermediate frame, and the reconstructed third frame to generate a new frame, the new frame for display in place of the reconstructed third frame; and

displaying the new frame on the display.

16. The method of claim 15, wherein the generating the at least one intermediate frame further comprises:

determining a plurality of motion vectors from the plurality of vectors; and

creating the at least one intermediate frame from the reconstructed second frame and the plurality of motion vectors.

17. An apparatus for reconstructing a video stream, the video stream including at least a first frame, a second frame, and a third frame, the first frame preceding the second frame in the video stream, the second frame preceding the third frame in the video stream, the apparatus comprising:

a memory means for storing the video stream; and

a processing means for

accessing the first frame and a plurality of vectors describing changes from the first frame to the second frame;

decoding the first frame and the plurality of vectors to store a reconstructed second frame in the memory means, the reconstructed second frame corresponding to the second frame;

determining whether a number of vectors in the plurality of vectors exceeds a predetermined number;

responsive to the determining, encoding the first frame and the reconstructed second frame to generate a second plurality of vectors; and

using the second plurality of vectors to store at least one intermediate frame between the second frame and the third frame in the memory means.