REAL-TIME SUPERRESOLUTION AND VIDEO TRANSMISSION

• US 20100290529A1
• Filed: 04/14/2010
• Published: 11/18/2010
• Est. Priority Date: 04/14/2009
• Status: Active Grant

• Current Assignee: FastVDO LLC
• Original Assignee: Pankaj Topiwala
• Inventors: Topiwala, Pankaj
• Granted Patent: US 8,520,736 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 3/4069 by subpixel displacements
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A method and system of performing real-time video superresolution. A decoder receives a data stream representing a low resolution video and including global motion vectors relating to image motion between frames of the low resolution video. The decoder uses the global motion vectors from the received data stream and multiframe processing algorithms to derive a high resolution video from the low resolution video. The sharpness of the high resolution video may be enhanced.

Claims

1. A method of performing real-time video superresolution, comprising:

receiving at a decoder a data stream representing a low resolution video and global motion vectors relating to image motion between frames of the low resolution video;

using the global motion vectors and advanced multiframe processing to derive a high resolution video from the low resolution video; and

increasing the sharpness of the high resolution video.

2. The method of claim 1, wherein the low resolution video is a downsampled version of a higher resolution original video.

3. The method of claim 2, wherein the low resolution video and global motion vectors are encoded using a video coding standard selected from the group comprising MPEG-1, MPEG-2, MPEG-4;

H.263, H.264 and H.265.

4. The method of claim 3, wherein the global motion vectors are encoded in the data stream, either as auxiliary data in a stream wrapper such as an MPEG-2 transport stream or within the video bit stream such as in a supplemental enhancement information message field of the H.264 standard.

5. The method of claim 4, wherein the global motion vectors are encoded in the supplemental enhancement information message field using non-uniform quantization.

6. The method of claim 1, wherein each frame of the high resolution video is derived from a plurality of frames of the low resolution video and the global motion vectors.

7. The method of claim 6, wherein the plurality of low resolution frames is at least three.

8. The method of claim 6, wherein the derivation of the high resolution video uses partition weighted sum filtering.

9. The method of claim 6, wherein the derivation of the high resolution video uses compressive sensing.

10. The method of claim 9, wherein the compressive sensing does not require the decoder to store trained dictionaries.

11. The method of claim 9, wherein the compressive sensing is used to represent a vectorized block of pixels in a given frame as a linear combination of vectorized blocks of pixels from frames that are temporally proximate to the given frame.

12. The method of claim 1, wherein the sharpness of the high resolution video is increased by iteratively applying a back-projection algorithm.

13. The method of claim 12, wherein the applied back-projection algorithm uses nearest neighbor based fusion of low resolution frames.

14. The method of claim 1 wherein the video superresolution processing is achieved in real-time via computer readable software code.

15. The method of claim 1 wherein the video superresolution processing is achieved in real-time via hardware components including field programmable gate arrays (FPGAs).

16. A system of performing real-time video superresolution, comprising:

a decoder configured to receive a data stream representing a low resolution video and global motion vectors relating to image motion between frames of the low resolution video and to use the global motion vectors to derive a high resolution video from the low resolution video.

17. The system of claim 16, wherein the received low resolution video is a downsampled version of a higher resolution video.

18. The system of claim 17, wherein the low resolution video and global motion vectors are encoded according to a video coding standard selected from the group comprising MPEG-1, MPEG-2, MPEG-4;

H.263, 11.264 and H.265.

19. The system of claim 18, wherein the global motion vectors are encoded in the data stream, either as auxiliary data in a stream wrapper such as an MPEG-2 transport stream or within the video bit stream such as in a supplemental enhancement information message field of the H.264 standard.

20. The system of claim 19, wherein the global motion vectors are encoded in the supplemental enhancement information message field using non-uniform quantization.

21. The system of claim 16, wherein each frame of the high resolution video is derived from a plurality of frames of the low resolution video and the global motion vectors.

22. The system of claim 21, wherein the plurality of low resolution frames is at least three.

23. The system of claim 21, wherein the derivation of the high resolution video uses partition weighted sum filtering.

24. The system of claim 21, wherein the derivation of the high resolution video uses compressive sensing.

25. The system of claim 24, wherein the decoder does not include trained dictionaries stored prior to receipt of and corresponding to the received data stream.

26. The system of claim 24, wherein the compressive sensing is used to represent a vectorized block of pixels in a first frame as a linear combination of vectorized blocks of pixels from frames that are temporally proximate to the first frame.

27. The system of claim 16, wherein the decoder further enhances the sharpness of the high resolution video.

28. The system of claim 27, wherein the sharpness of the high resolution video is increased by iteratively applying back-projection.

29. The system of claim 28, wherein the applied back-projection uses nearest neighbor based fusion of low resolution frames.

30. A system of performing video real-time superresolution, comprising:

an encoder configured to encode a low resolution video by downsampling a high resolution video, deriving global motion vectors relating to image motion between frames of the low resolution video; and

a decoder configured to receive the encoded low resolution video and global motion vectors and to use the global motion vectors to derive a high resolution video from the low resolution video.

31. The system of claim 30, wherein the low resolution video and global motion vectors are encoded according to a video coding standard selected from the group comprising MPEG-1, MPEG-2, MPEG-4;

H.263, H.264 and H.265.

32. The system of claim 31, wherein the global motion vectors are encoded in the data stream, either as auxiliary data in a stream wrapper such as an MPEG-2 transport stream or within the video bit stream such as in a supplemental enhancement information message field of the H.264 standard.

33. The system of claim 30, wherein each frame of the high resolution video is derived from a plurality of frames of the low resolution video and the global motion vectors.

34. The system of claim 30, wherein the decoder does not include trained dictionaries stored prior to receipt of and corresponding to the received data stream.

35. The system of claim 30, wherein the decoder further enhances the sharpness of the high resolution video by iteratively applying a back-projection algorithm.