Apparatus and methods for video compression using multi-resolution scalable coding

• US 10,212,438 B2
• Filed: 10/31/2017
• Issued: 02/19/2019
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Abbas, Adeel, Adsumilli, Balineedu Chowdary, Newman, David
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US15/799,837
• Publication Number:
  US 20180054624A1
• Time in Patent Office: 476 Days
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

Apparatus and methods for digital video data compression via a scalable, multi-resolution approach. In one embodiment, the video content may be encoded using a multi-resolution and/or multi-quality scalable coding approach that reduces computational and/or energy load on a client device. In one implementation, a low fidelity image is obtained based on a first full resolution image. The low fidelity image may be encoded to obtain a low fidelity bitstream. A second full resolution image may be obtained based on the low fidelity bitstream. A portion of a difference image obtained based on the second full resolution image and the first full resolution may be encoded to obtain a high fidelity bitstream. The low fidelity bitstream and the high fidelity bitstream may be provided to e.g., a receiving device.

Claims

1. A method for providing video content, the method comprising:

obtaining a first panoramic image by decoding a first bitstream using a first decoder;

obtaining a view frame image by at least decoding a second bitstream using a second decoder;

obtaining a second panoramic image at least in part by up-sampling the first panoramic image to the resolution of the view frame image;

obtaining an output view frame image at least in part by combining the view frame image and the second panoramic image; and

displaying the obtained output view frame image via a rendering device.

2. The method of claim 1, wherein the obtaining of the first panoramic image comprises obtaining a low fidelity image characterized by at least a first resolution, and the obtaining of the second panoramic image comprises obtaining a high fidelity image characterized by at least a second resolution that is greater than the first resolution.

3. The method of claim 2, wherein a ratio of the first resolution to the second resolution is characterized by a resolution scalability parameter.

4. The method of claim 3, wherein:

the resolution scalability parameter is based at least on a display capability of a display device; and

the method further comprises displaying the output view frame image via the display device.

5. The method of claim 3, wherein the resolution scalability parameter is based at least on a processing or memory limitation of a display device.

6. The method of claim 1, further comprising:

obtaining an intermediate image by at least decoding an intermediate bitstream using at least a third decoder; and

wherein the up-sampling of the first panoramic image to the resolution of the view frame image further comprises combining the intermediate image with the first panoramic image.

7. A method for providing video content, the method comprising:

for a first viewport display, obtaining a first output image by combining a panoramic image and a first difference image for first coordinates of the first viewport display;

determining when a change occurs in the first viewport display;

responsive to the change, requesting a second difference image; and

obtaining a second output image for the first viewport display by combining the panoramic image and the second difference image.

8. The method of claim 7, further comprising scaling at least one of a resolution quality, a size, and/or a bitrate associated with the second difference image according to a Quality of Service (QoS) parameter.

9. A computerized apparatus for providing video content, the apparatus comprising:

an electronic storage apparatus configured to store a sequence of images;

a communications interface configured to enable communication of bitstreams to a client device; and

one or more processors configured to execute a plurality of computer readable instructions; and

a storage apparatus in data communication with the one or more processors and comprising at least one computer program, the at least one computer program comprising a plurality of instructions which are configured to, when executed by the one or more processors, cause the computerized apparatus to;

obtain a first panoramic image by a decode of a first bitstream using a first decoder;

obtain a view frame image by at least a decode of a second bitstream using a second decoder;

obtain a second panoramic image at least in part by up-sampling the first panoramic image to the resolution of the view frame image;

obtain an output view frame image at least in part by a combination of the view frame image and the second panoramic image; and

cause display of the obtained output view frame image via a rendering device.

10. The computerized apparatus of claim 9, wherein the first panoramic image comprises a low fidelity image characterized by at least a first resolution, and the second panoramic image comprises a high fidelity image characterized by at least a second resolution that is greater than the first resolution.

11. The computerized apparatus of claim 10, wherein a ratio of the first resolution to the second resolution is characterized by a resolution scalability parameter.

12. The computerized apparatus of claim 11, wherein:

the resolution scalability parameter is based at least on a display capability of a display device; and

the at least one computer program further comprises additional instructions which are configured to, when executed by the one or more processors, cause the computerized apparatus to;

display the output view frame image via the display device.

13. The computerized apparatus of claim 11, wherein the resolution scalability parameter is based at least on a processing or memory limitation of a display device.

14. The computerized apparatus of claim 9, wherein the at least one computer program further comprises additional instructions which are configured to, when executed by the one or more processors, cause the computerized apparatus to:

obtain an intermediate image by at least a decode of an intermediate bitstream using at least a third decoder; and

wherein the up-sampling of the first panoramic image to the resolution of the view frame image further comprises a combination of the intermediate image with the first panoramic image.

15. A non-transitory computer readable apparatus comprising a storage medium in data communication with one or more processors and comprising at least one computer program, the at least one computer program comprising a plurality of instructions which, when executed by the one or more processors, cause a computerized apparatus to:

obtain a first panoramic image by a decode of a first bitstream using a first decoder;

obtain a view frame image by at least a decode of a second bitstream using a second decoder;

obtain a second panoramic image at least in part by up-sampling the first panoramic image to the resolution of the view frame image;

obtain an output view frame image at least in part by a combination of the view frame image and the second panoramic image; and

cause display of the obtained output view frame image via a rendering device.

16. The non-transitory computer readable apparatus of claim 15, wherein the first panoramic image comprises a low fidelity image characterized by at least a first resolution, and the second panoramic image comprises a high fidelity image characterized by at least a second resolution that is greater than the first resolution.

17. The non-transitory computer readable apparatus of claim 16, wherein a ratio of the first resolution to the second resolution is characterized by a resolution scalability parameter.

18. The non-transitory computer readable apparatus of claim 17, wherein:

the resolution scalability parameter is based at least on a display capability of a display device; and

the at least one computer program further comprises additional instructions which are configured to, when executed by the one or more processors, cause the computerized apparatus to;

display the output view frame image via the display device.

19. The non-transitory computer readable apparatus of claim 17, wherein the resolution scalability parameter is based at least on a processing or memory limitation of a display device.

20. The non-transitory computer readable apparatus of claim 15, wherein the at least one computer program further comprises additional instructions which are configured to, when executed by the one or more processors, cause the computerized apparatus to:

obtain an intermediate image by at least a decode of an intermediate bitstream using at least a third decoder; and

wherein the up-sampling of the first panoramic image to the resolution of the view frame image further comprises a combination of the intermediate image with the first panoramic image.