Apparatus and methods for video compression using multi-resolution scalable coding

• US 10,652,558 B2
• Filed: 12/27/2018
• Issued: 05/12/2020
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Abbas, Adeel, Adsumilli, Balineedu Chowdary, Newman, David
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US16/234,214
• Publication Number:
  US 20190246126A1
• Time in Patent Office: 502 Days
• Field of Search: 37524002
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/29 involving scalability at the object level, e.g. video object layer [VOL]
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution

Abstract

Apparatus and methods for digital video data compression via a scalable, multi-resolution approach. In one embodiment, the video content may be encoded using a multi-resolution and/or multi-quality scalable coding approach that reduces computational and/or energy load on a client device. In one implementation, a low fidelity image is obtained based on a first full resolution image. The low fidelity image may be encoded to obtain a low fidelity bitstream. A second full resolution image may be obtained based on the low fidelity bitstream. A portion of a difference image obtained based on the second full resolution image and the first full resolution may be encoded to obtain a high fidelity bitstream. The low fidelity bitstream and the high fidelity bitstream may be provided to e.g., a receiving device.

Claims

1. A method of obtaining an image responsive to a change of a viewport, the method comprising:

obtaining a difference image based at least on a first full-resolution image and a second full-resolution image;

encoding a portion of the difference image to produce an encoded portion, the encoded portion corresponding to a position of the viewport; and

providing, via a communications interface to a client device, at least the encoded portion corresponding to the position of the viewport.

2. The method of claim 1, further comprising:

obtaining the first full-resolution image from a sequence of images, the first full-resolution having a first frame resolution;

downsampling the first full-resolution image to obtain a downsampled image having a second frame resolution, the second frame resolution being lower than the first frame resolution; and

encoding the downsampled image into a reduced-fidelity bitstream.

3. The method of claim 2, further comprising selecting an image resolution of the second frame resolution of the downsampled image based at least on a scaling factor ranging from two (2) to sixteen (16), inclusive.

4. The method of claim 2, further comprising:

decoding the reduced-fidelity bitstream into a reduced-fidelity image; and

upsampling the reduced-fidelity image to obtain the second full-resolution image;

wherein the encoding of the portion of the difference image comprises generating an increased-fidelity bitstream; and

wherein the providing the at least encoded portion of the difference image comprises providing at least the increased-fidelity bitstream to the client device.

5. The method of claim 3, wherein the providing the at least encoded portion of the difference image comprises providing the reduced-fidelity bitstream to the client device along with the increased-fidelity bitstream.

6. The method of claim 1, wherein the obtaining of the difference image comprises subtracting values of pixels of the second full-resolution image from pixels of the first full-resolution image for each of one or more image components.

7. The method of claim 1, further comprising modifying a bitrate for the at least encoded portion of the difference image based on a scalability parameter.

8. A non-transitory computer-readable apparatus comprising a storage medium, the storage medium including a computer program comprising a plurality of instructions which are configured to, when executed by one or more processor apparatus, cause a computerized apparatus to:

obtain a residual image based on a first full-resolution image and a second full-resolution image;

detect a portion of the obtained residual image, the portion being viewed by a user;

encode the portion of the obtained residual image;

produce a high-fidelity bitstream based at least on the encode of the portion of the obtained residual image; and

provide at least the produced high-fidelity bitstream to a client device of the user.

9. The non-transitory computer-readable apparatus of claim 8, wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

downsample the first full-resolution image to obtain a downsampled image having a resolution lower than that of the first full-resolution image; and

encode the downsampled image into a low-fidelity bitstream;

wherein a number of bits per pixel of the high-fidelity bitstream is greater than that of the low-fidelity bitstream.

10. The non-transitory computer-readable apparatus of claim 9, wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

determine the frame resolution of the downsampled image based at least on a downsampling factor selected from a range of two (2) and sixteen (16), inclusive.

11. The non-transitory computer-readable apparatus of claim 8, wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

decode the low-fidelity bitstream into a low-fidelity image; and

upsample the low-fidelity image to obtain the second full-resolution image.

12. The non-transitory computer-readable apparatus of claim 11, wherein the upsample of the low-fidelity image comprises a reciprocation of a downsampling factor, the downsampling factor being selected from a range of two (2) and sixteen (16), inclusive.

13. The non-transitory computer-readable apparatus of claim 8, wherein the residual image is obtained based on a subtraction of values of pixels of the second full-resolution image from pixels of the first full-resolution image for each of one or more image components.

14. The non-transitory computer-readable apparatus of claim 8, wherein the plurality of instructions are further configured to, when executed by the one or more processor apparatus, cause the computerized apparatus to:

obtain a plurality of source images from one or more capture devices; and

combine the plurality of source images to obtain the first full-resolution image.

15. A computerized apparatus for providing video content, the computerized apparatus comprising:

a processing apparatus;

a network interface in data communication with the processing apparatus;

a non-transitory computer-readable apparatus comprising a storage medium, the storage medium including a computer program comprising a plurality of instructions which are configured to, when executed by the processing apparatus, cause a computerized apparatus to;

derive a difference image based at least on a first image and a second image;

encode a portion of the derived difference image to generate an encoded portion, the encoded portion corresponding to a position of a viewport as reflected in data received from at least one client device via the network interface; and

provide, to the network interface for distribution to the at least one client device, at least the encoded portion of the difference image corresponding to the position of the viewport.

16. The computerized apparatus of claim 15, wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

downsample the first image to obtain a downsampled image having a second frame resolution, the second frame resolution being lower than a first frame resolution of the first image; and

encode the downsampled image into a first bitstream.

17. The computerized apparatus of claim 16, wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

decode the first bitstream into a reduced-fidelity image; and

upsample the reduced-fidelity image to obtain the second image;

wherein the encode of the portion of the difference image comprises generation of a second bitstream; and

wherein the provision the at least encoded portion of the difference image comprises provision of at least the high-fidelity bitstream to the at least one client device.

18. The computerized apparatus of claim 17, wherein the provision the at least encoded portion of the difference image comprises provision of the first bitstream along with the second bitstream to the at least one client device.

19. The computerized apparatus of claim 15, further comprising a storage device;

wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to;

cause storage of the at least encoded portion of the difference image on at least one of (i) the storage device of the computerized apparatus or (ii) a networked remote storage device.

20. The computerized apparatus of claim 15, wherein the plurality of instructions are further configured to, when executed by the processing apparatus, cause the computerized apparatus to:

obtain the first image based on one or more source images acquired from one or more capture devices.