Encoding and decoding selectively retrievable representations of video content

• US 9,171,577 B1
• Filed: 05/23/2011
• Issued: 10/27/2015
• Est. Priority Date: 04/25/2003
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Schunck, Brian G.
• Primary Examiner(s): HOLDER, ANNER N
• Application Number:
  US13/113,950
• Time in Patent Office: 1,618 Days
• Field of Search: 375/240.25
• US Class Current: 1/1
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 2020/00072 the compressed signal including a video signal
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Claims

1. A method for processing compressed video data, the method comprising:

storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution;

selecting a first display resolution at which to display a first frame of video based on a first processing load of a decoder;

retrieving a first subset of the image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video;

decoding the retrieved first subset of image components to generate a first modified frame of video comprising the image corresponding to the first frame of video at the first display resolution;

displaying the first modified frame of video;

in response to a change in processing load of the decoder to a second processing load, selecting a second display resolution at which to display a second frame of video based on the second processing load;

retrieving a second subset of the image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video;

decoding the retrieved second subset of image components to generate a second modified frame of video comprising the image corresponding to the second frame of video at the second display resolution; and

displaying the second modified frame of video.

2. The method of claim 1, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution.

3. The method of claim 1, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution.

4. The method of claim 3, wherein retrieving a subset of image components comprises:

identifying an image component associated with a selected display resolution; and

retrieving image components from the storage structure consecutively from a base image component to the identified image component.

5. The method of claim 1, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises:

identifying one or more image components with associated resolutions equal to or less than a selected display resolution;

identifying the one or more locations of the one or more identified image components in the storage structure using the using the image component index; and

retrieving the one or more identified image components from the one or more identified locations in the storage structure.

6. The method of claim 1, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame.

7. The method of claim 1, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame.

8. A system for processing compressed video data, the system comprising:

a memory configured to store, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution;

a decoder configured to, for each frame of video, decode a retrieved subset of image components corresponding to the frame of video to generate a modified frame of video comprising the image corresponding to the frame of video at a display resolution;

a resolution module configured to;

select a display resolution at which to display a first frame of video based on a processing load of the decoder; and

in response to a change in processing load of the decoder to a second processing load, select a second display resolution based on the second processing load;

a data component module configured to;

retrieve a first subset of image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video; and

retrieve a second subset of image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video; and

a display configured to display modified frames of video.

9. The system of claim 8, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution.

10. The system of claim 8, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution.

11. The system of claim 10, wherein retrieving a subset of image components comprises:

identifying an image component associated with a selected display resolution; and

retrieving image components from the storage structure consecutively from a base image component to the identified image component.

12. The system of claim 8, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises:

identifying one or more image components with associated resolutions equal to or less than a selected display resolution;

identifying the one or more locations of the one or more identified image components in the storage structure using the using the image component index; and

retrieving the one or more identified image components from the one or more identified locations in the storage structure.

13. The system of claim 8, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame.

14. The system of claim 8, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame.

15. A non-transitory computer-readable storage medium storing executable computer program instructions thereon for processing compressed video data that, in response to execution by a computing device, cause the computing device to perform operations comprising:

storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component associated with a lowest resolution and one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the lowest resolution;

selecting a first display resolution at which to display a first frame of video based on a first processing load of a decoder;

retrieving a first subset of the image components corresponding to the first frame of video from the storage structure selected based on the first display resolution, the first subset of image components including the base image component corresponding to the first frame of video;

decoding the retrieved first subset of image components to generate a first modified frame of video comprising the image corresponding to the first frame of video at the first display resolution;

displaying the first modified frame of video;

in response to a change in processing load of the decoder to a second processing load, selecting a second display resolution at which to display a second frame of video based on the second processing load;

retrieving a second subset of the image components corresponding to the second frame of video from the storage structure selected based on the second display resolution, the second subset of image components including the base image component corresponding to the second frame of video;

decoding the retrieved second subset of image components to generate a second modified frame of video comprising the image corresponding to the second frame of video at the second display resolution; and

displaying the second modified frame of video.

16. The non-transitory computer-readable storage medium of claim 15, wherein retrieving a subset of image components comprises retrieving all image components from the storage structure which are associated with a resolution equal to or less than a selected display resolution.

17. The non-transitory computer-readable storage medium of claim 15, wherein image components corresponding to a frame of video are stored in the storage structure consecutively based on an increase in the resolutions associated with the image components from the lowest resolution to the original resolution.

18. The non-transitory computer-readable storage medium of claim 17, wherein retrieving a subset of image components comprises:

identifying an image component associated with a selected display resolution; and

retrieving image components from the storage structure consecutively from a base image component to the identified image component.

19. The non-transitory computer-readable storage medium of claim 15, wherein the storage structure additionally comprises an image component index storing the locations of the image components in the storage structure, and wherein retrieving a subset of image components comprises:

identifying one or more image components with associated resolutions equal to or less than a selected display resolution;

identifying the one or more locations of the one or more identified data components in the storage structure using the using the image component index; and

retrieving the one or more identified image components from the one or more identified locations in the storage structure.

20. The non-transitory computer-readable storage medium of claim 15, wherein a retrieved subset of image components comprises all of the image components corresponding to a frame when a selected resolution comprises the original resolution of the image corresponding to the frame.

21. The non-transitory computer-readable storage medium of claim 15, wherein a retrieved subset of image components comprises less than all of the image components corresponding to a frame when a selected resolution comprises a resolution less than the original resolution of the image corresponding to the frame.