Encoding and decoding selectively retrievable representations of video content

• US 9,961,355 B2
• Filed: 10/07/2016
• Issued: 05/01/2018
• Est. Priority Date: 04/25/2003
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Schunck, Brian G.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/288,836
• Publication Number:
  US 20170026658A1
• Time in Patent Office: 571 Days
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 2020/00072 the compressed signal including a video signal
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Claims

1. A method for processing compressed video data, the method comprising:

storing, in a storage structure, for each of a plurality of frames of video having a corresponding image at an original display resolution, a corresponding plurality of image components representative of the frame of video, the plurality of image components including a base image component associated with a given display resolution, and one or more additional image components associated with display resolutions greater than or equal to the given resolution and less than or equal to the original resolution, and the base image component comprising the corresponding image at the given display resolution;

selecting, based at least on a first processing load of a decoder, a first display resolution at which to display a first of the plurality of frames of video;

retrieving from the storage structure a first subset of the plurality of image components corresponding to the first frame of video, the first subset of image components selected based at least on the first display resolution and including the base image component corresponding to the first frame of video;

decoding the retrieved first subset of image components to generate a first modified frame of video at the first display resolution, the first modified frame of video comprising the image corresponding to the first frame of video;

based at least on a change in processing load of the decoder to a second processing load, selecting, based on the second processing load, a second display resolution at which to display a second frame of video;

retrieving from the storage structure a second subset of the plurality of image components corresponding to the second frame of video, the second subset selected based at least on the second display resolution and including the base image component corresponding to the second frame of video; and

decoding the retrieved second subset of image components to generate a second modified frame of video at the second display resolution, the second modified frame of video comprising the image corresponding to the second frame of video.

2. The method of claim 1, wherein:

the retrieving the first subset of the plurality of image components corresponding to the first frame of video comprises retrieving image components corresponding to the first frame of video which are associated with a resolution equal to or less than the first display resolution; and

the retrieving the second subset of the plurality of image components corresponding to the second frame of video comprises retrieving image components corresponding to the second frame of video which are associated with a resolution equal to or less than the second display resolution.

3. The method of claim 1, wherein the corresponding plurality of image components for each of the plurality of frames of video are stored in the storage structure consecutively and in ascending order of resolution, starting with the base image component.

4. The method of claim 3, wherein the retrieving the first subset of the plurality of image components corresponding to the first frame of video comprises:

identifying one of said plurality of image components corresponding to the first frame of video associated with first selected display resolution; and

retrieving from the storage structure in consecutive order, those of said plurality of image components corresponding to the first frame of video beginning with said base image component ending with the identified one of said plurality of image components.

5. The method of claim 1, wherein the storage structure additionally comprises an image component index storing at least the locations of the plurality of image components corresponding to the first frame of video in the storage structure, and wherein the retrieving of the first subset of the plurality of image components comprises:

identifying one or more of the plurality of image components corresponding to the first frame of video with associated resolutions equal to or less than the first display resolution;

identifying the one or more locations of the one or more identified plurality of image components in the storage structure using the using the image component index; and

retrieving the one or more identified plurality of image components from the one or more identified locations in the storage structure.

6. The method of claim 1, wherein the first display resolution comprises the original resolution and the retrieved first subset of the plurality of image components comprises all of the image components corresponding to the first frame of video.

7. The method of claim 6, wherein the second display resolution comprises a resolution less than the original resolution but greater than the given resolution, and the retrieved second subset of the plurality of image components comprises less than all of the image components corresponding to the second frame of video.

8. A system for processing compressed video data, the system comprising:

a memory configured to store, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including a base image component comprising the corresponding image at a base resolution and one or more additional image components associated with resolutions greater than or equal to the base resolution and less than or equal to the original resolution;

a decoder in data communication with at least the memory and configured to, for each of the plurality of frames of video, decode a retrieved subset of the plurality of image components corresponding to the frame of video to generate a corresponding modified frame of video, the corresponding modified frame of video comprising the image corresponding to the frame of video at a selected display resolution;

a resolution selection apparatus in data communication with at least the decoder and configured to;

select a first display resolution at which to display a first of said plurality of frames of video, the selection based at least on a first processing load of the decoder; and

in response to a change in processing load of the decoder to a second processing load, select a second display resolution, the second display resolution selected based at least on the second processing load; and

a data component apparatus in data communication with at least the memory and configured to;

retrieve a first subset of a plurality image components corresponding to the first frame of video from the storage structure, the first subset selected based at least on the first display resolution, the first subset including a base image component corresponding to the first frame of video; and

retrieve a second subset of a plurality of image components corresponding to a second of said plurality of frames of video from the storage structure, the second subset selected based at least on the second display resolution, the second subset including a base image component corresponding to the second frame of video.

9. The system of claim 8, wherein the retrieval of at least one of (i) the first subset of the plurality of image components associated with the first frame of video, and/or (ii) the second subset of the plurality of image components associated with the second frame of video, comprises retrieval of image components associated with that frame of video which are associated with a resolution equal to or less than the respective selected first or second display resolution.

10. The system of claim 8, wherein the system is further configured to store the plurality of image components corresponding to at least one of the first frame of video and second frame of video in the storage structure such that the stored plurality of image components are retrievable by the system in order of increasing display resolution.

11. The system of claim 10, wherein the system is further configured such that retrieval of at least one of the first subset of image components and the second subset of image components comprises:

identification of an image component associated with the respective first or second selected display resolution; and

consecutive retrieval from the storage structure of the plurality of image components corresponding to the first or second frame of video, respectively, the consecutive retrieval comprising (i) retrieval of the a base image corresponding to the first or second frame of video, respectively, and (ii) retrieval of all other of the plurality of image components up to and including the identified image component.

12. The system of claim 8, wherein the system is further configured to generate an image component index within the storage structure, the component index configured to store locations of the plurality of image components associated with the first and second frames of video in the storage structure, and wherein the retrieval of the first and second subsets of image components each comprises:

identification of one or more of the plurality of image components having associated resolutions equal to or less than the selected display resolution;

identification of one or more locations in the storage structure of the identified one or more image components, the identification of the one or more locations using at least the image component index; and

retrieval of the identified one or more image components from the identified one or more locations in the storage structure.

13. The system of claim 8, wherein the selected first display resolution comprises the original resolution, and the retrieved first subset of image components comprises all of the image components corresponding to the first frame of video.

14. The system of claim 13, wherein the selected second display resolution comprises the a resolution less than the original resolution, and the retrieved second subset of image components comprises less than all of the image components corresponding to the second frame of video.

15. A non-transitory computer-readable storage medium storing executable computer program instructions thereon, the program instructions configured for processing video data and that, when executed by a computerized device, cause the computerized device to perform operations comprising:

storage, in a storage structure, for a first of a plurality of frames of video comprising a corresponding image at an original resolution, a corresponding first plurality of image components representative of the first frame of video, the first plurality of image components including;

(i) a base image component associated with a base resolution, and (ii) one or more additional image components associated with resolutions greater than the base resolution and less than or equal to the original resolution;

storage, in the storage structure, for a second of the plurality of frames of video comprising a corresponding image at an original resolution, a corresponding second plurality of image components representative of the second frame of video, the second plurality of image components including;

(i) a base image component associated with a base resolution, and (ii) one or more additional image components associated with resolutions greater than the base resolution and less than or equal to the original resolution;

select a first display resolution at which to display the first frame of video, the selection based at least on a first processing load;

retrieve a subset of the first plurality of image components from the storage structure, the subset of the first plurality of image components being selected based at least on the first display resolution, the subset of the first plurality of image components including the base image component corresponding to the first frame of video;

decode the retrieved subset of the first plurality of image components to generate a first modified frame of video comprising the image corresponding to the first frame of video represented at the first display resolution;

based at least on a change in the processing load to a second processing load, select a second display resolution at which to display a second frame of video;

retrieve a subset of the second plurality of image components corresponding to the second frame of video from the storage structure, the subset of the second plurality of image components being selected based at least on the second display resolution, the subset of the second plurality of image components including the base image component corresponding to the second frame of video; and

decode the retrieved subset of the second plurality of image components to generate a second modified frame of video comprising the image corresponding to the second frame of video represented at the second display resolution.

16. The non-transitory computer-readable storage medium of claim 15, wherein the retrieval of the subset of the first plurality of image components comprises retrieval of image components from the storage structure which are associated with a resolution equal to or less than the selected first display resolution.

17. The non-transitory computer-readable storage medium of claim 15, wherein the first plurality of image components and the second plurality of image components are each stored in the storage structure consecutively from an image component corresponding to the base resolution to an image component corresponding to the original resolution.

18. The non-transitory computer-readable storage medium of claim 17, wherein the retrieval of the subsets of the first and second pluralities of image components comprises:

identification of an image component associated with the respective selected first or second display resolution; and

retrieval of at least a portion of the respective first or second plurality of image components from the storage structure, the retrieval of the at least portion comprising consecutive retrieval beginning with the respective base image component and ending with the identified image component.

19. The non-transitory computer-readable storage medium of claim 15, wherein the plurality of program instructions are further configured to, when executed;

create an image component index associated with the storage structure, the index configured to store respective locations of the first and second pluralities of image components within the storage structure; and

wherein said retrieval of the subsets of the first and second pluralities of the image components comprises;

identification of one or more image components within the first or second pluralities with associated resolutions equal to or less than the selected first or second display resolutions, respectively;

identification of locations of the one or more identified image components in the storage structure using at least the image component index; and

retrieval of the one or more identified image components from the identified locations in the storage structure.

20. A non-transitory computer-readable storage medium storing executable computer program instructions thereon, the program instructions configured for processing video data and that, when executed by a first computerized device, cause the first computerized device to perform operations on a plurality of frames of video data, the plurality of frames each comprising a corresponding original resolution, the operations comprising:

selection of a first display resolution at which to display a first of said plurality of frames of video data, the selection based at least on a first processing load of the first computerized device;

retrieval of a subset of a plurality of image components associated with a first frame of the video data from a storage apparatus in data communication with the first computerized device, the subset selected based at least on the first display resolution, the subset of the first plurality of image components including at least a base image component corresponding to the first frame;

decode of the retrieved subset to generate a first modified frame of video data comprising the image corresponding to the first frame, the image represented at the first display resolution;

detection of at least a change of processing load of the first computerized device from the first processing load to a second processing load;

selection of a second display resolution at which to display a second frame of video, the selection of the second frame of video based at least on the second processing load;

retrieval of a subset of a plurality of image components associated with the second frame of the video data from the storage apparatus, the subset of the plurality of image components associated with the second frame of the video data selected based at least on the second display resolution, the subset of the second plurality of image components including at least a base image component corresponding to the second frame; and

decode of the retrieved subset of the plurality of image components associated with the second frame of the video data to generate a second modified frame of video data comprising the image corresponding to the second frame, the image represented at the second display resolution;

wherein the pluralities of image components representative of the respective first and second frames of video data each include;

(i) a base image component associated with a prescribed resolution of the respective first or second frame, and (ii) one or more additional image components associated with resolutions greater than the prescribed resolution and less than or equal to the original resolution of the respective first or second frame.

21. The non-transitory computer-readable storage medium of claim 20, wherein the retrieval of the subset of the first plurality of image components comprises retrieval of image components from the storage structure which are associated with a resolution equal to or less than the selected first display resolution.

22. The non-transitory computer-readable storage medium of claim 20, wherein the first plurality of image components and the second plurality of image components are each stored in the storage structure consecutively from an image component corresponding to the prescribed resolution to an image component corresponding to the original resolution.

23. The non-transitory computer-readable storage medium of claim 22, wherein the retrieval of the subsets of the first and second pluralities of image components comprises:

identification of an image component associated with the respective selected first or second display resolution; and

retrieval of at least a portion of the respective first or second plurality of image components from the storage structure, the retrieval of the at least portion comprising consecutive retrieval beginning with the respective base image component and ending with the identified image component.

24. The non-transitory computer-readable storage medium of claim 20, wherein the prescribed resolution comprises a lowest resolution.

25. The non-transitory computer-readable storage medium of claim 20, wherein the plurality of program instructions are further configured to, when executed;

create an image component index associated with the storage structure, the index configured to store respective locations of the first and second pluralities of image components within the storage structure; and

wherein said retrieval of the subsets of the first and second pluralities of the image components comprises;

identification of one or more image components within the first or second pluralities with associated resolutions equal to or less than the selected first or second display resolutions, respectively;

identification of locations of the one or more identified image components in the storage structure using at least the image component index; and

retrieval of the one or more identified image components from the identified locations in the storage structure.

26. The non-transitory computer-readable storage medium of claim 20, wherein the (i) base image component, and the (ii) one or more additional image components associated with resolutions greater than the prescribed resolution and less than or equal to the original resolution of the respective first or second frame, are each generated by a second computerized device.

27. The non-transitory computer-readable storage medium of claim 26, wherein the first computerized device comprises an encoder apparatus, and the second computerized device comprises a decoder apparatus.

28. The non-transitory computer-readable storage medium of claim 27, wherein the encoder apparatus is at least part of a video camera apparatus, and the decoder apparatus comprises a commoditized personal computer.

29. The non-transitory computer-readable storage medium of claim 20, wherein the (i) base image component, and the (ii) one or more additional image components associated with resolutions greater than the prescribed resolution and less than or equal to the original resolution of the respective first or second frame, are each generated by the first computerized device as part of an encoding process which precedes at least said retrieval of a subset of a plurality of image components associated with a first frame of the video data.