Encoding and decoding selectively retrievable representations of video content

• US 9,967,580 B2
• Filed: 11/17/2016
• Issued: 05/08/2018
• Est. Priority Date: 04/25/2003
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Schunck, Brian G.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US15/354,933
• Publication Number:
  US 20170142435A1
• Time in Patent Office: 537 Days
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 2020/00072 the compressed signal including a video signal
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Claims

1. A computer-readable apparatus comprising at least one computer program having a plurality of instructions, the plurality of instructions configured to, when executed on a processing apparatus:

store video data so as to enable selection of one of a plurality of different rendering resolutions at decode by at least;

production of a plurality of data structures from the video data, each of the plurality of data structures associated with a respective portion of the video data;

obtainment of a plurality of components for each of the plurality of data structures, each of the plurality of components corresponding to a respective different rendering resolution; and

storage of data relating to the plurality of components for each data structure.

2. The computer-readable apparatus of claim 1, wherein the storage of data relating to the plurality of components for each data structure comprises storage of data relating to the plurality of components for each data structure in a prescribed order so as to enable access by a decoder of the plurality of components in ascending resolution order, the access based at least on said selection.

3. The computer-readable apparatus of claim 1, wherein the plurality of instructions are further configured to, when executed:

associate at least first and second ones of the plurality of components of each of the portions with its respective portion, the first and second components for each portion corresponding to respective first and second rendering resolutions of the video data of that portion; and

wherein the storage of data relating to the plurality of components for each data structure comprises storage of the associated at least first and second components of each portion individually so as to enable individual access thereof.

4. The computer-readable apparatus of claim 3, wherein said production, association, obtainment and storage of data collectively enable simultaneous decode of corresponding ones of said first and second components associated with different ones of said portions without decode of any other components thereof.

5. The computer-readable apparatus of claim 4, wherein said simultaneous decode of corresponding ones of said first and second components associated with different ones of said portions comprises simultaneous decode using at least one processor apparatus.

6. The computer-readable apparatus of claim 1, wherein the production of a plurality of data structures comprises processing the video data into a plurality of individual files.

7. The computer-readable apparatus of claim 1, wherein the obtainment of a plurality of components for each of the plurality of data structures, each of the plurality of components corresponding to a respective different rendering resolution, comprises performance of at least a respective spatial compression on video data of each said data structures.

8. The computer-readable apparatus of claim 7, wherein the performance of at least a respective spatial compression on video data of each said data structures comprises performance of at least one wavelet-based filter operation in at least one spatial dimension of the video data of each said data structures.

9. The computer-readable apparatus of claim 8, wherein the performance of at least one wavelet-based filter operation comprises performance of a series of successive wavelet-based filter operations until a desired rendering resolution is achieved.

10. The computer-readable apparatus of claim 8, wherein the performance of at least one wavelet-based filter operation comprises performance of at least one wavelet-based filter operation that halves an original resolution associated with the video data in at least one of a vertical and/or horizontal dimension.

11. The computer-readable apparatus of claim 1, wherein said video data comprises a plurality of frames, and the plurality of instructions are further configured to, when executed, temporally compress a plurality of frames of said video data into a lesser number of frames prior to said processing the video data into a plurality of data structures.

12. The computer-readable apparatus of claim 1, wherein:

each of the components for each of the plurality of data structures is non-redundant to others of the plurality of components within at least a same one of the plurality of data structures; and

the storage of data relating to the plurality of components for each data structure comprises storage of the data relating to the plurality of components for each of the data structures such that, upon a subsequent decoding of the stored video data, two or more of the plurality of components within each of the plurality of data structures can be combined to obtain video data at a desired resolution, the desired resolution being determinable at time of said subsequent decoding.

13. The computer-readable apparatus of claim 1, wherein:

said storage of data relating to the plurality of components for each of the data structures comprises storage of;

(i) a first component having a lowest resolution associated therewith, and (ii) at least one second component; and

wherein a combination of said first component and said at least one second component produces a rendering having resolution higher than said lowest resolution, yet below a full resolution of said video data before said encoding.

14. The computer-readable apparatus of claim 1, wherein:

said storage of data relating to the plurality of components for each of the data structures comprises storage of;

(i) a first component having a lowest resolution associated therewith, and (ii) at least one second component enabling enhancement of said first component to a higher resolution than said lowest resolution.

15. The computer-readable apparatus of claim 14, wherein said first component comprises a thumbnail image of video data within one of said data structures.

16. A computerized electronic device, comprising:

video data acquisition apparatus;

data processing apparatus in data communication with the video data acquisition apparatus; and

storage apparatus in data communication with the data processing apparatus, the storage apparatus comprising at least one computer program having a plurality of instructions, the plurality of instructions configured to, when executed on the data processing apparatus;

store video data so as to enable selection of one of a plurality of different rendering resolutions at decode by at least;

processing the video data into a plurality of data structures, each of the plurality of data structures associated with a respective portion of the video data;

obtaining a plurality of components for each of the plurality of data structures, each of the plurality of components corresponding to a respective different rendering resolution; and

storing data relating to the plurality of components for each data structure.

17. The computerized electronic device of claim 16, wherein:

the computerized electronic device comprises a consumer electronics device configured for video data capture and processing, and the video data acquisition apparatus comprises one or more digital video cameras each capable of generating at least a portion of the video data; and

the generated at least a portion of the video data comprises uncompressed high-definition (HD) video data comprising at least one of;

(i) 1080i resolution and/or (ii) 720p resolution.

18. The computerized electronic device of claim 17, wherein the at least one computer program comprises application software configured to generate a user interface on a display device in data communication with the data processing apparatus, the user interface configured to enable a user of the computerized electronic device to perform one or more video data editing operations using at least a portion of the stored data relating to the plurality of components for one or more of the data structures.

19. A method of parallel encoding of video data, the method comprising:

obtaining a plurality of frames of said video data;

simultaneously processing at least first and second ones of said frames using at least a processing apparatus so as to provide at least one of spatial and/or temporal compression of the plurality of frames, said simultaneous processing further comprising outputting a plurality of data components relating to the spatially and/or temporally compressed plurality of frames; and

storing the plurality of data components in independently accessible storage locations within a storage apparatus;

wherein said plurality of data components are configured such that each can be independently accessed and combined with one or more others of the plurality of data components to provide renderable compressed images corresponding to the plurality of frames.

20. The method of claim 19, wherein the simultaneously processing at least first and second ones of said plurality of frames using at least a processing apparatus so as to provide at least one of spatial and/or temporal compression of the plurality of frames comprises simultaneously processing:

(i) said at least first and second ones of said plurality of frames; and

(ii) at least third and fourth ones of said plurality of frames, so as to provide at least said at least one of spatial and/or temporal compression.