Encoding and decoding selectively retrievable representations of video content

• US 9,800,885 B2
• Filed: 07/25/2014
• Issued: 10/24/2017
• Est. Priority Date: 04/25/2003
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Schunck, Brian G.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US14/340,873
• Publication Number:
  US 20150010286A1
• Time in Patent Office: 1,187 Days
• Field of Search: 37524025
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 2020/00072 the compressed signal including a video signal
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise be required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Claims

1. A method for previewing video edits comprising:

storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including (i) a base image component associated with a lowest resolution of a plurality of available resolutions and (ii) one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, the base image component comprising the corresponding image at the lowest resolution;

selecting a preview resolution at which to display the frame of video, the selected preview resolution being automatically determined based on one or more runtime factors;

selectively decoding at least a portion of the plurality of image components based on the determined preview resolution, the selective decoding comprising;

when the determined preview resolution matches the lowest resolution, accessing a first subset of stored image components corresponding to the frame of video, the first subset of stored image components including the base image component corresponding to the frame of video at the lowest resolution;

when the determined preview resolution does not match the lowest resolution, accessing a second subset of stored image components, the second subset of stored image components including (i) the base image component associated with the lowest resolution and (ii) at least one of the one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution; and

decoding the accessed subset of stored image components to generate a decoded frame of video comprising an image corresponding to the frame of video at the preview resolution;

receiving, from a user, an edit to the frame of video at the determined preview resolution; and

applying the received edit to the decoded frame of video to produce an edited frame of video at the preview resolution; and

displaying the edited frame of video to the user.

2. The method of claim 1, wherein the preview resolution is selected based on the received edit to the frame of video.

3. The method of claim 1, wherein the preview resolution comprises the original resolution.

4. The method of claim 1, wherein the received edit comprises one or more of:

scene selection, color correction, timeline playback, transition timing, and clip length trimming.

5. The method of claim 1, wherein the original resolution comprises 4k resolution.

6. The method of claim 1, wherein the frame of video is a frame within a video clip, and wherein the received edit comprises an edit to the video clip.

7. The method of claim 6, wherein displaying the edited frame of video comprises displaying the edited frame of video within a preview of the video clip.

8. The method of claim 7, wherein the video clip comprises an original frame rate, and

wherein the preview of the video clip comprises a preview frame rate lower than the original frame rate.

9. A system for previewing video edits comprising:

a non-transitory computer-readable storage medium configured to store, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including (i) a base image component associated with a lowest resolution of a plurality of resolutions and (ii) one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, the base image component comprising the corresponding image at the lowest resolution;

an input configured to receive, from a user, an edit to a frame of video;

a video editor configured to;

select a preview resolution at which to display the frame of video;

selectively decode at least a portion of the plurality of image components based on the selected preview resolution, the selective decoding being configured to;

access a first subset of stored image components corresponding to the frame of video when the preview resolution matches the lowest resolution, the first subset of stored image components including the base image component associated with the lowest resolution;

access a second subset of stored image components when the preview resolution does not match the lowest resolution, the second subset of stored image components including (i) the base image component associated with the lowest resolution and (ii) at least one of the one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution; and

decode the accessed subset of stored image components to generate a decoded frame of video comprising an image corresponding to the frame of video at the preview resolution; and

apply the received edit to the decoded frame of video to produce an edited frame of video at the preview resolution; and

an output configured to output a preview video comprising the edited frame of video at the preview resolution to the user, the preview video comprising a sequence of edited frames at a temporal resolution different than that of a video comprising said plurality of non-edited frames of video.

10. The system of claim 9, wherein the preview resolution is selected based on the received edit to the frame of video.

11. The system of claim 9, wherein the preview resolution is selected by the user.

12. The system of claim 11, wherein the preview resolution comprises the original resolution.

13. The system of claim 9, wherein the received edit comprises one or more of:

scene selection, color correction, timeline playback, transition timing, and clip length trimming.

14. The system of claim 9, wherein the original resolution comprises 4k resolution.

15. The system of claim 9, wherein the frame of video is a frame within a video clip comprising the preview video, and wherein the received edit comprises an edit to the video clip.

16. The system of claim 15, wherein outputting the edited frame of video comprises displaying the edited frame of video within the preview video of the video clip.

17. The system of claim 16, wherein the video clip comprises an original frame rate, and

wherein the preview of the video clip comprises a preview frame rate lower than the original frame rate.

18. An apparatus configured to preview an edit to a frame of video, the apparatus comprising:

means for storing, in a storage structure, for each of a plurality of frames of video, a corresponding plurality of image components representative of the frame of video, the frame of video comprising a corresponding image at an original resolution, the plurality of image components including (i) a base image component associated with a lowest resolution of a plurality of available resolutions and (ii) one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution, the base image component comprising the corresponding image at the lowest resolution;

means for determining a preview resolution at which to display the frame of video;

means for selectively decoding at least a portion of the plurality of image components based on the determined preview resolution, the means for selectively decoding comprising;

means for accessing a first subset of stored image components corresponding to the frame of video when the determined preview resolution matches the lowest resolution, the first subset of stored image components including the base image component corresponding to the frame of video at the lowest resolution;

means for accessing a second subset of stored image components when the determined preview resolution does not match the lowest resolution, the second subset of stored image components including (i) the base image component associated with the lowest resolution and (ii) at least one of the one or more additional image components associated with resolutions greater than or equal to the lowest resolution and less than or equal to the original resolution; and

means for decoding the accessed subset of stored image components to generate a decoded frame of video comprising an image corresponding to the frame of video at the preview resolution;

means for receiving, from a user, an edit to the frame of video at the determined preview resolution;

means for applying the received edit to the decoded frame of video to produce an edited frame of video at the preview resolution; and

means for displaying the edited frame of video to the user.

19. The apparatus of claim 18, wherein the means for determining the preview resolution at which to display the frame of video comprises an automatic selection based on one or more computational factors.

20. The apparatus of claim 18, wherein the preview resolution excludes the original resolution.