Encoding and decoding selectively retrievable representations of video content

• US 9,854,263 B2
• Filed: 09/14/2015
• Issued: 12/26/2017
• Est. Priority Date: 04/25/2003
• Status: Expired due to Term

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Schunck, Brian G.
• Primary Examiner(s): Holder, Anner N
• Application Number:
  US14/852,868
• Publication Number:
  US 20150382001A1
• Time in Patent Office: 834 Days
• Field of Search: 37524025
• CPC Class Codes: G11B 20/00007 Time or data compression or expansion audio compression based on psychoacoustics G10L19/00; data processing for reproducing audio data at different playback speeds G10L21/04; video compression H04N19/00; data compression per se H03M7/30
• CPC Class Codes: G11B 2020/00072 the compressed signal including a video signal
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise be required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Claims

1. A method for processing compressed video data, the method comprising:

accessing, by a computer system, a plurality of encoded video frames, each video frame comprising a plurality of data structure components encoded from said each video frame, each of the plurality of data structure components being associated with a different one of a plurality of image resolutions such that the video frame can be displayed at one image resolution of the plurality of image resolutions in part by combining all of a subset of data structure components associated with an image resolution that is equal to or less than the one image resolution, each of the subset of data structure components comprising a data size that is equal to or less than that associated with the one image resolution, such that the subset of data structure components comprises data sufficient to reconstruct the video frame at a corresponding one of the plurality of image resolutions;

identifying a hardware display communicatively coupled to the computer system;

determining an available bandwidth between the computer system and the hardware display;

selecting the one image resolution from the plurality of image resolutions based on the determined available bandwidth;

selecting, for each video frame of the plurality of video frames, a subset of data structure components of the video frame, the selected subset of data structure components comprising all of the data structure components associated with the image resolution that is equal to or less than the one image resolution selected from the plurality of available image resolutions; and

transmitting, for each video frame, the selected subset of data structure components to the hardware display.

2. The method of claim 1, wherein the selected one image resolution is based on a selection input received from a user.

3. The method of claim 1, wherein the selected one image resolution is selected by the computer system automatically in response to the determination of the available bandwidth.

4. The method of claim 1, wherein the selected one image resolution is selected in response to a determination that the determined available bandwidth is lower than a threshold bandwidth required to support a current image resolution, the selected one image resolution being less than the current image resolution.

5. A system for processing compressed video data, the system comprising:

a non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a processor, cause the system to;

access a plurality of encoded video frames, each video frame comprising a plurality of data components each associated with a different one of a plurality of available image resolutions such that the video frame can be displayed at one image resolution of the plurality of available image resolutions in part by combining all of a subset of data components associated with an image resolution that is equal to or less than the one image resolution, each of the subset of data components comprising a data size that is equal to or less than that associated with the one image resolution, such that the subset of data components comprises data sufficient to reconstruct the video frame at a corresponding one of the plurality of image resolutions;

identify a hardware display communicatively coupled to the system;

determine an available bandwidth between the system and the hardware display;

select one image resolution from the plurality of available image resolutions based on the determined available bandwidth;

for each video frame of the plurality of video frames;

select a subset of data components of the video frame, the selected subset of data components comprising all of the data components associated with the image resolution selected from the plurality of available image resolutions, said all of the data components associated with the image resolution being equal to or less than the selected one image resolution; and

transmit the selected subset of data components to the hardware display.

6. The system of claim 5, wherein the selected one image resolution is selected by a user.

7. The system of claim 5, wherein the selected one image resolution is selected by the system automatically in response to the determination of the available bandwidth.

8. The system of claim 5, wherein the selected one image resolution is selected in response to a determination that the determined available bandwidth is lower than a threshold bandwidth required to support a current image resolution, the selected image resolution being less than the current image resolution.

9. The system of claim 5, further comprising a network interface configured to communicate with the hardware display that is located remotely from the system.

10. The system of claim 5, further comprising a network interface configured to communicate with the hardware display via the internet.

11. The system of claim 5, further comprising a network interface configured to communicate with the hardware display via a wireless network.

12. The system of claim 5, wherein the plurality of image resolutions comprises a 4k resolution, and wherein the selected one image resolution is less than the 4k resolution.

13. The system of claim 5, further comprising a hardware display that is configured to, for each video frame:

decode the transmitted one or more image components corresponding to the video frame to produce decoded image components corresponding to the video frame;

combine the decoded image components corresponding to the video frame to produce a result video frame at the selected one image resolution; and

display the result video frame at the selected one image resolution.

14. A non-transitory computer-readable storage medium storing executable computer instructions for processing compressed video data that when executed by a processor of a computer system, causes the processor to:

access a plurality of encoded video frames, each video frame comprising a plurality of data components each associated with a different one of a plurality of available image resolutions such that the video frame can be displayed at a selected one image resolution of the plurality of available image resolutions in part by combining all of a subset of data components associated with an image resolution that is equal to or less than the one image resolution, each of the subset of data components comprising a data size that is equal to or less than that associated with the selected one image resolution, such that the subset of data components comprises data sufficient to reconstruct the video frame at a corresponding one of the plurality of available image resolutions;

identify a hardware display communicatively coupled to the computer system;

determine an available bandwidth between the computer system and the hardware display;

select one image resolution from the plurality of available image resolutions based on the determined available bandwidth;

for each video frame of the plurality of video frames;

select one or more data components of the video frame, the selected one or more data components comprising all of the data components of the video frame associated with the image resolution that is equal to or less than the one image resolution selected from the plurality of available image resolutions; and

transmit the selected one or more data components to the hardware display.

15. The computer-readable storage medium of claim 14, further comprising executable computer instructions which when executed by the processor cause the hardware display to:

for each video frame;

decode the transmitted one or more data components corresponding to the video frame to produce decoded data components corresponding to the video frame;

combine the decoded data components corresponding to the video frame to produce a result video frame at the selected one image resolution; and

display the video frame at the selected one image resolution.

16. A method for processing compressed video data, the method comprising:

receiving, by a computer system, a plurality of data components, each of the data components associated with a different one of a plurality of accessible image resolutions of a video frame;

decoding the received plurality of data components corresponding to the video frame to produce a plurality of decoded data components corresponding to the video frame;

combining all of a subset of decoded data components that are associated with an image resolution that is equal to or less than a display resolution of the video frame such that the subset of decoded data components comprises data sufficient to reconstruct the video frame at the display resolution, the display resolution being one of a plurality of accessible image resolutions; and

displaying the video frame at the display resolution.

17. The method of claim 16, further comprising automatically selecting the display resolution based on one or more runtime factors.

18. The method of claim 17, wherein the one or more runtime factors comprises a currently available bandwidth.

19. The method of claim 16, further comprising enabling a manual selection of the display resolution.

20. The method of claim 19, further comprising receiving the manual selection from a user interface.

21. A method for providing compressed video data, the method comprising:

accessing, by a computer system, a plurality of video frames;

identifying a hardware display communicatively coupled to the computer system;

determining an available bandwidth between the computer system and the hardware display;

selecting a desired image resolution out of one or more usable image resolutions based on the determined available bandwidth;

for each video frame of the plurality of video frames;

iteratively encoding the video frame into a plurality of data components associated with a plurality of respective image resolutions;

wherein each iteration of the iterative encoding process comprises separating a component of the plurality of data components into (i) one or more low pass components associated with a lower image resolution, and (ii) one or more high pass components associated with a higher image resolution; and

transmitting, to the hardware display, all of the data components of the plurality of data components that have an image resolution that is equal to or lower than the one selected image resolution.

22. The method of claim 21, wherein the transmitting to the hardware display comprises transmitting via a wireless transmission.

23. The method of claim 21, wherein the transmitting to the hardware display comprises transmitting via an internet.

24. The method of claim 21, wherein the separating the component into one or more low pass components comprises down-sampling the component.

25. The method of claim 21, wherein the separating the component into one or more high pass components comprises applying a horizontal or vertical edge filter.