HIERARCHICAL VIDEO COMPRESSION SUPPORTING SELECTIVE DELIVERY OF TWO-DIMENSIONAL AND THREE-DIMENSIONAL VIDEO CONTENT

• US 20110157309A1
• Filed: 12/30/2010
• Published: 06/30/2011
• Est. Priority Date: 12/31/2009
• Status: Abandoned Application

• Current Assignee: Avago Technologies General IP PTE Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Bennett, James D., Karaoguz, Jeyhan
• Application Number:
  US12/982,053
• Publication Number:
  US 20110157309A1
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G02B 6/00 Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
• CPC Class Codes: G03B 35/24 using apertured or refractive resolving means on screens or between screen and eye
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 2300/023 Display panel composed of stacked panels
• CPC Class Codes: G09G 2320/028 by changing the viewing angle properties, e.g. widening the viewing angle, adapting the viewing angle to the view direction
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/20 for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/312 the parallax barriers being placed behind the display panel, e.g. between backlight and spatial light modulator [SLM]
• CPC Class Codes: H04N 13/315 the parallax barriers being time-variant
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/351 : for displaying simultaneously
• CPC Class Codes: H04N 13/359 : Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/361 : Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/403 : the images being monoscopic
• CPC Class Codes: H04N 2013/405 : the images being stereoscopic or three dimensional
• CPC Class Codes: H04N 21/235 : Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/4122 : additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/435 : Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04S 7/303 : Tracking of listener position or orientation

Abstract

Systems, methods and apparatuses are described herein for encoding a plurality of video frame sequences, wherein each video frame sequence corresponds to a different perspective view of the same subject matter. In accordance with various embodiments, the encoding is performed in a hierarchical manner that leverages referencing between frames of different ones of the video frame sequences (so-called “external referencing”), but that also allows for encoded representations of only a subset of the video frame sequences to be provided when less than all sequences are required to support a particular viewing mode of a display system that is capable of displaying the subject matter in a two-dimensional mode, a three-dimensional mode, or a multi-view three-dimensional mode.

Claims

1. A method for encoding a first video frame sequence and a second video frame sequence, the first video frame sequence corresponding to a first perspective view, the second frame sequence corresponding to a second perspective view, the method comprising:

applying a first tier of hierarchical encoding to the first video frame sequence to produce first tier encoded data, the first tier of hierarchical encoding using first internal referencing, the first internal referencing involving multiple frames within the first video frame sequence;

applying a second tier of hierarchical encoding to the second video frame sequence to produce second tier encoded data, the second tier of hierarchical encoding using both second internal referencing and first external referencing, the second internal referencing involving multiple frames within the second video frame sequence, the first external referencing involving references between a frame of the second video frame sequence and a frame of the first video frame sequence; and

the first tier encoded data being independently decodable, and the second tier encoded data being decodable with access to the first tier encoded data.

2. The method of claim 1, further comprising delivering the first tier encoded data to service a two-dimensional viewing environment.

3. The method of claim 1, further comprising delivering both the first tier encoded data and the second tier encoded data to service a three-dimensional viewing environment.

4. The method of claim 1, further comprising applying a third tier of hierarchical encoding to a third video frame sequence corresponding to a third perspective view, the third tier of hierarchical encoding using both third internal referencing and second external referencing, the third internal referencing involving multiple frames within the third video frame sequence, the second external referencing involving references between a frame of the third video frame sequence and a frame of the second video frame sequence.

5. The method of claim 1, further comprising applying a third tier of hierarchical encoding to a third video frame sequence corresponding to a third perspective view, the third tier of hierarchical encoding using both third internal referencing and second external referencing, the third internal referencing involving multiple frames within the third video frame sequence, the second external referencing involving references between a frame of the third video frame sequence and a frame of the first video frame sequence.

6. The method of claim 1, further comprising applying a third tier of hierarchical encoding to a third video frame sequence corresponding to a third perspective view, the third tier of hierarchical encoding using both third internal referencing and second external referencing, the third internal referencing involving multiple frames within the third video frame sequence, the second external referencing involving references between a first frame of the third video frame sequence and a frame of the first video frame sequence and references between a second frame of the third video frame sequence and a frame of the second video frame sequence.

7. A method used to encode a plurality of video elements, each of the plurality of video elements representing a video frame sequence from a selected perspective view, the method comprising:

identifying a first of the plurality of video elements, the first of the plurality of video elements having a first video frame sequence corresponding to a first perspective view;

producing an encoded version of the first video frame sequence, the encoded version of the first video frame sequence being independently decodable;

identifying a second of the plurality of video elements, the second of the plurality of video elements having a second video frame sequence corresponding to a second perspective view; and

producing an encoded version of the second video frame sequence, the encoded version of the second video frame sequence being decodable with reference to at least a representation of a frame of the first video frame sequence.

8. The method of claim 7, wherein the encoded version of the first video frame sequence can be decoded to produce a two-dimensional output.

9. The method of claim 7, wherein the encoded versions of both the first video frame sequence and the second video frame sequence can be decoded to produce a three-dimensional output.

10. The method of claim 7, further comprising producing an encoded version of a third video frame sequence, the encoded version of the third video frame sequence being decodable with reference to at least a representation of a frame of the second video frame sequence.

11. The method of claim 7, further comprising producing an encoded version of a third video frame sequence, the encoded version of the third video frame sequence being decodable with reference to at least a representation of a frame of the first video frame sequence.

12. The method of claim 7, further comprising producing an encoded version of a third video frame sequence, the encoded version of the third video frame sequence being decodable with reference to at least a representation of a frame of the first video frame sequence and a representation of a frame of the second video frame sequence.

18. The decoding system of claim 11, wherein the first predefined decoding approach comprises first tier hierarchical decoding and the second predefined decoding approach comprises second tier hierarchical decoding.

19. The decoding system of claim 11, wherein the encoded video content also has a third encoded portion relating to a third perspective view of three-dimensional content;

wherein the second predefined decoding approach supports external referencing between frames represented by the second encoded portion of the encoded video content and frames represented by the first encoded portion of the video content;

wherein the processing circuit, using a third predefined decoding approach, operates on the third encoded portion of the encoded video content to produce third video data, the third video data representative of the third perspective view of the three-dimensional content, the third predefined decoding approach supporting external referencing between frames represented by the third encoded portion of the encoded video content and frames represented by the second encoded portion of the encoded video content and external referencing between frames represented by the third encoded portion of the encoded video content and frames represented by the first encoded portion of the encoded video content.

13. An encoding system that receives video content, the video content having both first video data representative of a first perspective view of three-dimensional content and second video data representative of a second perspective view of three-dimensional content, the encoding system comprising:

a processing circuit that, using a first predefined encoding approach, operates on the first video data to produce a first encoded portion of an encoded version of the video content;

the processing circuit that, using a second predefined encoding approach, operates on the second video data to produce a second encoded portion of the encoded version of the video content, the second predefined encoding approach differing from the first predefined encoding approach in at least that the second predefined encoding approach supports external referencing; and

output circuitry, coupled to the processing circuitry, through which the first encoded portion and the second encoded portion are communicated.

14. The encoding system of claim 13, wherein the first predefined encoding approach comprises first tier hierarchical encoding and the second predefined encoding approach comprises second tier hierarchical encoding.

15. The encoding system of claim 13, wherein the video content also has third video data representative of a third perspective view of three-dimensional content;

wherein the second predefined encoding approach supports external referencing between frames of the second video data and frames of the first video data;

wherein the processing circuit, using a third predefined encoding approach, operates on the third video data to produce a third encoded portion of the encoded version of the video content, the third predefined encoding approach supporting external referencing between frames of the third video data and frames of the second video data and external referencing between frames of the third video data and frames of the first video data.

16. The encoding system of claim 15, wherein the first predefined encoding approach comprises first tier hierarchical encoding, the second predefined encoding approach comprises second tier hierarchical encoding, and the third predefined encoding approach comprises third tier hierarchical encoding.

20. The decoding system of claim 13, wherein the first predefined decoding approach comprises first tier hierarchical decoding, the second predefined decoding approach comprises second tier hierarchical decoding, and the third predefined decoding approach comprises third tier hierarchical decoding.

17. A decoding system that receives encoded video content, the encoded video content having both a first encoded portion relating to a first perspective view of three-dimensional content and a second encoded portion relating to a second perspective view of three-dimensional content, the decoding system comprising:

a processing circuit that, using a first predefined decoding approach, operates on the first encoded portion of the encoded video content to produce first video data, the first video data representative of the first perspective view of the three-dimensional content;

the processing circuit that, using a second predefined decoding approach, operates on the second encoded portion of the encoded video content to produce second video data, the second video data representative of the second perspective view of the three-dimensional content, and the second predefined decoding approach differing from the first predefined decoding approach in at least that the second predefined decoding approach supports external referencing; and

output circuitry, coupled to the processing circuitry, through which the first video data and the second video data are communicated.