INTERPOLATION OF THREE-DIMENSIONAL VIDEO CONTENT

• US 20110157315A1
• Filed: 12/30/2010
• Published: 06/30/2011
• Est. Priority Date: 12/31/2009
• Status: Abandoned Application

• Current Assignee: Avago Technologies General IP PTE Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Bennett, James D., Karaoguz, Jeyhan
• Application Number:
  US12/982,248
• Publication Number:
  US 20110157315A1
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G02B 6/00 Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings
• CPC Class Codes: G03B 35/24 using apertured or refractive resolving means on screens or between screen and eye
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G09G 2300/023 Display panel composed of stacked panels
• CPC Class Codes: G09G 2320/028 by changing the viewing angle properties, e.g. widening the viewing angle, adapting the viewing angle to the view direction
• CPC Class Codes: G09G 2370/04 Exchange of auxiliary data, i.e. other than image data, between monitor and graphics controller
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 3/20 for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/312 the parallax barriers being placed behind the display panel, e.g. between backlight and spatial light modulator [SLM]
• CPC Class Codes: H04N 13/315 the parallax barriers being time-variant
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/351 : for displaying simultaneously
• CPC Class Codes: H04N 13/359 : Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/361 : Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/403 : the images being monoscopic
• CPC Class Codes: H04N 2013/405 : the images being stereoscopic or three dimensional
• CPC Class Codes: H04N 21/235 : Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/4122 : additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/435 : Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04S 7/303 : Tracking of listener position or orientation

Abstract

Techniques are described herein for interpolating three-dimensional video content. Three-dimensional video content is video content that includes portions representing respective frame sequences that provide respective perspective views of a given subject matter over the same period of time. For example, the three-dimensional video content may be analyzed to identify one or more interpolation opportunities. If an interpolation opportunity is identified, frame data that is associated with the interpolation opportunity may be replaced with an interpolation marker. In another example, a frame that is not directly represented by data in the three-dimensional video content may be identified. For instance, the frame may be represented by an interpolation marker or corrupted data. The interpolation marker or corrupted data may be replaced with an interpolated representation of the frame.

Claims

1. An encoding system servicing three-dimensional video content, the three-dimensional video content having both a first portion representing a first sequence of frames that provide a first perspective view and a second portion representing a second sequence of frames that provide a second perspective view, the encoding system comprising:

processing circuitry;

input circuitry through which the processing circuitry receives both the first portion that represents the first sequence of frames that provide the first perspective view and the second portion that represents the second sequence of frames that provide the second perspective view;

the processing circuitry encodes the first portion and the second portion received, the encoding involving at least in part analyzing the first portion and the second portion to identify an interpolation opportunity, and, upon so identifying, the processing circuitry replaces frame data with an interpolation marker; and

output circuitry through which the processing circuitry delivers an encoded representation of the three-dimensional video content.

2. The encoding system of claim 1, wherein the processing circuitry compares a current frame with frames that neighbor the current frame to identify the interpolation opportunity.

3. The encoding system of claim 2, wherein the interpolation opportunity is identified in a first frame of the first portion while the neighboring frames include a second frame from the second portion.

4. The encoding system of claim 1, wherein the interpolation marker is accompanied by an interpolation instruction.

5. The encoding system of claim 1, wherein the processing circuitry determines that an accuracy of an estimate of the frame data is greater than a threshold accuracy;

and

wherein the processing circuitry analyzes the first portion and the second portion to identify the interpolation opportunity in response to determination that the accuracy of the estimate is greater than the threshold accuracy.

6. The encoding system of claim 1, wherein the processing circuitry determines that an error occurs with respect to the frame data;

and

wherein the processing circuitry analyzes the first portion and the second portion to identify the interpolation opportunity in response to determination that the error occurs.

7. The encoding system of claim 1, wherein the processing circuitry determines that a source that generates the three-dimensional video content has at least one specified characteristic;

and

wherein the processing circuitry analyzes the first portion and the second portion to identify the interpolation opportunity in response to determination that the source has the at least one specified characteristic.

8. The encoding system of claim 1, wherein the processing circuitry determines that a communication channel via which the three-dimensional video content is to be transmitted has at least one specified characteristic;

and

wherein the processing circuitry analyzes the first portion and the second portion to identify the interpolation opportunity in response to determination that the communication channel has the at least one specified characteristic.

9. The encoding system of claim 1, wherein the interpolation marker specifies a type of interpolation to be performed to generate the frame data.

10. A decoding system servicing encoded three-dimensional video content, the encoded three-dimensional video content having both a first encoded portion of a first encoded sequence of frames that represent a first perspective view and a second encoded portion of a second encoded sequence of frames that represent a second perspective view, the encoding system comprising:

processing circuitry;

input circuitry through which the processing circuitry receives both the first encoded portion of the first encoded sequence of frames that represent the first perspective view and the second encoded portion of the second encoded sequence of frames that represent the second perspective view;

the processing circuitry decodes the first encoded portion and the second encoded portion received, the decoding involving responding to an interpolation marker by generating frame data to replace the interpolation marker; and

output circuitry through which the processing circuitry delivers a decoded representation of the encoded three-dimensional video content.

11. The decoding system of claim 10, wherein the processing circuitry determines that a number of perspective views that a display is capable of processing is greater than a number of perspective views that is initially represented by the encoded three-dimensional video content;

wherein the processing circuitry provides an interpolation request to an encoder through the output circuitry, the interpolation request requesting inclusion of the interpolation marker in the encoded three-dimensional video content, in response to determination that the number of perspective views that the display is capable of processing is greater than the number of perspective views that is initially represented by the encoded three-dimensional video content; and

wherein the processing circuitry interpolates between a decoded version of the first encoded portion and a decoded version of the second encoded portion to generate the frame data that corresponds to a third sequence of frames that represent a third perspective view, the third perspective view not being initially represented by the encoded three-dimensional video content.

12. The decoding system of claim 10, wherein the processing circuitry receives an interpolation instruction from an upstream device through the input circuitry;

and

wherein the processing circuitry generates the frame data in accordance with the interpolation instruction.

13. The decoding system of claim 10, wherein the processing circuitry decodes the first encoded portion to provide a first decoded portion of a first decoded sequence of frames that represents the first perspective view;

wherein the processing circuitry decodes the second encoded portion to provide decoded data that represents the second perspective view, the decoded data including the interpolation marker; and

wherein the processing circuitry interpolates between the first decoded portion and a third decoded portion of a third decoded sequence of frames that represents a third perspective view to generate the frame data to replace the interpolation marker in the decoded data.

14. The decoding system of claim 13, wherein the processing circuitry receives a weight indicator from an upstream device through the input circuitry, the weight indicator specifying an extent to which the first decoded portion is to be weighed with respect to the third decoded portion;

and

wherein the processing circuitry generates the frame data based on the extent that is specified by the weight indicator.

15. A method used in decoding encoded three-dimensional video content, the encoded three-dimensional video content having both first encoded data relating to a first sequence of frames representing a first perspective view and second encoded data relating to a second sequence of frames representing a second perspective view, the method comprising:

retrieving at least a portion of the first encoded data that relates to the first sequence of frames representing the first perspective view;

retrieving at least a portion of the second encoded data that relates to the second sequence of frames representing the second perspective view;

identifying a first frame within the first sequence of frames not directly represented by the first encoded data retrieved; and

producing an interpolation of the first frame.

16. The method of claim 15, wherein the interpolation is based at least in part on the second encoded data.

17. The method of claim 15, wherein identifying the first frame comprises:

identifying an interpolation marker that is associated with the first frame.

18. The method of claim 17, wherein the interpolation marker is accompanied by interpolation instructions.

19. The method of claim 15, wherein the first frame comprising a missing frame.

20. The method of claim 15, wherein producing the interpolation of the first frame comprises:

producing the interpolation of the first frame based on at least the portion of the first encoded data and at least a portion of third encoded data that relates to a third sequence of frames representing a third perspective view based on a weight indicator, the weight indicator specifying an extent to which at least the portion of the first encoded data is to be weighed with respect to at least the portion of the third encoded data.