METHODS AND APPARATUS FOR RECEIVING AND/OR PLAYING BACK CONTENT

• US 20160065947A1
• Filed: 09/03/2015
• Published: 03/03/2016
• Est. Priority Date: 09/03/2014
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M
• Granted Patent: US 11,122,251 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information may be in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information may be communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.

Claims

1. A content playback method comprising:

receiving first correction information corresponding to a first camera;

receiving a first encoded image including image content captured by said first camera;

decoding the first encoded image to generate a first decoded image; and

performing a first rendering operation using the first correction information, the first decoded image and an environmental mesh model to generate a first image for display.

2. The content playback of claim 1, further comprising:

receiving second correction information corresponding to a second camera;

receiving a second encoded image including image content captured by said second camera;

decoding the second encoded image to generate a second decoded image; and

performing a second rendering operation using the second correction information, the second decoded image and the environmental mesh model to generate a second image for display.

3. The content playback method of claim 2, wherein the first and second cameras are part of a first stereoscopic camera pair and wherein the first encoded image is a left eye image and the second encoded image is a right eye image.

4. The content playback method of claim 3, wherein said first and second rendering operations are further performed as a function of a first UV map, said first correction information providing information on corrections to be made to node positions in said first UV map when performing said first rendering operation to compensate for distortions introduced into the first image by a lens of said first camera.

5. The content playback method of claim 4, wherein said first correction information includes information identifying a node in the first UV map, a U offset to be applied to said node and a V offset to be applied to said node.

6. The content playback method of claim 4, wherein said second correction information provides information on corrections to be made to node positions in said first UV map when performing said second rendering operation to compensate for distortions introduced into the second image by a lens of said second camera.

7. The content playback method of claim 6, further comprising:

using a first mask to determine how portions of said first image are combined with portions of a first image corresponding to a different field of view as part of said first rendering operation when applying portions of said first image to a surface of said environmental mesh model as part of said first rendering operation.

8. The content playback method of claim 7, wherein said portions of a first image corresponding to a different field of view correspond to a sky or ground field of view.

9. The content playback method of claim 8, further comprising:

using the first mask to determine how portions of said second image are combined with a portions of a second image corresponding to the different field of view as part of said second rendering operation when applying portions of said second image to the surface of said environmental mesh model as part of said second rendering operation.

10. The content playback method of claim 8, wherein said first image is a left eye image corresponding to a forward field of view and wherein said first image corresponding to a different field of view is a left eye image captured by a third camera corresponding to a side field of view adjacent said forward field of view.

11. The content playback method of claim 10, wherein said second image is a right eye image corresponding to a forward field of view and wherein said second image corresponding to a different field of view is a right eye image captured by a fourth camera corresponding to a side field of view adjacent said forward field of view.

12. The content playback method of claim 1, further comprising:

receiving additional mesh correction information corresponding to a plurality of different cameras; and

storing said additional mesh correction information.

13. The content playback method of claim 12, further comprising:

using mesh correction information corresponding to a fourth camera when rendering an image corresponding to said fourth camera, said fourth camera being one of said plurality of different cameras.

14. The content playback method of claim 14, further comprising:

determining which mesh correction information to use when performing a rendering operation based on which camera captured image content being used in the rendering operation or an indication from a server indicating which mesh correction information should be used when rendering images corresponding to a received content stream.

15. A content playback device comprising:

an interface configured to;

receive first correction information corresponding to a first camera; and

receive a first encoded image including image content captured by said first camera;

a decoder configured to decode the first encoded image to generate a first decoded image; and

a rendering engine configured to perform a first rendering operation using the first correction information, the first decoded image and an environmental mesh model to generate a first image for display.

16. The content playback device of claim 15,

wherein said interface is further configured to:

receive second correction information corresponding to a second camera; and

receive a second encoded image including image content captured by said second camera;

wherein said decoder is further configured to decode the second encoded image to generate a second decoded image; and

wherein said rendering engine is further configured to perform a second rendering operation using the second correction information, the second decoded image and the environmental mesh model to generate a second image for display.

17. The content playback device of claim 16, wherein said rendering engine is further configured to use a first UV map to perform the first and second rendering operations, said first correction information providing information on corrections to be made to node positions in said first UV map when said first rendering operation is performed to compensate for distortions introduced into the first image by a lens of said first camera.

18. The content playback device of claim 45,

wherein said rendering engine is further configured to use a first mask to determine how portions of said first image are combined with portions of a first image corresponding to a different field of view as part of said first rendering operation when applying portions of said first image to a surface of said environmental mesh model as part of said first rendering operation.

19. The content playback device of claim 18, further comprising:

wherein said rendering engine is further configured to use the first mask to determine how portions of said second image are combined with a portions of a second image corresponding to the different field of view as part of said second rendering operation when applying portions of said second image to the surface of said environmental mesh model as part of said second rendering operation.

20. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control a playback system to:

receive first correction information corresponding to a first camera;

receive a first encoded image including image content captured by said first camera;

decode the first encoded image to generate a first decoded image; and

perform a first rendering operation using the first correction information, the first decoded image and an environmental mesh model to generate a first image for display.