METHODS AND APPARATUS FOR CAPTURING, STREAMING AND/OR PLAYING BACK CONTENT

• US 20160065946A1
• Filed: 09/03/2015
• Published: 03/03/2016
• Est. Priority Date: 09/03/2014
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M
• Granted Patent: US 10,397,543 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information maybe in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information maybe communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.

Claims

1. A method of providing stereoscopic content, the method comprising:

storing first correction information for a first camera of a first stereoscopic camera pair;

storing second correction information for a second camera of the first stereoscopic camera pair; and

operating a server to provide said first correction information and said second correction information to a playback device, said first correction information being for use in rendering image content captured by said first camera, said second correction information being for use in rendering image content captured by said second camera.

2. The method of claim 1, where said first correction information is first camera dependent mesh correction information;

wherein said second correction information is second camera dependent mesh correction information;

wherein said first stereoscopic camera pair corresponds to a first direction, the method further comprising;

operating said server to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras.

3. The method of claim 2, further comprising:

transmitting to said playback device an environmental mesh model to be used in rendering image content.

8. The method of claim 2, wherein said stereoscopic content stream communicates encoded image content including encoded left and right eye images captured by said first and second cameras of said first stereoscopic camera pair.

9. The method of claim 8, further comprising:

operating the server to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

10. The method of claim 9, wherein said first and second stereoscopic camera pairs correspond to a forward viewing direction but different locations at an area or event location where content is being captured for streaming.

11. The method of claim 9, further comprising:

indicating to the playback device that said third and fourth correction mesh information should be used when content captured by said second camera pair is streamed to the playback device instead of content from said first camera pair.

12. The method of claim 11, wherein said step of indicating to the playback device that said third and fourth correction mesh information should be used is performed in response to:

i) a network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair (e.g., as part of a broadcaster camera controlled switch in source cameras) or ii) a user control change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

13. The method of claim 12, further comprising:

storing fifth mesh correction information for a first camera of a third stereoscopic camera pair;

storing sixth correction mesh information for a second camera of the third stereoscopic camera pair; and

operating the server to provide said fifth and sixth correction mesh information to the playback device, said fifth correction mesh information being for use in rendering image content captured by said first camera of said third stereoscopic camera pair, said sixth correction mesh information being for use in rendering image content captured by said second camera of the third stereoscopic camera pair.

4. The method of 3,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction mesh includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

5. The method of claim 4, further comprising:

transmitting to said playback device a first UV map to be used for mapping portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of a image rendering operation.

6. The method of claim 5, wherein the first correction information includes information indicating one or more adjustments to be made to node locations in said first UV map as part of rendering portions of images captured by said first camera of said first stereoscopic camera pair.

7. The method of claim 6, wherein the second correction information includes information indicating one or more adjustments to be made to node locations in said first UV map as part of rendering portions of images captured by said second camera of said first stereoscopic camera pair.

14. A content delivery system, comprising:

a memory including stored first correction information for a first camera of a first stereoscopic camera pair and stored second correction information for a second camera of the first stereoscopic camera pair; and

a processor configured to control said system to provide said first correction information and said second correction information to a playback device, said first correction information being for use in rendering image content captured by said first camera, said second correction information being for use in rendering image content captured by said second camera.

15. The system of claim 14, wherein said first stereoscopic camera pair corresponds to a first direction;

and

wherein said processor is further configured to control said system to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras; and

wherein said processor is further configured to transmit to said playback device an environmental mesh model to be used in rendering image content.

18. The system of claim 14, wherein said processor is further configured to control said system to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

19. The system of claim 18, wherein said processor is configured to indicate to the playback device that said third and fourth correction mesh information should be in response to said system detecting:

i) a network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair or ii) a user controlled change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

16. The system of 15,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction mesh includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

17. The system of claim 16, wherein said processor is further configured to transmit to said playback device a first UV map to be used for mapping portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of a image rendering operation.

20. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control a content delivery system to:

store, in a memory, first correction information for a first camera of a first stereoscopic camera pair;

store, in the memory, second correction information for a second camera of the first stereoscopic camera pair; and

provide said first correction information and said second correction information to a playback device, said first correction information being for use in rendering image content captured by said first camera, said second correction information being for use in rendering image content captured by said second camera.