Methods and apparatus for capturing, streaming and/or playing back content

• US 10,397,543 B2
• Filed: 09/03/2015
• Issued: 08/27/2019
• Est. Priority Date: 09/03/2014
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, McKay Moss, Alan, Medina, Hector M
• Primary Examiner(s): Volentine, Rebecca A
• Application Number:
  US14/845,202
• Publication Number:
  US 20160065946A1
• Time in Patent Office: 1,454 Days
• Field of Search: None
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/172 image signals comprising non-image signal components, e.g. headers or format information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information maybe in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information maybe communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.

Claims

1. A method of providing stereoscopic content, the method comprising:

storing, in a content delivery system,

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including;

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and

second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map;

operating the content delivery system to provide said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair to a playback device prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content second stereoscopic pair; and

in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair, operating the content delivery system to indicate to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.

2. The method of claim 1,

wherein said first stereoscopic camera pair corresponds to a first direction, the method further comprising:

operating said content delivery system to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras.

3. The method of claim 2, further comprising:

transmitting to said playback device an environmental mesh model to be used in rendering image content using said UV map.

4. The method of claim 3,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction information includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

5. The method of claim 4 wherein said UV map maps portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of an image rendering operation.

6. The method of claim 5, wherein the first correction information includes information identifying a subset of nodes in the UV map and provides node position correction information for the subset of nodes for which corrections are to be performed, said subset of nodes being less than all the nodes in the UV map.

7. The method of claim 6, wherein the second correction information includes information identifying a subset of nodes in the UV map and provides node position correction information for the subset of nodes for which corrections are to be performed, said subset of nodes being less than all the nodes in the UV map.

8. The method of claim 2, wherein said stereoscopic content stream communicates encoded image content including encoded left and right eye images captured by said first and second cameras of said first stereoscopic camera pair.

9. The method of claim 8, further comprising:

operating the content delivery system to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

10. The method of claim 9, wherein said first and second stereoscopic camera pairs correspond to a forward viewing direction but different locations at an area or event location where content is being captured for streaming.

11. The method of claim 9, further comprising:

indicating to the playback device that said third and fourth correction mesh information should be used when content captured by said second camera pair is streamed to the playback device instead of content from said first camera pair.

12. The method of claim 11, wherein said step of indicating to the playback device that said third and fourth correction mesh information should be used is performed in response to:

a user control change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

13. The method of claim 12, further comprising:

storing fifth correction mesh information for a first camera of a third stereoscopic camera pair;

storing sixth correction mesh information for a second camera of the third stereoscopic camera pair; and

operating the content delivery system to provide said fifth and sixth correction mesh information to the playback device, said fifth correction mesh information being for use in rendering image content captured by said first camera of said third stereoscopic camera pair, said sixth correction mesh information being for use in rendering image content captured by said second camera of the third stereoscopic camera pair.

14. A content delivery system, comprising:

a memory including;

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including;

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and

second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map; and

a processor configured to;

control said system to provide, to a playback device, said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair, prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detect a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to second stereoscopic pair; and

indicate, in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to said streaming content corresponding to second stereoscopic pair, to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.

15. The content delivery system of claim 14, wherein said first stereoscopic camera pair corresponds to a first direction;

wherein said processor is further configured to control said system to transmit a stereoscopic content stream including encoded images generated from image content captured by said first and second cameras; and

wherein said processor is further configured to transmit to said playback device an environmental mesh model to be used in rendering image content.

16. The content delivery system of claim 15,

wherein said first camera of the first stereoscopic camera pair captures left eye images and said second camera of the first stereoscopic camera pair captures right eye images;

wherein said first correction information includes information generated based on measurement of one or more optical characteristics of a first lens of said first camera of the first stereoscopic camera pair; and

wherein said second correction information includes information generated based on measurement of one or more optical characteristic of a second lens of said second camera of the first stereoscopic camera pair.

17. The content delivery system of claim 16, wherein said processor is further configured to transmit to said playback device a first UV map to be used for mapping portions of images captured by said first stereoscopic camera pair to a portion of said environmental mesh model as part of an image rendering operation.

18. The content delivery system of claim 14, wherein said processor is further configured to control said system to provide third correction mesh information and fourth correction mesh information to the playback device, said third correction mesh information being for use in rendering image content captured by a first camera of a second stereoscopic camera pair, said fourth correction mesh information being for use in rendering image content captured by a second camera of the second stereoscopic camera pair.

19. The content delivery system of claim 18, wherein said processor is configured to indicate to the playback device that said third and fourth correction mesh information should be in response to said system detecting:

i) a network controlled switch from streaming content from said first stereoscopic camera pair to said second stereoscopic pair or ii) a user controlled change from receiving a first content stream including content from said first stereoscopic camera pair to receiving a second content stream including encoded content from the second stereoscopic camera pair.

20. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control a content delivery system to:

store, in the content delivery system,

i) first stereoscopic camera pair correction information corresponding to a first stereoscopic camera pair, said correction information corresponding to the first stereoscopic camera pair including;

first correction information for a first camera of the first stereoscopic camera pair, said first correction information being first camera dependent mesh correction information, indicating corrections to be made to node positions in a UV map when rendering an image captured by said first camera; and

second correction information for a second camera of the first stereoscopic camera pair, said second correction information being second camera dependent mesh correction information, indicating corrections to be made to node positions in the UV map when rendering an image captured by said second camera;

ii) second stereoscopic camera pair correction information corresponding to a second stereoscopic camera pair, said second stereoscopic camera pair including a third camera and a fourth camera, said correction information corresponding to the second stereoscopic camera pair including third correction information for images captured by the third camera and fourth correction information for images captured by the fourth camera; and

iii) said UV map;

provide said UV map, said first correction information corresponding to the first stereoscopic camera pair and said second correction information corresponding to the second stereoscopic camera pair to a playback device prior to detecting a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic camera pair;

detect a network controlled switch from streaming content from the first stereoscopic camera pair to streaming content corresponding to the second stereoscopic pair; and

indicate, in response to detecting the network controlled switch from streaming content from said first stereoscopic camera pair to streaming content corresponding to the said second stereoscopic pair, to the playback device that the second stereoscopic camera pair correction information should be used for rendering operations.