Methods and apparatus for requesting, receiving and/or playing back content corresponding to an environment

• US 10,038,889 B2
• Filed: 03/01/2016
• Issued: 07/31/2018
• Est. Priority Date: 03/01/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Primary Examiner(s): Ma, Tize
• Assistant Examiner(s): Elbinger, Steven
• Application Number:
  US15/058,131
• Publication Number:
  US 20160253809A1
• Time in Patent Office: 882 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09G 2352/00 Parallel handling of streams of display data
• CPC Class Codes: G09G 2360/02 Graphics controller able to handle multiple formats, e.g. input or output formats
• CPC Class Codes: G09G 2360/125 using unified memory architecture [UMA]
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/232 : using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 21/4345 : Extraction or processing of SI, e.g. extracting service information from an MPEG stream

Abstract

Methods and apparatus for receiving content including images of surfaces of an environment visible from a default viewing position and images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are received in content streams that can be in a variety of stream formats. In one stream format non-occluded image content is packed into a frame with occluded image content with the occluded image content normally occupying a small portion of the frame. In other embodiments occluded image portions are received in an auxiliary data stream which is multiplexed with a data stream providing frames of non-occluded image content. UV maps which are used to map received image content to segments of an environmental model are also supplied with the UV maps corresponding to the format of the frames which are used to provide the images that serve as textures.

Claims

1. A method of operating a content playback device, the method comprising:

receiving a first UV map indicating a mapping between portions of a first frame, including non-occluded image content, to a first set of segments of a first model of an environment, said first set of segments corresponding to portions of said environment visible from a first location;

receiving an auxiliary UV map indicating a mapping between portions of an auxiliary frame including occluded image content to a second set of segments of the first model of an environment, said second set of segments corresponding to portions of said environment which are not visible from said first location;

receiving said first frame in a primary content stream;

receiving said auxiliary frame in an auxiliary content stream, said auxiliary frame including occluded image content corresponding to said first frame and at least one additional frame in a set of M frames included in said primary content stream, M being greater than one;

detecting a head position of a user;

rendering an image by;

i) using the first UV map to map image portions included in the first frame to segments of the first set of segments of the first model of the environment and ii) using the auxiliary UV map to map image portions included in the auxiliary frame to segments of the second set of segments of the first model of the environment, said rendered image including a portion of the environment which would be visible to a user in the environment having the detected head position; and

outputting said portion of the rendered image to a display.

2. The method of claim 1, wherein outputting said portion of the rendered image to the display includes outputting image portions corresponding to at least some portions of the environment which are not visible from the first location, when a detected head position of the user corresponds to a location offset from said first location.

3. The method of claim 2,

wherein receiving said first frame and receiving said auxiliary frame are performed as part of receiving a multiplexed content stream including said primary content stream and said auxiliary content stream.

4. The method of claim 3, further comprising:

demultiplexing said multiplexed content stream to recover said primary content stream and said auxiliary content stream.

5. The method of claim 4, wherein said primary content stream includes M frames for each frame in said auxiliary content stream.

6. The method of claim 1, wherein said first frame and said auxiliary frame are received in encoded form, the method further comprising:

using a first decoder to decode said first frame prior to rendering the image; and

using a second decoder, which is different from said first decoder, to decode said auxiliary frame prior to rendering the image.

7. The method of claim 1, wherein said auxiliary frame is the same size and includes the same number of pixels as said first frame.

8. The method of claim 7, wherein said first frame and said auxiliary frame are received in encoded form, the method further comprising:

using a decoder on a time shared basis to decode said first frame and said auxiliary frame prior to rendering the image.

9. The method of claim 1, further comprising, prior to rendering the image, performing the steps of:

receiving a second environmental model including segments corresponding to portions of the environment visible from said first location in the environment;

receiving auxiliary environmental model information providing information on segments corresponding to occluded portions of said environment which are not visible from said first location in the environment; and

using the auxiliary environmental model information to modify the second environmental model to generate said first model of the environment, said first model of the environment including segments corresponding to occluded portions of said environment and segments corresponding to non-occluded portions of said environment.

10. The method of claim 1, wherein said display is a head mounted display.

11. A content playback device, comprising:

a receiver configured to;

receive a first UV map indicating a mapping between portions of a first frame, including non-occluded image content, to a first set of segments of a first model of an environment, said first set of segments corresponding to portions of said environment visible from a first location;

receive an auxiliary UV map indicating a mapping between portions of an auxiliary frame including occluded image content to a second set of segments of the first model of the environment, said second set of segments corresponding to portions of said environment which are not visible from said first location;

receive said first frame in a primary content stream;

receive said auxiliary frame in an auxiliary content stream, said auxiliary frame including occluded image content corresponding to said first frame and at least one additional frame in a set of M frames included in said primary content stream, M being greater than one;

a memory for storing received frames; and

a processor configured to;

detect a head position of a user;

render an image by;

i) using the first UV map to map image portions included in the first frame to segments of the first set of segments of the first model of the environment and ii) using the auxiliary UV map to map image portions included in the auxiliary frame to segments of the second set of segments of the first model of the environment, said rendered image including a portion of the environment which would be visible to a user in the environment having the detected head position; and

output said portion of the rendered image to a display.

12. The content playback device of claim 11, further comprising:

said display.

13. The content playback device of claim 12, wherein said display is a head mounted display.

14. The content playback device of claim 13, wherein the processor is configured to:

output, as part of said portion of the rendered image, image portions corresponding to portions of the environment which are not visible from said first location.

15. The content playback device of claim 11, wherein the receiver receives said first frame and said auxiliary frame in a multiplexed content stream including a primary content stream including said first frame and an auxiliary content stream including said auxiliary frame;

and

wherein the content playback device further comprises;

a demultiplexer for demultiplexing said multiplexed content stream to recover said primary content stream and said auxiliary content stream.

16. The content playback device of claim 15, wherein said primary content stream includes M frames for each frame in said auxiliary content stream.

17. The content playback device of claim 16, wherein said first frame and said auxiliary frame are received in encoded form, and wherein said content playback device further includes:

a first decoder to decode said first frame prior to image rendering; and

a second decoder, which is different from said first decoder, to decode the said auxiliary frame prior to image rendering.

18. A non-transitory computer readable medium having computer executable instructions stored thereon which, when executed by a processor of a content playback device control the content playback device to perform the steps of:

receiving a first UV map indicating a mapping between portions of a first frame, including non-occluded image content, to a first set of segments of a first model of an environment, said first set of segments corresponding to portions of said environment visible from a first location;

receiving an auxiliary UV map indicating a mapping between portions of an auxiliary frame including occluded image content to a second set of segments of the first model of the environment, said second set of segments corresponding to portions of said environment which are not visible from said first location;

receiving said first frame in a primary content stream;

receiving said auxiliary frame in an auxiliary content stream, said auxiliary frame including occluded image content corresponding to said first frame and at least one additional frame in a set of M frames included in said primary content stream, M being greater than one;

detecting a head position of a user;

rendering an image by;

i) using the first UV map to map image portions included in the first frame to segments of the first set of segments of the first model of the environment and ii) using the auxiliary UV map to map image portions included in the auxiliary frame to segments of the second set of segments of the first model of the environment said rendered image including a portion of the environment which would be visible to a user in the environment having the detected head position; and

outputting said portion of the rendered image to a display.