METHODS AND APPARATUS FOR REQUESTING, RECEIVING AND/OR PLAYING BACK CONTENT CORRESPONDING TO AN ENVIRONMENT

• US 20160253810A1
• Filed: 03/01/2016
• Published: 09/01/2016
• Est. Priority Date: 03/01/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Granted Patent: US 10,574,962 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09G 2352/00 Parallel handling of streams of display data
• CPC Class Codes: G09G 2360/02 Graphics controller able to handle multiple formats, e.g. input or output formats
• CPC Class Codes: G09G 2360/125 using unified memory architecture [UMA]
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/232 : using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 21/4345 : Extraction or processing of SI, e.g. extracting service information from an MPEG stream

Abstract

Methods and apparatus for receiving content including images of surfaces of an environment visible from a default viewing position and images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are received in content streams that can be in a variety of stream formats. In one stream format non-occluded image content is packed into a frame with occluded image content with the occluded image content normally occupying a small portion of the frame. In other embodiments occluded image portions are received in an auxiliary data stream which is multiplexed with a data stream providing frames of non-occluded image content. UV maps which are used to map received image content to segments of an environmental model are also supplied with the UV maps corresponding to the format of the frames which are used to provide the images that serve as textures.

Claims

1. A method of operating a content playback device, the method comprising:

receiving a first frame including non-occluded image content corresponding to a portion of an environment visible from a first location in the environment and occluded image content corresponding to a portion of the environment which is not visible from said first location;

detecting a head position of a user; and

outputting to a display an image of portions of the environment as a function of the detected head position.

2. The method of claim 1, wherein said outputting to a display an image includes:

outputting image portions corresponding to portions of the environment which are visible from the first location when said users head position is in a default location; and

outputting image portions corresponding to at least some portions of the environment which are not visible from the first location when said users head position indicates a location offset from said default location.

3. The method of claim 1, further comprising:

receiving a UV map indicating a mapping between portions of a frame and segments of a model of the environment; and

wherein outputting to a display an image includes outputting a portion of a rendered image generated by using the UV map to map image portions included in the first frame to segments of the model of the environment.

4. The method of claim 3, further comprising:

receiving said model of the environment, said model being a mesh model including segments corresponding to surfaces in said environment which are visible from the first location and at least some segments corresponding to occluded portions of said environment which are not visible from the first location.

5. The method of claim 3, wherein said non-occluded image content includes content captured by a first camera at said first location in the environment and wherein the occluded image content was captured by an additional camera located in said environment at a location offset from said first location.

6. The method of claim 4, the method further comprising:

receiving a list of available programs, said list including for a first program a list of streams corresponding to said program but in different stream formats, a first one of said different stream formats including a stream format which does not include occluded image content and a second stream format which includes occluded image content;

receiving user input selecting said first program; and

selecting, based on device capability information, which stream corresponding to the user selected program to request.

7. The method of claim 6,

wherein the playback device supports display of occluded image content;

and

wherein said step of selecting includes selecting a stream that includes occluded image content; and

wherein the method further comprises;

sending a request for the selected stream corresponding to the user selected program to a content server.

8. The method of claim 4, further comprising:

receiving a list of available programs, said list including a first program;

receiving user input selecting said first program; and

sending a request to a content server for the stream.

9. The method of claim 8, further comprising:

communicating to said content server device capability information indicating that said playback device is capable of supporting the display of occluded image content corresponding to portions of said environment which are not visible from said first location.

10. The method of claim 9, wherein communicating to said content server device capability information includes at least one of:

i) providing an identifier to the content server which can be used to determine capabilities of the playback device or ii) indicating to the content server content stream formats that can be supported by the playback device.

11. The method of claim 10, wherein communicating to said content server device capability information includes indicating to the content server that the playback device supports a content stream format in which occluded image data is transmitted in a frame which also includes non-occluded image data.

12. A non-transitory computer readable medium for use in a system, said non-transitory computer readable medium including computer executable instructions which, when executed by a computer, control the system to:

receive a first frame including non-occluded image content corresponding to a portion of an environment visible from a first location in the environment and occluded image content corresponding to a portion of the environment which is not visible from said first location;

detect a head position of a user; and

output to a display an image of portions of the environment as a function of the detected head position.

13. A content playback device comprising:

a receiver configured to receive a first frame including non-occluded image content corresponding to a portion of an environment visible from a first location in the environment and occluded image content corresponding to a portion of the environment which is not visible from said first location;

a processor configured to control said content playback device to;

detect a head position of a user, and

output to a display an image of portions of the environment as a function of the detected head position; and

a memory for storing said image.

14. The content playback device of claim 13, wherein output to a display an image includes:

outputting image portions corresponding to portions of the environment which are visible from the first location when said users head position is in a default location; and

outputting image portions corresponding to at least some portions of the environment which are not visible from the first location when said users head position indicates a location offset from said default location.

15. The content playback device of claim 13,

wherein said receiver is further configured to receive a UV map indicating a mapping between portions of a frame and segments of a model of the environment;

and

wherein outputting to a display an image includes outputting a portion of a rendered image generated by using the UV map to map image portions included in the first frame to segments of the model of the environment.

16. The content playback device of claim 15, wherein said receiver is further configured to receive said model of the environment, said model being a mesh model including segments corresponding to surfaces in said environment which are visible from the first location and at least some segments corresponding to occluded portions of said environment which are not visible from the first location.

17. The content playback device of claim 15, wherein said non-occluded image content includes content captured by a first camera at said first location in the environment and wherein the occluded image content was captured by an additional camera located in said environment at a location offset from said first location.

18. The content playback device of claim 16,

wherein said receiver is further configured to receive a list of available programs, said list including for a first program a list of streams corresponding to said program but in different stream formats, a first one of said different stream formats including a stream format which does not include occluded image content and a second stream format which includes occluded image content;

and

said processor is further configured to operate said content playback device to;

receive user input selecting said first program; and

select, based on device capability information, which stream corresponding to the user selected program to request.

19. The content playback device of claim 18, further comprising:

a transmitter configured to send a request for the selected stream corresponding to the user selected program to a content server;

wherein the playback device supports display of occluded image content; and

wherein said processor as part of being configured to operate said content playback device to select which stream corresponding to the user selected program to request is configured to operate said content playback device to select a stream that includes occluded image content.

20. The content playback device of claim 16 further comprising a transmitter configured to send a request to a content server for the stream;

wherein said receiver is further configured to receive a list of available programs, said list including a first program; and

wherein said processor is further configured to operate said content playback device to receive user input selecting said first program.