METHODS AND APPARATUS FOR MAKING ENVIRONMENTAL MEASUREMENTS AND/OR USING SUCH MEASUREMENTS IN 3D IMAGE RENDERING

• US 20160253839A1
• Filed: 03/01/2016
• Published: 09/01/2016
• Est. Priority Date: 03/01/2015
• Status: Abandoned Application

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Application Number:
  US15/057,210
• Publication Number:
  US 20160253839A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G09G 2352/00 Parallel handling of streams of display data
• CPC Class Codes: G09G 2360/02 Graphics controller able to handle multiple formats, e.g. input or output formats
• CPC Class Codes: G09G 2360/125 using unified memory architecture [UMA]
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/232 : using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 : wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/282 : for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 19/44 : Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 21/4345 : Extraction or processing of SI, e.g. extracting service information from an MPEG stream

Abstract

Methods and apparatus for making and using environmental measurements are described. Environmental information captured using a variety of devices is processed and combined to generate an environmental model which is communicated to customer playback devices. A UV map which is used for applying, e.g., wrapping, images onto the environmental model is also provided to the playback devices. A playback device uses the environmental model and UV map to render images which are then displayed to a viewer as part of providing a 3D viewing experience. In some embodiments updated environmental model is generated based on more recent environmental measurements, e.g., performed during the event. The updated environmental model and/or difference information for updating the existing model, optionally along with updated UV map(s), is communicated to the playback devices for use in rendering and playback of subsequently received image content. By communicating updated environmental information improved 3D simulations are achieved.

Claims

1. A method of operating a playback device, the method comprising:

receiving information communicating a first mesh model of a 3D environment generated based on measurements of a portion of said environment made using a light field camera at a first time;

receiving image content; and

rendering, using said first mesh model at least some of the received image content.

2. The method of claim 1, further comprising:

receiving updated mesh model information, said updated mesh model information including at least some updated mesh model information generated based on measurements of said portion of said environment using said light field camera at a second time.

3. The method of claim 2, further comprising:

receiving additional image content; and

rendering, using said updated mesh model information at least some of the received additional image content.

4. The method of claim 3, wherein said information communicating a first mesh model of the 3D environment includes information defining a complete mesh model.

5. The method of claim 4, wherein said updated mesh model information communicates a complete updated mesh model.

6. The method of claim 5, wherein said updated mesh model information provides new mesh information for portions of said 3D environment which have changed between said first and second time periods.

7. The method of claim 6, wherein said updated mesh model information is difference information indicating a difference between said first mesh model and an updated mesh model.

8. The method of claim 7, wherein said first mesh model information includes a first set of coordinate triples, each coordinate triple indicating a coordinate in X, Y, Z space of a node in the first mesh model.

9. The method of claim 8, wherein said updated mesh model information includes at least one of:

i) new sets of mesh coordinates for at least some nodes in said first mesh model information, said new coordinates being intended to replace coordinates of corresponding nodes in said first mesh model;

or ii) a new set of coordinate triples to be used for at least a portion of said first mesh model in place of a previous set of coordinate triples, said new set of coordinate triples including the same or a different number of coordinate triples than the previous set of coordinate triples to be replaced.

10. The method of claim 9, further comprising:

receiving a first map mapping a 2D image space to said first mesh model; and

wherein rendering, using said first mesh model at least some of the received image content, includes using said first map to determine how to wrap an image included in said received image content onto said first mesh model.

11. The method of claim 10, further comprising:

receiving updated map information corresponding to said updated mesh model information; and

wherein rendering, using said updated mesh model information at least some of the received additional image content, includes using said updated map information to determine how to wrap an additional image included in said received additional image content onto said updated mesh model.

12. The method of claim 11, wherein the updated map information includes map difference information, the method further comprising:

generating an updated map by applying said map difference information to said first map to generate an updated map; and

wherein rendering, using said updated mesh model information, at least some of the received additional image content, includes using said updated map to determine how to wrap an additional image included in said received additional image content onto said updated mesh model.

13. A computer readable medium including computer executable instructions which, when executed by a computer, control the computer to:

receive information communicating a first mesh model of a 3D environment generated based on measurements of a portion of said environment made using a light field camera at a first time;

receive image content; and

render, using said first mesh model at least some of the received image content.

14. A playback apparatus, comprising:

a processor configured to control said playback apparatus to;

receive information communicating a first mesh model of a 3D environment generated based on measurements of a portion of said environment made using a light field camera at a first time;

receive image content; and

render, using said first mesh model at least some of the received image content.

15. The playback apparatus of claim 14, wherein the processor is further configured to control the playback apparatus to:

receive updated mesh model information, said updated mesh model information including at least some updated mesh model information generated based on measurements of the portion of said environment using said light field camera at a second time.

16. The playback apparatus of claim 15, wherein the processor is further configured to control the playback apparatus to:

receive additional image content; and

render, using said updated mesh model information, at least some of the received additional image content.

17. The playback apparatus of claim 14, wherein the processor is further configured to control the playback apparatus to:

receive a first map mapping a 2D image space to said first mesh model; and

use said first map to determine how to wrap an image included in said received image content onto said first mesh model as part of being configured to render, using said first mesh model, at least some of the received image content.

18. The playback apparatus of claim 17, wherein the processor is further configured to control the playback apparatus to:

receive updated map information corresponding to said updated mesh model information; and

use said updated map information to determine how to wrap an additional image included in said received additional image content onto said updated mesh model as part of being configured to render, using said updated mesh model information, at least some of the received additional image content.

19. The playback apparatus of claim 18, wherein the updated map information includes map difference information;

and

wherein the processor is further configured to control the playback apparatus to;

generate an updated map by applying said map difference information to said first map to generate an updated map; and

use said updated map to determine how to wrap an additional image included in said received additional image content onto said updated mesh model as part of rendering, using said updated mesh model information, at least some of the received additional image content.

20. The playback apparatus of claim 16, wherein said information communicating a first mesh model of the 3D environment includes information defining a complete mesh model.