Method and system for rendering virtual content

• US 10,408,613 B2
• Filed: 05/05/2015
• Issued: 09/10/2019
• Est. Priority Date: 07/12/2013
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Abovitz, Rony, Schowengerdt, Brian T., Watson, Mathew D.
• Primary Examiner(s): Wang, Jin Cheng
• Application Number:
  US14/704,765
• Publication Number:
  US 20150235441A1
• Time in Patent Office: 1,589 Days
• Field of Search: 345633, 715852, 382103, 382190, 707609, 707722
• CPC Class Codes: A61B 2034/101 Computer-aided simulation of surgical operations
• CPC Class Codes: A61B 2562/0204 Acoustic sensors
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 2562/0247 Pressure sensors
• CPC Class Codes: A61B 3/0008 provided with illuminating means A61B3/117 takes precedence; goniolenses used for laser treatment A61F9/009; illuminating means for optical instruments G02B27/00
• CPC Class Codes: A61B 3/022 for testing contrast sensitivity
• CPC Class Codes: A61B 3/024 for determining the visual field, e.g. perimeter types
• CPC Class Codes: A61B 3/028 for testing visual acuity; for determination of refraction, e.g. phoropters
• CPC Class Codes: A61B 3/063 for testing light sensitivity, i.e. adaptation
• CPC Class Codes: A61B 3/066 for testing colour vision
• CPC Class Codes: A61B 3/08 for testing binocular or stereoscopic vision, e.g. strabismus
• CPC Class Codes: A61B 3/085 for testing strabismus
• CPC Class Codes: A61B 3/10 Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
• CPC Class Codes: A61B 3/1015 for wavefront analysis
• CPC Class Codes: A61B 3/102 for optical coherence tomography [OCT]
• CPC Class Codes: A61B 3/1035 for measuring astigmatism testing astigmatism A61B3/036
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 3/12 for looking at the eye fundus, e.g. ophthalmoscopes A61B3/13 takes precedence
• CPC Class Codes: A61B 3/1216 for diagnostics of the iris
• CPC Class Codes: A61B 3/13 Ophthalmic microscopes
• CPC Class Codes: A61B 3/14 : Arrangements specially adapted for eye photography apparatus or arrangements for taking photographs per se G03B
• CPC Class Codes: A61B 3/165 : Non-contacting tonometers
• CPC Class Codes: A61B 34/10 : Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 5/0059 : using light, e.g. diagnosis by transillumination, diascopy, fluorescence A61B5/0093, A61B5/1455, A61B5/02416 and A61B5/0261 take precedence
• CPC Class Codes: A61B 5/0066 : Optical coherence imaging
• CPC Class Codes: A61B 5/0077 : Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/01 : Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue clinical contact thermometers G01K13/20
• CPC Class Codes: A61B 5/14532 : for measuring glucose, e.g. by tissue impedance measurement
• CPC Class Codes: A61B 5/1455 : using optical sensors, e.g. spectral photometrical oximeters
• CPC Class Codes: A61B 5/14555 : specially adapted for the eye fundus
• CPC Class Codes: A61B 5/369 : Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/398 : Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/6803 : Head-worn items, e.g. helmets, masks, headphones or goggles
• CPC Class Codes: A63F 13/00 : Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 : comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 : involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G01B 11/303 : using photoelectric detection means
• CPC Class Codes: G02B 2027/0105 : Holograms with particular structures
• CPC Class Codes: G02B 2027/0127 : comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 : comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 : comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0185 : Displaying image at variable distance
• CPC Class Codes: G02B 2027/0187 : slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0101 : characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0179 : Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G02B 27/42 : Diffraction optics , i.e. systems including a diffractive element being designed for providing a diffractive effectG02B27/60 takes precedence
• CPC Class Codes: G02B 27/4205 : having a diffractive optical element [DOE] contributing to image formation, e.g. whereby modulation transfer function MTF or optical aberrations are relevant
• CPC Class Codes: G02B 27/4227 : in image scanning systems
• CPC Class Codes: G02B 6/10 : of the optical waveguide type G02B6/02, G02B6/24 take precedence; devices or arrangements for the control of light by electric, magnetic, electro-magnetic or acoustic means G02F1/00; transferring the modulation of modulated light G02F2/00; optical logic elements G02F3/00; optical analogue/digital converters G02F7/00
• CPC Class Codes: G02B 6/34 : utilising prism or grating G02B6/293 takes precedence
• CPC Class Codes: G06F 16/7837 : using objects detected or recognised in the video content
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/005 : Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 : for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 30/0643 : Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2200/24 : involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/10148 : Varying focus
• CPC Class Codes: G06T 2207/10152 : Varying illumination
• CPC Class Codes: G06T 2207/30041 : Eye; Retina; Ophthalmic
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2210/41 : Medical
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06V 10/20 : Image preprocessing
• CPC Class Codes: G06V 10/40 : Extraction of image or video features
• CPC Class Codes: G06V 10/758 : Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/113 : Recognition of static hand signs
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G16H 40/00 : ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
• CPC Class Codes: G16H 40/20 : for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
• CPC Class Codes: G16H 40/67 : for remote operation
• CPC Class Codes: H04B 10/25891 : Transmission components H04B10/40 takes precedence

Abstract

A waveguide apparatus includes a planar waveguide and at least one optical diffraction element (DOE) that provides a plurality of optical paths between an exterior and interior of the planar waveguide. A phase profile of the DOE may combine a linear diffraction grating with a circular lens, to shape a wave front and produce beams with desired focus. Waveguide apparati may be assembled to create multiple focal planes. The DOE may have a low diffraction efficiency, and planar waveguides may be transparent when viewed normally, allowing passage of light from an ambient environment (e.g., real world) useful in AR systems. Light may be returned for temporally sequentially passes through the planar waveguide. The DOE(s) may be fixed or may have dynamically adjustable characteristics. An optical coupler system may couple images to the waveguide apparatus from a projector, for instance a biaxially scanning cantilevered optical fiber tip.

Claims

1. A method of rendering virtual content, comprising:

rendering a virtual world for a user by using at least a virtual world model based at least in part upon a location of the user at least by performing a first process, the first process comprising;

determining the location of a user in the virtual world at least by matching a signature of the location with a predetermined signature of a first node in a topological map, without having to process geometries at the location, wherein nodes in the topological map correspond to respective distinct signatures and do not include keyframes of corresponding locations;

retrieving a first set of data associated with a first part of the virtual world by using the virtual world model, the first set of data corresponding to both the first node in a topological map and the location of the user, wherein the virtual world model does not render virtual contents that are displayed to the user; and

rendering, based at least in part or in whole on the first set of data, the first part of the virtual world to a user device of the user at least by generating the virtual contents for the first part;

retrieving a second part of the virtual world by using at least the virtual world model based at least in part upon an anticipated location from the location of the user at least by performing a second process, the second process comprising;

determining an anticipated location for the user at the location at least by;

identifying an area around the user;

communicating an intended direction of the user and one or more locations within the area to a server storing the virtual world model; and

determining, at the server, the anticipated location from the world model based at least in part upon the intended location of the user and the one or more locations within the area;

retrieving a second set of data associated with the second part of the virtual world that corresponds to a second node representing the anticipated location of the user in the topological map; and

sharing a space at the anticipated location with a plurality of users including the user at least by updating the virtual world for the user, wherein updating the virtual world comprises rendering, based on the location and the anticipated location of the user, the second part of the virtual world using at least the second set of data, the second part comprising a virtual object that was previously generated in response to an action of a different user in the second part of the virtual world.

2. The method of claim 1, further comprising detecting the location of the user at least by determining a first histogram for the location and a second histogram for the anticipated location.

3. The method of claim 1, further comprising determining a radius around the location of the user, the radius denoting the part of the virtual world.

4. The method of claim 1, wherein the first set of data or the second set of data includes keyframes and images.

5. The method of claim 1, wherein the virtual world model comprises a persistent digital representation of the physical environment in which the user is located.

6. A method of rendering virtual content, comprising:

rendering a virtual world for a user by using at least a virtual world model based at least in part upon a first location of the user at least by performing a first process, the first process comprising;

determining the first location of the user at least by matching a signature with a predetermined signature of a first node in a topological map, without having to process geometries at the location, wherein nodes in the topological map correspond to respective distinct signatures and do not include keyframes of corresponding locations;

retrieving a first set of data associated with a first part of the virtual world by using the virtual world model, the first set of data corresponding to both the first node in a topological map and the first location of the user, wherein the virtual world model does not render the virtual contents that are displayed to the user; and

rendering, based in whole or in part on the first set of data, the first part of the virtual world to a user device of the user at least by generating the virtual contents for the first part;

retrieving a second part of the virtual world by using at least the virtual world model based at least in part upon an anticipated location from the location of the user at least by performing a second process, the second process comprising;

determining an anticipated location of the user at least by;

identifying an area around the first node in the topological map;

communicating an intended direction of the user and one or more locations within the area to a server storing the virtual world model; and

determining, at the server, the anticipated location from the world model based at least in part upon the intended location of the user and the one or more locations within the area;

retrieving a second set of data associated with the second part of the virtual world that corresponds to a second node representing the anticipated location of the user in the topological map; and

rendering, based at least in part on the second set of data, second virtual content to the user device of the user, the second virtual content comprising a virtual object that was previously generated in response to an action of a different user in the second part of the virtual world.

7. The method of claim 6, further comprising detecting the first and second locations of the user at least by determining a first histogram for the location and a second histogram for the anticipated location.

8. The method of claim 6, further comprising determining a radius around the first and second location of the user, the radius denoting the part of the virtual world.

9. The method of claim 6, wherein the set of data includes keyframes and images.

10. The method of claim 6, wherein the virtual world model comprises an updatable persistent digital representation of the physical environment in which the user is located.

11. A method of rendering virtual content, comprising:

rendering a virtual world for a user by using at least a virtual world model based at least in part upon a location of the user at least by performing a first process, the first process comprising;

determining the location of a first user in the virtual world at least by matching a signature of the location with a predetermined signature of a first node in a topological map, without having to process geometries at the location, wherein nodes in the topological map correspond to respective distinct signatures and do not include keyframes of corresponding locations;

retrieving a first set of data associated with a first part of the virtual world by using the virtual world model, the first set of data corresponding to both the first node in a topological map and the location of the first user;

retrieving a second set of data associated with a second part of the virtual world by using the virtual world model, the second set of data corresponding to a location of a second user and a second node in the topological map, wherein the virtual world model does not render virtual contents that are displayed to the first user;

rendering, based in whole or in part on the first and second sets of data, the first part and the second part of the virtual world to a user device of the first user at least by generating virtual contents for the first part and the second part; and

retrieving a second part of the virtual world by using at least the virtual world model based at least in part upon an anticipated location from the location of the user at least by performing a second process, the second process comprising;

determining an anticipated location for the first user at the location at least by;

identifying an area around the first user;

communicating an intended direction of the first user and one or more locations within the area to a server storing the virtual world model; and

determining, at the server, the anticipated location from the world model based at least in part upon the intended location of the first user and the one or more locations within the area;

retrieving a third set of data associated with a third part of the virtual world that corresponds to a third node representing the anticipated location of the first user in the topological map; and

sharing a space at the anticipated location with a plurality of users including the first user and the second user at least by updating the virtual world for the user, wherein updating the virtual world comprises rendering, based on the location and the anticipated location of the first user, the second part of the virtual world using at least the third set of data, the third part comprising a virtual object that was previously generated in response to an action of a different user in the third part of the virtual world.

12. The method of claim 11, further comprising detecting the first location of the first user at least by determining a first histogram for the location and a second histogram for the anticipated location.

13. The method of claim 11, further comprising determining a radius around the first location of the first user, the radius denoting the first part of the virtual world.

14. The method of claim 11, wherein the first and second sets of data each includes keyframes and images.

15. The method of claim 11, wherein the virtual world model comprises an updatable persistent digital representation of the physical environment.

16. An AR system, comprising:

a processor;

a sensor operatively coupled to the processor to render a virtual world for a user by using at least a virtual world model based at least in part upon a location of the user at least by performing a first process, the first process comprising;

determining the a location of a user in the virtual world at least by matching a signature of the location with a predetermined signature of a first node in a topological map, without having to process geometries at the location, wherein nodes in the topological map correspond to respective distinct signatures and do not include keyframes of corresponding locations; and

retrieve a first set of data associated with a first part of the virtual world by using the virtual world model, the first set of data corresponding to both the first node in a topological map and the location of the user, wherein the virtual world model does not render virtual contents that are displayed to the user;

render, based at least in part or in whole on the first set of data, the first part of the virtual world to a user device of the user at least by generating the virtual content for the first part;

the processor further configured to retrieve a second part of the virtual world by using at least the virtual world model based at least in part upon an anticipated location from the location of the user at least by performing a second process, the second process comprising;

determining an anticipated location for the user at the location at least by;

identifying an area around the user;

communicating an intended direction of the user and one or more locations within the area to a server storing the virtual world model; and

determining, at the server, the anticipated location from the world model based at least in part upon the intended location of the user and the one or more locations within the area;

retrieving a second set of data associated with the second part of the virtual world that corresponds to a second node representing the anticipated location of the user in the topological map; and

the processor further configured to share a space at the anticipated location with a plurality of users including the user at least by updating the virtual world for the user, wherein updating the virtual world comprises rendering, based on the location and anticipated location of the user, the second part of the virtual world using at least the second set of data, the second part comprising a virtual object that was previously generated in response to an action of a different user in the second part of the virtual world.

17. A computer program product comprising a non-transitory computer readable storage medium embodying one or more instructions executable by a processor to perform a process for rendering virtual content, the processing comprising:

rendering a virtual world for a user by using at least a virtual world model based at least in part upon a location of the user at least by performing a first process, the first process comprising;

determining the a location of a user in the virtual world at least by matching a signature of the location with a predetermined signature of a first node in a topological map, without having to process geometries at the location, wherein nodes in the topological map correspond to respective distinct signatures and do not include keyframes of corresponding locations;

retrieving a first set of data associated with a first part of the virtual world by using the virtual world model, the first set of data corresponding to both the first node in a topological map and the location of the user, wherein the virtual world model does not render virtual contents that are displayed to the user;

rendering, based at least in part or in whole on the first set of data, the first part of the virtual world to a user device of the user at least by generating the virtual contents for the first part;

retrieving a second part of the virtual world by using at least the virtual world model based at least in part upon an anticipated location from the location of the user at least by performing a second process, the second process comprising;

determining an anticipated location for the user at the location at least by;

identifying an area around the user;

communicating an intended direction of the user and one or more locations within the area to a server storing the virtual world model; and

determining, at the server, the anticipated location from the world model based at least in part upon the intended location of the user and the one or more locations within the area;

retrieving a second set of data associated with the second part of the virtual world that corresponds to the a second node representing the anticipated location of the user in the topological map; and

sharing a space at the anticipated location with a plurality of users including the user at least by updating the virtual world for the user, wherein updating the virtual world comprises rendering, based on the location and anticipated location of the user, the second part of the virtual world using at least the second set of data, the second part comprising a virtual object that was previously generated in response to an action of a different user in the second part of the virtual world.