Interacting with totems in augmented or virtual reality systems

• US 9,922,462 B2
• Filed: 05/07/2015
• Issued: 03/20/2018
• Est. Priority Date: 04/18/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Miller, Samuel A.
• Primary Examiner(s): Nguyen, Vu
• Application Number:
  US14/706,888
• Publication Number:
  US 20150317839A1
• Time in Patent Office: 1,048 Days
• Field of Search: None
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/577 using determination of contact between game characters or objects, e.g. to avoid collision between virtual racing cars
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/02 : Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 13/40 : of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 13/80 : 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 15/10 : Geometric effects
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 : involving 3D image data
• CPC Class Codes: G06T 2213/08 : Animation software package
• CPC Class Codes: G06T 2215/16 : Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

An augmented reality display system comprises a passable world model data comprises a set of map points corresponding to one or more objects of the real world. The augmented reality system also comprises a processor to communicate with one or more individual augmented reality display systems to pass a portion of the passable world model data to the one or more individual augmented reality display systems, wherein the piece of the passable world model data is passed based at least in part on respective locations corresponding to the one or more individual augmented reality display systems.

Claims

1. An augmented reality display system, comprising:

one or more sensors configured to identify an inanimate physical object or a virtual object as a totem and to capture data pertaining to an interaction of the user of the augmented reality display system with the totem;

a processor configured to determine a user input based at least in part on the data pertaining to the interaction of the user with the totem; and

in response to the user input, an optical sub-system configured to emit light to at least one eye of the user to render a virtual content perceived by the user in relation to the totem that is also perceived by the user through the augmented reality display system that is wearable by the user, wherein

the virtual content includes one or more virtual input structures that are operatively coupled to the processor and appear to emanate from the totem to assimilate one or more physical input structures of one or more physical input devices that change their appearances as perceived by the user in response to interactions by the user;

the optical sub-system is further configured to present one or more first virtual input elements of the virtual content to the user in response to a first user input that manipulates the inanimate physical object or the virtual object into a first position or orientation with respect to a reference frame; and

the optical sub-system is further configured to present one or more second virtual input elements of the virtual content to the user in response to a second user input that manipulates the inanimate physical object or the virtual object into a second position or orientation with respect to the reference frame.

2. The augmented reality display system of claim 1, further comprising a database to store a correlation map, wherein the correlation map comprises a set of predetermined characteristics of the interaction with the totem and a corresponding set of user input commands, wherein the user input is determined based at least in part on the stored correlation map.

3. The augmented reality display system of claim 2, wherein the at least one characteristic pertains to a movement of the totem, a direction of the movement of the totem, or placement of the totem relative to a real-world, and the data pertains to a relative movement between the user and the totem.

4. The augmented reality display system of claim 2, wherein the optical sub-system is further configured to emit the light to the eyes of the user to render the virtual content so that the virtual content appears to be locked to a reference frame of the totem.

5. The augmented reality display system of claim 1, wherein the optical sub-system is further configured to emit the light to the eyes of the user to render the virtual content by rendering one or more virtual structures as one or more virtual input structures on the totem, and the one or more virtual structures correspond to one or more respective functions, one or more respective categories of functions, one or more respective categories of contents, one or more respective media types, one or more respective tools, or one or more respective applications.

6. The augmented reality display system of claim 5, wherein the one or more sensors are further configured to identify the interaction of the user with the one or more virtual structures rendered on the totem to determine the user input, and the totem includes no physical input structures or electronics to provide the user input.

7. The augmented reality display system of claim 1, wherein a predetermined reference frame is consulted to determine the interaction of the user with the totem, and the predetermined reference frame comprises a head-centric reference frame, a hand-centric reference frame, a body-centric reference frame, or a world-centric reference frame.

8. The augmented reality display system of claim 1, wherein the optical sub-system is further configured to emit the light to the eyes of the user to render the virtual content by rendering one or more virtual user interfaces according to a movement, an orientation, or a position or a change in the movement, the orientation, or the position of the totem.

9. The augmented reality display system of claim 1, wherein the totem includes one or more first devices that are used to replicate a physical sensation of the interaction of the user with the totem.

10. The augmented reality display system of claim 1, further comprising:

one or more first devices that are used to generate tactile feedback to the user in response to the interaction that causes the virtual content rendered in relation to the real-world image of the totem to interact with an object.

11. The augmented reality display system of claim 1, further comprising:

detecting a position or a movement of the totem in response to the interaction of the user;

determining a mapping between the interaction of the user and the position or the movement of the totem;

determining one or more control functions based at least in part upon the mapping; and

providing one or more corresponding inputs that correspond to the one or more control functions to a computing device.

12. The augmented reality display system of claim 1, wherein a real object is pre-designated as the totem.

13. The augmented reality display system of claim 1, further comprising an optical apparatus to render a virtual user interface in relation to the totem.

14. The augmented reality display system of claim 1, wherein the data pertains to a number of interactions of the user with the totem that comprises a virtual object rendered by the optical sub-system, and the one or more sensors comprises at least one of an image-based sensor, a haptic sensor, or a depth sensor.

15. A machine implemented method for rendering virtual contents in relation to a totem, comprising:

identifying, with at least one or more sensors, an inanimate physical object or a virtual object as a totem;

capturing, at the one or more sensors, data pertaining to an interaction of the user of an augmented reality display system with the totem;

determining a user input based at least in part on the data pertaining to the interaction of the user with the totem; and

in response to the user input,

emitting light to at least one eye of the user to render a virtual content perceived by the user in relation to the totem that is also perceived by the user through the augmented reality display system that is wearable by the user, wherein

the virtual content includes one or more virtual input structures that are operatively coupled to a processor and appear to emanate from the totem to assimilate one or more physical input structures of one or more physical input devices that change their appearances as perceived by the user in response to interactions by the user;

rendering the virtual content at least by presenting one or more first virtual input elements of the virtual content to the user in response to a first user input that manipulates the inanimate physical object or the virtual object into a first position or orientation with respect to a reference frame; and

rendering the virtual content at least by presenting one or more second virtual input elements of the virtual content to the user in response to a second user input that manipulates the inanimate physical object or the virtual object into a second position or orientation with respect to the reference frame.

16. The machine implemented method of claim 15, emitting the light to the eyes of the user to render the virtual content comprising:

determining a reference frame of the totem; and

rendering the virtual content that appears to be locked to a reference frame of the totem.

17. The machine implemented method of claim 15, emitting the light to the eyes of the user to render the virtual content comprising:

rendering one or more virtual structures as one or more virtual input structures on the totem, wherein

the one or more virtual structures correspond to one or more respective functions, one or more respective categories of functions, one or more respective categories of contents, one or more respective media types, one or more respective tools, or one or more respective applications.

18. The machine implemented method of claim 17, further comprising:

identifying, with the one or more sensors, the interaction of the user with the one or more virtual structures that are rendered on the totem;

determining, at a micro-processor, the user input based in part or in whole upon the interaction of the user with the one or more virtual structures, wherein the totem includes no physical input structures or electronics to provide the user input; and

transmitting one or more signals corresponding to the user input to a computing system.

19. The machine implemented method of claim 15, further comprising:

identifying a movement, an orientation, or a position or a change in the movement, the orientation, or the position of the totem; and

rendering one or more virtual user interfaces according to the position or the change in the movement, the orientation, or the position of the totem.

20. The machine implemented method of claim 15, further comprising:

generating one or more additional user interface elements in relation to the totem in the virtual content in response to a user interaction with at least one virtual input structure of the one or more virtual input structures.

21. The machine implemented method of claim 15, further comprising:

rendering one or more user interface elements on the totem or on one or more corresponding physical features of the totem as if the one or more user interface elements reside on the totem.