Methods and systems for creating virtual and augmented reality

• US 10,203,762 B2
• Filed: 06/13/2015
• Issued: 02/12/2019
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Bradski, Gary R., Miller, Samuel A., Abovitz, Rony
• Primary Examiner(s): Nakhjavan, Shervin
• Application Number:
  US14/738,877
• Publication Number:
  US 20160026253A1
• Time in Patent Office: 1,340 Days
• Field of Search: None
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 30/52 the 3D volume being constructed from a stack or sequence of 2D planes, e.g. depth sampling systems
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 13/167 : Synchronising or controlling image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/395 : with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 21/414 : Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Claims

1. An augmented reality display system, comprising:

an image capturing device of the augmented reality display system to capture at least one image, wherein

the image capturing device comprises one or more image capturing sensors,

at least a portion of the at least one image is perceived within a field of view of a user, and

the at least one image captures at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality display system to the user; and

a processor coupled directly with no intervening elements or indirectly with one or more intervening elements to the image capturing device to recognize the at least one gesture as at least one recognized gesture, the processor configured to recognize the at least one gesture as the at least one recognized gesture is further configured to;

identify a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;

determine whether the at least one image includes one or more identifiable depth points at least by performing a line search for the at least one image with one or more lines or line segments;

determine an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon the plurality of computational resource utilization or expense requirements;

during one or more earlier stages in the order of processing, generate one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements; and

during one or more later stages in the order of processing, determine and recognize the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational expense criterion and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein

the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and

the processor further configured to determine a user input based at least in part on the at least one recognized gesture.

2. The augmented reality display system of claim 1, wherein the processor is configured to generate a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures and to recognize the at least one gesture when the scoring value exceeds a threshold value.

3. The augmented reality display system of claim 1, further comprising a database to store predetermined gestures, wherein the computational resource utilization or expense requirement includes reducing or minimizing computational resource utilization for the gesture recognition for the at least one gesture, the first computational resource or expense requirement corresponds to a relatively lower computation resource utilization, and the second computational resource or expense requirement corresponds to a relatively higher computational resource utilization when compared to the first computational resource or expense requirement.

4. The augmented reality display system of claim 3, further comprising a networked memory to access the database of predetermined gestures.

5. The augmented reality display system of claim 1, wherein the processor is further configured to recognize the at least one gesture that comprises a hand gesture or motion or a finger gesture or a finger motion.

6. The augmented reality display system of claim 1, wherein the augmented reality display system comprises a user wearable apparatus to display a virtual world as well as at least a portion of a physical environment in which the user is located.

7. The augmented reality display system of claim 1, where the processor is further configured to recognize the at least one gesture that comprises an inter-finger interaction.

8. The augmented reality display system of claim 1, wherein the processor is further configured to recognize the at least one gesture comprising at least one of inter-finger interactions, pointing, tapping, or rubbing.

9. The augmented reality display system of claim 1, further comprising a spatial light modulator that is coupled directly without intervening elements or indirectly with one or more elements to the processor, and the processor controls the spatial light modulator in a manner such that one or more virtual objects are displayed to the user based at least in part on the user input.

10. The augmented reality display system of claim 9, further comprising a virtual user interface to receive the user input or a user interaction with the virtual user interface or with the one or more virtual objects.

11. A method for determining user input, comprising:

capturing an image corresponding to a field of view of a user through an augmented reality system, wherein

the image comprises a gesture image of at least one gesture that is created by the user and interacts with virtual content projected by the augmented reality system to the user, and

at least a portion of the image is perceived by the user within the field of view provided by the augmented reality system;

identifying a plurality of candidate gestures and a plurality of computational utilization or expense requirements for gesture recognition of the at least one gesture;

determining whether the image includes one or more identifiable depth points at least by performing a line search for the image with one or more lines or line segments;

determining an order of processing in which a plurality of analysis nodes is executed to perform respective gesture identification processes on the at least one gesture with respect to the plurality of candidate gestures based at least in part upon a plurality of computational resource utilization or expense requirements;

during one or more earlier stages in the order of processing, generating one or more reduced sets of candidate gestures from the plurality of candidate gestures at least by executing one or more first gesture identification processes of the respective gesture identification processes that analyze the at least one image with a first analysis node to remove one or more candidate gestures from the plurality of candidate gestures based at least in part upon a first computational resource utilization or expense requirement of the plurality of computational resource utilization or expense requirements;

during one or more later stages in the order of processing, determining and recognizing the at least one gesture from the one or more reduced sets of candidate gestures as the at least one recognized gesture at least by analyzing the at least one image based at least in part on the one or more reduced sets of candidate gestures with a second analysis node corresponding to a second computational resource or expense requirement and at least by executing one or more second gesture identification processes on the at least one gesture and the one or more reduced sets of candidate gestures, wherein

the one or more second gesture identification processes consume a larger amount of processing power than the one or more first gesture identification processes; and

determining a user input based in part or in whole upon the at least one recognized gesture.

12. The method of claim 11, further comprising generating a scoring value for a set of points for the at least one gesture based in part or in whole on results of comparing the set of points to a first set of points associated with a database including predetermined gestures.

13. The method of claim 12, further comprising recognizing the at least one gesture when the scoring value exceeds a threshold value.

14. The method of claim 11, further comprising overlaying a virtual world with at least a portion of a physical environment in which the user is located.

15. The method of claim 14, further comprising accessing a networked memory to access a database including predetermined gestures.

16. The method of claim 11, determining the user input further comprising recognizing the at least one gesture that comprises a hand gesture, a hand motion, a finger gesture, or a finger motion.

17. The method of claim 11, further comprising capturing and recognizing a sensory input from the user or from a physical environment in which the user is located.

18. The method of claim 11, determining the user input further comprising recognizing the at least one gesture that comprises an inter-finger interaction.

19. The method of claim 11, determining the user input further comprising recognizing the at least one gesture that comprises at least one of inter-finger interactions, pointing, tapping, or rubbing.

20. The method of claim 11, further comprising displaying one or more virtual objects to the user based at least in part on the user input.

21. The method of claim 20, further comprising recognizing a user interaction with the one or more virtual objects or with a virtual world in which the one or more virtual objects are displayed in a virtual user interface based in part or in whole upon the at least one gesture.

22. The method of claim 11, further comprising:

generating, at a processor of the augmented reality system, a scoring value for a set of points identified for the at least one gesture based at least in part on comparison between the set of points and predetermined gestures; and

recognizing the at least one gesture as the at least one recognized gesture when the scoring value exceeds a threshold value.