METHODS AND SYSTEMS FOR CREATING VIRTUAL AND AUGMENTED REALITY

• US 20160026253A1
• Filed: 06/13/2015
• Published: 01/28/2016
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Bradski, Gary R., Miller, Samuel A., Abovitz, Rony
• Granted Patent: US 10,203,762 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 30/52 the 3D volume being constructed from a stack or sequence of 2D planes, e.g. depth sampling systems
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G10L 19/00 Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis in musical instruments G10H
• CPC Class Codes: H04N 13/128 : Adjusting depth or disparity
• CPC Class Codes: H04N 13/167 : Synchronising or controlling image signals
• CPC Class Codes: H04N 13/194 : Transmission of image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/332 : Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/395 : with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 21/414 : Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Claims

1-24. -24. (canceled)

25. An augmented reality display system, comprising:

an image capturing device to capture one or more images, wherein the one or more images corresponding to a field of view of a user, and at least one image of the one or more images captures at least one gesture created by the user; and

a processor coupled directly with no intervening elements or indirectly with one or more intervening elements to the image capturing device to identify a set of points as associated with the at least one gesture, to compare the set of points against a database including predetermined gestures, to recognize the at least one gesture based at least in part on comparison results, and to determine a user input based at least in part on the at least one gesture that has been recognized.

26. The augmented reality display system of claim 25, wherein the processor generates a scoring value for the set of points based on the comparison.

27. The augmented reality display system of claim 26, wherein the processor recognizes the at least one gesture when the scoring value exceeds a threshold value.

28. The augmented reality display system of claim 25, further comprising the database to store the predetermined gestures.

29. The augmented reality display system of claim 28, further comprising a networked memory to access the database of predetermined gestures.

30. The augmented reality display system of claim 25, wherein the at least one gesture comprises a hand gesture or motion or a finger gesture or a finger motion.

31. The augmented reality display system of claim 25, wherein the augmented reality display system comprises a user wearable apparatus to display a virtual world as well as at least a portion of a physical environment in which the user is located.

32. The augmented reality display system of claim 25, where the at least one gesture comprises an inter-finger interaction.

33. The augmented reality display system of claim 25, wherein the at least one gesture comprising at least one of inter-finger interactions, pointing, tapping, or rubbing.

34. The augmented reality display system of claim 25, further comprising a spatial light modulator, wherein the spatial light modulator is coupled directly without intervening elements or indirectly with one or more elements to the processor, and the processor controls the spatial light modulator in a manner such that one or more virtual objects are displayed to the user based at least in part on the user input.

35. The augmented reality display system of claim 34, further comprising a virtual user interface to receive the user input or a user interaction with the virtual user interface or with the one or more virtual objects.

36. A method for determining user input, comprising:

capturing an image of a field of view of a user, the image comprising a gesture created by the user;

identifying a set of points associated with the gesture at least by analyzing the image that has been captured;

comparing the set of points to a first set of points associated with a database including predetermined gestures; and

determining a user input based in part or in whole on the gesture.

37. The method of claim 36, further comprising generating a scoring value for the set of points based in part or in whole on results of comparing the set of points to the first set of points.

38. The method of claim 37, further comprising recognizing the gesture when the scoring value exceeds a threshold value.

39. The method of claim 36, further comprising overlaying a virtual world with at least a portion of a physical environment in which the user is located.

40. The method of claim 39, further comprising accessing a networked memory to access the database including predetermined gestures.

41. The method of claim 36, wherein the gesture comprises a hand gesture, a hand motion, a finger gesture, or a finger motion.

42. The method of claim 36, further comprising capturing and recognizing a sensory input from the user or from a physical environment in which the user is located.

43. The method of claim 36, wherein the gesture comprises an inter-finger interaction.

44. The method of claim 36, wherein the gesture comprises at least one of inter-finger interactions, pointing, tapping, or rubbing.

45. The method of claim 36, further comprising displaying one or more virtual objects to the user based at least in part on the user input.

46. The method of claim 45, further comprising recognizing a user interaction with the one or more virtual objects or with a virtual world in which the one or more virtual objects are displayed in a virtual user interface based in part or in whole upon the gesture.

47-447. -447. (canceled)