Method and apparatus for immersive system interfacing

• US 10,048,760 B2
• Filed: 05/24/2013
• Issued: 08/14/2018
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Abercrombie, Nathan
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US13/902,740
• Publication Number:
  US 20140351770A1
• Time in Patent Office: 1,908 Days
• Field of Search: 715863, 715763, 715781
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Disclosed are methods and systems for immersive system interfacing. A stimulus is defined in the form of a gesture or posture with an end-effector such as a hand. The stimulus is sensed within an environment and communicated to a processor. In response to the stimulus the processor generates a user interface, the interface being outputted near the end-effector. The stimulus may be identical for all times, all places, and all conditions in the environment. The interface may be generated and outputted in response to the stimulus for all times, places, and conditions in the environment. The apparatus includes a processor adapted to define a stimulus in the form of an end-effector gesture or posture, and to generate a user interface in response to the stimulus. A sensor is adapted to sense the stimulus, and a display is adapted to output the interface to an environment.

Claims

1. A method comprising:

defining, by a processor of a head-mounted device, a first hand gesture by a user for a first augmented-reality user interface;

sensing, by a sensor of the head-mounted device, the first hand gesture in a three-dimensional environment;

in response to sensing the first hand gesture, generating, by the processor, the first augmented-reality user interface, wherein the first augmented-reality user interface includes;

a first manipulation site configured to receive a first input from a first digit of a hand of the user; and

a second manipulation site configured to receive a second input from a second digit of the hand of the user;

displaying, by a display of the head-mounted device, the first manipulation site at a first point proximate to and aligned with the first digit of the hand; and

displaying, by the display of the head-mounted device, the second manipulation site at a second point proximate to and aligned with the second digit of the hand.

2. The method of claim 1, wherein the three-dimensional environment comprises a real-world environment.

3. The method of claim 1, wherein the three-dimensional environment comprises an augmented reality environment.

4. The method of claim 1, wherein the three-dimensional environment comprises a virtual reality environment.

5. The method of claim 1, wherein the first hand gesture comprises a hand posture.

6. The method of claim 1, wherein the first hand gesture comprises at least three digits of the hand being substantially extended.

7. The method of claim 6, wherein the three digits include a thumb, a first finger, and a second finger of the hand.

8. The method of claim 6 wherein sensing the first hand gesture comprises determining a position of at least three digits of the hand.

9. The method of claim 6 wherein the first hand gesture is a substantially static posture.

10. The method of claim 6 wherein the first hand gesture is as a dynamic gesture.

11. The method of claim 1, wherein the sensor is an optical sensor.

12. The method of claim 1, wherein the first hand gesture comprises at least three digits of the hand being at least partially spread apart.

13. The method of claim 12, wherein the at least partial spreading apart comprises at least two gaps of at least partial spreading between at least three groups of at least one of the three digits.

14. The method of claim 1, wherein the first hand gesture comprises transitioning from a first configuration of the hand to a second configuration of the hand.

15. The method of claim 1, wherein the first hand gesture comprises transitioning from a first configuration of the hand wherein at least a thumb, a first finger, and a second finger of the hand are substantially clenched together to a second configuration of the hand wherein at least the thumb, the first finger, and the second finger of the hand are substantially extended.

16. The method of claim 1, wherein the first hand gesture comprises transitioning from a first configuration of the hand wherein at least a thumb, a first finger, and a second finger of the hand are pinched together so as to be in contact at tips thereof to a second configuration of the hand wherein at least the thumb, the first finger, and the second finger of the hand are substantially extended.

17. The method of claim 1, further comprising outputting the first augmented-reality user interface so as to be substantially aligned with a plane defined by tips of the first digit and the second digit.

18. The method of claim 1, wherein the first augmented-reality user interface is substantially aligned with a plane defined by a palm of the hand.

19. The method of claim 1, further comprising displaying the first augmented-reality user interface in front of the hand.

20. The method of claim 1, further comprising displaying at least a portion of the first augmented-reality user interface so as to be in contact with at least one of the digits of the hand.

21. The method of claim 1, further comprising

defining, by the processor, an input manipulation for the manipulation site;

and

in response to the input manipulation being applied to the manipulation site, executing an instruction corresponding to the input manipulation.

22. The method of claim 1, wherein the first manipulation site comprises a contact site.

23. The method of claim 1, wherein the first augmented-reality user interface comprises a graphical user interface.

24. The method of claim 1, wherein the first augmented-reality user interface comprises a visible marking corresponding with the first manipulation site.

25. The method of claim 1, wherein the first augmented-reality user interface includes a third manipulation site configured to receive a third input from the second digit of the hand of the user.

26. The method of claim 1, wherein the first manipulation site and the second manipulation site are disposed substantially in at least a partial ring substantially circumscribing at least a portion of the hand.

27. The method of claim 1, wherein the first manipulation site and the second manipulation site are disposed substantially adjacent so as to be substantially continuous.

28. The method of claim 1, wherein the first manipulation site and the second manipulation site are discrete manipulation sites.

29. The method of claim 1, further comprising:

defining, by the processor, an active condition for the first manipulation site;

generating, by the processor, a input when the sensor detects an input manipulation is applied at the first manipulation site; and

defining, by the processor, an inactive condition for the first manipulation site, wherein the processor does not generate the input when the sensor detects the input manipulation applied at the first manipulation site.

30. The method according to claim 29, further comprising switching the first manipulation site from the inactive condition to the active condition.

31. The method according to claim 29, further comprising:

defining, by the processor, a first switch manipulation to switch the first manipulation site from the active condition to the inactive condition; and

defining, by the processor, a second switch manipulation to switch the first manipulation site from the inactive condition to the active condition.

32. The method according to claim 29, further comprising displaying the first manipulation site when the first manipulation site is in the inactive condition.

33. The method of claim 1, further comprising:

displaying the first manipulation site in at least a first partial ring; and

displaying the second manipulation site in at least a second partial ring, wherein the first partial ring and the second partial ring are disposed substantially in substantially concentric partial rings substantially circumscribing at least a portion of said hand.

34. The method of claim 1, further comprising displaying the first manipulation site proximate a tip of one of the first digit.

35. The method of claim 1, further comprising configuring the first augmented-reality user interface based at least in part on at least one of:

a status of the three-dimensional environment,

a content of the three-dimensional environment,

a location within the three-dimensional environment of the first augmented-reality user interface,

a location of the three-dimensional environment,

a time,

conditions within the three-dimensional environment,

conditions external to the three-dimensional environment,

an identity of the user,

a condition of the user, or

a non-input behavior of the user.

36. The method of claim 1, further comprising generating content of the first user interface based at least in part on at least one of:

a status of the three-dimensional environment,

a content of the three-dimensional environment,

a location within the three-dimensional environment of the first user interface,

a location of the three-dimensional environment,

a time,

conditions within the three-dimensional environment,

conditions external to the three-dimensional environment,

an identity of the user,

a condition of the user, or

a non-input behavior of the user.

37. The method of claim 1, further comprising defining the first hand gesture to be substantially identical for substantially all times, places, and conditions in the three-dimensional environment.

38. The method of claim 1, further comprising:

defining, by the processor of the head-mounted device, a second hand gesture by the user for a second augmented-reality user interface;

sensing, by the sensor of the head-mounted device, the second hand gesture in the three-dimensional environment;

in response to sensing the second hand gesture, generating, by the processor, the second augmented-reality user interface, wherein the second augmented-reality user interface includes;

a third manipulation site configured to receive a third input from the first digit of the hand of the user, wherein the third manipulation site is different than the first manipulation site; and

a fourth manipulation site configured to receive a fourth input from the second digit of the hand of the user, wherein the fourth manipulation site is different than the second manipulation site;

displaying, by the display of the head-mounted device, the third manipulation site at a third point proximate to and aligned with the first digit of the hand; and

displaying, by the display of the head-mounted device, the fourth manipulation at a fourth point proximate to and aligned with the second digit of the hand.

39. The method of claim 38, wherein the first augmented reality user interface and the second augmented reality user interface are different.

40. The method of claim 1, wherein the first augmented-reality user interface includes:

a third manipulation site configured to receive a third input from the first digit of the first hand; and

a fourth manipulation site configured to receive a fourth input from the second digit of the first hand.

41. An apparatus, comprising:

a sensor adapted to sense a first hand gesture;

a processor coupled to the sensor, the processor adapted to;

define the first hand gesture by a user for a first augmented-reality user interface;

in response to sensing the first hand gesture, generate a first user interface,

wherein the first augmented-reality user interface includes;

a first manipulation site configured to receive a first input from a first digit of a hand of the user; and

a second manipulation site configured to receive a second input from a second digit of the hand of the user;

a display coupled to the processor, the display operable to;

display the first manipulation site at a first point proximate to and aligned with the first digit of the hand; and

display the second manipulation site at a second point proximate to and aligned with the second digit of the hand.

42. The apparatus of claim 41, wherein the display is adapted to display the first augmented-reality user interface in a physical environment.

43. The apparatus of claim 41, wherein the display comprises a stereo display.

44. The apparatus of claim 41, wherein the sensor comprises a stereo sensor.

45. The apparatus of claim 41, wherein the sensor comprises an image sensor.

46. The apparatus of claim 41, wherein the processor, the display, and the sensor are disposed on a wearable device.

47. The apparatus of claim 41, wherein the processor, the display, and the sensor are disposed on a head-mounted display.

48. An apparatus comprising:

means for defining a first hand gesture by a user for an augmented-reality user interface;

means for sensing the first hand gesture;

means for generating a user interface in response to sensing the first hand gesture, wherein the augmented-reality user interface includes;

a first manipulation site configured to receive a first input from a first digit of a hand of the user; and

a second manipulation site configured to receive a second input from a second digit of the hand of the user;

means for displaying the first manipulation site at a first point proximate to and aligned with the first digit of the hand; and

means for displaying the second manipulation site at a second point proximate to and aligned with the second digit of the hand.

49. A method comprising:

defining, by a processor, a first hand gesture by a user for a first augmented-reality user interface;

defining, by the processor, a second hand gesture by the user for a second augmented-reality user interface;

sensing, by a sensor, the first hand gesture in a three-dimensional environment;

sensing, by the sensor, the second hand gesture in the three-dimensional environment;

in response to sensing the first hand gesture, generating, by the processor, the first augmented-reality user interface, wherein the first augmented-reality user interface includes;

a first manipulation site configured to receive a first input from a first digit of a hand of the user; and

a second manipulation site configured to receive a second input from a second digit of the hand of the user;

in response to sensing the second hand gesture, generating, by the processor, the second user augmented-reality interface, wherein the second augmented-reality user interface includes;

a third manipulation site configured to receive a third input from the first digit of a hand of the user, wherein the third manipulation site is different than the first manipulation site; and

a fourth manipulation site configured to receive a fourth input from the second digit of the hand of the user, wherein the fourth manipulation site is different than the second manipulation site;

displaying, by a display, the first manipulation site at a first point proximate to and aligned with the first digit of the hand;

displaying, by the display, the second manipulation site at a second point proximate to and aligned with the second digit of the hand;

displaying, by the display, the third manipulation site at a third point proximate to and aligned with the first digit of the hand; and

displaying, by the display, the fourth manipulation at a fourth point proximate to and aligned with the second digit of the hand.

50. An apparatus, comprising:

a processor adapted to;

define a hand gesture by a user for a augmented-reality user interface;

in response to sensing the hand gesture, generate a first user interface, wherein the augmented-reality user interface includes;

a first manipulation site configured to receive a first input from a digit of a first hand of the user; and

a second manipulation site configured to receive a second input from a digit of a second hand of the user; and

a display coupled to the processor, the display operable to;

display the first manipulation site at a first point proximate to the digit of the first hand; and

display the second manipulation site at a second point proximate to the digit of the second hand.