Gesture based user interface for augmented reality

• US 9,547,438 B2
• Filed: 06/21/2011
• Issued: 01/17/2017
• Est. Priority Date: 06/21/2011
• Status: Expired due to Fees

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Bromer, Nicholas S.
• Primary Examiner(s): Ghebretinsae, Temesghen
• Assistant Examiner(s): LANDIS, LISA S
• Application Number:
  US13/320,904
• Publication Number:
  US 20120326961A1
• Time in Patent Office: 2,037 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Technologies are generally described for systems and methods effective to provide a gesture keyboard that can be utilized with a virtual display. In an example, the method includes receiving sensory information associated with an object in proximity to, or in contact with, an input device including receiving at least one level of interaction differentiation detected from at least three levels of interaction differentiation, interpreting a command from the sensory information as a function of the at least one level of interaction differentiation, and outputting an action indication based on the command.

Claims

1. A method, comprising:

receiving sensory information associated with an object in proximity to, or in contact with, an input device including receiving at least three different levels of interaction differentiation comprising a proximity level of interaction differentiation, an electrical continuity level of interaction differentiation, and an actuation level of interaction differentiation;

analyzing the at least one level of interaction differentiation, comprising;

determining the proximity level of interaction differentiation as a function of capacitance between the object and the input device, wherein the proximity level of interaction differentiation identifies a lateral position of the object with respect to the input device and a height of the object above a plane of the input device based on a trigonometric function of a first beat frequency in a first resonant circuit associated with a first character input component of the input device and a second beat frequency in a second resonant circuit associated with a second character input component of the input device,

sensing the electrical continuity level of interaction differentiation based on detection of an electrical contact between a surface of at least one component of the input device and the object, and

determining the actuation level of interaction differentiation based on movement of at least a portion of the input device by the object;

determining a command from the sensory information based on a combination of the at least three levels of interaction differentiation, wherein a respective command for each combination of the at least three levels of interaction differentiation is distinguishable from other respective commands for other combinations of the at least three levels of interaction differentiation, comprising interpreting the command selected from different sets of commands based on whether the interaction differentiation is the proximity level, the electrical continuity level, or the actuation level; and

outputting an action indication based on the command.

2. The method of claim 1, wherein the ascertaining the height comprises:

comparing a first resonant frequency of a first component of the input device to a second resonant frequency of a second component of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency; and

measuring a distance between the object and the input device based on the lowest resonant frequency including determining that the object is closer to the first component or the second component based on the lowest resonant frequency.

3. The method of claim 1, wherein the ascertaining the height comprises:

comparing responses of components of the input device; and

applying the trigonometric function to the responses to ascertain the height of the object indirectly.

4. The method of claim 1, wherein the locating the lateral position comprises:

determining a first resonant frequency of a first component of the input device and a second resonant frequency of a second component of the input device; and

determining the lateral position as a function of a local minima between the first resonant frequency and the second resonant frequency.

5. The method of claim 1, wherein the locating the lateral position comprises:

detecting a difference between a first resonant frequency of a first component and a second resonant frequency of a second component adjacent to the first component; and

determining that the object is closer to the first component or the second component as a function of the difference.

6. The method of claim 1, wherein the outputting the action indication comprises highlighting an item on a virtual display that comprises a first portion and a second portion, wherein the first portion comprises the command associated with the action indication and the second portion comprises a virtual representation of the input device.

7. The method of claim 6, wherein the highlighting the item is performed in response to the object being in physical contact with the input device.

8. The method of claim 1, wherein the receiving sensory information comprises:

detecting two or more objects in proximity to, or in contact with, the input device at a same time; and

identifying the two or more objects as a single object for purposes of a gesture command, wherein the receiving the sensory information includes receiving the sensory information associated with the single object.

9. The method of claim 1, wherein the outputting the action indication comprises transmitting, to a display device, a signal including the action indication and an interpretation of the command for output by the display device.

10. The method of claim 1, wherein the determining the command comprises interpolating motion of the object.

11. The method of claim 10, wherein the interpolating comprises:

receiving data from a plurality of components of the input device; and

based on the data, constructing a smooth line to locate the object.

34. The method of claim 1, wherein the determining the command comprises selecting the command from a plurality of command gestures, wherein each command gesture from the plurality of command gestures corresponds to a different command.

35. The method of claim 34, wherein the plurality of command gestures comprise a set of proximity-level gesture commands and a set of command gestures.

37. The method of claim 1, further comprising:

measuring a beat frequency that represents an extent to which the object is closer to a first key than to a second key in a key pair.

38. The method of claim 3, wherein the applying the trigonometric function further comprises applying the trigonometric function based on a location of the components of the input device.

12. A system, comprising:

a processor, coupled to a memory, configured to execute or facilitate execution of computer-executable instructions to perform operations, the operations comprising;

receiving at least one gesture within a space relative to a keyboard comprising a plurality of keys;

identifying a command based on the at least one gesture, wherein the command is selected from a set of alternative commands determined as a function of a combination of inputs received from one or more of an electrical continuity level sensed based on ground information related to a grounding of a surface of a key of the plurality of keys by an object, a proximity detected from movement of the object based on identifying a lateral location of the object with respect to the keyboard and a height of the object above a plane of the keyboard based on a trigonometric function of a first beat frequency of a first resonant circuit associated with a first key of the plurality of keys and a second beat frequency of a second resonant circuit associated with a second key of the plurality of keys, and an actuation detected from movement of the key of the plurality of keys by the object, wherein the set of alternative commands are different for different combinations of the electrical continuity level, the proximity, and the actuation; and

rendering information of the at least one gesture and a result of the command, wherein the information is configured to be rendered on a virtual display.

13. The system of claim 12, wherein the operations further comprise ascertaining the proximity by detecting cursor control and is configured to ascertain that a mechanical actuation indicates manual commands, and wherein the electrical continuity facilitates a highlighting of a key in the virtual display.

14. The system of claim 12, wherein the operations further comprise detecting the movement in the space relative to and near the keyboard and detect pressure applied to the keyboard.

15. The system of claim 14, wherein the operations further comprise highlighting at least a portion of the virtual display in response to detection of the pressure applied to the keyboard.

16. The system of claim 12, further comprising a sensor configured to measure a capacitance, wherein the operations further comprise determining the height and the lateral location of the object above the keyboard based on the capacitance.

17. The system of claim 16, wherein the object and a key of the keyboard form a capacitor of a resonant circuit, and wherein the capacitance between the object and the key is proportional to 1/d, where d is a separation in units of one-half a size of the object and the key, and wherein a frequency of the resonant circuit is a measure of distance.

18. The system of claim 16, wherein the operations further comprise measuring a beat frequency that represents an extent to which the object is closer to a first key than to a second key in a key pair.

19. The system of claim 18, wherein the beat frequency of the first key and the second key disappears in response to the object being between the first key and the second key.

20. The system of claim 12, wherein the virtual display comprises augmented reality spectacles.

21. A non-transitory computer-readable storage device storing executable instructions that, in response to execution, cause a system comprising a processor to perform operations, comprising:

detecting a gesture that indicates at least one command to be performed comprising receiving data indicative of an electrical continuity level based on detection of an electrical contact between a surface of a conductive key on a keyboard and an external element, wherein the data is received in response to the conductive key being determined not to be actuated for typing;

determining the gesture is the at least one command selected from a plurality of commands associated with the electrical continuity level, wherein the plurality of commands are different from a set of commands associated a combination of inputs associated with actuation of the conductive key and data indicative of a proximity level based on a capacitance between the external element and the keyboard, wherein the proximity level identifies a lateral position of the external element with respect to the keyboard and a height of the external element above a plane of the keyboard based on a trigonometric function of a first beat frequency in a first resonant circuit associated with a first key on the keyboard and a second beat frequency in a second resonant circuit associated with a second key on the keyboard; and

initiating a result of the at least one command as a perceivable event within a virtual space.

22. The non-transitory computer-readable storage device of claim 21, wherein the detecting the gesture further comprises:

distinguishing the gesture that indicates the at least one command from a set of common proximity-level gestures, wherein the set of common proximity-level gestures are ignored.

23. The non-transitory computer-readable storage device of claim 21, wherein the detecting the gesture further comprises:

receiving an actuation that indicates the electrical contact between the external element and the conductive key of the keyboard.

24. The non-transitory computer-readable storage device of claim 21, wherein the detecting the gesture further comprises:

comparing a first resonant frequency of a first conductive actuation key of an input device with a second resonant frequency of a second conductive actuation key of the input device to determine a lowest resonant frequency of the first resonant frequency and the second resonant frequency; and

measuring a distance between the external element and the input device based on the lowest resonant frequency including determining that the external element is closer to the first conductive actuation key or the second conductive actuation key based on the lowest resonant frequency.

25. The non-transitory computer-readable storage device of claim 21, wherein the detecting the gesture further comprises:

determining a first resonant frequency of a first conductive actuation key of an input device and a second resonant frequency of a second conductive actuation key of the input device; and

determining the lateral position of the external element as a function of a local minima between the first resonant frequency and the second resonant frequency.

26. The non-transitory computer-readable storage device of claim 21, wherein the interpreting the gesture comprises ascertaining the gesture is at least one of a sweep gesture, a push-pull gesture, a circle gesture, a deflection gesture, an expansion-contraction gesture, or a rotating gesture.

27. The non-transitory computer-readable storage device of claim 21, wherein the initiating the result comprises highlighting an item on a virtual display that comprises a first portion and a second portion, illustrating a command associated with the gesture by the first portion, and presenting a representation of an input device by the second portion.

28. The non-transitory computer-readable storage device of claim 27, wherein the highlighting the item is in response to physical contact between the external element and the input device.

36. The non-transitory computer-readable storage device of claim 21, wherein the operations further comprise:

measuring a beat frequency that represents an extent to which the object is closer to a first key than to a second key in a key pair.

29. A device, comprising:

a keyboard comprising;

an array of keys, wherein at least a subset of keys of the array of keys comprise a respective displacement actuated switch configured to detect pressure applied to a respective key of at least the subset of keys and wherein at least the subset of keys comprise respective conductive touch surfaces; and

at least one capacitive sensor comprising two or more resonant circuits, configured to detect an object near the keyboard based on identifying a lateral position of the object with respect to the keyboard and a height of the object above a plane of the keyboard based on a trigonometric function of a first beat frequency of a first resonant circuit associated with a key of the array of keys and a second beat frequency of a second resonant circuit associated with another key of the array of keys;

an electrical contact sensor that detects an electrical continuity in relation to the object touching a conductive touch surface without application of at least a threshold pressure to the keyboard;

a processor, coupled to a memory, configured to execute or facilitate execution of computer-executable instructions to perform operations, the operations comprising;

translating a gesture near, or applied to, the keyboard into a command, wherein the command is based on a combination of inputs associated with the pressure, the object, and the electrical continuity selected from a set of commands associated with the pressure, the object, and the object touching the conductive surface, wherein respective commands of the first set of commands are based on different combinations of input; and

changing a display as a function of the command.

30. The device of claim 29, wherein the keyboard is configured to detect an electrical contact between the object and at least the subset of keys to receive interaction information from a capacitive interaction, a conductive interaction, or a mechanical interaction.

31. The device of claim 29, wherein the display is a remote virtual display and the processor is further configured to generate signals and transmit the signals to the remote virtual display.

32. The device of claim 29, wherein the operations further comprise accessing a data store that comprises information representing a set of command gestures corresponding to different commands.

33. The device of claim 32, wherein the data store further comprises other information representing a set of common proximity-level gestures that are distinguishable from the set of command gestures.