Systems, devices, and methods for touch-free typing

• US 10,203,812 B2
• Filed: 10/09/2014
• Issued: 02/12/2019
• Est. Priority Date: 10/10/2013
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay
• Primary Examiner(s): Soto Lopez, Jose
• Application Number:
  US15/028,098
• Publication Number:
  US 20160253044A1
• Time in Patent Office: 1,587 Days
• CPC Class Codes: G06F 2203/04108 Touchless 2D- digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface without distance measurement in the Z direction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Systems, devices, methods, and non-transitory computer-readable media are provided for receiving data input via touch-free gestures and movements. For example, a data input device includes at least one processor for receiving information from a sensor. The processor may be configured to receive sensor data from the sensor of a user'"'"'s hand spaced a distance from a displayed keyboard and in non-contact with the displayed keyboard, and track, using the received sensor data, one or more fingers in air a distance from the displayed keyboard image. The processor may also be configured to correlate locations of the one or more fingers in the air with images of a plurality of keys in the displayed keyboard, and select keys from the keyboard image based on the correlated locations of the one or more fingers in the air, and a detection of a predefined gesture performed by the user.

Claims

1. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

displaying an image of a keyboard, the displayed keyboard image including a plurality of keys;

receiving, from at least one sensor, sensor data of a user'"'"'s hand spaced a distance from the displayed keyboard image, the user'"'"'s hand being in non-contact with the displayed keyboard image;

tracking, using the received sensor data, one or more fingers of the user'"'"'s hand in air a distance from the displayed keyboard image;

correlating locations of the one or more fingers in the air with images of the plurality of keys, wherein the correlation does not require any indication of the one or more fingers touching the displayed keyboard image; and

selecting keys from the image of the keyboard based on detection of a predefined finger gesture performed by the user'"'"'s hand, and at least one of the correlated locations of the finger in the air before the detected predefined finger gesture or a location of the user'"'"'s hand.

2. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to display the image of the keyboard on a display device.

3. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to display the keyboard image in air, separated from any physical device or surface.

4. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to:

display an image of a virtual mouse at a distance from the user;

detect a first hand pose and a second hand pose;

enable a keyboard mode for selection of at least one key from the displayed keyboard when the first hand pose is detected;

enable a mouse mode for manipulation of the virtual mouse when the second hand pose is detected; and

enable the user to switch between the keyboard mode and the mouse mode by changing between the first hand pose and the second hand pose.

5. The non-transitory computer-readable medium of claim 4, wherein the second hand pose is a generally upward pointing finger.

6. The non-transitory computer-readable medium of claim 4, wherein the second hand pose is a generally closed hand.

7. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to assign areas in space to alphanumeric keys on the keyboard, and wherein the areas in space assigned to each key vary in size depending on criteria including an expected frequency of key usage.

8. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to assign larger areas in space for keys that are more frequently used, and to assign smaller areas in space for keys that are less frequently used.

9. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to dynamically vary an area in space assigned to a particular key based on a prediction that the particular key will be selected next.

10. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor assign larger areas in space to one or more keys when the user hand is still.

11. The non-transitory computer-readable medium of claim 1, wherein the at least one processor avoids overshooting by predicting a next key that will be selected using motion features of the user'"'"'s hand.

12. The non-transitory computer-readable medium of claim 11, further configured to cause the at least one processor to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with at least one location of the user hand or one or more fingers.

13. The non-transitory computer-readable medium of claim 11, further configured to cause the at least one processor to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with at least one change of location, of the user'"'"'s hand location or one or more finger.

14. The non-transitory computer-readable medium of claim 11, further configured to cause the at least one processor to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with the motion features of the user moving hand.

15. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to selectively display the keyboard image and wherein the display of the keyboard image is triggered by detection of at least one raised hand.

16. The non-transitory computer-readable medium of claim 15, further configured to cause the at least one processor to cease display of the keyboard image when the raised hand is lowered.

17. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to monitor a hand position and a finger position in the received sensor data, and wherein the monitored hand position is used to determine a selected keyboard row and the monitored finger position is used to determine a selected key.

18. The non-transitory computer-readable medium of claim 17, wherein the selected keyboard row is determined based on a horizontal motion in the monitored hand position representing forward or backward movement.

19. The non-transitory computer-readable medium of claim 17, wherein the selected keyboard row is determined based on a vertical motion in the monitored hand position representing upward or downward movement.

20. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to determine in advance of a word completion, a likely word that the user is typing, display the likely word, recognize a hand orientation indicative of a user'"'"'s acceptance of the likely word, and to accept the likely word in response to the recognized hand orientation.

21. The non-transitory computer-readable medium of claim 20, wherein the recognized hand orientation includes an animated gesture.

22. The non-transitory computer-readable medium of claim 1, further configured to cause the at least one processor to determine in advance of a word completion, a likely word that the user is typing, display the likely word, recognize a hand orientation indicative of a user'"'"'s rejection of the likely word, and to reject the likely word in response to the recognized hand orientation.

23. The non-transitory computer-readable medium of claim 20, wherein the recognized hand orientation includes an animated gesture.

24. A data input device, comprising:

at least one processor configured to display an image of a keyboard, the displayed keyboard image including a plurality of keys;

receive, from at least one sensor, sensor data of a user'"'"'s hand spaced a distance from the displayed keyboard image, the user'"'"'s hand in non-contact with the displayed keyboard image;

track, using the received sensor data, one or more fingers of the user'"'"'s hand in air a distance from the displayed keyboard image;

correlate locations of the one or more fingers in the air with images of the plurality of keys, wherein the correlation does not require any indication of the one or more fingers touching the displayed keyboard image; and

select keys from the image of the keyboard based on the correlated locations of the one or more fingers in the air, and a detection of a predefined gesture performed by the finger.

25. The data input device of claim 24, wherein the at least one processor is further configured to display the image of the keyboard on a display device.

26. The data input device of claim 24, wherein the at least one processor is further configured to display the keyboard image in air, separated from any physical device or surface.

27. The data input device of claim 24, wherein the at least one processor is further configured to:

display an image of a virtual mouse at a distance from the user;

detect a first hand pose and a second hand pose;

enable a keyboard mode for selection of at least one key from the displayed keyboard when the first hand pose is detected;

enable a mouse mode for manipulation of the virtual mouse when the second hand pose is detected; and

enable the user to switch between the keyboard mode and the mouse mode by changing between the first hand pose and the second hand pose.

28. The data input device of claim 27, wherein the second hand pose is a generally upward pointing finger.

29. The data input device of claim 27, wherein the second hand pose is a generally closed hand.

30. The data input device of claim 24, wherein the at least one processor is further configured to assign areas in space to alphanumeric keys on the keyboard, and wherein the areas in space assigned to each key vary in size depending on criteria including an expected frequency of key usage.

31. The data input device of claim 24, wherein the at least one processor is further configured to cause the at least one processor to assign larger areas in space for keys that are more frequently used, and to assign smaller areas in space for keys that are less frequently used.

32. The data input device of claim 24, wherein the at least one processor is further configured to dynamically vary an area in space assigned to a particular key based on a prediction that the particular key will be selected next.

33. The data input device of claim 24, further configured to cause the at least one processor assign larger areas in space to one or more keys when the user hand is still.

34. The data input device of claim 24, wherein the at least one processor is further configured to predict a next key that will be selected, using motion features of the received images to avoid overshooting.

35. The data input device of claim 34, wherein the at least one processor is further configured to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with at least one location of the user hand or one or more fingers.

36. The data input device of claim 34, wherein the at least one processor is further configured to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with at least one change of location, of the user'"'"'s hand location or one or more finger.

37. The data input device of claim 34, wherein the at least one processor is further configured to predict the next key using at least one of a word completion algorithm and a prior key selected using information associated with the motion features of the user moving hand.

38. The data input device of claim 24, wherein the at least one processor is further configured to selectively cause the keyboard image to be displayed and wherein the display of the keyboard image is triggered by detection of at least one raised hand.

39. The data input device of claim 38, wherein the at least one processor is further configured to cause the keyboard image display to cease when the raised hand is lowered.

40. The data input device of claim 24, wherein the at least one processor is further configured to monitor a hand position and a finger position in the received sensor data, and wherein the monitored hand position is used to determine a selected keyboard row and the monitored finger position is used to determine a selected key.

41. The data input device of claim 40, wherein the selected keyboard row is determined based on a horizontal motion in the monitored hand position representing forward or backward movement.

42. The data input device of claim 40, wherein the selected keyboard row is determined based on a vertical motion in the monitored hand position representing upward or downward movement.

43. The data input device of claim 24, wherein the at least one processor is further configured to determine in advance of a word completion, a likely word that the user is typing, to cause a display of the likely word, to recognize a hand orientation indicative of a user'"'"'s acceptance of the likely word, and to accept the likely word in response to the recognized hand orientation.

44. The data input device of claim 43, wherein the recognized hand orientation includes an animated gesture.

45. The data input device of claim 24, wherein the at least one processor is further configured to determine in advance of a word completion, a likely word that the user is typing, display the likely word, recognize a hand orientation indicative of a user'"'"'s rejection of the likely word, and to reject the likely word in response to the recognized hand orientation.

46. The data input device of claim 45, wherein the recognized hand orientation includes an animated gesture.