Dynamic text input using on and above surface sensing of hands and fingers

• US 9,430,145 B2
• Filed: 04/06/2011
• Issued: 08/30/2016
• Est. Priority Date: 04/06/2011
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Treskunov, Anton, Kim, Seung Wook, Marti, Stefan
• Primary Examiner(s): Bashore, William
• Assistant Examiner(s): SHREWSBURY, NATHAN K
• Application Number:
  US13/081,361
• Publication Number:
  US 20120260207A1
• Time in Patent Office: 1,973 Days
• Field of Search: 715/765, 715/773
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A virtual keyboard is displayed on a touch screen display surface of a computing device. Partial images of the keyboard are displayed, where a partial image may be one key, referred to as the most probable key that the user will touch, or a group of keys, which may include some less probable or surrounding keys that may be touched. Sensors under or near the display surface detect an outline of the user hands and determines which finger is the fastest moving finger, which is presumed to be the finger used to touch a key. The most probable key is determined based on the fastest moving finger and may be displayed before the finger touches the surface. If the most probable key is not touched, a user profile containing user typing habits may be updated to reflect that a less probable key was touched.

Claims

1. A method of implementing a virtual keyboard, the method comprising:

detecting a hand above a display surface;

tracking at least one finger movement of at least one finger of the hand before the at least one finger touches the display surface;

determining a fastest moving finger of the at least one finger;

retrieving predetermined key configuration data corresponding to the fastest moving finger;

determining, based on the predetermined key configuration data corresponding to the fastest moving finger, a most probable key that the fastest moving finger is likely to touch before the fastest moving finger touches the display surface;

adjusting the display surface by displaying one or more virtual keys on the display surface before the fastest moving finger touches the display surface, wherein the one or more virtual keys displayed includes the most probable key;

detecting contact on the display surface by the fastest moving finger; and

selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected;

wherein the predetermined key configuration data corresponding to the fastest moving finger is updated in response to the fastest moving finger touching a different virtual key than the most probable key; and

wherein each finger has corresponding predetermined key configuration data comprising information relating to a most probable key that the finger is likely to touch.

2. The method of claim 1, further comprising:

tracking at least one downward finger movement of at least one finger of the hand before the at least one finger touches the display surface;

determining whether the fastest moving finger touches a less probable key instead of the most probable key; and

selectively updating a user typing profile including data representing one or more user typing habits based on the contact detected, wherein the user typing profile is updated to incorporate data that the less probable key was touched in response to determining that the fastest moving finger touched the less probable key, thereby using the one or more user typing habits to modify the virtual keyboard;

wherein the virtual keyboard is displayable anywhere on the display surface.

3. The method of claim 1, wherein selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected comprises:

determining whether the fastest moving finger touches a less probable key instead of the most probable key; and

updating the predetermined key configuration data corresponding to the fastest moving finger in response to determining that the fastest moving finger touched the less probable key to incorporate data that the less probable key was touched.

4. The method of claim 1, further comprising:

storing user typing historical data; and

adjusting position and size of at least one virtual key on the display surface based on the historical data.

5. The method of claim 1, wherein displaying one or more virtual keys on the display surface before the fastest moving finger touches the display surface includes displaying at least one virtual key that is not already displayed on the display surface.

6. The method of claim 1, wherein displaying one or more virtual keys on the display surface before the fastest moving finger touches the display surface includes displaying at least one less probable key.

7. The method of in claim 1, further comprising outlining the hand and one or more fingers of the hand using one or more sensors below the display surface.

8. The method of claim 1, further comprising:

retrieving a user typing profile, wherein the user typing profile includes data representing one or more user typing habits.

9. The method of claim 1, further comprising:

storing one or more coordinates associated with one or more fingertips of the hand.

10. The method of claim 1, wherein, for each finger, corresponding predetermined key configuration data comprises at least one of a character and a coordinate relating to a most probable key that the finger is likely to touch.

11. The method of claim 9, further comprising:

assigning a unique identifier to each fingertip.

12. The method of claim 11, further comprising:

retrieving an identifier assigned to the fastest moving finger.

13. The method of claim 1, further comprising:

determining a horizontal position of the fastest moving finger.

14. The method of claim 1, further comprising:

retrieving partial keyboard data using the predetermined key configuration data corresponding to the fastest moving finger; and

based on the partial keyboard data retrieved, displaying a partial keyboard including one or more virtual keys on the display surface before the fastest moving finger touches the display surface;

wherein the partial keyboard minimizes the amount of display space utilized on the display surface; and

wherein size and position of the partial keyboard is determined based on size and position of the hand.

15. The method of claim 1, further comprising:

determining a most probable key that the fastest moving finger is likely to touch utilizing data related to user typing muscle memory.

16. A method of entering text using a virtual keyboard, the method comprising:

outlining a hand and one or more fingertips of the hand;

determining a keyboard size and a keyboard position on a display surface;

identifying the one or more fingertips of the hand;

tracking at least one finger movement of at least one finger of the hand before the at least one finger touches the display surface;

determining a fastest moving finger of the at least one finger;

retrieving predetermined key configuration data corresponding to the fastest moving finger;

based on the keyboard size, the keyboard position, the fastest moving finger and the predetermined key configuration data corresponding to the fastest moving finger, displaying a partial virtual keyboard on the display surface before the fastest moving finger touches the display surface, wherein the partial virtual keyboard displayed includes a most probable key that the fastest moving finger is likely to touch before the fastest moving finger touches the display surface;

detecting contact on the display surface by the fastest moving finger; and

selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected;

wherein the predetermined key configuration data corresponding to the fastest moving finger is updated in response to the fastest moving finger touching a different virtual key than the most probable key; and

wherein each finger has corresponding predetermined key configuration data comprising information relating to a most probable key that the finger is likely to touch.

17. The method of claim 16, further comprising:

tracking at least one downward finger movement of at least one finger of the hand before the at least one finger touches the display surface.

18. The method of claim 17, further comprising:

determining whether the fastest moving finger touches a less probable key instead of the most probable key; and

selectively updating a user typing profile including data representing one or more user typing habits based on the contact detected, wherein the user typing profile is updated to incorporate data that the less probable key was touched in response to determining that the fastest moving finger touched the less probable key, thereby using the one or more user typing habits to modify the virtual keyboard.

19. The method of claim 17, wherein selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected comprises:

determining whether the fastest moving finger touches a less probable key instead of the most probable key; and

updating the predetermined key configuration data corresponding to the fastest moving finger in response to determining that the fastest moving finger touched the less probable key to incorporate data that the less probable key was touched.

20. The method of claim 16, further comprising:

storing user typing historical data; and

adjusting the keyboard size and the keyboard position based on the historical data.

21. The method of claim 17, wherein the partial virtual keyboard displayed includes at least one virtual key that is not already displayed on the display surface.

22. The method of claim 16, wherein the partial virtual keyboard displayed further includes one or more less probable keys.

23. The method of claim 16, wherein one or more sensors below the display surface are used for outlining the hand and the one or more fingertips of the hand.

24. The method of claim 16, further comprising:

retrieving a user typing profile, wherein the user typing profile includes data representing one or more user typing habits.

25. The method of claim 17, wherein, for each finger, corresponding predetermined key configuration data comprises at least one of a character and a coordinate relating to a most probable key that the finger is likely to touch.

26. The method of claim 25, further comprising:

retrieving partial keyboard data using the predetermined key configuration data corresponding to the fastest moving finger;

wherein the partial keyboard is further based on the partial keyboard data retrieved; and

wherein the partial keyboard minimizes the amount of display space utilized on the display surface.

27. The method of claim 16, further comprising:

assigning a unique identifier to each fingertip.

28. The method of claim 17, further comprising:

determining a most probable key that the fastest moving finger is likely to touch utilizing data related to user typing muscle memory.

29. A computing system for implementing a virtual keyboard, the system comprising:

a processor;

a touch screen display;

a sensor configured to track a hand and one or more fingers of the hand above the touch screen display; and

a data storage device storing instructions that when executed by the processor causes the processor to m operations including;

storing one or more user typing profiles and predetermined key position data for the one or more fingers of the hand, wherein each finger has corresponding predetermined key configuration data comprising information relating to a most probable key that the finger is likely to touch;

determining a fastest moving finger of the hand based on at least one finger movement of at least one finger of the hand before the at least one finger touches the touch screen display;

retrieving predetermined key configuration data corresponding to the fastest moving finger;

determining, based on the predetermined key configuration data corresponding to the fastest moving finger, a most probable key that the fastest moving finger is likely to touch before the fastest moving finger touches the touch screen display;

adjusting the touch screen display by displaying one or more virtual keys on the touch screen display before the fastest moving finger touches the touch screen display, wherein the one or more virtual keys displayed includes the most probable key;

detecting contact on the touch screen display by the fastest moving finger; and

selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected;

wherein the predetermined key configuration data corresponding to the fastest moving finger is updated in response to the fastest moving finger touching a different virtual key than the most probable key.

30. The system of claim 29, wherein the operations further include:

measure measuring a finger speed of the fastest moving finger.

31. The system of claim 29, wherein the operations further include:

processing data on which key of the virtual keyboard was touched in response to determining that the most probable key was not touched; and

selectively updating a user typing profile including data representing one or more user typing habits based on the contact detected, wherein the user typing profile is updated to incorporate data that the most probable key was not touched, thereby using the one or more user typing habits to modify the virtual keyboard.

32. The system of claim 29, wherein the operations further include:

operating and managing data from the sensor.

33. The system of claim 29, wherein the sensor utilizes wedge optics technology.

34. The system of claim 29, wherein selectively updating the predetermined key configuration data corresponding to the fastest moving finger based on the contact detected comprises:

determining whether the fastest moving finger touches a less probable key instead of the most probable key; and

updating the predetermined key configuration data corresponding to the fastest moving finger in response to determining that the fastest moving finger touched the less probable key to incorporate data that the less probable key was touched.

35. The system of claim 29, wherein the operations further include:

storing finger coordinate data for the one or more fingers of the hand, and keyboard size and position data for the virtual keyboard; and

determining size and position of the virtual keyboard on the touch screen display based on the keyboard size and position data for the virtual keyboard.

36. The system of claim 29, wherein the sensor is under the touch screen display and is able to detect finger movements above the touch screen display.