Finger hover detection for improved typing

• US 9,489,086 B1
• Filed: 04/29/2014
• Issued: 11/08/2016
• Est. Priority Date: 04/29/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Chaplinsky, Robert C., Edgar, William Scott, Gerpheide, George E., Marsden, Randal J.
• Primary Examiner(s): Sasinowski, Andrew
• Application Number:
  US14/265,340
• Time in Patent Office: 924 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Typing on touchscreens is improved by dynamically and automatically positioning the desired home-row keys of an onscreen keyboard below the user'"'"'s fingers while their fingers are hovering above the surface, thus reducing the need for the user to look at the onscreen keyboard while typing. The location of each of the user'"'"'s fingers while hovering and/or resting is monitored. Correlation between the hover sensor data and touch sensor data increases the confidence level by which the system can determine which of the user'"'"'s fingers were used to select a key. By determining which finger was used to make the selection, the system accurately disambiguates which letter the user intended to type.

Claims

1. A method of operation within a computing device having a touchscreen and one or more touch sensors, the method comprising:

rendering a keyboard on the touchscreen, including rendering individual keys of the keyboard in respective positions on the touchscreen;

sensing, via signals received from the one or more touch sensors, one or more fingers of a user hovering at respective hover locations above a surface of the touchscreen;

while sensing the one or more fingers of the user hovering above the surface of the touchscreen, adjusting the respective positions of the individual keys on the touchscreen based on the respective hover locations of the one or more fingers of the user.

2. The method of claim 1, wherein the one or more touch sensors comprise one or more capacitive touch sensors.

3. The method of claim 1, further comprising:

before rendering the keyboard;

detecting contact with the touchscreen by a predetermined number of the user'"'"'s fingers, wherein the respective positions of the individual keys for rendering on the keyboard are determined based on locations of the predetermined number of the user'"'"'s fingers during the detected contact; and

while detecting the contact with the touchscreen, generating, a handprint for the user, wherein adjusting the respective positions of the individual keys is also based at least in part on a comparison of the respective hover locations of the one or more fingers with the handprint.

4. The method of claim 1 wherein rendering individual keys of the keyboard in the respective positions on the touchscreen comprises displaying, for each of the individual keys, a graphical image of a respective individual key, the graphical image having a shape, area, locus and angular orientation, and wherein adjusting the respective positions of the individual keys comprises, changing at least one of the shape, area, locus or angular orientation of the respective individual key.

5. The method of claim 1 wherein rendering individual keys of the keyboard in respective positions on the touch screen comprises rendering, for each of the individual keys, a virtual depiction of a respective individual key, without a corresponding displayed graphical image, but having a shape, area, locus and angular orientation, and wherein adjusting the respective positions of the individual keys comprises, changing at least one of the shape, area, locus or angular orientation of the respective individual key.

6. The method of claim 1 wherein sensing the one or more fingers includes comprises sensing both hands of the user hovering at respective first and second locations above the surface of the touchscreen, wherein the first and second locations are offset from one another along an axis parallel to the surface of the touchscreen, and wherein adjusting the respective positions of the individual keys on the touchscreen surface comprises offsetting respective positions of a first set of the individual keys associated with one of the user'"'"'s hands from respective positions of a second set of the individual keys associated with the other of the user'"'"'s hands in accordance with the offset between the first and second locations of the user'"'"'s hands.

7. The method of claim 1 further comprising generating a handprint of at least one hand of the user touching the touchscreen, and wherein sensing the respective fingers of the user hovering at the respective hover locations above the surface of the touchscreen comprises determining locations of the one or more fingers of the user'"'"'s hand based at least in part on the handprint.

8. The method of claim 1 wherein rendering the keyboard on the touchscreen comprises at least one of (i) displaying the keyboard on the touchscreen or (ii) enabling a region of the touchscreen to function as a keyboard without displaying the keyboard, and wherein rendering individual keys of the keyboard in respective positions on the touchscreen comprises at least one of displaying the individual keys of the keyboard in respective positions on the touchscreen or (ii) enabling respective regions of the touchscreen to function as the individual keys of the keyboard without displaying the individual keys.

9. A computing device comprising:

a touchscreen;

one or more processors;

one or more touch sensors that are communicably coupled with the one or more processors and the touchscreen; and

a memory having program code stored therein which, when executed by one or more processors, causes the one or more processors to;

render a keyboard on the touchscreen, including placing individual keys of the keyboard in respective positions on the touchscreen;

sense, via signals received from the one or more touch sensors, one or more fingers of a user hovering at respective hover locations above a surface of the touchscreen;

while sensing the one or more fingers of the user hovering above the surface of the touchscreen, adjust the respective positions of the individual keys on the touchscreen based on the respective hover locations of the one or more fingers of the user.

10. A method of operation within a computing device having a touchscreen, the method comprising:

tracking respective locations of a user'"'"'s fingers above the touchscreen;

detecting that the user unambiguously types, using the user'"'"'s first finger, a first key of a plurality of keys on a keyboard that is displayed on the touchscreen;

generating, based on (i) the respective tracked locations of the user'"'"'s fingers and (ii) the user'"'"'s use of the first finger to unambiguously type the first key, finger-to-key correlation information that indicates, for one or more keys of the plurality of keys on the keyboard, which of the user'"'"'s fingers is most likely to be used to type each of the one or more keys; and

detecting an ambiguous input that is provided by the user'"'"'s second finger, wherein the ambiguous input is detected near to two or more keys of the plurality of keys on the keyboard; and

determining that the ambiguous input corresponds to selection of a particular key of the two or more keys based at least in part on (i) the generated finger-to-key correlation information and (ii) the user'"'"'s use of the second finger to provide the ambiguous input.

11. The method of claim 10 wherein tracking respective locations of the user'"'"'s fingers above the touchscreen comprises tracking locations of the user'"'"'s fingers while the user'"'"'s hands are hovering above a surface of the touchscreen.

12. The method of claim 10, wherein:

the computing device further includes one or more capacitive touch sensors that are communicably coupled with the touchscreen, and

tracking locations of the user'"'"'s fingers while the user'"'"'s hands are hovering above a surface of the touchscreen comprises sensing locations of the user'"'"'s hands via signals received from the one or more capacitive touch sensors.

13. The method of claim 10, further comprising:

generating a handprint of at least one hand of the user, wherein tracking respective locations of the user'"'"'s fingers above the touchscreen comprises tracking locations of the user'"'"'s fingers based at least in part on the handprint.

14. The method of claim 10 further comprising:

detecting an unambiguous selection of a third character that is associated with a respective key;

disambiguating which of the user'"'"'s fingers was used to select the third character based at least in part on the finger-to-key correlation information.

15. The method of claim 10, further comprising:

determining a typing style for the user, wherein the generated finger-to-key correlation information is further based on the determined typing style for the user.

16. The method of claim 10 further comprising, wherein:

the ambiguous input is provided before the user types the first key, and

determining that the ambiguous input corresponds to selection of the particular key is performed after the ambiguous input is provided.

18. The method of claim 10, wherein the computing device includes one or more touch sensors and the method further comprises:

before tracking the respective locations of the user'"'"'s fingers, sensing, via signals received from the one or more touch sensors, that one or more of the user'"'"'s fingers are hovering above the touchscreen, wherein tracking the respective locations includes tracking the respective locations via signals received from the one or more touch sensors.

19. The method of claim 18, wherein the one or more touch sensors comprise one or more capacitive touch sensors.

17. A computing device comprising:

a touchscreen;

one or more processors; and

a memory having program code stored therein which, when executed by one or more processors, causes the one or more processors to;

track respective locations of a user'"'"'s fingers above the touchscreen;

detect that the user unambiguously types, using the user'"'"'s first finger, a first key of a plurality of keys on a keyboard that is displayed on the touchscreen;

generate, based on (i) the respective tracked locations of the user'"'"'s fingers and (ii) the user'"'"'s use of the first finger to unambiguously type the first key, finger-to-key correlation information that indicates, for one or more keys of the plurality of keys on the keyboard, which of the user'"'"'s fingers is most likely to be used to type each of the one or more keys; and

detect an ambiguous input that is provided by the user'"'"'s second finger, wherein the ambiguous input is detected near to two or more keys of the plurality of keys on the keyboard; and

determine that the ambiguous input corresponds to selection of a particular key of the two or more keys based at least in part on (i) the generated finger-to-key correlation information and (ii) the user'"'"'s use of the second finger to provide the ambiguous input.