Intelligent touchscreen keyboard with finger differentiation

• US 9,448,651 B2
• Filed: 01/08/2013
• Issued: 09/20/2016
• Est. Priority Date: 01/09/2012
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhai, Shumin
• Primary Examiner(s): Patel, Sanjiv D
• Application Number:
  US13/736,341
• Publication Number:
  US 20130176227A1
• Time in Patent Office: 1,351 Days
• Field of Search: 345173-178
• US Class Current: 1/1
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Implementations of the present disclosure include actions of displaying a plurality of keys on a touchscreen of a computing device, receiving user input to the touchscreen, the user input including a contact with the touchscreen, in response to receiving the user input, determining spatial information associated with the contact and generating an image of the contact, determining an output based on the image and the spatial information, and executing the output.

Claims

1. A computing device, comprising:

a touchscreen;

one or more processors in communication with the touchscreen; and

a non-transitory computer-readable medium that is coupled to the one or more processors and that has instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

displaying a plurality of virtual keys on the touchscreen, wherein each of the plurality of virtual keys (i) corresponds to a different character among a plurality of characters and (ii) is displayed at a different location on the touchscreen;

receiving user input to the touchscreen, the user input comprising a contact with the touchscreen;

determining, based at least on one or more characteristics of the contact with the touchscreen, a set of finger probabilities that indicates, for each of a plurality of user fingers, a likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers;

accessing conversion data that indicates, for each of the plurality of user fingers, likelihoods of the respective user finger being used to target different characters among the plurality of characters;

identifying that the contact with the touchscreen occurred at a first location on the touchscreen;

identifying a particular character targeted by the user input from among multiple of the plurality of characters that are located in proximity of the first location on the touchscreen at which the contact occurred, wherein the particular character is identified based at least on;

(i) the set of finger probabilities that indicates, for each of the plurality of user fingers, the likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers,

(ii) the conversion data that indicates, for each of the plurality of user fingers, the likelihoods of the respective user finger being used to target different characters among the plurality of characters, and

(iii) the identification that the contact with the touchscreen occurred at the first location on the touchscreen; and

in response to identifying the particular character targeted by the user input, selecting from among the plurality of virtual keys a particular virtual key that corresponds to the particular character and performing an action with the computing device that corresponds to the particular virtual key.

2. The computing device of claim 1, wherein the touchscreen is a surface acoustic wave (SAW) touchscreen, a resistive touchscreen, a capacitive touchscreen, an infrared touchscreen, or an optical sensing touchscreen, wherein the optical sensing touchscreen is configured to sense touch based on Frustrated Total Internal Reflection.

3. The computing device of claim 1, wherein determining the set of finger probabilities comprises:

identifying a contact image that results from the user input contacting the touchscreen, wherein the contact image indicates a shape of the contact between a user finger and the touchscreen; and

analyzing the contact image, using a model that includes respective representations of the plurality of user fingers, to determine similarities between the contact image and the respective representations of the plurality of user fingers.

4. The computing device of claim 1, wherein the operations further comprise:

updating the model based on the contact image; and

at a later time, using the updated model to determine similarities between (i) a second contact image that results from subsequent user input contacting the touchscreen and (ii) the respective representations of the plurality of user fingers.

5. The computing device of claim 1, wherein performing the action with the computing device that corresponds to the particular virtual key comprises typing the particular character in a user interface displayed on the touchscreen of the computing device.

6. The computing device of claim 1, wherein the operations comprise:

generating, based on the set of finger probabilities and the conversion data, a first set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input;

generating, based on (i) a keyboard geometry model that identifies a layout of the plurality of virtual keys on the touchscreen and (ii) identifying that the contact with the touchscreen occurred at the first location on the touchscreen, a second set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input,

wherein identifying the particular character targeted by the user input comprises combining the first set of character probabilities and the second set of character probabilities to generate a third set of character probabilities and identifying the particular character based on the third set of character probabilities.

7. A computer-implemented method, comprising:

displaying, on a touchscreen of a computing device, a plurality of virtual keys, wherein each of the plurality of virtual keys (i) corresponds to a different character among a plurality of characters and (ii) is displayed at a different location on the touchscreen;

receiving user input to the touchscreen, the user input comprising a contact with the touchscreen;

determining, based at least on one or more characteristics of the contact with the touchscreen, a set of finger probabilities that indicates, for each of a plurality of user fingers, a likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers;

accessing conversion data that indicates, for each of the plurality of user fingers, likelihoods of the respective user finger being used to target different characters among the plurality of characters;

identifying that the contact with the touchscreen occurred at a first location on the touchscreen;

identifying a particular character targeted by the user input from among multiple of the plurality of characters that are located in proximity of the first location on the touchscreen at which the contact occurred, wherein the particular character is identified based at least on;

(i) the set of finger probabilities that indicates, for each of the plurality of user fingers, the likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers,

(ii) the conversion data that indicates, for each of the plurality of user fingers, the likelihoods of the respective user finger being used to target different characters among the plurality of characters, and

(iii) the identification that the contact with the touchscreen occurred at the first location on the touchscreen; and

in response to identifying the particular character targeted by the user input, selecting from among the plurality of virtual keys a particular virtual key that corresponds to the particular character and performing an action with the computing device that corresponds to the particular virtual key.

8. The computer-implemented method of claim 7, wherein determining the set of finger probabilities comprises:

identifying a contact image that results from the user input contacting the touchscreen, wherein the contact image indicates a shape of the contact between a user finger and the touchscreen; and

analyzing the contact image, using a model that includes respective representations of the plurality of user fingers, to determine similarities between the contact image and the respective representations of the plurality of user fingers.

9. The computer-implemented method of claim 7, wherein a second virtual key among the plurality of virtual keys is displayed at the first location on the touchscreen where the contact with the touchscreen is identified to have occurred, the second virtual key being different than the particular virtual key that corresponds to the identified particular character targeted by the user input.

10. The computer-implemented method of claim 7, wherein the plurality of user fingers comprises a left index finger and a right index finger.

11. The computer-implemented method of claim 7, wherein displaying the plurality of virtual keys on the touchscreen of the computing device comprises displaying a virtual keyboard.

12. The computer-implemented method of claim 7, wherein performing the action with the computing device that corresponds to the particular virtual key comprises typing the particular character in a user interface displayed on the touchscreen of the computing device.

13. The computer-implemented method of claim 8, further comprising:

updating the model based on the contact image; and

at a later time, using the updated model to determine similarities between (i) a second contact image that results from subsequent user input contacting the touchscreen and (ii) the respective representations of the plurality of user fingers.

14. The computer-implemented method of claim 7, comprising:

generating, based on the set of finger probabilities and the conversion data, a first set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input;

generating, based on (i) a keyboard geometry model that identifies a layout of the plurality of virtual keys on the touchscreen and (ii) identifying that the contact with the touchscreen occurred at the first location on the touchscreen, a second set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input,

wherein identifying the particular character targeted by the user input comprises combining the first set of character probabilities and the second set of character probabilities to generate a third set of character probabilities and identifying the particular character based on the third set of character probabilities.

15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause performance of operations comprising:

displaying, on a touchscreen of a computing device, a plurality of virtual keys, wherein each of the plurality of virtual keys (i) corresponds to a different character among a plurality of characters and (ii) is displayed at a different location on the touchscreen;

receiving user input to the touchscreen, the user input comprising a contact with the touchscreen;

determining, based at least on one or more characteristics of the contact with the touchscreen, a set of finger probabilities that indicates, for each of a plurality of user fingers, a likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers;

accessing conversion data that indicates, for each of the plurality of user fingers, likelihoods of the respective user finger being used to target different characters among the plurality of characters;

identifying that the contact with the touchscreen occurred at a first location on the touchscreen;

identifying a particular character targeted by the user input from among multiple of the plurality of characters that are located in proximity of the first location on the touchscreen at which the contact occurred, wherein the particular character is identified based at least on;

(i) the set of finger probabilities that indicates, for each of the plurality of user fingers, the likelihood that the contact with the touchscreen was performed by the respective user finger among the plurality of user fingers,

(ii) the conversion data that indicates, for each of the plurality of user fingers, the likelihoods of the respective user finger being used to target different characters among the plurality of characters, and

(iii) the identification that the contact with the touchscreen occurred at the first location on the touchscreen; and

in response to identifying the particular character targeted by the user input, selecting from among the plurality of virtual keys a particular virtual key that corresponds to the particular character and performing an action with the computing device that corresponds to the particular virtual key.

16. The non-transitory computer-readable medium of claim 15, wherein determining the set of finger probabilities comprises:

identifying a contact image that results from the user input contacting the touchscreen, wherein the contact image indicates a shape of the contact between a user finger and the touchscreen; and

analyzing the contact image, using a model that includes respective representations of the plurality of user fingers, to determine similarities between the contact image and the respective representations of the plurality of user fingers.

17. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise:

updating the model based on the contact image; and

at a later time, using the updated model to determine similarities between (i) a second contact image that results from subsequent user input contacting the touchscreen and (ii) the respective representations of the plurality of user fingers.

18. The non-transitory computer-readable medium of claim 15, wherein a second virtual key among the plurality of virtual keys is displayed at the first location on the touchscreen where the contact with the touchscreen is identified to have occurred, the second virtual key being different than the particular virtual key that corresponds to the identified particular character targeted by the user input.

19. The non-transitory computer-readable medium of claim 15, wherein performing the action with the computing device that corresponds to the particular virtual key comprises typing the particular character in a user interface displayed on the touchscreen of the computing device.

20. The non-transitory computer-readable medium of claim 15, wherein the operations comprise:

generating, based on the set of finger probabilities and the conversion data, a first set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input;

generating, based on (i) a keyboard geometry model that identifies a layout of the plurality of virtual keys on the touchscreen and (ii) identifying that the contact with the touchscreen occurred at the first location on the touchscreen, a second set of character probabilities that indicate likelihoods that one or more characters were targeted by the user input,

wherein identifying the particular character targeted by the user input comprises combining the first set of character probabilities and the second set of character probabilities to generate a third set of character probabilities and identifying the particular character based on the third set of character probabilities.