Intelligent touchscreen keyboard with finger differentiation

• US 10,372,328 B2
• Filed: 09/19/2016
• Issued: 08/06/2019
• Est. Priority Date: 01/09/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Zhai, Shumin
• Primary Examiner(s): Patel, Sanjiv D.
• Application Number:
  US15/268,928
• Publication Number:
  US 20170003878A1
• Time in Patent Office: 1,051 Days
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04186 Touch location disambiguation
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

Implementations of the present disclosure include actions of displaying a plurality of keys on a touchscreen of a computing device, receiving user input to the touchscreen, the user input including a contact with the touchscreen, in response to receiving the user input, determining spatial information associated with the contact and generating an image of the contact, determining an output based on the image and the spatial information, and executing the output.

Claims

1. A computer-implemented method, comprising:

displaying, on a touchscreen of a computing device, a plurality of virtual keys;

receiving, by the computing device, a contact with the touchscreen while the plurality of virtual keys were displayed on the touchscreen, wherein the contact was with a contacted area of the touchscreen, the contacted area of the touchscreen being a portion of the touchscreen at which the computing device displayed multiple candidate virtual keys of the plurality of virtual keys;

determining, by the computing device and based on an electrical or optical change to the touchscreen that results from the contact, a contact image that depicts the contacted area of the touchscreen;

determining, by the computing device, a particular finger that provided the contact with the contacted area of the touchscreen from among multiple fingers of a same user hand, based on analysis of the contact image with multiple pre-defined contact images that model touchscreen contacts by different, respective user fingers of the same user hand;

selecting, by the computing device, a particular virtual key, of the multiple candidate virtual keys displayed at the contacted area of the touchscreen, to assign to the contact with the touchscreen, the particular virtual key being selected from among the multiple candidate virtual keys, based on;

(i) a location of the contacted area of the touchscreen, and

(ii) the computing device having determined that the particular finger provided the contact with the touchscreen in distinction to another one of the multiple fingers of the same user hand, based on the analysis of the contact image; and

performing, by the computing device in response to the computing device selecting the particular virtual key of the multiple candidate virtual keys to assign to the contact with the touchscreen, an action that corresponds to user selection of the particular virtual key.

5. The computer-implemented method of claim 1, wherein selecting the particular virtual key from the multiple candidate virtual keys to assign to the contact with the touchscreen comprises comparing a characteristic of the contacted area of the touchscreen with pre-defined characteristics of contacted areas for one or more user fingers of the plurality of user fingers of the same user hand.

6. The computer-implemented method of claim 5, further comprising updating the pre-defined characteristics of contacted areas for one or more user fingers of the plurality of user fingers of the same user hand based on the determined characteristic of the contacted area of the touchscreen.

7. The computer-implemented method of claim 1, wherein performing, by the computing device in response to the computing device selecting the particular virtual key to assign to the contact with the touchscreen, the action that corresponds to user selection of the particular virtual key comprises typing in a user interface displayed on the touchscreen of the computing device a particular character assigned to the particular virtual key.

8. The computer-implemented method of claim 1, wherein determining, by the computing device, the particular finger that provided the contact with the touchscreen from among multiple fingers of the same user hand, based on analysis of the contact image to multiple pre-defined contact images that model touchscreen contacts by different, respective user fingers of the same user hand, includes comparing a shape of the contact image to one or more shapes of the multiple pre-defined contact images.

18. The computer-implemented method of claim 1, wherein the contact image indicates variation in a characteristic of the contact that occurred in an interior portion of the contacted area of the touchscreen, the contacted area of the touchscreen being the portion of the touchscreen at which the computing device displayed multiple candidate virtual keys of the plurality of virtual keys.

2. A computer-implemented method, comprising:

displaying, on a touchscreen of a computing device, a plurality of virtual keys, wherein each of the plurality of virtual keys (i) corresponds to a different character of a plurality of characters and (ii) is displayed at a particular location on the touchscreen that is different from the locations at which any other virtual key of the plurality of virtual keys is displayed on the touchscreen;

receiving, by the computing device, a contact with the touchscreen while the plurality of virtual keys are displayed on the touchscreen, wherein the contact is with a contacted area of the touchscreen, the contacted area of the touchscreen being a portion of the touchscreen at which the computing device displayed multiple candidate virtual keys of the plurality of virtual keys;

determining, by the computing device, a characteristic of the contacted area of the touchscreen that resulted from the contact with the touchscreen;

determining, by the computing device, a particular finger that provided the contact with the contacted area of the touchscreen from among multiple fingers of a same user hand, based on analysis of the contact image with multiple pre-defined contact images that model the touchscreen contacts by different, respective fingers of the same user hand;

selecting, by the computing device, a particular virtual key of the multiple candidate virtual keys associated with the contacted area of the touchscreen, to assign to the contact with the touchscreen, the particular virtual key being selected from among the multiple candidate virtual keys, based on;

(i) a location of the contacted area of the touchscreen

(ii) the computing device having determined the characteristic of the contacted area of the touchscreen that resulted from the contact with the touchscreen; and

(iii) the computing device having determined that the particular finger provided the contact with the touchscreen in distinction to another one of the multiple fingers from the same user hand, based on the analysis of the contact image; and

performing, by the computing device, in response to the computing device selecting the particular virtual key of the multiple candidate virtual keys to assign to the contact with the touchscreen, an action that corresponds to user selection of the particular virtual key.

3. The computer-implemented method of claim 2, wherein selecting the particular virtual key of the multiple candidate virtual keys to assign to the contact with the touchscreen comprises:

determining a set of finger probabilities based on the characteristic of the contacted area of the touchscreen, wherein each finger probability of the set of finger probabilities (i) corresponds to a different user finger of the plurality of user fingers of the same user hand and (ii) indicates a respective likelihood that the contact with the touchscreen was performed by the particular user finger that corresponds to the finger probability; and

identifying conversion data that indicates, for each user finger of the plurality of user fingers of the same user hand, a respective set of character probabilities, wherein each respective set of character probabilities indicates a respective likelihood that the user finger would be used to target a particular character that corresponds to a respective character probability.

4. The computer-implemented method of claim 3, wherein selecting the particular virtual key from the multiple candidate virtual keys to assign to the contact with the touchscreen comprises selecting the particular virtual key from among the multiple candidate virtual keys using the set of finger probabilities, the conversion data, and the location of the touchscreen at which the contact with the touchscreen occurred.

9. A non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors of a computing device, cause the one or more processors to perform operations comprising:

displaying, on a touchscreen of a computing device, a plurality of virtual keys;

receiving, by the computing device, a contact with the touchscreen while the plurality of virtual keys were displayed on the touchscreen, wherein the contact was with a contacted area of the touchscreen, the contacted area of the touchscreen being a portion of the touchscreen at which the computing device displayed candidate virtual keys of the plurality of virtual keys;

determining, by the computing device and based on an electrical or optical change to the touchscreen that results from the contact, a contact image that depicts the contacted area of the touchscreen;

determining, by the computing device, a particular finger that provided the contact with the touchscreen from among multiple fingers of a same user hand, based on analysis of the contact image to multiple pre-defined contact images that model touchscreen contacts by different, respective user fingers from the same user hand;

selecting, by the computing device, a particular virtual key of the multiple candidate virtual keys associated with the contact area of the touchscreen, to assign to the contact with the touchscreen, the particular virtual key being selected from among the multiple candidate virtual keys, based on;

(i) a location of the contacted area of the touchscreen, and

(ii) the computing device having determined that the particular finger provided the contact with the touchscreen in distinction to another one of the multiple fingers of the same user hand providing the contact with the touchscreen, based on the analysis of the contact image; and

performing, by the computing device in response to the computing device selecting the particular virtual key of the multiple candidate virtual keys to assign to the contact with the touchscreen, an action that corresponds to user selection of the particular virtual key.

10. The computer-readable medium of claim 9, wherein selecting the particular virtual key, of the multiple candidate virtual keys associated with the contacted area of the touchscreen, to assign to the contact with the touchscreen comprises:

determining, based on a characteristic of the contacted area of the touchscreen, a particular user finger, from among multiple fingers of a same user hand, that provided the contact with the touchscreen; and

selecting the particular key, from among the multiple fingers of the same user hand, to assign to the contact with the touchscreen based at least on the particular user finger from among the multiple fingers of the same user hand that provided the contact with the touchscreen.

11. The computer-readable medium of claim 9, wherein each of the plurality of virtual keys (i) corresponds to a different character of a plurality of characters and (ii) is displayed at a particular location on the touchscreen that is different from the locations at which any other virtual key of the plurality of virtual keys is displayed on the touchscreen.

12. The computer-readable medium of claim 11, wherein selecting the particular virtual key of the multiple candidate virtual keys to assign to the contact with the touchscreen comprises:

determining a set of finger probabilities based on the comparison of the contact image to the one or more pre-defined images that respectively model touchscreen contact by different user fingers, wherein each finger probability of the set of finger probabilities (i) corresponds to a different user finger of a plurality of user fingers of a same user hand and (ii) indicates a respective likelihood that the contact with the touchscreen was performed by the particular user finger that corresponds to the finger probability; and

identifying conversion data that indicates, for each user finger of the plurality of user fingers of the same user hand, a respective set of character probabilities, wherein each respective set of character probabilities indicates a respective likelihood that the user finger would be used to target a particular character that corresponds to a character probability.

13. The computer-readable medium of claim 12, wherein selecting the particular virtual key from the multiple candidate virtual keys to assign to the contact with the touchscreen comprises selecting the particular virtual key from among the multiple candidate virtual keys using the set of finger probabilities, the conversion data, and the location of the touchscreen at which the contact with the touchscreen occurred.

14. The computer-readable medium of claim 9, wherein selecting the particular virtual key from among multiple candidate virtual keys to assign to the contact with the touchscreen comprises comparing a characteristic of the contacted area of the touchscreen with pre-defined characteristics of contacted areas for one or more user fingers of the plurality of user fingers of the same user hand.

15. The computer-readable medium of claim 14, wherein the operations further comprise updating the pre-defined characteristics of contacted areas for one or more user fingers of the plurality of user fingers of the same user hand based on the determined characteristic of the contacted area of the touchscreen.

16. The computer-readable medium of claim 9, wherein performing, by the computing device in response to the computing device selecting the particular virtual key to assign to the contact with the touchscreen, the action that corresponds to user selection of the particular virtual key comprises typing in a user interface displayed on the touchscreen of the computing device a particular character assigned to the particular virtual key.

17. The computer-readable medium of claim 9, wherein determining, by the computing device, the particular finger that provided the contact with the touchscreen from among multiple fingers of the same user hand, based on analysis of the contact image to multiple pre-defined contact images that model touchscreen contacts by different, respective user fingers of the same user hand, includes comparing a shape of the contact image to one or more shapes of the multiple pre-defined contact images.