Translating user interaction with a touch screen into text

• US 8,463,731 B2
• Filed: 02/17/2010
• Issued: 06/11/2013
• Est. Priority Date: 02/17/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Li, Yang
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): COUGHLAN, PETER D
• Application Number:
  US12/707,558
• Publication Number:
  US 20110202493A1
• Time in Patent Office: 1,210 Days
• Field of Search: None
• US Class Current: 706/50
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for translating user interaction with a touchscreen into text are provided. In one embodiment, a method comprises receiving information characterizing a first user interaction with a touchscreen; receiving information characterizing a second user interaction with the touchscreen; assigning a probability to each of two or more interpretations of a combination of the first user interaction and the second user interaction, each of the interpretations construing the combination as a different pair of characters, each of the probabilities embodying a likelihood that the first user interaction and the second user interaction represent the respective pair of characters; using the two or more interpretations to search a collection of data for objects that are each identifiable by one of the two or more interpretations; and outputting identifiers of at least some of the objects.

Claims

1. A method performed by system of one or more data processing devices, the method comprising:

receiving, at the system, information characterizing a first user interaction with a touchscreen;

receiving, at the system, information characterizing a second user interaction with the touchscreen;

the system assigning a probability to each of two or more interpretations of a combination of the first user interaction and the second user interaction, each of the interpretations construing the combination as a different pair of characters, each of the probabilities embodying a likelihood that the first user interaction and the second user interaction represent the respective pair of characters, wherein the system assigning the probability to each of the two or more interpretations of the combination of the first user interaction and the second user interaction includes;

assigning first probabilities to interpretations of the first user interaction,

independently assigning second probabilities to interpretations of the second user interaction, and

determining the probabilities assigned to the interpretations of the combination using the first probabilities and the second probabilities;

the system using the two or more interpretations to search a collection of data for objects that are each identifiable by one of the two or more interpretations; and

the system outputting identifiers of at least some of the objects.

2. The method of claim 1, wherein outputting the identifiers of the objects comprising selecting, for output, a proper subset of the objects that are identifiable by one of the two or more interpretations, wherein the objects in the proper subset are selected based on the probability assigned to the interpretations by which the objects are identifiable.

3. The method of claim 1, wherein outputting the identifiers of the objects comprises:

selecting a proper subset of the objects that are identifiable by a first of the two or more interpretations; and

selecting a proper subset of the objects that are identifiable by a second of the two or more interpretations, wherein a total number of objects selected for output is smaller than the number of objects identifiable by the first of the two or more interpretations, and the first of the two or more interpretations is more likely to be correct than the second of the two or more interpretations.

4. The method of claim 1, wherein:

the system is a handheld device that comprises the touchscreen and one or more data storage devices;

the collection of data comprises data stored in the one or more data storage devices; and

outputting the identifiers of the objects comprises displaying the identifiers on the touchscreen.

5. The method of claim 4, wherein the second user interaction comprises user interaction overlaid on a display of identifiers of objects that are each identifiable by one of the two or more interpretations of the first user interaction with the touchscreen.

6. The method of claim 4, wherein searching the collection of data comprises searching an index of identifiers of objects that are stored locally on the handheld device.

7. The method of claim 1, further comprising:

receiving, at the system, information characterizing a third user interaction with the touchscreen;

the system assigning a probability to each of two or more interpretations of a combination of the first user interaction, the second user interaction, and the third user interaction, each of the interpretations construing the combination as a different group of characters, each of the probabilities embodying a likelihood that the first user interaction, the second user interaction, and the third user interaction represent the respective group of characters;

the system using the two or more interpretations to search the collection of data for objects that are each identifiable by one of the two or more interpretations of the combination of the first user interaction, the second user interaction, and the third user interaction; and

the system outputting identifiers of the objects that are each identifiable by one of the two or more interpretations of the combination of the first user interaction, the second user interaction, and the third user interaction.

8. The method of claim 1, further comprising receiving, at the system, a command specifying that the first user interaction is to be used to search for a first set of terms that begin with interpretations of the first user interaction and the second user interaction is to be used to search for a second set of terms that begin with interpretations of the second user interaction, wherein the terms in the second set are associated with the terms in the first set.

9. The method of claim 1, wherein determining the probabilities assigned to the interpretations of the combination of the first user interaction and the second user interaction comprises:

multiplying each of the first probabilities by each of the second probabilities to determine the probabilities assigned to the interpretations of the combination.

10. The method of claim 1, further comprising:

receiving, at the system, information characterizing an access history at a device that includes the touchscreen;

the system scoring the objects using the access history, the score of each object embodying how often the object has been accessed by the device that includes the touchscreen;

the system selecting a proper subset of the objects identifiable by one of the two or more interpretations based on the scores of the objects; and

the system outputting identifiers of the objects in the proper subset.

11. The method of claim 1, wherein outputting the identifiers of the objects comprises marking the identifiers of the objects on a display screen to indicate why the objects are identifiable by the combination of the first user interaction and the second user interaction.

12. A device comprising:

a touchscreen;

a data store comprising one or more data storage devices storing a collection of local data, the local data including personal contacts or media files; and

a data processor programmed to perform data processing activities, the activities including;

a drawn user input translator to translate a first user interaction with the touchscreen into two or more characters, wherein the drawn user input translator is further configured to;

assign first probabilities to each of the two or more characters into which the first user interaction with the touchscreen is translated, each of the probabilities embodying a likelihood that the first user interaction represents the respective character;

independently assign second probabilities to each of two or more characters into which a second user interaction with the touchscreen is translated, each of the probabilities embodying a likelihood that the second user interaction represents the respective character; and

determine the probabilities assigned to the interpretations of the combination using the first probabilities and the second probabilities;

a search engine to identify the personal contacts or the media files that are identifiable by the two or more characters into which the first user interaction is translated by the drawn user input translator, and

a display manager to present identifiers of at least some of the personal contacts or the media files that are identified by the search engine on the touchscreen, wherein a first of the identifiers presented on the touchscreen identifies a first contact or a first media file that is identifiable by a first of the characters into which the first user interaction is translated, and a second of the identifiers presented on the touchscreen identifies a second contact or a second media file that is identifiable by a second of the characters into which the first user interaction is translated.

13. The device of claim 12, wherein:

a third contact or a third media file is identifiable by one of the characters into which the first user interaction is translated; and

the data processor is further programmed to;

score the first contact or the first media file using a historical record of access of the first contact or the first media file;

score the third contact or the third media file using a historical record of access of the third contact or the third media file; and

select the first of the identifiers for presentation on the touchscreen in response to the first contact or the first media file having been accessed more often than the third contact or the third media file.

14. The device of claim 12, wherein:

the first of the identifiers comprises a first marking indicating why the first contact or the first media file is identifiable by the first of the characters; and

the second of the identifiers comprises a second marking indicating why the second contact or the second media file is identifiable by the second of the characters.

15. The device of claim 12, wherein the data processor is further programmed to

determine the probabilities assigned to the interpretations of the combination by multiplying each of the first probabilities by each of the second probabilities.

16. The device of claim 12, wherein the data processor is further programmed to:

select a proper subset of the contacts or media files that are identifiable by the first of the characters for a presentation on the touchscreen; and

select a proper subset of the objects that are identifiable by the second of the characters for the presentation on the touchscreen, wherein a total number of objects selected for output is smaller than the number of objects identifiable by the first of the characters, and the first of the characters is more likely to be correct than the second of the second of the characters.

17. A handheld device comprising:

a system of one or more data processing devices;

a data store of one or more data storage devices; and

a touchscreen, the touchscreen displaying a visual presentation of a collection of identifiers of candidate objects that are identifiable by multiple interpretations of user interaction representing one or more characters;

wherein the system of one or more data processing devices is configured to;

assign first probabilities to each of two or more characters into which a first user interaction with the touchscreen is translated, each of the probabilities embodying a likelihood that the first user interaction represents the respective character;

independently assign second probabilities to each of two or more characters into which a second user interaction with the touchscreen is translated, each of the probabilities embodying a likelihood that the second user interaction represents the respective character; and

determine the probabilities assigned to the interpretations of the combination using the first probabilities and the second probabilities;

wherein at least two of the candidate objects are identifiable by different interpretations of the user interaction representing the one or more characters; and

wherein at least a portion of the visual presentation of the identifier collection is operative to receive a subsequent user interaction representing a further character.

18. The handheld device of claim 17, wherein the touchscreen further comprises a user interaction display area displaying smaller-sized likenesses of the user interaction.

19. The handheld device of claim 17, wherein the visual presentation comprises an identifier of a first candidate object that has been selected from amongst a collection of candidate objects that are identifiable by the multiple interpretations of the user interaction based on a historical record of access of the first candidate object.

20. The handheld device of claim 19, wherein the visual presentation further comprises an identifier of a second candidate object that has been selected from amongst a collection of candidate objects that are identifiable by the multiple interpretations of the user interaction without using a historical record of access of the second candidate object.

21. The handheld device of claim 17, wherein the data store comprises an index of the identifiers of the candidate objects, the candidate objects being stored in the data store.

22. The handheld device of claim 21, wherein the candidate objects comprise contacts and applications.