Natural input recognition system and method using a contextual mapping engine and adaptive user bias

• US 7,246,060 B2
• Filed: 11/06/2001
• Issued: 07/17/2007
• Est. Priority Date: 11/06/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Winkler, David V., Geidl, Erik M.
• Primary Examiner(s): Harper; V. Paul
• Assistant Examiner(s): Vo; Huyen X.
• Application Number:
  US09/993,331
• Publication Number:
  US 20030088410A1
• Time in Patent Office: 2,079 Days
• Field of Search: 707/102, 707/2, 704/235, 704/254, 704/270, 704/275, 704/243, 704/255, 704/251, 704/231, 704/246, 704/276, 704/9, 704/10, 382/187
• US Class Current: 704/235
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/262 using context analysis, e.g. lexical, syntactic or semantic context
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/18 using natural language modelling

Abstract

A natural (e.g., handwriting or speech) input recognition system and method that uses contextual mapping to improve recognition accuracy by biasing recognition based on the context of an input field. As natural input data is being entered into an application field, the context (type) of the field is determined and used to locate context-based validation rules and context-based user bias data. When entry is complete, the context-based validation rules and context-based user bias data are provided to a recognition engine with the natural input data. The recognizer biases its recognition result by using the rules and the user bias data to recognize the natural input. A field signature generator is described that determines each field'"'"'s context, independent of the application, and a data harvesting engine is described that automatically collects user bias data from various data stores.

Claims

1. In a computing device, a method for recognizing natural human input, the method comprising:

an act of detecting, at a system component, natural input data directed to an input field from a plurality of input fields of an executing program, wherein the natural input data comprises an input other than textual input, the natural input data entered into the computing device by a user of the computing device;

an act of calling a field signature engine to obtain a field signature for the input field;

an act of receiving a field signature for the input field from the field signature engine, the field signature representing the context of the input field, the field signature having been constructed from an analysis of the attributes of the input field;

an act of using the field signature as a key to query a field mapping database for a factoid for the represented context;

an act of receiving a factoid from the field mapping database, the factoid containing contextual rules to more accurately recognize that natural input data;

an act of using the factoid as a key to query a user bias database for user bias data;

an act of receiving using bias data that can be used to further refine the recognition of the natural input data, within the represented context and in combination with the contextual rules, to more accurately recognize that natural data input;

an act of submitting the natural input data, the factoid, and the user bias data to a recognition engine so that the recognition engine can use the contextual rules contained in the factoid and the user bias data to perform context-based recognition of the natural data input; and

an act of receiving a recognition result of the natural input data from the recognition engine for inclusion in the input field, the recognition result biased by the user-specific biasing information within the represented context.

2. The method of claim 1 wherein the user biasing information comprises previously entered user data that is more likely to be a recognition result of the natural input data based on the represented context of the input field.

3. The method of claim 2 wherein the factoid is developed based on communicating with the executing program.

4. The method of claim 2 wherein receiving the recognition result to the executing program includes providing the factoid to a recognition engine.

5. The method of claim 1 wherein the user biasing information comprises a set of user bias data.

6. The method of claim 5, further comprising:

maintaining the set of user bias data in a user bias database.

7. The method of claim 5 further comprising, harvesting the user bias data from at least one data store accessible to the computing device.

8. The method of claim 1 wherein the input field corresponds to a window, and wherein generating a field signature includes acquiring window attribute data.

9. The method of claim 1 wherein the natural input data comprises speech or handwriting data.

10. A computer system comprising:

one or more processors;

system memory; and

one or more computer-readable media having stored thereon computer-executable instructions representing the following;

a human input recognition engine configured to convert natural input data to recognition results, wherein the natural input data comprises an input other than textual input, each recognition result comprising at least one computer code;

a field determination mechanism that determines contexts for at least some input fields of executable programs as determined contexts;

at least one field mapping database that maintains rules corresponding to determined contexts;

at least one user bias database that maintains biasing information for a plurality of the determined contexts; and

an input system configured to;

1) detect natural input data directed to an input field from among a plurality of input fields of the executing program, the natural input data entered into the computing device by a user of the computing device;

2) call the field determination mechanism to obtain the determined context of the input field to which the natural input data is directed;

3) receive a field signature for the input field from the field determination mechanism, the field signature representing the determined context of the input field, the field signature having been constructed from an analysis of the attributes of the input field;

4) use the field signature as a key to query a field mapping database for a factoid for the represented context;

5) receive a factoid from the file mapping database, the factoid containing contextual rules to more accurately recognize that natural input data;

6) use the factoid as a key to query a user bias database for user bias data;

7) receive using bias data that can be used to further refine the recognition of the natural input data, within the represented context and in combination with the contextual rules, to more accurately recognize that natural data input;

8) communicate the natural input data, the factoid, and the user bias data to the recognition engine so that the recognition engine can use the rules contained in the factoid and the user bias data to perform context-based recognition of the natural data input;

9) receive a recognition result from the recognition engine, the recognition result from analyzing the natural input data based on the rules in the factoid and the user-specific biasing information for the input field; and

10) provide to the executing program at least one computer code corresponding to the recognition result received from the recognition engine.

11. The system of claim 10 wherein the field determination mechanism includes a field signature engine that generates a field signature corresponding to the context based on characteristics of the input field.

12. The system of claim 11 wherein the characteristics of the input field include text displayed proximate the field.

13. The system of claim 10 wherein the natural input data comprises speech or handwriting data.

14. The system of claim 10 wherein the field determination mechanism includes a field signature engine that generates a field signature corresponding to the field type based on characteristics of the input field.

15. The system of claim 10 wherein the at least one database of biasing information comprises a database of sets of user bias data, and wherein the input system communicates the biasing information including a set of user bias data to the recognition engine.

16. The system of claim 15, wherein the user bias data set communicated to the recognition engine is retrieved from the database of sets of user bias data based on the field type determined by the field determination mechanism.

17. The system of claim 15 wherein the database is securely maintained on the computing device.

18. The system of claim 15 further comprising a data harvesting engine that obtains at least some of the user bias data from at least one data store accessible to the computing device.

19. The system of claim 18 wherein the database of sets of user bias data includes at least some data that was not harvested by the harvesting engine.

20. The system of claim 18 wherein the data harvesting engine operates as a background process.

21. The system of claim 10 wherein the at least one database of biasing information comprises a first database of factoids and a second database of sets of user bias data, and wherein the input system communicates the biasing information including a factoid and a set of user bias data to the recognition engine.

22. The system of claim 21 wherein the factoid and the user bias data are retrieved from their respective databases based on the field type.

23. The system of claim 21 wherein the factoid includes information corresponding to at least one criterion with which the recognition result should comply.

24. In a computing device, a system comprising:

a field determination mechanism that determines a context of an input field from among a plurality of input fields in an executable program and provides a field signature representative thereof;

a field mapping database indexed by field signatures that contains contextual rules to more accurately recognize natural input data and provide a factoid of rules for a context represented by a field signature;

a database of biasing information indexed by factoid that includes sets of user bias data used to further refine the recognition of natural input data, within a represented context and in combination with the contextual rules, to more accurately recognize natural input data and to provide user bias data for a represented context;

an input system configured to receive natural input data directed towards the input field, to obtain a factoid from the field mapping database, and to obtain user bias data corresponding to the factoid, wherein the natural input data comprises an input other than textual input, the user bias data corresponding to the factoid for the input field differing from user bias data corresponding to factoids for one or more other input fields of the executable program;

a human input recognizer that converts natural input data to computer codes, the recognizer configured to receive the factoid, the user bias data and the natural input data from the input system, to analyze the natural input data based on the user bias data and the rules contained in the factoid, and to provide a recognition result comprising a set of at least one computer code to the input system based on the natural input data, the rules contained in the factoid and the user bias data, the recognition result biased by the user bias data; and

the input system returning data to the executable program that corresponds to the recognition result.

25. The system of claim 24 wherein the field determination mechanism comprises a field signature engine that generates a field signature for the field.

26. The system of claim 24 wherein the factoid includes information corresponding to at least one criterion with which the recognition result should comply.