Natural language input method and apparatus

• US 6,975,983 B1
• Filed: 10/31/2000
• Issued: 12/13/2005
• Est. Priority Date: 10/29/1999
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Fortescue, Nicholas David, Keiller, Robert Alexander
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US09/699,510
• Time in Patent Office: 1,869 Days
• Field of Search: 704/9, 704/257
• US Class Current: 704/9
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks

Abstract

A system is disclosed for generating a modified data structure defining modified grammar rules for the recognition of a natural language input with pauses in which grammar rules for recognition of a natural language are analysed to identify positions in the grammar rules at which pauses can occur in the natural language. A modified data structure is generated defining the modified grammar rules in dependence upon the analysis. The modified data structure is used to improve the accuracy of recognition of a natural language input with pauses. Where the natural language input is used in conjunction with a second modality input, the analysis is performed to identify the positions of pauses in dependence upon the second modality inputs.

Claims

1. Data processing apparatus comprising:

receiving means for receiving a data structure defining grammar rules for recognition of a natural language input;

analysing means for analysing the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

generating means for generating a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein said generating means is adapted to fragment the grammar rules in accordance with said identified positions to generate sub grammar rules to form said modified data structure.

2. Data processing apparatus according to claim 1, wherein said analysing means is adapted to identify the positions in accordance with pause criteria for the natural language input.

3. Data processing apparatus according to claim 1 or claim 2, wherein said generating means is adapted to add marker means to the identified positions in the grammar rules at which pauses can occur in the natural language input to generate the modified data structure.

4. Data processing apparatus according to claim 1, wherein said generating means is adapted to form a hierarchical structure using said sub grammar rules to form said modified data structure.

5. Data processing apparatus according to claim 1 or 2, wherein said receiving means is adapted to receive a data structure defining grammar rules for use in speech recognition of a natural language speech input, and said generating means is adapted to generate said modified data structure defining modified grammar rules for speech recognition of a natural language speech input with pauses therein.

6. Data processing apparatus comprising:

receiving means for receiving a data structure defining grammar rules for recognition of a natural language input;

analysing means for analysing the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

generating means for generating a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein said receiving means is adapted to receive said data structure defining grammar rules for recognition of a natural language input as a first modality input in conjunction with associated events in at least one further modality input, said data structure defining the association between events in each modality input, and events in said first modality input comprising units in the natural language.

7. Data processing apparatus according to claim 6, wherein said analysing means is adapted to identify said positions in the grammar rules based on events in at least one said further modality input.

8. Data processing apparatus comprising:

receiving means for receiving a data structure defining grammar rules for recognition of a natural language input;

analysing means for analysing the data structure to identify positions in the grammar rules at which events in at least one further modality input can occur in the natural language input as a first modality input; and

generating means for generating a modified data structure defining modified grammar rules for recognition of a natural language input as the first modality input in conjunction with associated events in a said further modality input, said data structure defining the association between events in each modality input, and events in said first modality input comprising units in the natural language.

9. Data processing apparatus according to claim 8, wherein said analysing means is adapted to identify positions in the grammar rules at which pauses can occur in the natural language input based on events in at least one said further modality input.

10. Data processing apparatus according to claim 8 or 9, wherein said generating means is adapted to generate a further modified data structure defining said modified grammar rules and the relationships with events in each further modality input.

11. A data processing method comprising the steps of:

receiving a data structure defining grammar rules for recognition of a natural language input;

analysing the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

generating a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein the generating step includes fragmenting the grammar rules in accordance with said identified positions to generate sub grammar rules to form said modified data structure.

12. A data processing method according to claim 11, wherein the analysing step includes identifying the positions in accordance with pause criteria for the natural language input.

13. A data processing method according to claim 11 or claim 12, wherein the generating step includes adding marker means to said data structure to identify the positions in the grammar rules at which pauses can occur in the natural language input.

14. A data processing method according to claim 11, wherein the generating step includes forming a hierarchical structure using said sub grammar rules to form said modified data structure.

15. A data processing method according to claims 11 or 12, wherein the receiving step comprises receiving a data structure defining grammar rules for use in speech recognition of a natural language speech input, and the generating step comprises generating a modified data structure defining modified grammar rules for speech recognition of a natural language speech input with pauses therein.

20. Apparatus for generating data in a computer usable form, the apparatus comprising:

receiving means for receiving a natural language input with a number of pauses therein; and

recognition means for recognising said natural language input using the modified data structure generated using the method of any one of claims 11, 12, 18 or 19 to generate data in computer usable form.

21. Apparatus according to claim 20, wherein said recognising means comprises speech recognition means for recognising a natural language speech input.

31. Processor implementable instructions for controlling a processor to carry out the method of any one of claims 11, 18, 19, 25 or 26.

32. A carrier medium for carrying the processor implementable instructions according to claim 31.

16. A data processing method comprising the steps of:

receiving a data structure defining grammar rules for recognition of a natural language input;

analysing the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

generating a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein the receiving step comprises receiving a data structure defining grammar rules for recognition of a natural language input as a first modality input in conjunction with associated events in at least one further modality input, said data structure defining the association between events in each modality input, events in said first modality input comprising units in the natural language.

17. A data processing method according to claim 16, wherein the analysing step comprises identifying said positions in the grammar rules based on events in at least one said further modality input.

22. Apparatus for generating data in a computer usable form, the apparatus comprising:

first modality receiving data generated for a natural language input by the apparatus of claim 20, said data comprising recognised units of the natural language and comprising data of a first modality input;

further modality receiving means for receiving data identifying events in at least one further modality input;

data structure receiving means for receiving a further modified data structure defining modified grammar rules and the relationships with events in each further modality, said further modified data structure having been generated using the method of claim 16;

analysing means for analysing the first modality input data and each further modality input data to determine if they match with any said modified grammar rule and related events in each further modality; and

generating means for generating computer usable data in dependence upon said analysis by said analysing means.

23. Apparatus according to claim 22, wherein said first modality receiving means is adapted to receive recognition data comprising an ordered list of likely natural language units to accompany the most likely natural language for each natural language unit recognised, and said analysing means is adapted to use said ordered list when the most likely natural language units do not result in a match with any modified grammar rule and related events in each further modality.

24. Apparatus according to claim 22, wherein said first modality receiving means is adapted to receive speech recognition data.

18. A data processing method comprising:

receiving a data structure defining grammar rules for recognition of a natural language input;

analysing the data structure to identify positions in the grammar rules at which events in at least one further modality input can occur in the natural language input as a first modality input; and

generating a modified data structure defining modified grammar rules for recognition of a natural language input as the first modality input in conjunction with associated events in a said further modality input, said data structure defining the association between events in each modality input, and events in said first modality input comprising units in the natural language.

19. A data processing method according to claim 18, wherein the analysing step comprises identifying positions in the grammar rules at which pauses can occur in the natural language input based on events in at least one said further modality input.

25. A data processing method according to claim 19, wherein the generating step includes generating a further modified data structure defining said modified grammar rules and the relationships with events in each further modality input.

26. A method of generating data in a computer usable form, the method comprising:

receiving a natural language input with a number of pauses therein; and

recognising said natural language input using the modified data structure generated using the method of claim 18 to generate data in computer usable form.

27. A method according to claim 26, wherein the recognising step comprises speech recognition of a natural language speech input.

28. A method of generating data in a computer usable form, the method comprising:

a first receiving step of receiving data generated for a natural language input by the method of claim 26, said data comprising recognised units of the natural language, and comprising data of a first modality input;

a second receiving step of receiving data identifying events in at least one further modality input;

a third receiving step of receiving a further modified data structure defining modified grammar rules and the relationship with events in each further modality, said further modified data structure having been generated using the method of claim 25;

analysing the first modality input data and each further modality input data to determine if they match with any said modified grammar rule and related events in each further modality; and

generating computer usable data in dependence upon said analysis.

29. A method according to claim 28, wherein the first receiving step comprises receiving recognition data comprising an ordered list of likely natural language units to accompany the most likely natural language unit for each natural language unit recognised, and the analysis step includes using the ordered list when the most likely natural language units do not result in a match with any modified grammar rule and related events in each further modality.

30. A method according to claim 29, wherein the first receiving step receives speech recognition data.

33. Data processing apparatus comprising:

a receiver operable to receive a data structure defining grammar rules for recognition of a natural language input;

an analyser operable to analyse the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

a generator operable to generate a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein said generator is operable to fragment the grammar rules in accordance with said identified positions to generate sub grammar rules to form said modified data structure.

34. Data processing apparatus comprising:

a receiver operable to receive a data structure defining grammar rules for recognition of a natural language input;

an analyser operable to analyse the data structure to identify positions in the grammar rules at which pauses can occur in the natural language input; and

a generator operable to generate a modified data structure defining modified grammar rules for recognition of a natural language input with pauses therein,

wherein said receiver is operable to receive said data structure defining grammar rules for recognition of a natural language input as a first modality input in conjunction with associated events in at least one further modality input, said data structure defining the association between events in each modality input, and events in said first modality input comprising units in the natural language.

35. Data processing apparatus comprising:

a receiver operable to receive a data structure defining grammar rules for recognition of a natural language input;

an analyser operable to analyse the data structure to identify positions in the grammar rules at which events in at least one further modality input can occur in the natural language input as a first modality input; and

a generator operable to generate a modified data structure defining modified grammar rules for recognition of a natural language input as the first modality input in conjunction with associated events in a said further modality input, said data structure defining the association between events in each modality input, and events in said first modality input comprising units in the natural language.