Allowing spelling of arbitrary words

• US 10,229,109 B1
• Filed: 09/11/2017
• Issued: 03/12/2019
• Est. Priority Date: 01/06/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Cherepanov, Evgeny A., Skobeltsyn, Gleb, Foerster, Jakob Nicolaus, Aleksic, Petar, Michaely, Assaf Avner Hurwitz
• Primary Examiner(s): Patel, Shreyans A
• Application Number:
  US15/700,614
• Time in Patent Office: 547 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/086 Recognition of spelled words
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for natural language processing. One of the methods includes receiving a first voice input from a user device; generating a first recognition output; receiving a user selection of one or more terms in the first recognition output; receiving a second voice input spelling a correction of the user selection; determining a corrected recognition output for the selected portion; and providing a second recognition output that merges the first recognition output and the corrected recognition output.

Claims

1. A computer-implemented method comprising:

receiving, at a user device, a first voice input from a user of the user device, the first voice input comprising a search query;

transmitting the first voice input over a network from the user device to a search system, the first voice input when received by the search system causing the search system to;

generate a transcription of the search query based on the first voice input, the transcription of the search query including one or more terms; and

transmit the transcription of the search query over the network to the user device;

displaying, by the user device, the transcription of the search query in a user interface of the user device;

receiving, at the user device, a selection indication indicating a user selection in the user interface of one of the one or more terms of the transcription of the search query displayed in the user interface;

receiving, at the user device, a second voice input from the user, the second voice input including a series of letters that spells out a correction of the selected term of the transcription of the search query; and

transmitting the second voice input over the network from the user device to the search system, the second voice input when received by the search system causing the search system to;

generate an updated transcription of the search query based on the first voice input and the second voice input;

obtain one or more search results responsive to the updated transcription of the search query; and

transmit the updated transcription of the search query and the one or more search results over the network to the user device.

2. The method of claim 1, wherein receiving the indication of the user selection in the user interface comprises receiving a touch input by the user in the user interface via a finger or stylus that selects the one of the one or more terms of the transcription of the search query displayed in the user interface.

3. The method of claim 1, further comprising:

receiving, at the user device, the one or more search results responsive to the updated transcription of the search query; and

displaying, by the user device, the one or more obtained search results responsive to the updated transcription of the search query in the user interface of the user device.

4. The method of claim 1, further comprising:

receiving, at the user device, one or more initial search results responsive to the transcription of the search query;

displaying, by the user device, the one or more initial search results response to the transcription of the search query in the user interface of the user device; and

in response to receiving the updated transcription of the search query, updating, by the user device, the one or more initial search results of the search query displayed in the user interface with the one or more obtained search results responsive to the updated transcription of the search query.

5. The method of claim 1, further comprising, in response to receiving the updated transcription of the search query from the search system, displaying, by the user device, the updated transcription of the search query in the user interface of the user device.

6. The method of claim 5, wherein displaying the updated transcription of the search query comprises replacing the selected term of the transcription of the search query displayed in the user interface with the correction of the selected term spelled out by the series of words in the second voice input.

7. The method of claim 1, further comprising, in response to receiving the selection indication indicating the user selection in the user interface, superimposing, by the user device, a graphical indicator in the user interface that indicates the selected term of the one or more terms of the transcription of the search query.

8. A system comprising:

one or more computers of a user device and one or more storage devices of the user device storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving a first voice input from a user of the user device, the first voice input comprising a search query;

transmitting the first voice input over a network from the user device to a search system, the first voice input when received by the search system causing the search system to;

generate a transcription of the search query based on the first voice input, the transcription of the search query including one or more terms; and

transmit the transcription of the search query over the network to the user device;

displaying the transcription of the search query in a user interface of the user device;

receiving a selection indication indicating a user selection in the user interface of one of the one or more terms of the transcription of the search query displayed in the user interface;

receiving a second voice input from the user, the second voice input including a series of letters that spells out a correction of the selected term of the transcription of the search query; and

transmitting the second voice input over the network from the user device to the search system, the second voice input when received by the search system causing the search system to;

generate an updated transcription of the search query based on the first voice input and the second voice input;

obtain one or more search results responsive to the updated transcription of the search query; and

transmit the updated transcription of the search query and the one or more search results over the network to the user device.

9. The system of claim 8, wherein receiving the selection indication indicating the user selection in the user interface comprises receiving a touch input by the user in the user interface via a finger or stylus that selects the one of the one or more terms of the transcription of the search query displayed in the user interface.

10. The system of claim 8, wherein the operations further comprise:

receiving the one or more search results responsive to the updated transcription of the search query; and

displaying the one or more obtained search results responsive to the updated transcription of the search query in the user interface of the user device.

11. The system of claim 8, wherein the operations further comprise:

receiving one or more initial search results responsive to the transcription of the search query;

displaying the one or more initial search results response to the transcription of the search query in the user interface of the user device; and

in response to receiving the updated transcription of the search query, updating the one or more initial search results of the search query displayed in the user interface with the one or more obtained search results responsive to the updated transcription of the search query.

12. The system of claim 8, wherein the operations further comprise, in response to receiving the updated transcription of the search query from the search system, displaying the updated transcription of the search query in the user interface of the user device.

13. The system of claim 12, wherein displaying the updated transcription of the search query comprises replacing the selected term of the transcription of the search query displayed in the user interface with the correction of the selected term spelled out by the series of words in the second voice input.

14. The system of claim 8, wherein the operations further comprise, in response to receiving the user input in the user interface, superimposing, by the user device, a graphical indicator in the user interface that indicates the selected term of the one or more terms of the transcription of the search query.

15. One or more non-transitory computer-readable storage media of a user device encoded with instructions that, when executed by one or more computers of the user device, cause the one or more computers to perform operations comprising:

receiving a first voice input from a user of the user device, the first voice input comprising a search query;

transmitting the first voice input over a network from the user device to a search system, the first voice input when received by the search system causing the search system to;

generate a transcription of the search query based on the first voice input, the transcription of the search query including one or more terms; and

transmit the transcription of the search query over the network to the user device;

displaying the transcription of the search query in a user interface of the user device;

receiving a selection indication indicating a user selection in the user interface of one of the one or more terms of the transcription of the search query displayed in the user interface;

receiving a second voice input from the user, the second voice input including a series of letters that spells out a correction of the selected term of the transcription of the search query; and

transmitting the second voice input over the network from the user device to the search system, the second voice input when received by the search system causing the search system to;

generate an updated transcription of the search query based on the first voice input and the second voice input;

obtain one or more search results responsive to the updated transcription of the search query; and

transmit the updated transcription of the search query and the one or more search results over the network to the user device.

16. The non-transitory computer-readable storage media of claim 15, wherein receiving the selection indication indicating the user selection in the user interface comprises receiving a touch input by the user in the user interface via a finger or stylus that selects the one of the one or more terms of the transcription of the search query displayed in the user interface.

17. The non-transitory computer-readable storage media of claim 15, wherein the operations further comprise:

receiving the one or more search results responsive to the updated transcription of the search query; and

displaying the one or more obtained search results responsive to the updated transcription of the search query in the user interface of the user device.

18. The non-transitory computer-readable storage media of claim 15, wherein the operations further comprise:

receiving one or more initial search results responsive to the transcription of the search query;

displaying the one or more initial search results response to the transcription of the search query in the user interface of the user device; and

in response to receiving the updated transcription of the search query, updating the one or more initial search results of the search query displayed in the user interface with the one or more obtained search results responsive to the updated transcription of the search query.

19. The non-transitory computer-readable storage media of claim 15, wherein the operations further comprise, in response to receiving the updated transcription of the search query from the search system, displaying the updated transcription of the search query in the user interface of the user device.

20. The non-transitory computer-readable storage media of claim 15, wherein the operations further comprise, in response to receiving the user input in the user interface, superimposing, by the user device, a graphical indicator in the user interface that indicates the selected term of the one or more terms of the transcription of the search query.