Multi-Modal Input on an Electronic Device

• US 20160132293A1
• Filed: 01/05/2016
• Published: 05/12/2016
• Est. Priority Date: 12/23/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Ballinger, Brandon M., Schalkwyk, Johan, Cohen, Michael H., Byrne, William J., Hafsteinsson, Gudmundur, LeBeau, Michael J.
• Granted Patent: US 10,157,040 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A computer-implemented input-method editor process includes receiving a request from a user for an application-independent input method editor having written and spoken input capabilities, identifying that the user is about to provide spoken input to the application-independent input method editor, and receiving a spoken input from the user. The spoken input corresponds to input to an application and is converted to text that represents the spoken input. The text is provided as input to the application.

Claims

1-20. -20. (canceled)

21. A computer-implemented method, comprising:

displaying, on an electronic display of a computing device, a user interface that includes a voice input control and a virtual keyboard;

in response to receiving data that indicates user interaction with the voice input control of the user interface that includes the voice input control and the virtual keyboard, enabling the computing device to receive voice input; and

detecting, by the computing device, an utterance as a result of the computing device being enabled to receive voice input.

22. The computer-implemented method of claim 21, further comprising receiving data that indicates user interaction with a text field displayed on the electronic display of the computing device,

wherein the user interface that includes the voice input control and the virtual keyboard is displayed in response to receiving the data that indicates user interaction with the text field.

23. The computer-implemented method of claim 22, wherein the user interface is a user interface for a multi-modal input method editor that enables the computing device to receive voice input and typed input.

24. The computer-implemented method of claim 23, further comprising:

displaying, on the electronic display of the computing device and before receiving the data that indicates user interaction with the text field, a user interface for an application that provides the text field; and

in response to receiving the data that indicates user interaction with the text field, displaying the user interface for the multi-modal input method editor, including the voice input control and the virtual keyboard, over a first portion of the user interface for the application that provides the text field, such that the first portion of the user interface for the application is not visible while the user interface for the multi-modal input method editor is displayed,

wherein a second portion of the user interface for the application remains visible while the user interface for the multi-modal input method editor is displayed, wherein the second portion of the user interface for the application includes the text field.

25. The computer-implemented method of claim 24, further comprising displaying the user interface for the multi-modal input method editor on the electronic display of the computing device below the second portion of the user interface for the application.

26. The computer-implemented method of claim 22, wherein the text field is provided in a web page presented by the computing device.

27. The computer-implemented method of claim 21, further comprising, in response to receiving the data that indicates user interaction with the voice input control of the user interface:

removing the virtual keyboard from display in the user interface; and

displaying, in the user interface with the virtual keyboard removed, an indication that the computing device is enabled to receive voice input;

28. The computer-implemented method of claim 27, further comprising, in response to receiving the data that indicates user interaction with the voice input control of the user interface, removing the voice input control from display in the user interface.

29. The computer-implemented method of claim 27, wherein displaying the indication that the computing device is enabled to listen for voice input comprises displaying a graphical representation of a microphone.

30. The computer-implemented method of claim 27, further comprising:

in response to receiving the data that indicates user interaction with the voice input control of the user interface, displaying, in the user interface with the virtual keyboard removed, a voice input cancellation control; and

in response to receiving data that indicates user interaction with the voice input cancellation control, disabling a capability of the computing device to receive voice input so as to cease receiving voice input.

31. The computer-implemented method of claim 21, wherein the user interface is a user interface for an application-independent input method editor that is enabled to receive input directed to multiple different applications on the computing device.

32. The computer-implemented method of claim 21, wherein displaying the user interface comprises displaying a first virtual button that comprises the voice input control and a plurality of second virtual buttons that comprise the virtual keyboard.

33. The computer-implemented method of claim 32, wherein displaying the user interface comprises displaying the plurality of second virtual buttons that comprise the virtual keyboard in a plurality of rows of virtual buttons, wherein displaying the first virtual button that comprises the voice input control comprises displaying the first virtual button in one of the plurality of rows of virtual buttons.

34. The computer-implemented method of claim 21, wherein the virtual keyboard is a QWERTY keyboard.

35. The computer-implemented method of claim 21, wherein displaying the voice input control in the user interface comprises displaying a graphical representation of a microphone.

36. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more processors, cause performance of operations comprising:

displaying, on an electronic display of a computing device, a user interface that includes a voice input control and a virtual keyboard;

in response to receiving data that indicates user interaction with the voice input control of the user interface that includes the voice input control and the virtual keyboard, enabling the computing device to receive voice input; and

detecting, by the computing device, an utterance as a result of the computing device being enabled to receive voice input.

37. A computing device, comprising:

an electronic display;

one or more processors; and

one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause performance of operations comprising;

displaying, on the electronic display of the computing device, a user interface that includes a voice input control and a virtual keyboard;

in response to receiving data that indicates user interaction with the voice input control of the user interface that includes the voice input control and the virtual keyboard, enabling the computing device to receive voice input; and

detecting, by the computing device, an utterance as a result of the computing device being enabled to receive voice input.

38. A computer-implemented method, comprising:

displaying, on an electronic display of a computing device, a user interface that includes a voice input control and a virtual keyboard;

in response to receiving data that indicates user interaction with the voice input control of the user interface that includes the voice input control and the virtual keyboard, enabling the computing device to receive voice input;

detecting, by the computing device, an utterance as a result of the computing device being enabled to receive voice input;

providing, by the computing device and to a server system, audio data that corresponds to the utterance; and

providing, by the computing device and for display in the text field, a transcription of the utterance received from the server system.

39. The computer-implemented method of claim 38, further comprising:

determining a context associated with the utterance;

providing, by the computing device and to the server system, data that indicates the context associated with the utterance; and

receiving the transcription at the computing device and from the server system, wherein the transcription was generated by the server system based at least in part on the data that indicates the context associated with the utterance.

40. The computer-implemented method of claim 38, further comprising:

removing the virtual keyboard and the voice input control from display in the user interface; and

displaying, in the user interface with the virtual keyboard and the voice input control removed, and for a period of time between providing the audio to the server system and receiving the transcription from the server system, an indication that a transcription process is underway to convert the utterance to text.

41. The computer-implemented method of claim 48, further comprising:

receiving, by the computing device and from the server system, a plurality of candidate transcriptions of the utterance;

in response to receiving the plurality of candidate transcriptions of the utterance, displaying on the electronic display of the computing device at least some of the plurality of candidate transcriptions; and

identifying that user input selected a first candidate transcription among the at least some of the plurality of candidate transcriptions,

wherein providing the transcription of the utterance for display in the text field comprises providing the first candidate transcription for display in the text field.