Multi-modal input on an electronic device

• US 10,713,010 B2
• Filed: 10/24/2018
• Issued: 07/14/2020
• Est. Priority Date: 12/23/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Ballinger, Brandon M., Schalkwyk, Johan, Cohen, Michael H., Byrne, William J., Hafsteinsson, Gudmundur, Lebeau, Michael J.
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US16/169,279
• Publication Number:
  US 20190056909A1
• Time in Patent Office: 629 Days
• Field of Search: 704275, 7042701, 704251, 704231, 704257, 704260, 704270, 704 8, 704 9, 704 10
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A computer-implemented input-method editor process includes receiving a request from a user for an application-independent input method editor having written and spoken input capabilities, identifying that the user is about to provide spoken input to the application-independent input method editor, and receiving a spoken input from the user. The spoken input corresponds to input to an application and is converted to text that represents the spoken input. The text is provided as input to the application.

Claims

1. A method comprising:

displaying, by a computing device, a text field on an electronic display of the computing device;

in response to receiving first interaction data indicating user interaction with the text field, invoking, by the computing device, an Input Method Editor (IME) user interface for display on the electronic display of the computing device, the IME user interface comprising a virtual keyboard and a graphical microphone;

in response to receiving second interaction data indicating user interaction with the graphical microphone of the IME user interface;

enabling, by the computing device, the IME user interface to receive voice input while disabling an ability of the IME user interface to receive typed text input; and

displaying, by the computing device, a visual indication on the electronic display that indicates the IME user interface is enabled to receive voice input; and

recording, by the computing device, an utterance spoken by a user of the computing device while the IME user interface is enabled to receive voice input.

2. The method of claim 1, further comprising, after recording the utterance spoken by the user, providing, by the computing device, a transcription of the utterance spoken by the user in the text field displayed on the electronic display of the computing device.

3. The method of claim 2, further comprising, after recording the utterance spoken by the user and prior to providing the transcription of the utterance spoken by the user in the text field, transcribing, by the computing device, the recorded utterance spoken by the user using a speech recognition system executing on the computing device.

4. The method of claim 2, further comprising, after recording the utterance spoken by the user and prior to providing the transcription of the utterance spoken by the user in the text field:

transmitting audio data corresponding to the recorded utterance spoken by the user to a remote server system having a speech recognition system, the speech recognition system of the remote server system configured to transcribe the recorded utterance spoken by the user using the audio data; and

receiving, at the computing device, the transcription of the utterance spoken by the user from the remote server system.

5. The method of claim 1, wherein the IME user interface comprises a user interface for a multi-modal IME that enables the computing device to receive voice input and typed input.

6. The method of claim 1, wherein the IME user interface comprises an application-independent IME user interface that is enabled to receive input directed to multiple different applications executing on the computing device.

7. The method of claim 1, wherein the text field is displayed in a first portion of the electronic display and the virtual keyboard of the IME user interface is displayed in a second portion of the electronic display, the virtual keyboard comprising a QWERTY keyboard.

8. The method of claim 7, wherein the graphical microphone of the IME user interface is displayed in the first portion of the electronic display next to or in the text field.

9. The method of claim 7, wherein the graphical microphone of the IME user interface is displayed in the second portion of the electronic display.

10. The method of claim 1, further comprising, in response to receiving the second interaction data indicating user interaction with the graphical microphone of the IME user interface, removing the graphical microphone from display in the IME user interface.

11. The method of claim 1, wherein displaying the visual indication on the electronic display comprises displaying the visual indication in place of the virtual keyboard.

12. The method of claim 1, further comprising:

in response to receiving the second interaction data indicating user interaction with the graphical microphone of the IME user interface, displaying, by the computing device, a voice input cancellation control on the electronic display; and

in response to receiving third interaction data indicating user interaction with the voice input cancellation control displayed on the electronic display, disabling, by the computing device, an ability of the IME user interface to receive voice input.

13. The method of claim 1, further comprising:

executing, by the computing device, software that provides the text field and displays a graphical user interface on the electronic display, the graphical user interface for the software comprising a first portion including the text field and a second portion,

wherein invoking the IME user interface for display on the electronic display comprises displaying the IME user interface over the second portion of the graphical user interface for the software providing the text field, the second portion of the graphical user interface for the software not visible while the IME user interface is displayed.

14. The method of claim 13, wherein the first portion of the graphical user interface for the software remains visible while the IME user interface is displayed.

15. The method of claim 13, wherein the software executing on the computing device comprises an application or an operating system for the computing device.

16. A computing device:

data processing hardware;

an electronic display in communication with the data processing hardware; and

memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising;

displaying a text field on the electronic display;

in response to receiving first interaction data indicating user interaction with the text field, invoking an Input Method Editor (IME) user interface for display on the electronic display, the IME user interface comprising a virtual keyboard and a graphical microphone;

in response to receiving second interaction data indicating user interaction with the graphical microphone of the IME user interface;

enabling the IME user interface to receive voice input while disabling an ability of the IME user interface to receive typed text input; and

displaying a visual indication on the electronic display that indicates the IME user interface is enabled to receive voice input; and

recording an utterance spoken by a user of the computing device while the IME user interface is enabled to receive voice input.

17. The computing device of claim 16, wherein the operations further comprise, after recording the utterance spoken by the user, providing a transcription of the utterance spoken by the user in the text field displayed on the electronic display of the computing device.

18. The computing device of claim 17, wherein the operations further comprise, after recording the utterance spoken by the user and prior to providing the transcription of the utterance spoken by the user in the text field, transcribing the recorded utterance spoken by the user using a speech recognition system executing on the computing device.

19. The computing device of claim 17, wherein the operations further comprise, after recording the utterance spoken by the user and prior to providing the transcription of the utterance spoken by the user in the text field:

transmitting audio data corresponding to the recorded utterance spoken by the user to a remote server system having a speech recognition system, the speech recognition system of the remote server system configured to transcribe the recorded utterance spoken by the user using the audio data; and

receiving the transcription of the utterance spoken by the user from the remote server system.

20. The computing device of claim 16, wherein the IME user interface comprises a user interface for a multi-modal IME that enables the computing device to receive voice input and typed input.

21. The computing device of claim 16, wherein the IME user interface comprises an application-independent IME user interface that is enabled to receive input directed to multiple different applications executing on the computing device.

22. The computing device of claim 16, wherein the text field is displayed in a first portion of the electronic display and the virtual keyboard of the IME user interface is displayed in a second portion of the electronic display, the virtual keyboard comprising a QWERTY keyboard.

23. The computing device of claim 22, wherein the graphical microphone of the IME user interface is displayed in the text field in the first portion of the electronic display.

24. The computing device of claim 22, wherein the graphical microphone of the IME user interface is displayed in the second portion of the electronic display.

25. The computing device of claim 16, wherein the operations further comprise, in response to receiving the second interaction data indicating user interaction with the graphical microphone of the IME user interface, removing the graphical microphone from display in the IME user interface.

26. The computing device of claim 16, wherein displaying the visual indication on the electronic display comprises displaying the visual indication in place of the virtual keyboard.

27. The computing device of claim 16, wherein the operations further comprise:

in response to receiving the second interaction data indicating user interaction with the graphical microphone of the IME user interface, displaying a voice input cancellation control on the electronic display; and

in response to receiving third interaction data indicating user interaction with the voice input cancellation control displayed on the electronic display, disabling an ability of the IME user interface to receive voice input.

28. The computing device of claim 16, wherein the operations further comprise:

executing software that provides the text field and displays a graphical user interface for on the electronic display, the graphical user interface for the software comprising a first portion including the text field and a second portion,

wherein invoking the IME user interface on the electronic display comprises displaying the IME user interface over the second portion of the graphical user interface for the software providing the text field, the second portion of the graphical user interface for the software not visible while the IME user interface is displayed.

29. The computing device of claim 28, wherein the first portion of the graphical user interface for the software remains visible while the IME user interface is displayed.

30. The computing device of claim 28, wherein the software executing on the computing device comprises an application or an operating system for the computing device.