User interface

• US 8,830,165 B1
• Filed: 01/24/2012
• Issued: 09/09/2014
• Est. Priority Date: 01/24/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Heath, Taliver Brooks, Parker, Evan H.
• Primary Examiner(s): Chow, Van
• Application Number:
  US13/356,728
• Time in Patent Office: 959 Days
• Field of Search: 345/156
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03541 Mouse/trackball convertible devices, in which the same ball is used to track the 2D relative movement
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/0489 using dedicated keyboard keys or combinations thereof
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs

Abstract

A method includes receiving a first input through a hands-based input device. A prompt is then output, and input responsive to the prompt may be received through either of a hands-based input device or a hands-free input device. Responsive to receiving a second input through a hands-free input device, a determination that the second input is responsive to the prompt may be made, an operation based on the second input may be performed, and cessation of the output of the prompt may occur.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device, typed input from a keyboard operatively coupled to the computing device;

outputting, by the computing device and while continuing to receive typed input from the keyboard, a prompt that requests user selection of an action to be performed by the computing device, wherein the typed input received while outputting the prompt is not responsive to the prompt;

receiving, by the computing device while outputting the prompt and while continuing to receive typed input from the keyboard, spoken input through a microphone operatively coupled to the computing device or visual input through a camera operatively coupled to the computing device;

determining whether the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt; and

in response to a determination that the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt, performing, by the computing device, an action corresponding to the spoken input or the visual input and ceasing to output the prompt.

2. The computer-implemented method of claim 1, wherein:

the computing device comprises a mobile computing device that includes a touch-sensitive display,

the keyboard is a virtual keyboard that is displayed on the touch-sensitive display, and

outputting the prompt comprises displaying the prompt on the touch-sensitive display.

3. The computer-implemented method of claim 2, further comprising:

identifying a term in the typed input that is determined to be likely misspelled,

wherein outputting the prompt that requests user selection of the action to be performed by the computing device comprises (i) displaying an indication that the identified term has been determined to be likely misspelled, and (ii) providing a suggested spell-corrected term to replace the identified term, and

wherein determining whether the spoken input or the visual input is responsive to the prompt comprises determining whether the spoken input or the visual input indicates user selection of an action to replace the identified term with the spell-corrected term.

4. The computer-implemented method of claim 3, wherein the prompt, the virtual keyboard, and a text entry area that receives the typed input are concurrently displayed on the touch-sensitive display of the mobile computing device.

5. The computer-implemented method of claim 1, wherein the prompt identifies a particular action to be performed by the computing device, and wherein determining whether the spoken input or the visual input is responsive to the prompt comprises determining whether the user has provided one of (i) an affirmance for the computing device to perform the particular action, and (ii) a selection to not perform the particular action identified by the prompt.

6. The computer-implemented method of claim 5, wherein determining whether the user has provided one of (i) an affirmance for the computing device to perform the particular action, and (ii) a selection to not perform the particular action identified by the prompt comprises identifying, from the visual input, a nodding motion of a head or a shaking motion of the head, respectively.

7. The computer-implemented method of claim 1, wherein determining whether the visual input is responsive to the prompt comprises determining whether a head movement is one of a nodding gesture and a shaking gesture.

8. The computer-implemented method of claim 1, wherein determining whether the spoken input is responsive to the prompt comprises performing speech recognition on audio received through the microphone of the computing device.

9. The computer-implemented method of claim 1, further comprising providing the typed input to a particular application that is executing on the computing device, wherein outputting the prompt comprises generating the prompt by the particular application that is executing on the computing device.

10. The computer-implemented method of claim 1, wherein outputting the prompt does not interfere with receiving typed input through the keyboard of the computing device that is not responsive to the prompt.

11. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more computers, cause performance of operations comprising:

receiving, by the one or more computers, typed input from a keyboard operatively coupled to the one or more computers;

outputting, by the one or more computers and while continuing to receive typed input from the keyboard, a prompt that requests user selection of an action to be performed by the one or more computers, wherein the typed input received while outputting the prompt is not responsive to the prompt;

receiving, by the one or more computers while outputting the prompt and while continuing to receive typed input from the keyboard, spoken input through a microphone operatively coupled to the one or more computers or visual input through a camera operatively coupled to the one or more computers;

determining whether the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt; and

in response to a determination that the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt, performing, by the one or more computers, an action corresponding to the spoken input or the visual input and ceasing to output the prompt.

12. The one or more non-transitory computer-readable media of claim 11, wherein:

the one or more computers comprises a mobile computing device that includes a touch-sensitive display,

the keyboard is a virtual keyboard that is displayed on the touch-sensitive display, and

outputting the prompt comprises displaying the prompt on the touch-sensitive display.

13. The one or more non-transitory computer-readable media of claim 12, wherein the operations further comprise:

identifying a term in the typed input that is determined to be likely misspelled,

wherein outputting the prompt that requests user selection of the action to be performed by the one or more computers comprises (i) displaying an indication that the identified term has been determined to be likely misspelled, and (ii) providing a suggested spell-corrected term to replace the identified term, and

wherein determining whether the spoken input or the visual input is responsive to the prompt comprises determining whether the spoken input or the visual input indicates user selection of an action to replace the identified term with the spell-corrected term.

14. The one or more non-transitory computer-readable media of claim 13, wherein the prompt, the virtual keyboard, and a text entry area that receives the typed input are concurrently displayed on the touch-sensitive display of the mobile computing device.

15. The one or more non-transitory computer-readable media of claim 11, wherein the prompt identifies a particular action to be performed by the one or more computers, and wherein determining whether the spoken input or the visual input is responsive to the prompt comprises determining whether the user has provided one of (i) an affirmance for the one or more computers to perform the particular action, and (ii) a selection to not perform the particular action identified by the prompt.

16. The one or more non-transitory computer-readable media of claim 15, wherein determining whether the user has provided one of (i) an affirmance for the one or more computers to perform the particular action, and (ii) a selection to not perform the particular action identified by the prompt comprises identifying, from the visual input, a nodding motion of a head or a shaking motion of the head, respectively.

17. The one or more non-transitory computer-readable media of claim 11, wherein determining whether the visual input is responsive to the prompt comprises determining whether a head movement is one of a nodding gesture and a shaking gesture.

18. The one or more non-transitory computer-readable media of claim 11, wherein determining whether the spoken input is responsive to the prompt comprises performing speech recognition on audio received through the microphone of the one or more computers.

19. The one or more non-transitory computer-readable media of claim 11, wherein the operations further comprise providing the typed input to a particular application that is executing on the one or more computers, wherein outputting the prompt comprises generating the prompt by the particular application that is executing on the one or more computers.

20. The one or more non-transitory computer-readable media of claim 11, wherein outputting the prompt does not interfere with receiving typed input through the keyboard of the one or more computers that is not responsive to the prompt.

21. A system comprising:

one or more computers; and

one or more non-transitory computer-readable media having instructions stored thereon that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

receiving, by the one or more computers, typed input from a keyboard operatively coupled to the one or more computers;

outputting, by the one or more computers and while continuing to receive typed input from the keyboard, a prompt that requests user selection of an action to be performed by the one or more computers, wherein the typed input received while outputting the prompt is not responsive to the prompt;

receiving, by the one or more computers while outputting the prompt and while continuing to receive typed input through the keyboard, spoken input through a microphone operatively coupled to the one or more computers or visual input through a camera operatively coupled to the one or more computers;

determining whether the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt; and

in response to a determination that the spoken input or the visual input, that was received while the prompt was being output and while typed input was continuing to be received, is responsive to the prompt, performing, by the one or more computers, an action corresponding to the spoken input or the visual input and ceasing to output the prompt.