User interface

• US 9,563,308 B1
• Filed: 12/08/2015
• Issued: 02/07/2017
• Est. Priority Date: 01/24/2012
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Heath, Taliver Brooks, Parker, Evan H.
• Primary Examiner(s): Chow, Van
• Application Number:
  US14/962,611
• Time in Patent Office: 427 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/0227 Cooperation and interconnection of the input arrangement with other functional units of a computer G06F3/023 - G06F3/037 take precedence
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03541 Mouse/trackball convertible devices, in which the same ball is used to track the 2D relative movement
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/0489 using dedicated keyboard keys or combinations thereof
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs

Abstract

A method includes receiving a first input through a hands-based input device. A prompt is then output, and input responsive to the prompt may be received through either of a hands-based input device or a hands-free input device. Responsive to receiving a second input through a hands-free input device, a determination that the second input is responsive to the prompt may be made, an operation based on the second input may be performed, and cessation of the output of the prompt may occur.

Claims

1. A computer-implemented method, comprising:

outputting, by a computing device, a prompt that requests user selection of an action to be performed by the computing device;

while outputting the prompt,

monitoring, by the computing device, typed input entered using a keyboard to determine whether the typed input is responsive to the prompt, and

monitoring, by the computing device, hands-free input entered using a camera or a microphone to determine whether the hands-free input is responsive to the prompt; and

determining that one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt, and in response;

performing, by the computing device, a particular action indicated by the one of the typed input and the hands-free input that is determined to be responsive to the prompt, and

ceasing to output the prompt.

2. The computer-implemented method of claim 1, wherein:

the typed input entered using the keyboard is not responsive to the prompt, and

determining that the one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt comprises determining that the hands-free input is responsive to the prompt.

3. The computer-implemented method of claim 2, wherein the hands-free input comprises visual input entered using the camera.

4. The computer-implemented method of claim 3, wherein determining that the hands-free input is responsive to the prompt comprises:

determining that the visual input entered using the camera shows a motion of a person'"'"'s head; and

identifying that the motion of the person'"'"'s head corresponds to a pre-defined nodding gesture that indicates a selection to perform a particular action identified by the prompt, or that the motion of the person'"'"'s head corresponds to a pre-defined shaking gesture that indicates a selection to not perform the particular action identified by the prompt.

5. The computer-implemented method of claim 2, wherein the hands-free input comprises spoken input entered using the microphone.

6. The computer-implemented method of claim 5, wherein determining that the hands-free input is responsive to the prompt comprises performing speech recognition on the spoken input to (i) determine whether the spoken input includes one or more terms that indicate a selection to perform a particular action identified by the prompt, or (ii) determine whether the spoken input includes one or more terms that indicate a selection to not perform the particular action identified by the prompt.

7. The computer-implemented method of claim 2, further comprising, while outputting the prompt and before determining that the hands-free input is responsive to the prompt, receiving a first portion of the typed input and determining that the first portion of the typed input is not responsive to the prompt.

8. The computer-implemented method of claim 1, wherein:

the hands-free input entered using the camera or the microphone is not responsive to the prompt, and

determining that the one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt comprises determining that the typed input is responsive to the prompt.

9. The computer-implemented method of claim 1, wherein the typed input is directed to a particular activity running on the computing device;

further comprising beginning to receive the typed input that is directed to the particular activity before outputting the prompt, and continuing to receive the typed input that is directed to the particular activity while outputting the prompt, the typed input not being re-directed from the particular activity as a result of outputting the prompt.

10. The computer-implemented method of claim 1, wherein:

the computing device is a mobile computing device, and

outputting the prompt comprises displaying the prompt on a touch-sensitive display of the mobile computing device.

11. The computer-implemented method of claim 10, wherein the keyboard is a virtual keyboard that is displayed on the touch-sensitive display.

12. The computer-implemented method of claim 1, further comprising:

determining that a particular term indicated by the typed input is likely misspelled,

wherein outputting the prompt that requests user selection of the action to be performed by the computing device comprises (i) presenting, on an electronic display of the computing device, an indication that the particular term is likely misspelled, and (ii) presenting, on the electronic display of the computing device, a suggested spell-corrected term to replace the particular term,

wherein determining that the one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt comprises determining that the one of the typed input and the hands-free input indicates a selection to replace or to not replace the particular term with the suggested spell-corrected term.

13. The computer-implemented method of claim 8, further comprising, while outputting the prompt and before determining that the typed input is responsive to the prompt, receiving a first portion of the hands-free input and determining that the first portion of the hands-free input is not responsive to the prompt.

14. One or more non-transitory computer-readable media having instructions stored thereon that, when executed by one or more processors, cause performance of operations comprising:

outputting, by a computing device, a prompt that requests user selection of an action to be performed by the computing device;

while outputting the prompt,

monitoring, by the computing device, typed input entered using a keyboard to determine whether the typed input is responsive to the prompt, and

monitoring, by the computing device, hands-free input entered using a camera or a microphone to determine whether the hands-free input is responsive to the prompt; and

determining that one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt, and in response;

performing, by the computing device, a particular action indicated by the one of the typed input and the hands-free input that is determined to be responsive to the prompt, and

ceasing to output the prompt.

15. The one or more non-transitory computer-readable media of claim 14, wherein:

the typed input entered using the keyboard is not responsive to the prompt, and

determining that the one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt comprises determining that the hands-free input is responsive to the prompt.

16. The one or more non-transitory computer-readable media of claim 15, wherein the hands-free input comprises visual input entered using the camera.

17. The one or more non-transitory computer-readable media of claim 16, wherein determining that the hands-free input is responsive to the prompt comprises:

determining that the visual input entered using the camera shows a motion of a person'"'"'s head; and

identifying that the motion of the person'"'"'s head corresponds to a pre-defined nodding gesture that indicates a selection to perform a particular action identified by the prompt, or that the motion of the person'"'"'s head corresponds to a pre-defined shaking gesture that indicates a selection to not perform the particular action identified by the prompt.

18. The one or more non-transitory computer-readable media of claim 15, wherein the hands-free input comprises spoken input entered using the microphone.

19. The one or more non-transitory computer-readable media of claim 18, wherein determining that the hands-free input is responsive to the prompt comprises performing speech recognition on the spoken input to (i) determine whether the spoken input includes one or more terms that indicate a selection to perform a particular action identified by the prompt, or (ii) determine whether the spoken input includes one or more terms that indicate a selection to not perform the particular action identified by the prompt.

20. The one or more non-transitory computer-readable media of claim 15, wherein the operations further comprise, while outputting the prompt and before determining that the hands-free input is responsive to the prompt, receiving a first portion of the typed input and determining that the first portion of the typed input is not responsive to the prompt.

21. The one or more non-transitory computer-readable media of claim 14, wherein:

the hands-free input entered using the camera or the microphone is not responsive to the prompt, and

determining that the one of the typed input entered using the keyboard and the hands-free input entered using the camera or the microphone is responsive to the prompt comprises determining that the typed input is responsive to the prompt.