Multi mode voice assistant for the hearing disabled

• US 10,468,022 B2
• Filed: 04/03/2017
• Issued: 11/05/2019
• Est. Priority Date: 04/03/2017
• Status: Active Grant

• Current Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Min, Jun-ki, Ali, Mir Farooq, Dabhi, Navin Tulsibhai
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US15/477,918
• Publication Number:
  US 20180286392A1
• Time in Patent Office: 946 Days
• Field of Search: 704275
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A voice assistant (VA) can switch between a voice input mode, in which the VA produces audible responses to voice queries, and a gesture input mode that can be triggered by a predetermined gesture, in which the VA produces visual responses to gesture-based queries.

Claims

1. A device, comprising:

at least one processor; and

storage accessible to the at least one processor, the storage comprising instructions executable by the processor to;

establish at least a first input mode to a voice assistant (VA), the first input mode comprising voice input to at least one microphone;

receive at least one input;

responsive to the at least one input, switch to at least a second input mode to the VA, the second input mode comprising gesture input to at least one sensor; and

present on a display at least one user interface (UI) comprising at least first and second elements, the first and second elements being at least one of selector elements and prompts that are operable to enable respective first and second output modes in response to input in the first input mode, the UI further comprising a human-perceptible indication of a recommended output mode juxtaposed with one of the first or second elements and not the other second or first element to indicate that the output mode associated with the first or second element by which the human-perceptible indication of a recommended output mode is juxtaposed is recommended over the output mode associated with the second or first element by which the human-perceptible indication of a recommended output mode is not juxtaposed, and information in a current output mode and a third element selectable to select both output modes in response to input in the second input mode comprising gesture input.

2. The device of claim 1, wherein the at least one input comprises a gesture in free space.

3. The device of claim 1, wherein the at least one sensor comprises a camera.

4. The device of claim 1, wherein the at least one sensor comprises a radar device.

5. The device of claim 1, wherein the sensor is on a wearable device attachable to a person.

6. The device of claim 1, wherein the sensor is on the VA.

7. The device of claim 1, wherein the VA outputs responses on at least one speaker in the first input mode.

8. The device of claim 1, wherein the VA outputs responses on at least one visual display in the second input mode.

9. A method, comprising:

configuring a voice assistant (VA) in an audible mode in which audible output is generated responsive to voice input;

identifying a first gesture to switch to a gesture mode;

responsive to identifying the first gesture, configuring the VA in a gesture mode in which a response embodied by visible output is generated and visibly presented responsive to gesture input and the response is also embodied by audible output and presented responsive to the gesture input;

identifying a second gesture to switch to the audible mode;

responsive to identifying the second gesture, configuring the VA in the audible mode; and

the first gesture to configure the VA in the gesture mode being identified responsive to identifying at least one hand being held up, the second gesture to switch to the audible mode being identified responsive to identifying at least one hand being held down.

10. The method of claim 9, comprising:

emitting radar signals toward a user; and

receiving reflections of the radar signals using a sensor on a clip-type device attachable to clothing, the reflections being used to identify the first gesture.

11. The method of claim 9, comprising:

emitting radar signals toward a user; and

receiving reflections of the radar signals using a sensor on the VA, the reflections being used to identify the first gesture.

12. The method of claim 9, comprising:

emitting radar signals toward a user; and

receiving reflections of the radar signals using a sensor on a mobile communication device, the reflections being used to identify at least the first gesture.

13. The method of claim 9, comprising using input signals from at least two sensors to identify the first gesture as follows:

f_g(S)=C_g, where

f_g is a recognition function for gesture g given by w₁s₁+w₂s₂+ . . . w_ns_n, where w_i is a weight value for the i^th sensor output s_i;

S is a set of sensors, {s₁, s₂, . . . s_n};

C_g is a degree of confidence for gesture g.

14. A device, comprising:

at least one processor; and

storage accessible to the at least one processor, the storage comprising instructions executable by the processor to;

establish at least a first input mode to a voice assistant (VA), the first input mode comprising voice input to at least one microphone;

receive at least one input;

responsive to the at least one input, switch to at least a second input mode to the VA, the second input mode comprising gesture input; and

present on a display at least one user interface (UI) comprising at least first and second elements, the first and second elements being at least one of selector elements and prompts to enable respective first and second output modes in response to input in the first input mode, the UI further comprising a human-perceptible indication of a recommended output mode juxtaposed with one of the first or second elements and not the other second or first element to indicate that the output mode associated with the first or second element by which the human-perceptible indication of a recommended output mode is juxtaposed is recommended over the output mode associated with the second or first element by which the human-perceptible indication of a recommended output mode is not juxtaposed.

15. The device of claim 14, wherein the at least one input comprises a gesture in free space.

16. The device of claim 14, comprising at least one camera to image gestures.

17. The device of claim 14, comprising at least one radar device to sense gestures.

18. The device of claim 14, comprising at least one gesture sensor on a wearable device attachable to a person.

19. The device of claim 14, comprising at least one gesture sensor on the VA.

20. The device of claim 14, wherein the VA outputs responses on at least one speaker in the first input mode.