COORDINATED SPEECH AND GESTURE INPUT

• US 20150199017A1
• Filed: 01/10/2014
• Published: 07/16/2015
• Est. Priority Date: 01/10/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Murillo, Oscar, Stifelman, Lisa, Bastien, David, Schwesinger, Mark, Song, Margaret
• Application Number:
  US14/152,815
• Publication Number:
  US 20150199017A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

A method to be enacted in a computer system operatively coupled to a vision system and to a listening system. The method applies natural user input to control the computer system. It includes the acts of detecting verbal and non-verbal touchless input from a user of the computer system, selecting one of a plurality of user-interface objects based on coordinates derived from the non-verbal, touchless input, decoding the verbal input to identify a selected action from among a plurality of actions supported by the selected object, and executing the selected action on the selected object.

Claims

1. Enacted in a computer system operatively coupled to a vision system, a method to apply natural user input (NUI) to control the computer system, the method comprising:

detecting a gesture of a user of the computer system, the gesture characterized by a position of a hand with respect to a body of the user;

selecting, based on coordinates derived from the position of the hand, one of a plurality of user-interface (UI) objects displayed on a UI in sight of the user, the selected UI object supporting a plurality of actions;

detecting vocalization from the user;

decoding the vocalization to identify a selected action from among the plurality of actions supported by the selected UI object; and

executing the selected action on the selected UI object.

2. The method of claim 1, wherein the selected UI object represents an executable process in the computer system, the method further comprising:

launching the executable process after the vocalization is decoded; and

reporting the selected action to the executable process.

3. The method of claim 1, further comprising, prior to detecting the gesture and vocalization, identifying the plurality of actions supported by the selected UI object.

4. The method of claim 1, further comprising mapping the position of the hand of the user to the coordinates, and displaying a pointer graphic on the UI at the coordinates.

5. Enacted in a computer system operatively coupled to a vision system, a method to apply natural user input (NUI) to control the computer system, the method comprising:

detecting one of non-verbal, touchless input and verbal input as a first type of natural user input;

detecting a second type of natural user input, the second type being verbal input if the first type is non-verbal touchless input, the second type being non-verbal touchless input if the first type is verbal input;

using the first type of user input to constrain a return-parameter space of the second type of user input to reduce noise in the first type of input;

selecting a user-interface (UI) object based on the first type of user input;

determining a selected action for the selected UI object based on the second type of user input; and

executing the selected action on the selected UI object.

6. The method of claim 5, wherein selection of the UI object does not specify the selected action, and wherein determining the selected action does not specify a receiver of the selected action.

7. The method of claim 5, wherein the non-verbal touchless user input provides one or more of a pointing direction of the user, a head or body orientation of the user, a pose or posture of the user, and a gaze direction or focal point of the user.

8. The method of claim 5, wherein the non-verbal, touchless user input is used to constrain the return-parameter space of the verbal user input.

9. The method of claim 8, wherein the non-verbal, touchless user input selects a UI object that supports a subset of actions recognizable by a speech-recognition engine of the computer system, the method further comprising:

limiting a vocabulary of the speech-recognition engine to the subset of actions supported by the UI object.

10. The method of claim 5, wherein the UI object is selected based on the non-verbal, touchless user input and the selected action is determined based on the verbal user input.

11. The method of claim 10, wherein determining the selected action for the selected UI object includes:

decoding a generic term for a receiver of the selected action; and

instantiating the generic receiver term based on context derived from the non-verbal, touchless user input.

12. The method of claim 11, wherein the generic receiver term is instantiated differently for different forms of non-verbal, touchless user input.

13. The method of claim 5, wherein the verbal user input is used to constrain the return-parameter space of the non-verbal, touchless user input.

14. The method of claim 13, wherein the non-verbal, touchless user input is consistent with user selection of a plurality of nearby UI objects that differ with respect to supported actions, the method further comprising:

selecting, from the plurality of nearby UI objects, one that supports the action indicated by the verbal user input, while dismissing a UI object that does not support the indicated action.

15. The method of claim 5, wherein the UI object is selected based on the verbal user input and the selected action is determined based on the non-verbal, touchless user input.

16. Enacted in a computer system operatively coupled to a vision system, a method to apply natural user input (NUI) to control the computer system, the method comprising:

detecting non-verbal, touchless input;

computing, based on the non-verbal, touchless user input, coordinates on a user interface (UI) arranged in sight of the user;

detecting a vocalization;

if the coordinates are within a first range, operating a speech-recognition engine of the computer system to interpret the vocalization using a first set of vocabulary; and

if the coordinates are within a second range, different than the first range, operating the speech-recognition engine to interpret the vocalization using a second set of vocabulary, which differs from the first set.

17. The method of claim 16, wherein the non-verbal, touchless user input includes a position of a hand of the user with respect to the user'"'"'s body, and wherein computing the target coordinates includes mapping the hand position to the target coordinates.

18. The method of claim 16, wherein the first set of vocabulary includes actions supported by a UI object displayed within the first range.

19. The method of claim 18, wherein computing coordinates in the first range activates the first UI object.

20. The method of claim 18, wherein computing coordinates in the second range invokes an operating system (OS) of the computer system, and wherein the second set of vocabulary is a combined, OS-level vocabulary.