Supplementing speech commands with gestures

• US 9,390,726 B1
• Filed: 12/30/2013
• Issued: 07/12/2016
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Smus, Boris, Kauffmann, Alejandro Jose, Plagemann, Christian
• Primary Examiner(s): He, Jialong
• Application Number:
  US14/142,973
• Time in Patent Office: 925 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence

Abstract

Described is a system and technique to supplement speech commands with gestures. A user interface may be improved by providing the ability for a user to intuitively provide speech commands with the aid of gestures. By providing gestures contemporaneously with a speech command, the user may delimit the commencement and end of a command thereby allowing the system to provide an immediate response. In addition, gestures may be detected in order to determine a source of a provided speech command, and accordingly, user specific actions may be performed based on the identity of the source.

Claims

1. A device for interpreting a voice-based input, the device operatively coupled to a camera that has a field-of-view, the device comprising:

a processor configured to;

detect a presence of (i) a first set of one or more people that are within a field-of-view of the camera, and (ii) a second set of one or more people that are outside the field-of-view of the camera and are within a predetermined distance from the camera;

receive a speech command;

detect a first touchless gesture performed by a particular person from among the first set of one or more people that are within the field-of-view of the camera and that are within the predetermined distance from the camera;

determine, based on detecting the first touchless gesture performed by the particular person, that the particular person is a source of the received speech command;

determine one or more identities of the second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera;

disambiguate the received speech command based on the first touchless gesture and the determined one or more identities of the second set of one or more people that are outside the field-of-view of the camera and are within the predetermined distance from the camera; and

perform the disambiguated speech command.

2. The device of claim 1, wherein the disambiguated received speech command includes a personal pronoun.

3. The device of claim 2, wherein the disambiguated received speech command includes one of “

he,”

“

she,”

“

him,”

“

her,”

“

his,”

“

hers,”

“

they,”

“

them,”

“

their,” and

“

theirs”

.

4. The device of claim 1, wherein the disambiguated received speech command includes a name associated with one of the determined one or more identities of the second set of one or more people.

5. The device of claim 1, wherein the one or more identities of the second set of one or more people are determined in response to the received speech command.

6. The device of claim 5, wherein the first touchless gesture performed by the particular person includes a pointing gesture to a second person, and one of the one or more identities determined is an identity of the second person as a result of the first touchless gesture being determined to include the pointing gesture to the second person.

7. The device of claim 6, wherein the disambiguated received speech command includes a third-person personal pronoun.

8. The device of claim 1, wherein the processor is further configured to determine an identity of the particular person as a result of determining that the particular person is the source of the received speech command, and

wherein the disambiguated received speech command includes one of “

I,”

“

me,”

“

my,”

“

mine,”

“

we”

“

us,”

“

our,” and

“

ours”

.

9. The device of claim 1, wherein the one or more identities of the second set of one or more people are determined in response to determining the presence of the second set of one or more people that are outside the field-of-view of the camera and within a predetermined distance from the camera.

10. The device of claim 9, wherein the first touchless gesture provides an indication to the device to commence the receiving of the speech command.

11. The device of claim 9, wherein the processor is further configured to detect a second touchless gesture performed by the particular person, and wherein the second touchless gesture provides an indication to the device that the speech command is complete.

12. The device of claim 1, wherein the first touchless gesture is performed contemporaneously with the speech command.

13. The device of claim 1, wherein the determined one or more identities of the second set of one or more people are based on performing a facial recognition.

14. The device of claim 1, wherein the camera includes a plurality of cameras.

15. The device of claim 1, wherein the device is operatively coupled to a plurality of microphones.

16. The device of claim 15, wherein the processor is further configured to adjust the plurality of microphones based on a relative position of the particular person as a result of determining that the particular person is the source of the received speech command.

17. A device for interpreting a voice-based input, the device operatively coupled to a camera that has a field-of-view, the device comprising:

a processor configured to;

determine one or more identities of (i) a first set of one or more people that are within a field-of-view of the camera and that are within a predetermined distance from the camera, and (ii) a second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera;

receive a speech command from a particular person among the first set of one or more people that are within the field-of-view of the camera and that are within the predetermined distance from the camera;

disambiguate the received speech command based on the determined one or more identities of the second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera; and

perform the disambiguated speech command.

18. The device of claim 17, wherein the disambiguated received speech command includes a personal pronoun.

19. The device of claim 18, wherein the personal pronoun is one of “

he,”

“

she,”

“

him,”

“

her,”

“

his,”

“

hers,”

“

they,” and

“

them,”

“

their,” and

“

theirs”

.

20. The device of claim 17, wherein the disambiguated received speech command includes a name associated with one of the determined one or more identities of the second set of one or more people.

21. The device of claim 17, wherein the processor is further configured to determine that the particular person is a source of the received speech command based on detecting a first touchless gesture performed by the particular person.

22. The device of claim 17, wherein the one or more identities of the second set of one or more people are determined in response to receiving the speech command.

23. A computer-implemented method for a device to interpret a voice-based input, the device operatively coupled to a camera that has a field-of-view, the method comprising:

detecting a presence of (i) a first set of one or more people that are within a field-of-view of the camera and that are within a predetermined distance from the camera, and (ii) a second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera;

receiving a speech command;

detecting a first touchless gesture performed by a particular person from among the first set of one or more people that are within the field-of-view of the camera and that are within the predetermined distance from the camera;

determining, based on detecting the first touchless gesture performed by the particular person, that the particular person is a source of the received speech command;

determining one or more identities of the second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera;

disambiguate the received speech command based on the determined one or more identities of the second set of one or more people that are outside the field-of-view of the camera and that are within the predetermined distance from the camera; and

performing, by the device, the disambiguated speech command.

24. The method of claim 23, wherein the disambiguated received speech command includes a personal pronoun.

25. The method of claim 24, wherein the disambiguated received speech command includes one of “

he,”

“

she,”

“

him,”

“

her,”

“

his,”

“

hers,”

“

they,”

“

them,”

“

their,” and

“

theirs”

.

26. The method of claim 23, wherein the disambiguated received speech command includes a name associated with one of the determined one or more identities of second set of one or more people.

27. The method of claim 23, wherein the one or more identities of the second set of one or more people are determined in response to the received speech command.

28. The method of claim 27, wherein the first touchless gesture performed by the particular person includes a pointing gesture to a second person, and one of the one or more identities determined is an identity of the second person as a result of the first touchless gesture being determined to include the pointing gesture to the second person.

29. The method of claim 28, wherein the disambiguated received speech command includes a third-person personal pronoun.

30. The method of claim 23, wherein the processor is further configured to determine an identity of the particular person as a result of determining that the particular person is the source of the received speech command, and

wherein the disambiguated received speech command includes one of “

I,”

“

me,”

“

my,”

“

mine,”

“

we”

“

us,”

“

our,” and

“

ours”

.

31. The method of claim 23, wherein the one or more identities of the second set of one or more people are determined in response to determining the presence of the second set of one or more people that are outside the field-of-view of the camera and within a predetermined distance from the camera.

32. The method of claim 31, wherein the first touchless gesture provides an indication to the device to commence the receiving of the speech command.

33. The method of claim 31, further comprising detecting a second touchless gesture performed by the particular person, wherein the second touchless gesture provides an indication to the device that the speech command is complete.

34. The method of claim 23, wherein the first touchless gesture is performed contemporaneously with the speech command.

35. The method of claim 23, wherein the determined one or more identities of the second set of one or more people are based on performing a facial recognition.

36. The method of claim 23, wherein the camera includes a plurality of cameras.

37. The method of claim 23, wherein the device is operatively coupled to a plurality of microphones.

38. The method of claim 37, further comprising adjusting the plurality of microphones based on a relative position of the particular person as a result of determining that the particular person is the source of the received speech command.