Method, apparatus and computer-readable media for touch and speech interface with audio location

• US 10,845,909 B2
• Filed: 05/30/2019
• Issued: 11/24/2020
• Est. Priority Date: 06/06/2016
• Status: Active Grant

• Current Assignee: Nureva, Inc.
• Original Assignee: Nureva, Inc.
• Inventors: Popovich, David, Springgay, David Douglas, Gurnsey, David Frederick
• Primary Examiner(s): Pardo, Thuy N
• Application Number:
  US16/426,472
• Publication Number:
  US 20190346955A1
• Time in Patent Office: 544 Days
• Field of Search: 345156, 345173, 345174, 345 8, 704260, 704275, 711163, 726 26, 348 61
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

Method, apparatus, and computer-readable media for touch and speech interface, with audio location, includes structure and/or function whereby at least one processor: (i) receives a touch input from a touch device; (ii) establishes a touch-speech time window; (iii) receives a speech input from a speech device; (iii) determines whether the speech input is present in a global dictionary; (iv) determines a location of a sound source from the speech device; (v) determines whether the touch input and the location of the speech input are both within a same region; (vi) if the speech input is in the dictionary, determines whether the speech input has been received within the window; and (vii) if the speech input has been received within the window, and the touch input and the speech input are both within the same region, activates an action corresponding to both the touch input and the speech input.

Claims

1. Speech input with audio location apparatus configured for one or more users to provide input to (i) a display device and (ii) a speech input device, both in in a shared physical space, comprising:

at least one memory storing a plurality of words in a global dictionary; and

at least one processor configured to;

provide at least one display on the display device;

receive an input from the speech input device in the shared physical space;

determine whether the received input from the speech input device is present in the global dictionary;

determine a directionality of a sound source in the shared physical space with respect to the display device, based on the received input from the speech sensitive input device; and

if the received input from the speech input device is present in the global dictionary and if the directionality of the sound source is directed towards the display device, provide at least one processor action corresponding to the directionality, wherein the at least one processor action includes providing a display on the display device at a region corresponding to the directionality of the sound source.

2. The apparatus according to claim 1, wherein the at least one processor is configured to provide the at least one processor action corresponding to the intersection of the sound source directionality and the display device.

3. The apparatus according to claim 1, wherein:

the display comprises a touch sensitive input device;

the at least one processor is configured to receive an input from the touch sensitive input device in the shared physical space; and

the at least one processor is configured to provide a display on the touch sensitive input device at a region corresponding to the directionality.

4. The apparatus according to claim 3, wherein the at least one processor is configured to:

receive an input from the speech input device in the shared physical space, indicating that a sound source is at a first location; and

if the directionality of the sound source is directed towards a location different than the first location, provide at least one processor action at the different location.

5. The apparatus according to claim 4, wherein the at least one processor is configured to:

receive an input from the speech input device in the shared physical space, indicating that a sound source is at a second location; and

if the directionality of the sound source is directed towards a location different than the second location, provide at least one processor action at the location that is different than the second location.

6. The apparatus according to claim 3, wherein the at least one processor is configured to determine a vector from an origin of the sound source to the touch sensitive input device.

7. The apparatus according to claim 6, wherein the at least one processor is configured to determine an intersection of (i) the vector of the sound source directionality and (ii) a touch system application program on the touch sensitive input device.

8. The apparatus according to claim 6, wherein the at least one processor is configured to determine the vector of the sound source directionality when the sound source is moving.

9. The apparatus according to claim 6, wherein the at least one processor is configured to determine (i) the vector of the sound source directionality and (ii) a location of said sound source.

10. The apparatus according to claim 6, wherein the at least one processor is configured to determine an intersection of (i) a vector of sound source directionality and (ii) a location, for plural sound sources.

11. The apparatus according to claim 10, wherein the at least one processor is configured to determine the intersection where one of the sound sources is moving.

12. The apparatus according to claim 1, wherein the shared physical space includes an audio microphone system, and wherein the at least one processor is configured to calibrate the audio microphone system with respect to the display device for at least one of (i) X, Y and (ii) X,Y, and Z, audio source positional information.

13. Speech input with audio location method for at least one user to provide input to (i) a display device and (ii) a speech input device, both in in a shared physical space, comprising:

using at least one memory storing a plurality of words; and

using at least one processor to;

provide at least one display on the display device;

receive an input from the speech input device in the shared physical space;

determine whether the received input from the speech input device is present in the global dictionary;

determine a directionality of a sound source in the shared physical space with respect to the display device, based on the received input from the speech sensitive input device; and

if the received input from the speech input device is present in the global dictionary and if the directionality of the sound source is directed towards the display device, provide at least one processor action corresponding to the directionality, wherein the at least one processor action includes providing a display on the display device at a region corresponding to the directionality of the sound source.

14. The method according to claim 13, wherein the at least one processor is configured to provide the at least one processor action corresponding to the intersection of the sound source directionality and the display device.

15. The method according to claim 13, wherein:

the display comprises a touch sensitive input device;

the at least one processor is configured to receive an input from the touch sensitive input device in the shared physical space; and

the at least one processor is configured to provide a display on the touch sensitive input device at a region corresponding to the directionality.

16. The method according to claim 15, wherein the at least one processor:

receives an input from the speech input device in the shared physical space, indicating that a sound source is at a first location; and

if the directionality of the sound source is directed towards a location different than the first location, provides at least one processor action at the different location.

17. The apparatus according to claim 16, wherein the at least one processor:

receives an input from the speech input device in the shared physical space, indicating that a sound source is at a second location; and

if the directionality of the sound source is directed towards a location different than the second location, provides at least one processor action at the location that is different than the second location.

18. The method according to claim 15, wherein the at least one processor determines a vector from an origin of the sound source to the touch sensitive input device.

19. The method according to claim 18, wherein the at least one processor determines an intersection of (i) the vector of the sound source directionality and (ii) a touch system application program on the touch sensitive input device.

20. The method according to claim 18, wherein the at least one processor determines the vector of the sound source directionality when the sound source is moving.

21. The method according to claim 18, wherein the at least one processor determines (i) the vector of the sound source directionality and (ii) a location of said sound source.

22. The method according to claim 18, wherein the at least one processor determines an intersection of (i) a vector of sound source directionality and (ii) a location, for plural sound sources.

23. The method according to claim 22, wherein the at least one processor is configured to determine the intersection where one of the sound sources is moving.

24. The method according to claim 13, wherein the shared physical space includes an audio microphone system, and wherein the at least one processor calibrates the audio microphone system with respect to the display device for at least one of (i) X, Y and (ii) X,Y, and Z, audio source positional information.

25. Program code embodied in a non-transitory computer readable medium for speech input for at least one user to provide input to (i) a display device and (ii) a speech input device, both in a shared physical space, said program comprising instructions causing at least one processor to:

store a plurality of words at least one memory; and

provide at least one display on the display device;

receive an input from the speech input device in the shared physical space;

determine whether the received input from the speech input device is present in the global dictionary;

determine a directionality of a sound source in the shared physical space with respect to the display device, based on the received input from the speech sensitive input device; and

if the received input from the speech input device is present in the global dictionary and if the directionality of the sound source is directed towards the display device, provide at least one processor action corresponding to the directionality, wherein the at least one processor action includes providing a display on the display device at a region corresponding to the directionality of the sound source.

26. The program code according to claim 25, wherein the instructions cause the at least one processor to provide the at least one processor action corresponding to the intersection of the sound source directionality and the display device.

27. The program code according to claim 26, wherein the display comprises a touch sensitive input device, and wherein the instructions cause the at least one processor to:

receive an input from the touch sensitive input device in the shared physical space; and

provide a display on the touch sensitive input device at a region corresponding to the directionality.

28. The program code according to claim 27, wherein the instructions cause the at least one processor to:

receive an input from the speech input device in the shared physical space, indicating that a sound source is at a first location; and

if the directionality of the sound source is directed towards a location different than the first location, provide a display on the touch sensitive input device at the different location.

29. The program code according to claim 28, wherein the instructions cause the at least one processor to:

receive an input from the speech input device in the shared physical space, indicating that a sound source is at a second location; and

if the directionality of the sound source is directed towards a location different than the second location, provide a display on the touch sensitive input device at the location that is different than the second location.

30. The program code according to claim 27, wherein the instructions cause the at least one processor to determine a vector from an origin of the sound source to the touch sensitive input device.

31. The program code according to claim 30, wherein the instructions cause the at least one processor to determine an intersection of (i) the vector of the sound source directionality and (ii) a touch system application program on the touch sensitive input device.

32. The program code according to claim 31, wherein the instructions cause the at least one processor to determine the vector of the sound source directionality when the sound source is moving.

33. The program code according to claim 30, wherein the instructions cause the at least one processor to determine (i) the vector of the sound source directionality and (ii) a location of said sound source.

34. The program code according to claim 30, wherein the instructions cause the at least one processor to determine an intersection of (i) a vector of sound source directionality and (ii) a location, for plural sound sources.

35. The program code according to claim 31, wherein the instructions cause the at least one processor to determine the intersection where one of the sound sources is moving.

36. The program code according to claim 25, wherein the shared physical space includes an audio microphone system, and wherein the instructions cause the at least one processor to calibrate the audio microphone system with respect to the display device for at least one of (i) X, Y and (ii) X,Y, and Z, audio source positional information.