Method, apparatus and computer-readable media for touch and speech interface with audio location

• US 10,338,713 B2
• Filed: 06/06/2017
• Issued: 07/02/2019
• Est. Priority Date: 06/06/2016
• Status: Active Grant

• Current Assignee: Nureva, Inc.
• Original Assignee: Nureva, Inc.
• Inventors: Popovich, David, Springgay, David Douglas, Gurnsey, David Frederick
• Primary Examiner(s): Pardo, Thuy N
• Application Number:
  US15/615,108
• Publication Number:
  US 20170351367A1
• Time in Patent Office: 756 Days
• Field of Search: 345173, 345156, 345174, 345 8, 345419, 345473, 704260, 704275, 704 3, 704 8, 704235, 348 1401
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

Method, apparatus, and computer-readable media for touch and speech interface, with audio location, includes structure and/or function whereby at least one processor: (i) receives a touch input from a touch device; (ii) establishes a touch-speech time window; (iii) receives a speech input from a speech device; (iii) determines whether the speech input is present in a global dictionary; (iv) determines a location of a sound source from the speech device; (v) determines whether the touch input and the location of the speech input are both within a same region; (vi) if the speech input is in the dictionary, determines whether the speech input has been received within the window; and (vii) if the speech input has been received within the window, and the touch input and the speech input are both within the same region, activates an action corresponding to both the touch input and the speech input.

Claims

1. Touch and speech input with audio location apparatus configured for one or more users to provide input to (i) a touch sensitive input device and (ii) a speech input device in a shared physical space, comprising:

at least one memory storing a plurality of words in a global dictionary; and

at least one processor configured to;

receive an input from the touch sensitive input device in the shared physical space;

establish a touch time window with respect to the received input from the touch sensitive input device;

receive an input from the speech input device in the shared physical space;

determine whether the received input from the speech input device is present in the global dictionary;

determine a position location of a sound source in the shared physical space from the received input from the speech input device;

determine whether the received input from the touch sensitive input device and the position location of received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space;

if the received input from the speech input device is present in the global dictionary, determine whether the received input from the speech input device has been received within the established touch time window; and

if the received input from the speech input device has been received within the established touch time window, and the received input from the touch sensitive input device and the received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space, activate an action corresponding to both (i) the received input from the touch sensitive input device and (ii) the received input from the speech input device.

2. The apparatus according to claim 1, wherein the at least one processor is configured to (i) determine locations of a plurality of sound sources from the received input from the speech input device, (ii) determine which one sound source of the plurality of sound sources is within the same region of the touch sensitive input device as the received input from the touch sensitive input device, and (iii) disregard at least one of the other ones of the plurality of sound sources.

3. The apparatus according to claim 2, wherein the at least one processor is configured to determine an X,Y location of each of the plurality of sound sources from the received input from the speech input device.

4. The apparatus according to claim 1, wherein the at least one processor is configured to receive speech input from a plurality of microphones.

5. The apparatus according to claim 4, wherein the at least one processor is configured to calibrate at least one location of at least one sound source with respect to the touch sensitive input device.

6. The apparatus according to claim 1, wherein the at least one processor is configured to, when a new input from the touch sensitive input device is received, ignore said new input if said new input is determined to be outside said same region.

7. The apparatus according to claim 1, wherein the at least one processor is configured to continuously perform automatic speech recognition functions.

8. The apparatus according to claim 1, wherein the at least one processor configured to:

receive a second input from the touch sensitive input device;

establish a second touch time window with respect to the received second input from the touch sensitive input device;

receive a second input from the speech input device;

determine whether the received second input from the speech input device is present in the global dictionary;

determine a second position location of a second sound source from the received second input from the speech input device;

determine whether the received second input from the touch sensitive input device and the second position location of the received second input from the speech input device are both within a second same region of the touch sensitive input device;

if the received second input from the speech input device is present in the global dictionary, determine whether the received second input from the speech input device has been received within the established second touch time window; and

if the received second input from the speech input device has been received within the established second touch time window, and the received second input from the touch sensitive input device and the received second input from the speech input device are both within the second same region of the touch sensitive input device, activate a second action corresponding to both (i) the received second input from the touch sensitive input device and (ii) the received second input from the speech input device.

9. The apparatus according to claim 8, wherein the at least one processor is configured to activate the action and the second action substantially simultaneously.

10. Touch and speech input with audio location method for at least one user to provide input to (i) a touch sensitive input device and (ii) a speech input device, in a shared physical space, comprising:

storing a plurality of words in a global dictionary in at least one memory; and

using at least one processor to;

receive an input from the touch sensitive input device in the shared physical space;

establish a touch time window with respect to the received input from the touch sensitive input device;

receive an input from the speech input device in the shared physical space;

determine whether the received input from the speech input device is present in the global dictionary;

determine a position location of a sound source in the shared physical space from the received input from the speech input device;

determine whether the received input from the touch sensitive input device and the position location of received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space;

if the received input from the speech input device is present in the global dictionary, determine whether the received input from the speech input device has been received within the established touch time window; and

if the received input from the speech input device has been received within the established touch time window, and the received input from the touch sensitive input device and the received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space, activate an action corresponding to both (i) the received input from the touch sensitive input device and (ii) the received input from the speech input device.

11. The method according to claim 10, wherein the at least one processor is (i) determines locations of a plurality of sound sources from the received input from the speech input device, (ii) determines which one sound source of the plurality of sound sources is within the same region of the touch sensitive input device as the received input from the touch sensitive input device, and (iii) disregards the other ones of the plurality of sound sources.

12. The method according to claim 11, wherein the at least one processor determines an X,Y location of each of the plurality of sound sources from the received input from the speech input device.

13. The method according to claim 10, wherein the at least one processor receives speech input from a plurality of microphones.

14. The method according to claim 13, wherein the at least one processor calibrates locations of sound sources with respect to the plurality of microphones.

15. The method according to claim 10, wherein the at least one processor, when a new input from the touch sensitive input device is received, ignores said new input if said new input is determined to be outside said same region.

16. The method according to claim 10, wherein the at least one processor continuously performs automatic speech recognition functions.

17. The method according to claim 10, wherein the at least one processor:

receives a second input from the touch sensitive input device;

establishes a second touch time window with respect to the received second input from the touch sensitive input device;

receives a second input from the speech input device;

determines whether the received second input from the speech input device is present in the global dictionary;

determines a second position location of a second sound source from the received second input from the speech input device;

determines whether the received second input from the touch sensitive input device and the second position location of the received second input from the speech input device are both within a second same region of the touch sensitive input device;

if the received second input from the speech input device is present in the global dictionary, determines whether the received second input from the speech input device has been received within the established second touch time window; and

if the received second input from the speech input device has been received within the established second touch time window, and the received second input from the touch sensitive input device and the received second input from the speech input device are both within the second same region of the touch sensitive input device, activates a second action corresponding to both (i) the received second input from the touch sensitive input device and (ii) the received second input from the speech input device.

18. The method according to claim 17, wherein the at least one processor activates the action and the second action substantially simultaneously.

19. The method according to claim 17, wherein the at least one processor:

determines a directionality of a sound source with respect to the touch sensitive input device;

if the directionality of the sound source is changing with respect to the touch sensitive input device, moves said same region on the touch sensitive input device to a new region corresponding to the changing directionality of the sound source.

20. Program code embodied in a non-transitory computer readable medium for touch and speech input for at least one user to provide input to a touch sensitive input device and using a speech input device, said program comprising instructions causing at least one processor to:

store a plurality of words in a global dictionary in at least one memory;

receive an input from the touch sensitive input device in a shared physical space;

establish a touch time window with respect to the received input from the touch sensitive input device;

receive an input from the speech input device;

determine whether the received input from the speech input device is present in the global dictionary;

determine a position location of a sound source in the shared physical space from the received input from the speech input device;

determine whether the received input from the touch sensitive input device and the position location of received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space;

if the received input from the speech input device is present in the global dictionary, determine whether the received input from the speech input device has been received within the established touch time window; and

if the received input from the speech input device has been received within the established touch time window, and the received input from the touch sensitive input device and the received input from the speech input device are both within a same region of the touch sensitive input device in the shared physical space, activate an action corresponding to both (i) the received input from the touch sensitive input device and (ii) the received input from the speech input device.

21. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to:

(i) determines locations of a plurality of sound sources from the received input from the speech input device, (ii) determines which one sound source of the plurality of sound sources is within the same region of the touch sensitive input device as the received input from the touch sensitive input device, and (iii) disregards the other ones of the plurality of sound sources.

22. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to determine an X,Y location of each of the plurality of sound sources from the received input from the speech input device.

23. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to receive speech input from a plurality of microphones.

24. Program code according to claim 23, wherein said program code comprises instructions causing the at least one processor to calibrate locations of sound sources with respect to the plurality of microphones.

25. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to, when a new input from the touch sensitive input device is received, ignore said new input if said new input is determined to be outside said same region.

26. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to continuously perform automatic speech recognition functions.

27. Program code according to claim 20, wherein said program code comprises instructions causing the at least one processor to:

receive a second input from the touch sensitive input device;

establishes a second touch time window with respect to the received second input from the touch sensitive input device;

receive a second input from the speech input device;

determine whether the received second input from the speech input device is present in the global dictionary;

determine a second position location of a second sound source from the received second input from the speech input device;

determine whether the received second input from the touch sensitive input device and the second position location of the received second input from the speech input device are both within a second same region of the touch sensitive input device;

if the received second input from the speech input device is present in the global dictionary, determines whether the received second input from the speech input device has been received within the established second touch time window; and

if the received second input from the speech input device has been received within the established second touch time window, and the received second input from the touch sensitive input device and the received second input from the speech input device are both within the second same region of the touch sensitive input device, activates a second action corresponding to both (i) the received second input from the touch sensitive input device and (ii) the received second input from the speech input device.

28. Program code according to claim 27, wherein said program code comprises instructions causing the at least one processor to activate the action and the second action substantially simultaneously.

29. Program code according to claim 27, wherein said program code comprises instructions causing the at least one processor to:

determine a directionality of a sound source with respect to the touch sensitive input device; and

if the directionality of the sound source is changing with respect to the touch sensitive input device, move said same region on the touch sensitive input device to a new region corresponding to the changing directionality of the sound source.