Multimodal commands

• US 10,296,289 B2
• Filed: 06/01/2017
• Issued: 05/21/2019
• Est. Priority Date: 06/01/2017
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Chen, Peng-Wen
• Primary Examiner(s): Azad, Abul K
• Application Number:
  US15/611,278
• Publication Number:
  US 20180349092A1
• Time in Patent Office: 719 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

Technology for detecting multimodal commands that enhance the human-computer interaction of a computing device. In an illustrative implementation, a computing device may receive multiple input events from a plurality of input devices. The plurality of input devices may each correspond to a different computer input modalities and the computing device may correlate the input events across different modalities. The computing device may keep the input events in their native form (e.g., input device specific) or may transform the input events into modality independent events. In either example, the computing device may determine the events satisfy a definition for a multimodal command that identifies multiple events from different computer input modalities. Responsive to the determination, the computing device may invoke the multimodal command on the client device to perform one or more computing operations.

Claims

1. A computer-implemented method comprising:

receiving, by a processing device, data structures storing data associated with multiple input events from a plurality of input devices of a client device, wherein the plurality of input devices correspond to different computer input modalities and comprise a first input device and a second input device;

correlating, by the processing device, a first input event to first data received from the first input device and a second input event to second data received from the second input device, wherein the correlating includes temporally aligning the first input event and the second input event with a sliding time window, wherein a size of the sliding time window is based on a quantity of the multiple input events during a duration of time;

transforming, by the processing device, the first data of the first input event into a modality independent action data;

transforming, by the processing device, the second data of the second input event into the modality independent action data;

determining, by the processing device analyzing the modality independent action data transformed from the first data and the second data, the first input event and the second input event satisfy a definition of a multimodal command that comprises particular events from the different computer input modalities; and

invoking, by the processing device, the multimodal command on the client device responsive to a determination that the first and second input events satisfy the definition of the multimodal command, wherein invoking includes initiating a computing action on the client device that is identified within the multimodal command.

2. The computer-implemented method of claim 1, wherein the computer input modalities comprise one or more of a visual modality, an auditory modality, a tactile modality, or a kinesthetic modality.

3. The computer-implemented method of claim 1, wherein the multiple input events represent multimodal user input received by at least two devices from a set of devices comprising an optical input device, an acoustic input device, and a touch input device.

4. The computer-implemented method of claim 1, wherein receiving the multiple input events comprises receiving the multiple input events over a plurality of input streams corresponding to the different computing input modalities.

5. The computer-implemented method of claim 1, wherein the multimodal command overlaps with a native unimodal command and the multimodal command and the native unimodal command invoke an identical computing task.

6. The computer-implemented method of claim 1, wherein the multimodal command is created in response to user input identifying the particular events from the different computer input modalities.

7. A computer-system comprising:

a memory; and

a processing device communicatively coupled to said memory, said processing device configured to;

receive data structures storing data associated with multiple input events from a plurality of input devices of a client device, wherein the plurality of input devices correspond to different computer input modalities and comprise a first input device and a second input device;

correlate a first input event to first data received from the first input device and a second input event to second data received from the second input device, including to temporally align the first input event and the second input event with a sliding time window, wherein a size of the sliding time window is based on a quantity of the multiple input events during a duration of time;

transform the first data of the first input event into a modality independent action data;

transform the second data of the second input event into the modality independent action data;

determine, via analysis of the modality independent action data transformed from the first data and the second data, the first input event and the second input event satisfy a definition of a multimodal command that comprises particular events from the different computer input modalities; and

invoke the multimodal command on the client device responsive to a determination that the first and second input events satisfy the definition of the multimodal command, wherein to invoke includes to initiate a computing action on the client device that is identified within the multimodal command.

8. The computer-system of claim 7, wherein the computer input modalities comprise one or more of a visual modality, an auditory modality, a tactile modality, or a kinesthetic modality.

9. The computer-system of claim 7, wherein the multiple input events represent multimodal user input received by at least two devices from a set of devices comprising an optical input device, an acoustic input device, and a touch input device.

10. The computer-system of claim 7, wherein to receive the multiple input events comprise the processing device to receive the multiple input events over a plurality of input streams corresponding to the different computing input modalities.

11. The computer-system of claim 7, wherein the multimodal command is created in response to user input identifying the particular events from the different computer input modalities.

12. A non-transitory computer-readable storage medium programmed to include instructions that, when executed by a processing device, cause the processing device to perform a plurality of operations comprising:

receiving data structures storing data associated with multiple input events from a plurality of input devices of a client device, wherein the plurality of input devices correspond to different computer input modalities and comprise a first input device and a second input device;

correlating a first input event to first data received from the first input device and a second input event to second data received from the second input device, wherein the correlating includes temporally aligning the first input event and the second input event with a sliding time window, wherein a size of the sliding time window is based on a quantity of the multiple input events during a duration of time;

transforming the first data of the first input event into a modality independent action data;

transforming the second data of the second input event into the modality independent action data;

determining, by analyzing the modality independent action data transformed from the first data and the second data, the first input event and the second input event satisfy a definition of a multimodal command that comprises particular events from the different computer input modalities; and

invoking the multimodal command on the client device responsive to a determination that the first and second input events satisfy the definition of the multimodal command, wherein invoking includes initiating a computing action on the client device that is identified within the multimodal command.

13. The non-transitory computer-readable storage medium of claim 12, wherein the computer input modalities comprise one or more of a visual modality, an auditory modality, a tactile modality, or a kinesthetic modality.

14. The non-transitory computer-readable storage medium of claim 12, wherein the multiple input events represent multimodal user input received by at least two devices from a set of devices comprising an optical input device, an acoustic input device, and a touch input device.

15. The non-transitory computer-readable storage medium of claim 12, wherein receiving the multiple input events comprises receiving the multiple input events over a plurality of input streams corresponding to the different computing input modalities.

16. The non-transitory computer-readable storage medium of claim 12, wherein correlating the first input event and the second input event comprises temporally aligning the input events with a sliding time window, wherein a size of the sliding time window is based on a quantity of input events.