Using multi-modal input to control multiple objects on a display

• US 8,977,972 B2
• Filed: 12/31/2009
• Issued: 03/10/2015
• Est. Priority Date: 12/31/2009
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Anderson, Glen J., Doherty, Rina A.
• Primary Examiner(s): RIEGLER, PATRICK F
• Application Number:
  US12/655,583
• Publication Number:
  US 20110161890A1
• Time in Patent Office: 1,895 Days
• Field of Search: 715/863, 715/764, 715/765, 715/767, 715/831, 345/156, 345/157, 345/158
• US Class Current: 715/764
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42206 characterized by hardware details
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

Embodiments of the invention are generally directed to systems, methods, and machine-readable mediums for implementing gesture-based signature authentication. In one embodiment, a system may include several modal input devices. Each modal input device is capable of retrieving a stream of modal input data from a user. The system also includes modal interpretation logic that can interpret each of the retrieved modal input data streams into a corresponding of set of actions. The system additionally includes modal pairing logic to assign each corresponding set of actions to control one of the displayed objects. Furthermore, the system has modal control logic which causes each displayed object to be controlled by its assigned set of actions.

Claims

1. A modular multi-modal input system, comprising:

a plurality of modal input devices, each modal input device to retrieve a stream of modal input data from a user;

modal interpretation logic to interpret each of the plurality of retrieved modal input data streams into a corresponding of set of actions;

modal pairing logic to assign each corresponding set of actions to control one of a plurality of displayed objects, wherein modal pairing logic implements a user interface to enable the user to explicitly assign each stream of modal input to a display object;

and modal control logic to control each displayed object by its assigned set of actions,

wherein each of the plurality of modal input devices captures a different type of modal input data corresponding with a different type of gameplay action,

wherein the modal pairing logic is operable to dynamically change a control assignment of a first displayed object of the plurality of displayed objects from a first modal input data stream of a first type of input device having a first type of data to a second modal input data stream of a second type of input device having a second type of data different from the first type of data.

2. The modular multi-modal input system of claim 1, wherein the modal pairing logic is operable to dynamically change a first modal input data stream of the plurality of modal input data streams from controlling a first displayed object to a second displayed object.

3. The modular multi-modal input system of claim 1, further comprising:

modal input management logic to manage two or more of the plurality of modal input devices as a group for the user, the group of modal input devices allowing the user to provide simultaneous control of two or more of the plurality of displayed objects.

4. The modular multi-modal input system of claim 3, wherein at least two of the plurality of modal input devices in the group are hands-free modal input devices.

5. The modular multi-modal input system of claim 3, wherein a first modal input device in the group is of a different modal type than a second modal input device in the group.

6. The modular multi-modal input system of claim 1, wherein at least one modal input device of the plurality of modal input devices is one of a video camera, a microphone, and an accelerometer.

19. The modular multi-modal input system of claim 1, wherein the user interface comprises a first list of available streams of modal input and a second list of available display objects to control.

7. A method, comprising:

retrieving a plurality of streams of modal input data from a user, each stream of modal input data received from a different one of a plurality of modal input devices;

interpreting each of the plurality of retrieved modal input data streams into a corresponding of set of actions;

assigning each corresponding set of actions to control one of a plurality of displayed objects;

implementing a user interface to enable the user to explicitly assign stream of modal input to a display object;

controlling each displayed object by its assigned set of actions; and

dynamically changing a control assignment of a first displayed object of the plurality of displayed objects from a first modal input data stream of a first type of input device having a first type of data to a second modal input data stream of a second type of input device having a second type of data different from the first type of data,

wherein each of the plurality of modal input devices captures a different type of modal input data corresponding with a different type of gameplay action.

8. The method of claim 7, further comprising:

dynamically changing a first modal input data stream of the plurality of modal input data streams from controlling a first displayed object to a second displayed object.

9. The method of claim 7, further comprising:

managing two or more of the plurality of modal input devices as a group for the user, the group of modal input devices allowing the user to provide simultaneous control of two or more of the plurality of displayed objects.

10. The method of claim 9, wherein at least two of the plurality of modal input devices in the group are hands-free modal input devices.

11. The method of claim 9, wherein a first modal input device in the group is of a different modal type than a second modal input device in the group.

12. The method of claim 7, wherein at least one modal input device of the plurality of modal input devices is one of a video camera, a microphone, and an accelerometer.

20. The method of claim 7, wherein the user interface comprises a first list of available streams of modal input and a second list of available display objects to control.

13. A non-transitory, machine-readable medium having stored thereon instructions, which if executed by a machine causes the machine to perform a method comprising:

retrieving a plurality of streams of modal input data from a user, each stream of modal input data received from a different one of a plurality of modal input devices;

interpreting each of the plurality of retrieved modal input data streams into a corresponding of set of actions;

assigning each corresponding set of actions to control one of a plurality of displayed objects rendered on a display device;

implementing a user interface to enable the user to explicitly assign each stream of modal input to a display object;

controlling each displayed object by its assigned set of actions;

and dynamically changing a control assignment of a first displayed object of the plurality of displayed objects from a first modal input data stream of a second type of input device having a second type of data different from the first type of data to a second modal input data stream of a second type of input device having a second type of data different from the first type of data,

wherein each of the plurality of modal input devices captures a different type of modal input data corresponding with a different type of gameplay action.

14. The non-transitory, machine-readable medium of claim 13, wherein the performed method further comprises:

dynamically changing a first modal input data stream of the plurality of modal input data streams from controlling a first displayed object to a second displayed object.

15. The non-transitory, machine-readable medium of claim 13, wherein the performed method further comprises:

managing two or more of the plurality of modal input devices as a group for the user, the group of modal input devices allowing the user to provide simultaneous control of two or more of the plurality of displayed objects.

16. The non-transitory, machine-readable medium of claim 15, wherein at least two of the plurality of modal input devices in the group are hands-free modal input devices.

17. The non-transitory, machine-readable medium of claim 15, wherein a first modal input device in the group is of a different modal type than a second modal input device in the group.

18. The non-transitory, machine readable medium of claim 13, wherein at least one modal input device of the plurality of modal input devices is one of a video camera and a microphone.

21. The non-transitory, machine-readable medium of claim 13, wherein the user interface comprises a first list of available streams of modal input and a second list of available display objects to control.