Method and apparatus for a verbo-manual gesture interface

• US 20070033012A1
• Filed: 10/07/2005
• Published: 02/08/2007
• Est. Priority Date: 07/19/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Outland Research LLC
• Inventors: Rosenberg, Louis
• Granted Patent: US 7,519,537 B2
• Time in Patent Office: Days
• US Class Current: 704/206
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

An interface system including a manipulandum adapted to be moveable according to a manual gesture imparted by the user; a sensor adapted to detect a characteristic of the manual gesture imparted to the manipulandum and to generate a sensor signal representing the detected characteristic of the manual gesture; a microphone adapted to detect a characteristic of an utterance spoken by the user and to generate an audio signal representing the detected characteristic of the spoken utterance; and a control system adapted receive the generated sensor and audio signals and to transmit a command signal to an electronic device via a communication link, the command signal being based on the generated sensor and audio signals and the time synchronization between them.

Claims

1. An interface system, comprising:

a manipulandum adapted to be moveable according to a manual gesture imparted by the user;

a sensor adapted to detect a characteristic of the manual gesture imparted to the manipulandum and to generate a sensor signal representing the detected characteristic of the manual gesture;

a microphone adapted to detect a characteristic of an utterance spoken by the user and to generate an audio signal representing the detected characteristic of the spoken utterance; and

a control system adapted to receive the generated sensor and audio signals and to transmit a command signal to an electronic device via a communication link, the command signal being based on the generated sensor and audio signals.

2. The interface system of claim 1, wherein the sensor includes a position sensor.

3. The interface system of claim 1, wherein the sensor includes a motion sensor.

4. The interface system of claim 3, wherein the detected characteristic of the manual gesture includes a direction of motion imparted to the manipulandum.

5. The interface system of claim 3, wherein the detected characteristic of the manual gesture includes a speed of motion imparted to the manipulandum.

6. The interface system of claim 1, wherein the sensor includes an orientation sensor.

7. The interface system of claim 5, wherein the detected characteristic of the spoken utterance includes at least one of a volume, rhythm, speed, and accent of the spoken utterance.

8. The interface system of claim 1, wherein the control system is adapted to output the command signal when:

a portion of the sensor signal has a predetermined relationship with a reference gesture pattern;

a portion of the audio signal has a predetermined relationship with a reference audio pattern; and

the portions of the sensor and audio signals have a predetermined temporal proximity to each other.

9. The interface system of claim 1, wherein the control system includes control electronics within the manipulandum.

10. The interface system of claim 1, wherein the control system includes a host computer coupled to the manipulandum.

11. The interface system of claim 1, wherein the manipulandum includes a proximity sensor adapted to communicate with the electronic device.

12. The interface system of claim 1, wherein

the manipulandum represents a magic wand;

the utterance represents a magical incantation; and

the electronic device includes an entertainment system.

13. An interface method, comprising:

detecting a characteristic of a manual gesture imparted by a user to a manipulandum;

detecting a characteristic of an utterance spoken by the user; and

controlling an electronic device based on the detected characteristics of the manual gesture and spoken utterance.

14. The interface method of claim 13, wherein detecting the characteristic of the manual gesture includes detecting a position of the manipulandum.

15. The interface method of claim 13, wherein detecting the characteristic of the manual gesture includes detecting a speed of motion of the manipulandum.

16. The interface method of claim 13, wherein detecting the characteristic of the manual gesture includes detecting a direction of motion of the manipulandum.

17. The interface method of claim 13, wherein detecting the characteristic of the manual gesture includes detecting an orientation of the manipulandum.

18. The interface method of claim 13, further comprising:

generating a sensor signal representing the detected characteristic of the manual gesture;

comparing the sensor signal with a reference gesture pattern;

generating an audio signal representing the detected characteristic of the spoken utterance;

comparing the audio signal with a reference audio pattern; and

controlling the electronic device when portions of the sensor and audio signals each have a predetermined relationship with the reference gesture and audio patterns, respectively.

19. The interface method of claim 18, further comprising controlling the electronic device when the portions of the sensor and audio signals have a predetermined temporal proximity to each other.

20. The interface method of claim 19, further comprising controlling the electronic device when the portions of the sensor and audio signals generated are substantially simultaneously.

21. A method of controlling an electronic device, comprising:

storing a plurality of definitions, each definition comprising temporally proximate pairing of a reference gesture signal pattern and a reference audio signal pattern;

assigning a control command to each definition, the control command being adapted to generate a response by an electronic device;

determining whether received signals representing a detected characteristic of a manual gesture imparted to a manipulandum and a detected characteristic of a spoken utterance have a predetermined relationship with a stored definition; and

transmitting a control command to an electronic device, the transmitted control command being assigned to a definition determined to have a predetermined relationship with the received signals.