MULTIMODAL REMOTE CONTROL

• US 20120239396A1
• Filed: 03/15/2011
• Published: 09/20/2012
• Est. Priority Date: 03/15/2011
• Status: Abandoned Application

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Johnston, Michael James, Worsley, Marcelo
• Application Number:
  US13/048,669
• Publication Number:
  US 20120239396A1
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G08C 2201/32 Remote control based on movements, attitude of remote control device
• CPC Class Codes: G08C 23/04 using light waves, e.g. infrared
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

A method and system for operating a remotely controlled device may use multimodal remote control commands that include a gesture command and a speech command. The gesture command may be interpreted from a gesture performed by a user, while the speech command may be interpreted from speech utterances made by the user. The gesture and speech utterances may be simultaneously received by the remotely controlled device in response to displaying a user interface configured to receive multimodal commands.

Claims

1. A remote control method, comprising:

detecting an audio input including speech content from a user;

detecting a motion input representative of a gesture performed by the user;

performing speech-to-text conversion on the audio input to generate a speech command;

processing the motion input to generate a gesture command;

synchronizing the speech command and the gesture command to generate a multimodal command; and

executing the multimodal command at a processor.

2. The method of claim 1, further comprising displaying multimedia content specified by the multimodal command.

3. The method of claim 2, wherein the multimedia content is a television program.

4. The method of claim 1, wherein the detecting of the motion input includes receiving an infrared signal generated by a remote control.

5. The method of claim 1, wherein the motion input is indicative of movement of a source of an infrared signal.

6. The method of claim 1, wherein the motion input is representative of multiple gestures.

7. The method of claim 1, wherein the detecting of the motion input and the detecting of the audio input occur in response to displaying a user interface configured to accept the multimodal command.

8. A remotely controlled device for processing multimodal remote control commands, comprising:

a processor configured to access memory media;

an infrared receiver; and

a microphone;

wherein the memory media include instructions executable by the processor to;

capture a speech utterance from a user via the microphone;

capture a gesture performed by the user via the infrared receiver;

identify a speech command from the speech utterance;

identify a gesture command from the gesture; and

combine the speech command and the gesture command into a multimodal command.

9. The remotely controlled device of claim 8, wherein the memory media include instructions executable by the processor to capture the gesture by detecting a motion of an infrared source.

10. The remotely controlled device of claim 8, wherein the memory media include instructions executable by the processor to execute the multimodal command and output multimedia content associated with the multimodal command.

11. The remotely controlled device of claim 10, wherein the memory media include instructions executable by the processor to display, using a display device, a user interface configured to accept the multimodal command.

12. The remotely controlled device of claim 10, further comprising a display device configured to display the multimedia content.

13. The remotely controlled device of claim 8, further comprising:

an image sensor, wherein the memory media include instructions executable by the processor to capture, using the image sensor, the gesture by detecting a body motion of the user.

14. Computer-readable memory media, including instructions executable by a processor to:

capture, via an audio input device, a speech utterance from a user;

capture, via a motion detection device, a gesture performed by the user; and

identify a multimodal command based on a combination of the speech utterance and the gesture.

15. The memory media of claim 14, further comprising instructions executable by a processor to display multimedia content specified by the multimodal command.

16. The memory media of claim 14, wherein the multimodal command is associated with a user interface configured to accept multimodal commands.

17. The memory media of claim 14, further comprising instructions executable by a processor to perform speech-to-text conversion on the speech utterance.

18. The memory media of claim 14, wherein the motion detection device includes an infrared camera.

19. The memory media of claim 18, wherein the gesture is captured by detecting a motion of an infrared source included in a remote control.

20. The memory media of claim 18, wherein the gesture is captured by detecting a motion of the user.