SYSTEMS AND METHODS FOR LIP READING CONTROL OF A MEDIA DEVICE

• US 20100189305A1
• Filed: 01/23/2009
• Published: 07/29/2010
• Est. Priority Date: 01/23/2009
• Status: Active Grant

• Current Assignee: Dish Technologies LLC (Dish Network Corporation)
• Original Assignee: Eldon Technology Limited (Dish Network Corporation)
• Inventors: Capless, Jonathan
• Granted Patent: US 8,798,311 B2
• Time in Patent Office: Days
• US Class Current: 382/100
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/20 Speech recognition techniques specially adapted for robustness in adverse environments, e.g. in noise, of stress induced speech G10L21/02 takes precedence
• CPC Class Codes: G10L 15/25 using position of the lips, movement of the lips or face analysis
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4821 using a grid, e.g. sorted out by channel and broadcast time

Abstract

Systems and methods of generating device commands based upon spoken user commands are disclosed. An exemplary embodiment captures a series of images of a user of a media device, generates image information corresponding to the series of captured images, determines lip movement of the user from the generated image information, determines at least one spoken user command based upon the determined lip movement of the user, and determines a device command based upon the determined spoken user command. Then, the device command is communicated to, for example, a media presentation device, wherein an operation of the media presentation device is controlled in accordance with the determined spoken user command.

Claims

1. A media device configured to generate device commands based upon spoken user commands, comprising:

an image capture device configured to capture a series of images of a user, and configured to generate image information corresponding to the series of captured images; and

a processor system communicatively coupled to the image capture device and configured to receive the image information from the image capture device configured to cause the determine at least one spoken user command based upon identified lip movements of the user shown in the captured image information, and configured determine a device command based upon the determined spoken user command.

2. The media device of claim 1, further comprising:

a presentation device interface coupled to the processor system and at least one media presentation device, wherein the device command controls operation of the media presentation device.

3. The media device of claim 1, further comprising:

a tuner residing in a set top box and configured to receive programming from a remote source;

a display configured to display the received programming; and

a digital video recorder configured to store the received programming for a later presentation on the display.

4. The media device of claim 1, wherein the image capture device resides in the media device.

5. The media device of claim 1, further comprising:

an external image capture device communicatively coupled to the processor system and operable to generate image information corresponding to the series of captured images.

6. The media device of claim 1, wherein the image information from the external image capture device is used to determine a second user command based upon identified lip movements of the user shown in the captured image information from the external image capture device, and wherein the second user command is compared with the first user command to verify the first user command.

7. The media device of claim 1, further comprising:

a power switch controllably coupled to the processor system; and

at least one electrical outlet coupled to the power switch, wherein the electrical outlet is operable to provide power to a home device via a power cord of the home device that is coupled to the electrical outlet, and

wherein the spoken user command corresponding to a request to turn on the home device is recognized such that the device command generated by the processor system causes the power switch to provide power to the electrical outlet.

8. The media device of claim 1, wherein the image capture device is a video camera.

9. A method for generating device commands to control a media presentation device based upon spoken user commands, the method comprising:

capturing a series of images of a user of the media presentation device;

generating image information corresponding to the series of captured images;

determining lip movement of the user from the generated image information;

determining at least one spoken user command based upon the determined lip movement of the user; and

determining a media presentation device command based upon the determined spoken user command.

10. The method of claim 9, further comprising:

communicating the media presentation device command to the media presentation device, wherein an operation of the media presentation device is controlled in accordance with the determined spoken user command.

11. The method of claim 9, wherein the media presentation device is a television.

12. The method of claim 9, wherein the captured series of of images includes a plurality of different users, the method comprising:

determining a plurality of lip movements of the plurality of users from the generated image information; and

determining at least one spoken user command based upon the plurality of determined lip movements of the plurality of users.

13. The method of claim 12, further comprising:

identifying at least one of the users from the plurality of users as an unauthorized user; and

disregarding the spoken user command from the identified unauthorized user.

14. The method of claim 9, further comprising:

determining a language that is spoken by the user, wherein the spoken user command based upon the determined lip movement of the user is further based upon the determined language.

15. The method of claim 9, further comprising:

communicating the device command to a power switch, wherein a home device receives power from an electrical outlet controlled by the power switch.

16. The method of claim 9, wherein the determined spoken user command identifies a home device and an action for the home device, and further comprising:

generating the device command based upon the action; and

communicating the device command to the identified home device.

17. The method of claim 9, further comprising:

determining an enablement command from a first spoken user command;

determining the device command from a second spoken user command after determining the enablement command; and

communicating the device command to the media presentation device only after determining the enablement command, wherein an operation of the media presentation device is controlled in accordance with the determined second spoken user command.

18. The method of claim 17, further comprising:

determining a disablement command from a third spoken user command, wherein further determined user commands are disregarded.

19. The method of claim 9, further comprising:

learning a plurality of spoken user commands from the user, wherein a current spoken user command is determined from the plurality of learned spoken user commands and from the determined lip movement of the user.

20. The method of claim 9, further comprising:

capturing a second series of images of the user with an external image capture device;

generating second image information corresponding to the captured second series of images;

determining the lip movement of the user from the generated second image information; and

verifying the determined spoken user command based upon the determined lip movement determined from the generated second image information.

21. The method of claim 9, wherein capturing the series of images of the user is captured by a still image capture device.

22. The method of claim 9, wherein capturing the series of images of the user is captured by a video image capture device, and comprising:

selecting the series of images of the user from a video captured by the video image capture device.

24. The media device of claim 22, wherein the tuner, the image capture device and the processing system reside in a set top box that is communicatively coupled to the television.

25. The media device of claim 22, further comprising:

a digital video recorder configured to store the received programming for a later presentation on the display.

23. A television controller system that generates device commands to control a television based upon image of spoken user commands, comprising:

a tuner residing in a set top box and configured to receive programming from a remote source;

a display configured to display the received programming;

an image capture device configured to capture a series of images of a user, and configured to generate image information corresponding to the series of captured images; and

a processor system communicatively coupled to the image capture device and configured to receive the image information from the image capture device configured to cause the determine at least one spoken user command based upon identified lip movements of the user shown in the captured image information, and configured determine a device command that controls the display based upon the determined spoken user command.