Audio-visual control system

• US 20070016426A1
• Filed: 06/28/2005
• Published: 01/18/2007
• Est. Priority Date: 06/28/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Hershey, John
• Granted Patent: US 7,518,631 B2
• Time in Patent Office: Days
• US Class Current: 704/277
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A visual control system controls a controlled component. In one embodiment, the visual control system controls the controlled component based on a visual location of a user. In another embodiment, input from a visual perception device is used to provide focus control for an audio input device. In additional embodiments, the visual control system stops, starts or suppresses speech recognition or other audio functions when the direction of the sound detected by the audio input device is not coming from the user'"'"'s visual location.

Claims

1. A system for operating a controlled component which receives audio input from an audio receiving device, comprising:

a visual processing component adapted to receive input from a visual perception device and to locate a user based on the input to provide a visual control output; and

a controller component coupled to the controlled component and configured to receive the visual control output from the visual processing component and to control the controlled component based on the visual control output.

2. The system of claim 1 wherein the controller component is configured to activate the controlled component when the user is detected in a predetermined orientation.

3. The system of claim 1 wherein the controlled component comprises a speech recognition component.

4. The system of claim 3 wherein the controller component is configured to start or interrupt speech recognition by the speech recognition component based on an orientation of the user.

5. The system of claim 1 wherein the visual processing component includes a face detector.

6. The system of claim 5 wherein the visual processing component is configured to perform motion tracking and depth tracking.

7. The system of claim 1 wherein the controller component is configured to adjust an orientation or alignment of the audio receiving device based upon the location of the user detected by the visual processing component.

8. The system of claim 1 wherein the controller component includes an audio direction sensor to sense an audio direction from which the audio receiving device is receiving audio and wherein the controller component is configured to determine whether the audio direction corresponds to the location of the user determined by the visual processing component to control operation of the controlled component.

9. The system of claim 1 and further comprising an interface display device, wherein the user is located by the visual processing component relative to the interface display device.

10. The system of claim 9 wherein the visual perception device and the audio receiving device are in a predetermined position relative to the interface display device.

11. The system of claim 1 wherein the visual perception device includes stereo cameras.

12. The system of claim 1 wherein the audio receiving device includes a microphone array.

13. The system of claim 1 wherein the controlled component is a video telephone conference component and the controller component is configured to receive the visual control output from the visual processing component to control the video telephone conference component based on the visual control output.

14. The system of claim 1 wherein the controller component is configured to control the controlled component by providing a control input to the audio receiving device or by providing a control input to an audio function or component.

15. A method, comprising:

processing a visual input from a visual perception device to determine a location of a user; and

operating an audio component based upon the location of the user.

16. The method of claim 15 wherein processing the visual input comprises:

detecting a front face of the user, and wherein operation of the audio component comprises enabling the audio component based upon detection of the front face of the user.

17. The method of claim 16 wherein operating the audio component comprises:

enabling speech recognition to recognize speech from an audio input device.

18. The method of claim 17 and further comprising:

interrupting speech recognition or suppressing input from the audio input device when the front face of the user in not detected.

19. The method of claim 15 and further comprising:

tracking the location of the user; and

adjusting an orientation of an audio input device based upon the location of the user.

20. The method of claim 15 and further comprising:

determining a direction of audio input received by an audio input device;

determining whether the direction of the audio input is coming from the location of the user; and

interrupting operation of the audio component or suppressing audio input from the audio input device when the audio input is not coming from the location of the user.