Optical-based scene detection and audio extraction

• US 8,890,953 B1
• Filed: 06/27/2011
• Issued: 11/18/2014
• Est. Priority Date: 06/27/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Coley, Christopher
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Aghevli, Reza
• Application Number:
  US13/169,547
• Time in Patent Office: 1,240 Days
• Field of Search: None
• US Class Current: 348/143
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

In analyzing a scene, a projection device may project an image onto surfaces or objects within the scene. The image may include user interface elements as well as structured light patterns for analyzing shapes of the surfaces and objects. The intensity of the reflection of the projected light may be monitored and used to derive an audio signal representing sound that occurs within the scene. The audio signal may be analyzed to detect and recognize user utterances and speech, which may be interpreted as commands.

Claims

1. An augmented reality system comprising:

one or more processors;

memory;

a projector that projects a structured light pattern onto upon objects that reflect the structured light pattern;

a camera that captures the structured light pattern reflected from the objects;

a shape detection module stored in the memory and executable by the one or more processors, the shape detection module being responsive to the reflected structured light pattern captured by the camera to determine shape properties of the objects; and

an audio detection module stored in the memory and executable by the one or more processors, the audio detection module being responsive to one or more variations in the reflected structured light pattern, due to movement associated with one or more of the objects, to recognize user utterances.

2. The augmented reality system of claim 1, wherein the audio detection module obtains the reflected structured light pattern from the camera.

3. The augmented reality system of claim 1, wherein the audio detection module is further responsive to intensity of the reflected structured light pattern to recognize the user utterances.

4. The augmented reality system of claim 1, wherein the audio detection module is further responsive to intensity of the reflected structured light pattern that is reflected from an area of one of the objects to recognize the user utterances.

5. The augmented reality system of claim 1, further comprising:

an optical sensor that is responsive to reflected light intensity over an area of one or more of the objects to produce an intensity signal; and

wherein the audio detection module is responsive to the intensity signal to recognize the user utterances.

6. The augmented reality system of claim 1, wherein the audio detection module is further responsive to the reflected structured light pattern to recognize speech.

7. The augmented reality system of claim 1, wherein the audio detection module is further responsive to an averaged intensity of the reflected structured light pattern to recognize the user utterances.

8. The augmented reality system of claim 1, wherein the audio detection module is further responsive to time-based intensity modulations of the reflected structured light pattern to recognize the user utterances.

9. A method of interacting with a user, comprising:

projecting by a projector a structured light pattern onto a surface;

capturing by a camera the structured light pattern reflected from the surface;

detecting by a shape detection module stored in a memory and executed by one or more processors, shape properties of the surface in response the reflected structured light pattern captured by the camera;

converting by an audio detection module one or more variations in the reflected structured light pattern, due to movement associated with the surface, to an audio representation.

10. The method of claim 9, wherein the projected image comprises an image that is recognizable to the user as a microphone.

11. The method of claim 9, further comprising recognizing spoken utterances of the user based on the audio representation.

12. The method of claim 9, further comprising recognizing speech of the user based on the audio representation.

13. The method of claim 9, further comprising responding to spoken commands of the user based on the audio representation.

14. The method of claim 9, wherein the projected image includes a non-visible component and the reflected intensity is captured from the non-visible component of the projected image.

15. The method of claim 9, wherein the projected image includes a non-perceivable structured light pattern and the reflected intensity is captured from the non-perceivable structured light pattern.

16. A method of environment detection, comprising:

projecting a light pattern onto objects within an environment;

capturing the light pattern as it is reflected from the objects;

analyzing the captured reflected light pattern to determine shape properties of the objects; and

analyzing one or more variations in the captured reflected light pattern including analyzing time-based intensity modulations of at least a portion of the captured reflected light pattern, due to movement associated with one or more of the objects, to detect sound within the environment.

17. The method of claim 16, wherein the projected light pattern comprises non-visible light.

18. The method of claim 16, further comprising recognizing user utterances based on the detected sound.

19. The method of claim 16, further comprising recognizing speech based on the detected sound.

20. The method of claim 16, wherein analyzing the captured reflected light pattern to determine shape properties of the objects comprises analyzing deformation of the captured reflected light pattern.