Application specific tracking of projection surfaces

• US 9,196,067 B1
• Filed: 03/05/2013
• Issued: 11/24/2015
• Est. Priority Date: 03/05/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Freed, Ian W., Worley, William Spencer III
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US13/786,221
• Time in Patent Office: 994 Days
• Field of Search: 345/419, 700/249, 700/245, 700/250, 348/46, 348/745
• US Class Current: 1/1
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/46 Indirect determination of position data
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/481 Constructional features, e.g. arrangements of optical elements
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

A projection, image, and depth capture system projects content into a scene and captures images of the scene as the user interacts with the content. The system uses depth analysis to determine location and distance of available surfaces in the scene onto which the content can be projected. Due to the complexity of this analysis and the inherent imperfections of the electronic and optical components, depth analysis possesses inherent noise that may adversely affect the accuracy of the projected image onto the surface. The system is configured with noise compensation technology that averages depth information over multiple image frames captured from the scene. The averaged information leads to a more consistent measurement of the distance to the surface, which in turn allows for more accurate focus of the projected content.

Claims

1. A system comprising:

a memory;

a processor;

a projector to project content onto a surface in an environment;

an infrared (IR) emitter to emit IR light onto the surface in the environment;

a depth sensor to receive at least a portion of the IR light reflected from the surface and to produce depth data at discrete times for use in determining a distance to the surface;

an analysis module stored in the memory and executed on the processor to receive the depth data produced the depth sensor and to average the depth data produced over multiple discrete times to produce averaged depth data, wherein the averaged depth data is used at least in part to focus the projector for projection of the content onto the surface;

a first application stored in the memory and executed by the processor to provide first content for projection by the projector, wherein the analysis module is configured to average the depth data over a first number of the multiple discrete times when the first application is being executed to provide content for projection; and

a second application stored in the memory and executed by the processor to provide second content for projection by the projector, the second content being different from the first content, wherein the analysis module is configured to average the depth data over a second, different number of the multiple discrete times when the second application is being executed to provide content for projection.

2. The system of claim 1, wherein the depth sensor comprises a time-of-flight sensor.

3. The system of claim 1, further comprising user preferences stored in the memory, wherein the analysis module is configured to average the depth data over a number of the multiple discrete times, the number being selected based at least in part on the user preferences.

4. A system comprising:

one or more components to produce depth data indicative of a distance from the one or more components to at least one surface in an environment onto which content may be projected; and

an analysis module to;

determine one of (i) an application associated with content being projected by the system or (ii) a type of the content being projected by the system;

determine a time over which to average the depth data based at least in part on the application or the type of the content;

average the depth data over the time to produce averaged depth information; and

determine the distance to the surface based at least in part on the averaged depth information.

5. The system of claim 4, wherein the one or more components comprise an infrared (IR) emitter to emit IR light onto the surface and a depth sensor to receive at least a portion of the IR light reflected from the surface.

6. The system of claim 4, wherein the one or more components comprise a projector to project structured light onto the environment and a camera to capture images of the environment when the structured light is projected onto the environment.

7. The system of claim 4, wherein the one or more components comprise a time-of-flight sensor.

8. The system of claim 4, wherein the one or more components comprise multiple imaging devices to produce stereoscopic images of the environment.

9. The system of claim 4, further comprising a computing device to determine, based at least in part on the averaged depth information, when a user touches the surface or performs a gesture relative to the surface.

10. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

identifying an application that is used to provide content for projection onto a surface in an environment;

capturing a sequence of images over time, the images including at least part of the surface;

producing, using each image of the sequence, depth data for use in determining a distance to the surface;

selecting, based at least in part on the application, a number of the images over which to average the depth data to produce averaged depth information; and

averaging the depth data associated with each image of the sequence to produce averaged depth information, wherein the number of the images in the sequence over which the depth data is averaged is the number of images.

11. The one or more non-transitory computer-readable media of claim 10, wherein the number of images over which to average is greater when the content provided by the application is static content as compared to when the content provided by the application is dynamic content.

12. The one or more non-transitory computer-readable media of claim 11, wherein the number of images over which to average is more than five when the content provided by the application is static content.

13. The one or more non-transitory computer-readable media of claim 11, wherein the number of images over which to average is less than four when the content provided by the application is dynamic content.

14. The one or more non-transitory computer-readable media of claim 10, the acts further comprising adjusting projection of the content onto the surface in the environment based at least in part on the averaged depth information.

15. The one or more non-transitory computer-readable media of claim 14, the acts further comprising predicting movement of the surface within the environment, wherein the predicted movement is used in part to adjust projection of the content.

16. The one or more non-transitory computer-readable media of claim 10, the acts further comprising identifying, based at least in part on the averaged depth information, at least one of a user touching the surface or the user performing a gesture relative to the surface.

17. A method comprising:

projecting content onto a surface in an environment;

generating depth data indicative of a distance to the surface;

determining one of (i) an application associated with the projected content or (ii) a type of the projected content being projected by the system;

determining a time over which to average the depth data based at least in part on the application or the type of the projected content;

averaging the depth data over the time to produce averaged depth information;

determining the distance to the surface based at least in part on averaged depth information; and

modifying projection of the content based at least in part on the averaged depth information.

18. The method of claim 17, wherein the generating depth data comprises using stereoscopic imaging to produce the depth data.

19. The method of claim 17, wherein the generating depth data comprises using structured light projected into the environment to produce the depth data.

20. The method of claim 17, wherein the generating depth data comprises using time-of-flight sensors to produce the depth data.

21. The method of claim 17, wherein the generating depth data and averaging the depth data comprises:

capturing images of the environment;

generating the depth data using the images;

averaging the depth data generated using the images to produce the averaged depth information.

22. A method comprising:

projecting content onto a surface in an environment;

generating depth data indicative of a distance to the surface;

determining one of (i) an application associated with the projected content or (ii) a type of the projected content being projected by the system;

determining a time over which to average the depth data based at least in part on the application or the type of the projected content;

averaging the depth data over the time to produce averaged depth information;

determining the distance to the surface based at least in part on averaged depth information; and

identifying, based at least in part on the averaged depth information, when a user touches the surface or performs a gesture relative to the surface.

23. The method of claim 22, wherein the generating depth data comprises using at least one of (1) stereoscopic imaging to produce the depth data, (2) structured light projected into the environment to produce the depth data, or (3) time-of-flight sensors to produce the depth data.