Projection and camera system for augmented reality environment

• US 9,472,005 B1
• Filed: 06/19/2012
• Issued: 10/18/2016
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Marason, Eric G., Weatherford, William Thomas, Coley, Christopher D., Yuan, Robert A., Virgen, Miguel
• Primary Examiner(s): Thompson, James A
• Application Number:
  US13/527,503
• Time in Patent Office: 1,582 Days
• Field of Search: 345/633, 353/28
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4816 of receivers alone
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A device includes a projection and camera system to create an augmented reality environment in which images are projected onto a scene and user movement within the scene is captured. The projection and camera system have a camera to image scattered IR light from the scene and compute time of flight values used in depth mapping of objects in the room. The system also has a projector to project the images onto the scene. The projected images and scattered IR light use a common optical path through the same lens. Additionally, an illumination system used to illuminate the scene with the IR light may also use the common optical path through the same lens. In one implementation, the projection and camera system are mounted in a movable head of a table lamp.

Claims

1. A device comprising:

a head structure;

a base structure connected to the head structure by an arm mechanism;

a plurality of components housed in one or more of the head structure or the base structure, the plurality of components comprising;

a lens;

a projector to project images, through the lens, onto one or more objects in an environment external to the device;

an illumination component to illuminate, through the lens, the one or more objects in the environment into which at least a portion of the images from the projector are to be projected;

a time-of-flight sensor to receive at least a portion of light from the illumination component scattered by the environment, the light being received at the lens, the time-of-flight sensor configured to determine a time of flight value associated with one or more objects associated with the one or more objects in the environment; and

a processor to determine distance information of the one or more objects relative to the time of flight sensor, the distance information determined based at least in part on the determined time of flight value.

2. The device of claim 1, wherein the illumination component, the lens, the time-of-flight sensor, and the projector are arranged within the head structure.

3. The device of claim 1, wherein the illumination component, the lens, the time-of-flight sensor, and the projector are arranged within the head structure and the processor resides in the base structure.

4. The device of claim 1, wherein the illumination component comprises an infrared (IR) device to generate IR light.

5. The device of claim 1, wherein the illumination component generates an illumination field and the projector generates a projector field, the illumination field being separate from the projector field.

6. The device of claim 1, wherein the projector projects an image that covers a first area of the scene, and the illumination component illuminates a second area of the scene, wherein the second area is larger than the first area.

7. The device of claim 1, wherein the head structure is movably connected to the arm mechanism via a connector that permits rotation about two different axes.

8. The device of claim 1, wherein the head structure is movably connected to the arm mechanism via a connector that permits rotation about a pan axis and a tilt axis.

9. The device of claim 1, further comprising:

one or more speakers arranged in the base structure; and

a power component to provide power to at least the processor, the power component residing in the base structure.

10. The device of claim 1, further comprising one or more microphones, the one or more microphones being arranged in at least one of the base structure or the head structure.

11. A system comprising:

a lens;

a projector to project an image through the lens into an area external to the system;

an illumination component, independent of the projector, to illuminate through the lens an area external to the system and into which at least a portion of the image from the projector is being projected; and

a time-of-flight component to receive light scattered from the illuminated area and determine a time-of-flight value associated with one or more objects in the illuminated area, the light being received back through the lens.

12. The system of claim 11, wherein the lens comprises a zoom lens.

13. The system of claim 11, wherein the illumination component comprises one of an infrared (IR) device to generate IR light and a plurality of infrared (IR) light emitting diodes (LEDs) positioned adjacent the lens.

14. The system of claim 11, wherein the illumination component generates an illumination field and the projector generates a projector field, the illumination field being separate from the projector field.

15. The system of claim 11, further comprising a beam splitter to reflect the projected image from a first path onto the optical path, the beam splitter further passing the collected light along the optical path without reflection.

16. The system of claim 11, further comprising a beam splitter to pass the projected image along the optical path and to reflect the collected light from the optical path onto a first path.

17. The system of claim 11, wherein the light from the illumination component comprises infrared (IR) light, and further comprising:

an IR filter; and

a second lens to direct the received IR light onto the time-of-flight component through the IR filter.

18. A system comprising:

a lens;

a projector to project an image through the lens;

an illumination component to generate light that is directed out through the lens to illuminate an environment external to the system and into which at least a portion of the image from the projector is being projected; and

a time-of-flight component to receive light scattered from the illuminated area and determine a time-of-flight value associated with one or more objects in the illuminated area, the light being received through the lens.

19. The system of claim 18, wherein the lens comprises a zoom lens.

20. The system of claim 18, wherein the illumination component comprises an infrared (IR) device to generate IR light.

21. The system of claim 18, further comprising a beam splitter to reflect the projected image from a first path onto the lens, the beam splitter further passing the collected light onto the lens without reflection.

22. The system of claim 18, further comprising a beam splitter to pass the projected image onto the lens and to reflect the collected light from the lens onto a first path.

23. The system of claim 18, wherein the light from the illumination component comprises infrared (IR) light, and further comprising:

an IR filter; and

a second lens to direct the received IR light onto the time-of-flight component through the IR filter.

24. A system comprising:

a lens having an associated optical path;

a projector to project an image through the lens along at least a portion of the optical path into an environment external to the system;

an illumination component to project light through the lens along at least a portion of the optical path in order to illuminate the environment into which at least a portion of the image from the projector is being projected; and

a sensor to receive light scattered from the environment through the lens along at least a portion of the optical path, the sensor to use the received light for analysis of one or more objects in the environment.

25. The system of claim 24, wherein the lens comprises a zoom lens.

26. The system of claim 24, wherein the sensor comprises a time-of-flight sensor to image the received light for use in computing a time of flight value associated with the one or more objects in the environment.

27. The system of claim 24, wherein the illumination component is a laser.

28. The system of claim 24, wherein the light generated by the illumination component is the light that is scattered by the environment.

29. A method comprising:

illuminating an environment with infrared (IR) light;

capturing, through a lens, the IR light scattered by one or more objects in the environment;

determining a time of flight value based on the scattered IR light that is captured through the lens; and

projecting an image through the lens into the environment being illuminated by the IR light such that the projected image and the captured IR light pass through the lens.

30. The method of claim 29, wherein the illuminating comprises directing the IR light through the lens to illuminate the environment.

31. The method of claim 29, further comprising computing a distance to at least one object in the environment based at least in part on the time of flight value.