Projection and camera system for augmented reality environment

• US 9,462,255 B1
• Filed: 06/19/2012
• Issued: 10/04/2016
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Marason, Eric G., Weatherford, William Thomas, Coley, Christopher D., Yuan, Robert A., Virgen, Miguel
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): TRUONG, NGUYEN T
• Application Number:
  US13/527,519
• Time in Patent Office: 1,568 Days
• Field of Search: 348/136
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4816 of receivers alone
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G03B 35/10 having single camera with stereoscopic-base-defining system
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A device includes a projection and camera system to create an augmented reality environment in which images are projected onto a scene and user movement within the scene is captured. The projection and camera system have a camera to image scattered IR light from the scene and compute time of flight values used in depth mapping of objects in the room. The system also has a projector to project the images onto the scene. The projected images and scattered IR light use a common optical path through the same lens. Additionally, an illumination system used to illuminate the scene with the IR light may also use the common optical path through the same lens. In one implementation, the projection and camera system are mounted in a movable head of a table lamp.

Claims

1. A device comprising:

a head structure;

a base structure connected to the head structure by an arm mechanism;

a plurality of components housed in one or more of the head structure or the base structure, the plurality of components comprising;

an illumination component to illuminate at least a portion of an environment;

a prism arrangement comprising a plurality of optical elements, the prism arrangement having an associated optical path within the prism arrangement;

a projector to project images onto a surface of the environment, the projector configured to project the images along at least a first portion of the optical path within the prism arrangement;

a time-of-flight sensor to receive at least a portion of light from the illumination component scattered by the environment, the light being received along at least the first portion of the optical path within the prism arrangement, the time-of-flight sensor configured to determine a time of flight value associated with one or more objects in the illuminated portion of the environment; and

a processor to determine distance information of the one or more objects relative to the time of flight sensor, the distance information determined based at least in part on the determined time of flight value.

2. The device of claim 1 wherein the illumination component, the prism arrangement, the sensor, and the projector are arranged within the head structure.

3. The device of claim 1, wherein the illumination component, the prism arrangement, the sensor, and the projector are arranged within the head structure and the processor resides in the base structure.

4. The device of claim 1, further comprising a lens, wherein the optical path goes through the lens.

5. The device of claim 1, wherein the projector projects an image that covers a first area of the scene, and the illumination component illuminates a second area of the scene, wherein the second area is larger than the first area.

6. The device of claim 1, wherein the head structure is movably connected to the arm mechanism via a connector that permits rotation about two different axes.

7. The device of claim 1, wherein the prism arrangement comprises three prisms.

8. The device of claim 1, wherein the prism arrangement comprises two prisms.

9. The device of claim 1, further comprising:

one or more speakers arranged in the base structure; and

a power component to provide power to at least the processor, the power component residing in the base structure.

10. The device of claim 1, further comprising one or more microphones, the one or more microphones being arranged in at least one of the base structure or the head structure.

11. A system comprising:

a projector to project an image along a first portion of an optical path within a prism arrangement comprising a plurality of optical elements, the prism arrangement guiding light from the projector along the optical path;

an illumination component to illuminate an area external to the system; and

a time-of-flight component to receive light scattered from the illuminated area and determine a time-of-flight value associated with one or more objects in the illuminated area, the light being received and directed through the prism arrangement along the first portion of the optical path within the prism arrangement.

12. The system of claim 11, further comprising a lens, wherein the optical path passes through the lens.

13. The system of claim 11, further comprising a zoom lens, wherein the optical path passes through the zoom lens.

14. The system of claim 11, wherein the prism arrangement comprises three prisms.

15. The system of claim 11, wherein the prism arrangement comprises:

a first prism adjacent to a second prism, the first and second prisms spaced apart by an air gap; and

a third prism adjacent to the second prism.

16. The system of claim 15, wherein cross sections of the first and second prism, arranged adjacent to one another, together approximate a rectangle.

17. The system of claim 11, wherein the prism arrangement comprises:

a first prism having at least a first surface;

a second prism having at least a first and second surface, the first surface of the second prism being adjacent to, but spaced apart by an air gap, the first surface of the first prism; and

a third prism having at least a first surface that is adjacent to the second surface of the second prism.

18. The system of claim 11, wherein the prism arrangement comprises two prisms.

19. The system of claim 11, wherein the prism arrangement comprises a first prism with a triangular cross section and a second prism with a trapezoidal cross section.

20. An optical assembly to direct light within a projection system, the optical assembly comprising:

a first prism having at least a first surface;

a second prism having at least first and second surfaces, the first surface of the second prism being adjacent to, but spaced apart by an air gap, the first surface of the first prism, wherein the second prism is configured to;

project light along a first portion of an optical path within the second prism, and

receive light reflected from one or more objects from outside the optical assembly along at least the first portion of the optical path; and

a third prism having at least a first surface that is joined to the second surface of the second prism.

21. The optical assembly of claim 20, wherein:

the first prism has a triangular cross section with three internal angles, the angles of the first prism being substantially 34°

, 52°

, and 94°

;

the second prism has a triangular cross section with three internal angles, the angles of the second prism being substantially 35°

, 65°

, and 80°

; and

the third prism has a triangular cross section with three internal angles, the angles of the third prism being substantially 45°

, 66°

, and 69°

.

22. The optical assembly of claim 20, wherein the second and third prisms are integrally formed as a single member having a trapezoidal cross section.

23. An optical assembly to direct light within a projection system, the optical assembly comprising:

a first prism having at least a first surface; and

a second prism having at least a first surface, the first surface of the second prism being adjacent to, but spaced apart by an air gap, to the first surface of the first prism, wherein the second prism is configured to;

project light along a first portion of an optical path within the second prism, and

receive light reflected from one or more objects from outside the optical assembly along at least the first portion of the optical path.

24. The optical assembly of claim 23, wherein:

the first prism has a triangular cross section with three internal angles, the angles of the first prism being substantially 34°

, 52°

, and 94°

; and

the second prism has a trapezoidal cross section with four internal angles, the angles of the second prism being substantially 35°

, 110°

, 69°

, and 146°

.

25. An optical assembly for use in a projection system, the optical assembly comprising:

a lens having an associated first portion of an optical path within the lens along which;

image information from a projector is transmitted at least in part to an area of an environment; and

light scattered from the environment is received at least in part, wherein the image information is transmitted and the scattered light is received at least partly along the first portion of the optical path within the lens; and

a prism arrangement comprising a plurality of optical elements to direct, at least in part;

the image information from the projector to the lens; and

the received light from the lens to a depth sensor.

26. The optical assembly of claim 25, wherein the prism arrangement comprises at least first and second prisms.

27. The optical assembly of claim 25, wherein the prism arrangement comprises:

a first prism adjacent to a second prism, the first and second prisms spaced apart by an air gap; and

a third prism adjacent to the second prism.