BLENDED REALITY SYSTEMS AND METHODS

• US 20160292917A1
• Filed: 03/30/2015
• Published: 10/06/2016
• Est. Priority Date: 03/30/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Dorner, Charles Shearer, Sayre, Paul Barnhart III, Hazlewood, William R.
• Granted Patent: US 9,858,719 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/026 and a display device, e.g. CRT, LCD, for adding markings or signs or to enhance the contrast of the viewed object
• CPC Class Codes: G03B 21/2053 Intensity control of illuminating light controlled by video signal processing H04N5/74 or H04N9/31
• CPC Class Codes: G03B 21/2066 Reflectors in illumination beam in projection beam G03B21/28
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Systems and methods are provided for generating a blended reality view to a user, the blended reality view combining images reflected by a mirror with images transmitted from a screen behind the mirror. Systems for generating blended reality views can include a display device with a screen positioned behind a mirror. The display device can generate a pattern of illumination and non-illumination on the screen so that the illuminated portions of the screen substantially transmit through the mirror. Projectors can be used to illuminate objects in front of the mirror so that the illuminated objects are reflected by the mirror. In combination, the portions of the screen transmitted through the mirror and the illuminated objects reflected by the mirror can provide a blended reality view to a user viewing the mirror.

Claims

1. An apparatus to provide a view of reflected and transmitted light to create a visual effect that combines reflected images and transmitted images, the apparatus comprising:

a display configured to display images;

a mirror having a first side and a second side, the mirror being partially-reflective and partially-transmissive such that at least a portion of light provided by the display is visible from the first side of the mirror;

a frame surrounding the mirror;

a plurality of projectors positioned on the frame and configured to illuminate the user and object elements facing the first side of the mirror;

a camera configured to acquire image data of the user and object elements; and

an image blending system communicably coupled to the display, to the plurality of projectors, and to the camera, the image blending system configured to;

in a scanning phase, analyze image data received from the camera to determine locations of the user and object elements;

in a tracking phase, analyze the image data received from the camera of the user to determine a location of the user'"'"'s eyes, the location of the user'"'"'s eyes including a height and a distance of the user'"'"'s eyes relative to the mirror;

control the plurality of projectors to illuminate at least some of the user and object elements to be reflected by the mirror as viewed from a particular projector; and

control the display to display images that are visible from the first side of the mirror within a first portion of the mirror;

wherein light from the illuminated user and object elements is reflected by the mirror and is visible from the first side of the mirror within a second portion of the mirror and combine with the displayed images to form a composite image.

2. The apparatus of claim 1 further comprising a lighting fixture, wherein the image blending system is further configured to control the intensity of light output by the lighting fixture to enhance the composite image formed from a combination of the reflected images and the displayed images.

3. The apparatus of claim 1 further comprising an active transmission matrix comprising a two-dimensional array of pixels, the active transmission matrix positioned between a screen of the display and the mirror, wherein the image blending system is further configured to control the transmissive properties of the two-dimensional array of pixels of the active transmission matrix to control light transmitted through the mirror to improve control of a contrast between the reflected images and the displayed images.

4. A method of generating a combined image at a targeted location that combines reflected images and transmitted images, the method comprising:

acquiring image data of surfaces in an environment;

generating a virtual three dimensional model of the surfaces in the environment by analyzing the acquired image data;

acquiring image data of an object positioned within the environment;

identifying a target on the object by analyzing the acquired image data;

generating images that are transmitted through from a first side of a mirror to a second side of the mirror to the identified target on the object; and

projecting light onto selected surfaces in the environment to selectively illuminate the selected surfaces,

wherein a combined image can be formed at the identified target, the combined image comprising images of the selected surfaces reflected by the mirror and the images transmitted through the mirror.

5. The method of claim 4, wherein generating the virtual three dimensional model of the surfaces in the environment comprises determining distances to the surfaces.

6. The method of claim 4 further comprising modifying light within the environment to enhance the combined image.

7. The method of claim 4, wherein identifying the target on the object is performed in real time to modify the generated images and the projected light based on movements of the identified target.

8. The method of claim 4 further comprising determining for each of a plurality of projectors a pattern of light to project to selectively illuminate the selected surfaces.

9. An apparatus comprising:

a display device comprising a screen;

a planar reflective surface positioned adjacent to the screen of the display device, the planar reflective surface configured to allow transmission of light from the screen through the planar reflective surface to a viewing side of the planar reflective surface;

a plurality of projectors configured to project light onto surfaces on the viewing side of the planar reflective surface;

an image acquisition device positioned to acquire images of the surfaces on the viewing side of the planar reflective surface; and

an image blending system configured to;

receive acquired images from the image acquisition system;

control the plurality of projectors to selectively illuminate surfaces on the viewing side of the planar reflective surface; and

control the display device to illuminate a portion of the screen and to leave a portion of the screen blank,

wherein at a targeted location on the viewing side of the planar reflective surface a combined image forms, the combined image comprising reflections of the selectively illuminated surfaces and transmitted illuminated portions of the screen.

10. The apparatus of claim 9, wherein the plurality of projectors are adjacent to a periphery of the planar reflective surface.

11. The apparatus of claim 10, wherein the plurality of projectors includes four projectors positioned at corners of the planar reflective surface.

12. The apparatus of claim 9, wherein a surface of the screen is parallel to a surface of the planar reflective surface.

13. The apparatus of claim 9 further comprising a light configured to vary an intensity of light output based on instructions from the image blending system.

14. The apparatus of claim 9, wherein the screen has a width that is at least 90% of the width of the planar reflective surface.

15. The apparatus of claim 9, wherein the display device is attached to the planar reflective surface.

16. The apparatus of claim 9 further comprising an active transmission matrix positioned between the screen of the display device and the planar reflective surface.

17. The apparatus of claim 9, wherein the screen is configured to output polarized light.

18. The apparatus of claim 17, wherein the screen is configured to alternately output light of orthogonal polarizations.

19. The apparatus of claim 9 further comprising a lenticular array positioned between the screen and the planar reflective surface.

20. The apparatus of claim 9, wherein the image acquisition device comprises a camera configured to detect infrared light.