Augmented reality with direct user interaction

• US 9,891,704 B2
• Filed: 12/26/2016
• Issued: 02/13/2018
• Est. Priority Date: 11/05/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hilliges, Otmar, Kim, David, Izadi, Shahram, Molyneaux, David, Hodges, Stephen Edward, Butler, David Alexander
• Primary Examiner(s): AMIN, JWALANT B
• Application Number:
  US15/390,632
• Publication Number:
  US 20170109938A1
• Time in Patent Office: 414 Days
• CPC Class Codes: A63F 13/20 Input arrangements for video game devices
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0145 creating an intermediate image
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Augmented reality with direct user interaction is described. In one example, an augmented reality system comprises a user-interaction region, a camera that captures images of an object in the user-interaction region, and a partially transparent display device which combines a virtual environment with a view of the user-interaction region, so that both are visible at the same time to a user. A processor receives the images, tracks the object'"'"'s movement, calculates a corresponding movement within the virtual environment, and updates the virtual environment based on the corresponding movement. In another example, a method of direct interaction in an augmented reality system comprises generating a virtual representation of the object having the corresponding movement, and updating the virtual environment so that the virtual representation interacts with virtual objects in the virtual environment. From the user'"'"'s perspective, the object directly interacts with the virtual objects.

Claims

1. An augmented reality system, comprising:

an image sensor arranged to capture images of an object in a user-interaction region;

a display device which is at least partially transparent and arranged to combine a displayed virtual environment with a view of the user-interaction region, such that both the virtual environment and the object are concurrently visible to a viewing user using the display device;

a processor arranged to receive the images, calculate a corresponding position within the virtual environment that is visually aligned with position of the object when viewed by the viewing user using the display device, and control the display device to update the virtual environment based on the corresponding position to enable an interaction with the virtual environment by a touch or gesture of the user, the processor being further arranged to selectively control the display device to render black pixels at locations in the virtual environment that are coincident with at least one hand of the user.

2. An augmented reality system according to claim 1, wherein the object is a body part of the user.

3. An augmented reality system according to claim 1, further comprising a tracking sensor arranged to monitor at least one of an eye, face and head of the user, and wherein the processor is further arranged to use data from the tracking sensor to track the user'"'"'s viewing position, and adapt the display of the virtual environment in dependence on the viewing position such that the virtual environment and the user-interaction region are visually aligned from the user'"'"'s perspective.

4. An augmented reality system according to claim 1, further comprising a projector arranged to selectively project light into the user-interaction region responsive to commands from the processor.

5. An augmented reality system according to claim 4, wherein the projector is controlled by the processor to selectively omit lighting in one or more portions of the user-interaction region, such that the one or more portions are less visible to the user.

6. An augmented reality system according to claim 1, further comprising a light sensor arranged to provide an ambient light level measurement to the processor, and wherein the processor is further arranged to control an output light intensity value for the display device responsive to the ambient light level measurement.

7. An augmented reality system according to claim 1, wherein the display device comprises at least one of:

a transparent organic light emitting diode display;

a liquid crystal display;

an organic light emitting diode display;

a stereoscopic display;

an autostereoscopic display; and

a volumetric display.

8. An augmented reality system according to claim 1, wherein the display device is arranged to be switchable between a first state in which both the user-interaction region and the virtual environment is visible to the user, and a second state in which only the virtual environment is visible to the user.

9. An augmented reality system according to claim 1, wherein the display device comprises:

a display screen arranged to display the virtual scene; and

an optical beam splitter positioned to reflect light from the display screen and transmit light from the user-interaction region, such that both the virtual environment and the object are concurrently visible to a viewing user on the beam splitter.

10. An augmented reality system according to claim 9, wherein the display screen comprises a first portion and second portion;

and

wherein the display device further comprises;

a half-silvered mirror positioned to reflect light from the first portion of the display screen; and

a full-silvered mirror positioned to reflect light from the second portion of the display and through the half-silvered mirror, such that images shown on the first and second portions appear overlaid but at different depths when viewed by the user.

11. An augmented reality system according to claim 9, wherein the optical beam-splitter comprises a half-silvered mirror.

12. An augmented reality system according to claim 9, wherein the optical beam-splitter is arranged so that the user-interaction region is located between a table-top and the optical beam-splitter.

13. An augmented reality system according to claim 1, wherein the image sensor is a depth camera.

14. A computer-implemented method of direct user-interaction in an augmented reality system, comprising:

controlling, using a processor, an at least partially transparent display device to display a virtual environment combined with a view of a user-interaction region, such that both the virtual environment and an object are concurrently visible and visually aligned to a viewing user using the display device;

detecting movement of the object located in the user-interaction region using an image sensor;

generating a virtual representation of the object having a corresponding movement in the virtual environment that is visually aligned with the movement of the object when viewed by the viewing user using the display device;

updating the virtual environment using the display device such that the virtual representation interacts with one or more virtual objects in the virtual environment, such that, from the perspective of the user, the movement of the object viewed through the display device directly interacts with the one or more virtual objects in the virtual environment to enable an interaction with the virtual environment by allowing a touch or gesture of the user to manipulate the position of the one or more virtual objects; and

selectively controlling the at least partially transparent display device to render black pixels at locations in the virtual environment that are coincident with at least one hand of the user.

15. A method according to claim 14, wherein the virtual environment is a physics simulation-based user interface generated by the processor, and the virtual representation interacts with one or more virtual objects in the virtual environment by simulating forces imparted to the one or more virtual objects by the movement of the virtual representation.

16. A method according to claim 14, further comprising the step of the processor rendering the one or more virtual objects such that the virtual objects appear placed on top of the user-controlled object when viewed by the user on the display device, and the step of updating the virtual environment comprises moving the virtual objects in alignment with the movement of the user-controlled object.

17. A method according to claim 14, further comprising the step of receiving, at the processor, an image of the user'"'"'s face from a tracking sensor, and determining the user'"'"'s viewing position.

18. A method according to claim 17, wherein the step of controlling the display device comprises using the user'"'"'s viewing position to visually align the virtual environment with the user-interaction region from the user'"'"'s perspective.

19. A method according to claim 14, further comprising the step of the processor selectively controlling the display device to reduce the light emitted from at least a portion of the display device, such that a corresponding portion of the user-interaction region is more visible through the display device than surrounding portions.

20. An augmented reality system, comprising:

a display screen arranged to display a three-dimensional virtual environment comprising one or more virtual objects;

a depth camera arranged to capture depth images of a user-controlled object located within a user-interaction region;

a half-silvered mirror positioned to reflect light from the display screen and transmit light from within the user-interaction region, such that both the virtual environment and the user-controlled object are concurrently visible to the user using a surface of the half-silvered mirror;

a processor arranged to;

control the display screen to display the virtual environment such that the virtual environment is visually aligned with the user-interaction region on a surface of the half-silvered mirror from the perspective of the user;

receive the depth images and track movement of the user-controlled object in three dimensions in the user-interaction region;

generate a virtual representation of the user-controlled object having a corresponding movement in the virtual environment that is visually aligned with the movement of the user-controlled object when viewed by the user on the half-silvered mirror; and

update the virtual environment on the display screen such that the virtual representation interacts with the one or more virtual objects in the virtual environment, such that, from the perspective of the user, the movement of the user-controlled object viewed through the half-silvered mirror directly interacts with the virtual objects in the virtual environment to enable an interaction with the virtual environment by allowing a touch or gesture of the user to manipulate the position of the one or more virtual objects, the processor being further arranged to selectively control the display screen to render black pixels at locations in the virtual environment that are coincident with at least one hand of the user.