User Interaction in Augmented Reality

• US 20120113223A1
• Filed: 11/05/2010
• Published: 05/10/2012
• Est. Priority Date: 11/05/2010
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hilliges, Otmar, Izadi, Shahram, Molyneaux, David, Hodges, Stephen Edward, Butler, David Alexander, Kim, David
• Application Number:
  US12/940,383
• Publication Number:
  US 20120113223A1
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2210/28 Force feedback

Abstract

Techniques for user-interaction in augmented reality are described. In one example, a direct user-interaction method comprises displaying a 3D augmented reality environment having a virtual object and a real first and second object controlled by a user, tracking the position of the objects in 3D using camera images, displaying the virtual object on the first object from the user'"'"'s viewpoint, and enabling interaction between the second object and the virtual object when the first and second objects are touching. In another example, an augmented reality system comprises a display device that shows an augmented reality environment having a virtual object and a real user'"'"'s hand, a depth camera that captures depth images of the hand, and a processor. The processor receives the images, tracks the hand pose in six degrees-of-freedom, and enables interaction between the hand and the virtual object.

Claims

1. A computer-implemented method of direct user-interaction in an augmented reality system, comprising:

controlling, using a processor, a display device to display a three-dimensional augmented reality environment comprising a virtual object and a real first and second object controlled by a user;

receiving, at the processor, a sequence of images from at least one camera showing the first and second object, and using the images to track the position of the first and second object in three dimensions;

enabling interaction between the second object and the virtual object when the first and second object are in contact at the location of the virtual object from the perspective of the user.

2. A method according to claim 1, wherein the first object comprises at least one of:

an object held in a hand of the user;

a hand;

a forearm;

a palm of a hand; and

a fingertip of a hand.

3. A method according to claim 1, wherein the second object comprises a digit of a hand.

4. A method according to claim 1, wherein the virtual object is a user-actuatable control.

5. A method according to claim 4, wherein the user-actuatable control comprises at least one of:

a button;

a menu item;

a toggle; and

an icon.

6. A method according to claim 4, wherein the step of enabling interaction comprises the second object actuating the control.

7. A method according to claim 1, wherein the step of enabling interaction comprises the second object performing at least one of:

a rotation operation;

a scaling operation;

a translation operation;

a warping operation;

a shearing operation;

a deforming operation;

a painting operation; and

a selection operation on the virtual object.

8. A method according to claim 1, wherein step of enabling interaction comprises generating a touch plane coincident with a surface of the first object, determining from the position of the second object that the second object and the touch plane converge, and, responsive thereto, triggering the interaction between the second object and the virtual object.

9. A method according to claim 1, wherein the step of using the position and orientation of the first object to update the augmented reality environment to display the virtual object comprises rendering the virtual object on a surface of the first object from the perspective of the user.

10. A method according to claim 1, further comprising the step of updating the location of the virtual object in the augmented reality environment to move the virtual object in accordance with a corresponding movement of the first object to maintain a relative spatial arrangement from the perspective of the user.

11. A method according to claim 1, wherein the camera is a depth camera arranged to capture images having a plurality of image elements, each image element having a value indicating a distance between the camera and a corresponding portion of the first or second object.

12. An augmented reality system, comprising:

a display device arranged to display a three-dimensional augmented reality environment comprising a virtual object and a real hand of a user;

a depth camera arranged to capture images of the hand of the user having a plurality of image elements, each image element having a value indicating a distance between the camera and a corresponding portion of the hand;

a processor arranged to receive the depth camera images, track the movement and pose of the hand of the user in six degrees of freedom, monitor the pose of the hand to detect a predefined gesture, and, responsive to detecting the predefined gesture, trigger an associated interaction between the hand of the user and the virtual object.

13. An augmented reality system according to claim 12, wherein the predefined gesture comprises movement of a digit of the hand associated with a function into contact with the virtual object, and the associated interaction comprises applying the function to the virtual object.

14. An augmented reality system according to claim 13, wherein the function comprises at least one of:

a copy function;

a paste function;

a cut function;

a delete function;

a move function;

a warping operation;

a shearing operation;

a deforming operation;

a painting operation;

a rotate function; and

a scale function.

15. An augmented reality system according to claim 12, wherein the predefined gesture comprises a pinch gesture, and the associated interaction comprises extrusion of a 3D virtual item from the virtual object based on a two-dimensional cross-section traced by the user'"'"'s hand.

16. An augmented reality system according to claim 15, wherein the processor is further arranged to enable the user to manipulate the 3D virtual item in the augmented reality environment, responsive to release of the pinch gesture.

17. An augmented reality system according to claim 12, wherein the predefined gesture comprises an extension of a plurality of digits of the hand towards the virtual object, and the associated interaction comprises the drawing of the virtual object towards the user, despite the virtual object being out of reach of the user'"'"'s hand.

18. An augmented reality system according to claim 12, wherein the display device comprises:

a display screen arranged to display the virtual object; and

an optical beam-splitter positioned to reflect light from the display screen on a first side of the beam-splitter, and transmit light from an opposite side of the beam-splitter, such that when the hand of the user is located on the opposite side, both the virtual object and the hand are concurrently visible to the user on the first side of the beam-splitter.

19. An augmented reality system according to claim 12, wherein the display device comprises:

a video camera mountable on the user'"'"'s head and arranged to capture images in the direction of the user'"'"'s gaze; and

a display screen mountable on the user'"'"'s head and arranged to display the video camera images combined with the virtual object.

20. One or more tangible device-readable media with device-executable instructions that, when executed by a computing device, direct the computing device to perform steps comprising:

generating a three-dimensional augmented reality environment comprising a virtual object and a real first hand and second hand of one or more users;

controlling a display device to display the virtual object and the first hand and second hand;

receiving a sequence of images from a depth camera showing the first hand and second hand;

analyzing the sequence of images to determine a position and pose of each of the first hand and second hand in six degrees of freedom;

using the position and pose of the second hand to render the virtual object at a location in the augmented reality environment such that the virtual object appears to be located on the surface of the second hand from the perspective of the user, and moving the virtual object in correspondence with movement of the second hand; and

triggering interaction between the first hand and the virtual object at the instance when the position and pose of the first hand and second hand indicates that a digit of the first hand is touching the second hand at the location of the virtual object.