Configuring an interaction zone within an augmented reality environment

• US 9,292,085 B2
• Filed: 06/29/2012
• Issued: 03/22/2016
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bennett, Darren, Mount, Brian J., Scavezze, Michael J., McCulloch, Daniel J., Ambrus, Anthony J., Steed, Jonathan T., Tomlin, Arthur C., Geisner, Kevin A.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US13/539,043
• Publication Number:
  US 20140002444A1
• Time in Patent Office: 1,362 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Technology is described for automatically determining placement of one or more interaction zones in an augmented reality environment in which one or more virtual features are added to a real environment. An interaction zone includes at least one virtual feature and is associated with a space within the augmented reality environment with boundaries of the space determined based on the one or more real environment features. A plurality of activation criteria may be available for an interaction zone and at least one may be selected based on at least one real environment feature. The technology also describes controlling activation of an interaction zone within the augmented reality environment. In some examples, at least some behavior of a virtual object is controlled by emergent behavior criteria which defines an action independently from a type of object in the real world environment.

Claims

1. A method for adaptively configuring one or more interaction zones within an augmented reality environment including a real environment and at least one virtual feature, the method comprising:

an interaction zone including at least one virtual feature, real environment compatibility criteria for the at least one virtual feature, space dimensions and activation criteria for the at least one virtual feature;

automatically selecting one or more interaction zone candidates based on one or more real environment features of the real environment satisfying real environment compatibility criteria for candidates being identified in a (3D) mapping of the augmented reality environment;

automatically selecting one or more interaction zones which satisfy zone compatibility criteria for configuration within the augmented reality environment from the one or more candidates;

updating the 3D mapping with 3D space position data for the one or more interaction zones selected for configuration in the augmented reality environment;

displaying at least one virtual feature of at least one of the selected interaction zones by a near-eye, augmented reality display responsive to application execution criteria being satisfied, and the at least one virtual feature being dynamically animated and responsive to the real environment compatibility criteria, the space dimensions and the activation criteria when the at least one virtual feature is within a display field of view of the near-eye, augmented reality display; and

identifying a range of the space dimensions for the one or more interaction zone candidates and determining whether one or more of the space dimensions within the range satisfy spatial distance criteria with at least one other of the interaction zone candidates relative to size dimensions of the augmented reality environment, and

responsive to satisfying the spatial distance criteria, providing another activation criteria for the at least one other of the interaction zone candidates to avoid satisfying the activation criteria of the one or more interaction zone candidates.

2. The method of claim 1 wherein the real environment compatibility criteria for at least one virtual feature of the one or more interaction zone candidates comprises a surface with one or more physical properties.

3. The method of claim 1 wherein the real environment compatibility criteria further comprises lighting.

4. The method of claim 1 wherein the zone compatibility criteria includes the spatial distance criteria between at least two types of interaction zones relative to the size dimensions of the augmented reality environment.

5. The method of claim 1 further comprising:

assigning activation criteria selected for any of the selected one or more interaction zones for satisfying adaptive augmented reality environment criteria.

6. A system for adaptively configuring one or more interaction zones within an augmented reality environment including a real environment and at least one virtual feature comprising:

a near-eye, augmented reality display having a display field of view and being supported by a near-eye support structure;

one or more processors communicatively coupled to the near-eye, augmented reality display for controlling the display;

the one or more processors having access to memory storing a three-dimensional (3D) mapping of the augmented reality environment including 3D space positions for objects in the augmented reality environment;

the memory storing interaction zone data for an interaction zone including an identifier of a virtual feature, a storage location identifier of virtual data for display of the virtual feature, space dimensions for the interaction zone, real environment compatibility criteria, activation criteria for the interaction zone, and one or more actions for performance by the virtual feature including one or more real environment dependent actions; and

the one or more processors automatically configuring one or more interaction zones at 3D space positions in the 3D mapping of the augmented reality environment based on one or more real environment features satisfying the real environment compatibility criteria, such that the at least one virtual feature is configured to be animated and responsive to the real environment dependent actions based on the real environment compatibility criteria, the space dimensions and the activation criteria when the at least one virtual feature is within a display field of view of the near-eye, augmented reality display, wherein

the real environment compatibility criteria comprise one or more emergent behavior rules which identify a real environment feature based on real environment dependent actions and independently of a type of object.

7. The system of claim 6 further comprising:

the one or more processors being communicatively coupled to at least one camera supported by the support structure for receiving image data and depth data including the display field of view captured by the at least one camera;

the one or more processors automatically determining one or more real environment features including size dimensions of the augmented reality environment based on the captured image and depth data; and

the one or more processors automatically configuring one or more interaction zones at 3D space positions in the 3D mapping of the augmented reality environment based on one or more real environment features satisfying the real environment compatibility criteria further comprises the one or more processors automatically determining a number of the one or more interaction zones based on the size dimensions of the augmented reality environment.

8. The system of claim 6 wherein the one or more real environment dependent actions are controlled by one or more emergent behavior rules which identify a real environment feature independently of a type of object.

9. The system of claim 6 wherein the real environment feature comprises at least one of the following comprising:

one or more surface properties; and

a lighting value.

10. The system of claim 6 wherein the one or more processors automatically configuring one or more interaction zones at 3D space positions in the 3D mapping of the augmented reality environment based on one or more real environment features satisfying the real environment compatibility criteria further comprises:

identifying the one or more real environment features which satisfy the real environment compatibility and comfort criteria for at least one physical characteristic of a user wearing the near-eye, augmented reality display; and

registering the one or more interaction zones to the identified one or more real environment features.

11. The system of claim 6 further comprising:

the one or more processors modifying a performance parameter for one or more actions for performance by at least one virtual feature of the one or more interaction zones based on the 3D space positions in the 3D mapping and quality of experience criteria.

12. The system of claim 6 further comprising:

a natural user interface comprising an eye tracking system being positioned by the support structure and eye tracking software being executed by the one or more processors for identifying a user physical action of an eye based action based on eye data from the eye tracking system; and

the one or more processors activating the virtual feature of at least one interaction zone of the one or more interaction zones responsive to the eye based action satisfying activation criteria of the at least one interaction zone.

13. The system of claim 6 wherein at least one interaction zone of the one or more interaction zones is registered to a movable object within the augmented reality environment.

14. One or more processor readable storage devices comprising instructions which cause one or more processors to execute a method for controlling activation of one or more interaction zones within an augmented reality environment including a real environment and at least one virtual feature by a near-eye, augmented reality display device system, the method comprising:

an interaction zone including at least one virtual feature, space dimensions and one or more boundaries, activation criteria for the interaction zone, and deactivation criteria for the interaction zone;

adaptively configuring one or more interaction zones within an augmented reality environment and automatically changing a shape of the one or more interaction zones to conform with different features of the at least one virtual feature;

identifying natural user input data from data captured by a capture device of the near-eye, augmented reality display system indicating at least one user physical action by at least one body part of a user wearing the near-eye, augmented reality display device system; and

responsive to the at least one user physical action satisfying the activation criteria for the at least one interaction zone, displaying at least one change of one or more virtual features associated with the at least one interaction zone, and the at least one virtual feature is configured to be animated and responsive to the user physical action based on the real environment compatibility criteria, the space dimensions and the activation criteria.

15. The one or more processor readable storage devices of claim 14 wherein the activation criteria for the at least one interaction zone further comprises successful completion of at least one task associated with at least one other interaction zone.

16. The one or more processor readable storage devices of claim 15 wherein the activation criteria for the at least one interaction zone further comprises the near-eye augmented reality display system detecting a near-eye, augmented reality display device of the near-eye, augmented reality display device system has moved at least a re-start distance from at least one boundary of the at least one interaction zone after having satisfied deactivation criteria by exiting from the at least one interaction zone.

17. The one or more processor readable storage devices of claim 14 wherein adaptively configuring one or more interaction zones within the augmented reality environment further comprises nesting at least two interaction zones of the one or more interaction zones in which the at least two interaction zones at least partially share a 3D space position in the augmented reality environment, and successful completion of at least one task in at least one of the at least two interaction zones permits display of another of the at least two interaction zones.

18. The one or more processor readable storage devices of claim 14 wherein the at least one user physical action satisfying the activation criteria for the at least one interaction zone is a gesture.