Selection of objects in three-dimensional space

• US 10,268,266 B2
• Filed: 06/29/2016
• Issued: 04/23/2019
• Est. Priority Date: 06/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mathey-Owens, Cheyne Rory Quin, Tomlin, Arthur, Ghaly, Marcus
• Primary Examiner(s): Polo, Gustavo D
• Application Number:
  US15/197,484
• Publication Number:
  US 20180004283A1
• Time in Patent Office: 1,028 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points
• CPC Class Codes: G06T 7/579 from motion

Abstract

A user may select or interact with objects in a scene using gaze tracking and movement tracking. In some examples, the scene may comprise a virtual reality scene or a mixed reality scene. A user may move an input object in an environment and be facing in a direction towards the movement of the input object. A computing device may use sensors to obtain movement data corresponding to the movement of the input object, and gaze tracking data including to a location of eyes of the user. One or more modules of the computing device may use the movement data and gaze tracking data to determine a three-dimensional selection space in the scene. In some examples, objects included in the three-dimensional selection space may be selected or otherwise interacted with.

Claims

1. A system comprising:

a display to present a scene, the scene comprising at least one virtual object;

one or more sensors to capture data from a real world environment of a user;

one or more processors communicatively coupled to the one or more sensors and the display; and

memory having computer-executable instructions stored thereupon which, when executed by the one or more processors, cause the computing device to perform operations comprising;

detecting, by the one or more sensors, a starting action;

obtaining, from the one or more sensors, movement data corresponding to a movement of an input object in the real world environment of the user;

detecting, by the one or more sensors, an ending action;

identifying a shape corresponding to the movement of the input object between the starting action and the ending action;

obtaining, from the one or more sensors, gaze tracking data including a location of the eyes of the user in the real world environment;

determining, based at least in part on the shape and the gaze tracking data, a three-dimensional (3D ) selection space in the scene by;

identifying a first location and a second location along the shape;

calculating a first vector originating from a location at or near an eye of the user and intersecting the first location along the shape;

calculating a second vector originating from the location at or near the eye of the user and intersecting the second location along the shape; and

extending the first vector and the second vector in a direction substantially parallel to a third vector which extends from the location near or at the eyes of the user to a location at or near the shape;

identifying one or more objects in the scene located in or at least partially in the 3D selection space, the objects including at least one of a physical object in the real world environment or the at least one virtual object; and

performing an operation on the one or more objects.

2. The system of claim 1, wherein the scene comprises one of:

a mixed reality scene including the real world environment and at least one of the physical object or the at least one virtual object;

or

a virtual reality scene including the at least one virtual object and excluding the real world environment.

3. The system of claim 1, the determining the 3D selection space in the scene further comprising:

determining a distance from the eye of the user to the 3D selection space to create a near boundary by cutting the 3D selection space; and

creating a finite 3D selection space by bounding the 3D selection space at the near boundary, the bounding comprising cutting the 3D selection space with a plane perpendicular to the third vector at the distance.

4. The system of claim 1, the determining the 3D selection space in the scene further comprising:

determining a distance from the user to the 3D selection space to create a far boundary by preventing the 3D selection space from extending further than the far boundary; and

creating a finite 3D selection space by bounding the 3D selection space at the far boundary, the bounding comprising cutting the 3D selection space with a plane perpendicular to the third vector at the distance.

5. The system of claim 1, wherein the movement of the input object in the real world environment comprises at least one of a linear movement or a free form movement.

6. The system of claim 1, wherein the input object comprises at least one of a controller or a body part of the user.

7. The system of claim 1, the performing the operation on the one or more objects including:

selecting the one or more objects; and

visually indicating that the one or more objects are selected by virtually associating an indicator with the one or more objects.

8. A computer-implemented method comprising:

obtaining, from one or more sensors, movement data corresponding to a movement of an input object in a real world environment;

obtaining, from the one or more sensors, gaze tracking data including a location of eyes of a user in the environment;

presenting, on a display of a computing device associated with the user, a scene comprising at least one virtual object;

calculating, based at least in part on the gaze tracking data and the movement data, multiple vectors, each of the multiple vectors originating at or near an eye of the user and passing through a location along a shape defined by the movement of the input object;

defining a three-dimensional (3D) volume in the scene by extending the multiple vectors in a direction substantially parallel to a reference vector extending from a location at or near the eye of the user to a location at or near the shape, the extending the multiple vectors in the direction substantially parallel to the reference vector to define the 3D volume comprising presenting a virtual representation of the 3D volume on the display of the computing device associated with the user;

identifying one or more objects in the scene included in or at least partially in the 3D volume, the one or more objects including at least one of a physical object in the real world environment or the at least one virtual object; and

performing an operation on the one or more objects included in or at least partially in the 3D volume.

9. The computer-implemented method of claim 8, wherein the scene comprises one of a mixed reality scene including the real world environment and at least one of the physical object or the at least one virtual object, or a virtual reality scene including the at least one virtual object and excluding the real world environment.

10. The computer-implemented method of claim 8, the obtaining the movement data comprising:

detecting, by the one or more sensors, a starting action of the input object;

detecting, by the one or more sensors, an ending action of the input object; and

capturing the movement of the input object detected between the starting action and the ending action.

11. The computer-implemented method of claim 8, further comprising:

determining a distance from the user to the 3D volume to create a near boundary at which to bound the 3D volume; and

bounding the 3D volume at the near boundary by cutting the 3D volume with a plane perpendicular to the reference vector at the distance.

12. The computer-implemented method of claim 8, further comprising:

determining a distance from the user to the 3D volume to create a far boundary at which to bound the 3D volume by preventing the 3D volume from extending further than the far boundary; and

bounding the 3D volume at the far boundary by cutting the 3D volume with a plane substantially perpendicular to the reference vector at the distance.

13. The computer-implemented method of claim 8, wherein the 3D volume comprises a geometric shape defined by:

a front planar surface substantially perpendicular to the reference vector and having a boundary defined by the movement of the input object;

a rear planar surface spaced apart from and parallel to the front planar surface, the rear planar surface having a boundary that is geometrically similar in shape to the boundary of the front planar surface; and

a side wall extending from the front planar surface to the rear planar surface and extending around the boundaries of the front planar surface and the rear planar surface.

14. A wearable computing device comprising:

one or more sensors to obtain data from an environment, the environment comprising at least one of a virtual reality environment or a mixed reality environment;

one or more processors communicatively coupled to the one or more sensors; and

memory having computer-executable instructions stored thereupon which, when executed by the one or more processors, cause the computing device to perform operations comprising;

obtaining, from the one or more sensors, movement data corresponding to a movement of an input object in the environment;

obtaining, from the one or more sensors, gaze tracking data corresponding to a location of eyes of a user;

analyzing the movement data to calculate a shape defined by the movement of the input object;

calculating vectors originating at a location near or at an eye of the user and intersecting with points along the shape;

extending the vectors in a direction substantially parallel to a reference vector which extends from the location near or at the eye of the user to a location at or near the shape to create a three-dimensional (3D) volume;

identifying one or more objects included in or partially included in the 3D volume; and

selecting the one or more objects included in or partially included in the 3D volume, the one or more objects including at least one of a physical object or a virtual object.

15. The wearable computing device of claim 14, the operations further comprising:

determining a distance from the user to the 3D volume to create a near boundary by cutting the 3D volume;

creating a frustum of the 3D volume by bounding the 3D volume at the near boundary, the bounding comprising cutting the 3D volume with a plane substantially perpendicular to the reference vector at the near boundary.

16. The wearable computing device of claim 14, the analyzing the movement data to calculate the shape defined by the movement of the input object comprising:

analyzing the movement data to determine the movement of the input object comprises a diagonal line; and

calculating the shape having a diagonal corresponding to the diagonal line.

17. The wearable computing device of claim 14, the selecting the one or more objects included in or partially in the 3D volume including:

visually indicating that the one or more objects included in or partially included in the 3D volume are selected by virtually associated an indicator with each of the one or more objects.