Interactions with 3D virtual objects using poses and multiple-dof controllers

• US 10,733,806 B2
• Filed: 11/13/2019
• Issued: 08/04/2020
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Hamilton, IV, Frank Alexander, Fontaine, Marshal Ainsworth, Hoover, Paul Armistead
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US16/682,851
• Publication Number:
  US 20200082637A1
• Time in Patent Office: 265 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user'"'"'s pose. The wearable system can support various user interactions with objects in the user'"'"'s environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user'"'"'s poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.

Claims

1. A system comprising:

a display system of a wearable device configured to provide a three-dimensional (3D) view to a user and permit a user interaction with objects in a field of regard (FOR) of a user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the display system;

a hardware processor in communication with the display system, the hardware processor programmed to;

determine contextual information regarding the environment of the user;

determine, based at least in part on the contextual information, a dynamically-adjustable aperture of a virtual cone;

initiate a cone cast of the virtual cone with the dynamically-adjustable aperture;

detect a collision between the virtual cone and one or more objects in the environment; and

perform an action on the one or more objects.

2. The system of claim 1, wherein the contextual information comprises at least one of:

a type, a layout, a location, a size, a distance, or a density of objects within a field of view (FOV) of the user.

3. The system of claim 2, wherein to calculate the density of objects within the FOV of the user, the hardware processor is programmed to:

calculate a number of objects in the FOV;

calculate a fraction of the FOV that is covered by the objects;

or

calculate a contour map for the objects.

4. The system of claim 1, wherein the contextual information comprises at least one of:

a preference of the user, a physical condition associated with the user, or information associated with the environment.

5. The system of claim 1, wherein to determine the dynamically-adjustable aperture, the hardware processor is programmed to select the aperture between a minimum size and a maximum size.

6. The system of claim 5, wherein the minimum size is zero.

7. The system of claim 1, wherein the action comprises rendering a focus indicator associated with the one or more objects.

8. The system of claim 1, wherein to detect the collision between the virtual cone and the one or more objects, the hardware processor is programmed to determine that the one or more objects intersect with a virtual surface of the virtual cone or fall within the dynamically-adjustable aperture of the virtual cone.

9. The system of claim 1, wherein the one or more objects comprise multiple collided objects, and the hardware processor is programmed to apply an occlusion disambiguation technique to the multiple collided objects to identify an occluded object or to determine a depth ordering or position among occluded objects.

10. The system of claim 1, wherein the one or more objects comprise multiple collided objects, and the hardware processor is programmed to present a user interface element to select one or more of the multiple collided objects.

11. The system of claim 1, wherein the action with the one or more objects comprises one or more of:

a selection of the one or more objects;

a movement of the one or more objects;

display of a menu or toolbar associated with the one or more objects;

or

a game operation on a virtual avatar in a game.

12. The system of claim 1, wherein the virtual cone comprises a central ray, the dynamically-adjustable aperture is transverse to the central ray, and a direction of the central ray is based on a pose of the user.

13. The system of claim 1, wherein the virtual cone comprises a proximal end, and the hardware processor is programmed to anchor the proximal end to at least one of the following locations:

a location in-between the user'"'"'s eyes, a location on a portion of a user'"'"'s arm, or a location on a user input device.

14. The system of claim 1, wherein the virtual cone comprises a distal end, and to detect the collision, the hardware processor is programmed to scan for collisions with the one or more objects within the distal end of the virtual cone.

15. The system of claim 1, wherein the virtual cone comprises a proximal end, a distal end, or a depth, and the hardware processor is programmed to anchor the proximal end, the distal end, or the depth of the virtual cone based on:

the contextual information, user input, a body gesture, a body pose, a direction of gaze, or a voice command.

16. The system of claim 1, further comprising:

a pose sensor configured to acquire pose data associated with a pose of the user; and

wherein the hardware processor is programmed to;

translate the virtual cone based on the pose data associated with the pose of the user.

17. The system of claim 16, wherein the hardware processor is programmed to apply a multiplier to the pose of the user.

18. The system of claim 17, wherein the multiplier increases with distance from the user.

19. The system of claim 1, wherein the hardware processor is further programmed to:

determine contextual features of the one or more objects; and

resize the dynamically-adjustable aperture of the virtual cone based at least in part on the contextual features.

20. The system of claim 1, wherein to initiate the cone cast, the hardware processor is programmed to extend a distal end of the virtual cone until the distal end reaches a termination threshold.

21. The system of claim 20, wherein the termination threshold comprises a threshold distance or a boundary of the environment.

22. The system of claim 1, wherein the hardware processor is further programmed to:

render to the user, via the display system, a visual representation of at least a portion of the virtual cone.

23. The system of claim 22, wherein the visual representation of the virtual cone comprises a color of the virtual cone.

24. The system of claim 23, wherein the color is based at least partly on a user preference or the environment of the user.

25. The system of claim 22, wherein the hardware processor is programmed to adjust a contrast of the visual representation of the virtual cone.

26. The system of claim 22, wherein the visual representation of at least a portion of the virtual cone comprises a visual representation of an aperture of the virtual cone, a surface of the virtual cone, or a central ray of the virtual cone.

27. The system of claim 1, wherein the objects comprise virtual objects.