Interactions with 3D virtual objects using poses and multiple-DOF controllers

• US 10,510,191 B2
• Filed: 08/02/2019
• Issued: 12/17/2019
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Hamilton, Frank, Fontaine, Marshal A., Hoover, Paul Armistead
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US16/530,901
• Publication Number:
  US 20190355180A1
• Time in Patent Office: 137 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user'"'"'s pose. The wearable system can support various user interactions with objects in the user'"'"'s environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user'"'"'s poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.

Claims

1. A system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user and permit a user interaction with objects in a field of regard (FOR) of a user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the display system;

a sensor configured to acquire pose data associated with a pose of the user;

a hardware processor in communication with the sensor and the display system, the hardware processor programmed to;

determine a group of objects in the FOR of the user;

initiate a cone cast, wherein the cone cast comprises a cast of a virtual cone with a dynamically-adjustable aperture;

determine a field of view (FOV) of the user based at least partly on the pose data acquired by the sensor;

identify, from the group of objects in the FOR of the user, a subgroup of objects that are within the FOV of the user;

analyze one or more contextual features of the subgroup of objects that are within the FOV of the user;

dynamically resize the aperture of the virtual cone based at least in part on the one or more contextual features; and

render to the user, via the display system, a visual representation of at least a portion of the virtual cone.

2. The system of claim 1, wherein the contextual features comprise at least one of:

a type, a layout, a location, a size, a distance, or a density of objects within the FOV of the user.

3. The system of claim 2, wherein to calculate the density of objects within the FOV of the user, the hardware processor is programmed to:

calculate a number of objects in the FOV;

calculate a fraction of the FOV that is covered by the objects;

or

calculate a contour map for the objects.

4. The system of claim 1, wherein the contextual features comprise at least one of:

a preference of the user, a physical condition associated with the user, or information associated with the environment.

5. The system of claim 1, wherein to dynamically resize the aperture of the virtual cone, the hardware processor is programmed to resize the aperture between a minimum size and a maximum size.

6. The system of claim 5, wherein the minimum size is zero.

7. The system of claim 1, wherein the hardware processor is further programmed to:

scan for a collision between the virtual cone and one or more objects in the FOV of the user;

identify, in response to the collision, a collided object; and

perform a user interaction associated with the collided object.

8. The system of claim 7, wherein the hardware processor is further programmed to render a focus indicator associated with the collided object.

9. The system of claim 7, wherein to identify the collided object, the hardware processor is programmed to determine that the collided object intersects with a virtual surface of the virtual cone or falls within the virtual cone.

10. The system of claim 7, wherein the collided object comprises multiple collided objects, and the hardware processor is programmed to apply an occlusion disambiguation technique to the multiple collided objects to identify an occluded object or to determine a depth ordering or position among occluded objects.

11. The system of claim 7, wherein the collided object comprises multiple collided objects, and the hardware processor is programmed to present a user interface element to select one or more of the multiple collided objects.

12. The system of claim 7, wherein the user interaction with the collided object comprises one or more of:

a selection of the collided object;

a movement of the collided object;

display of a menu or toolbar associated with the collided object;

or

a game operation on a virtual avatar in a game.

13. The system of claim 1, wherein the virtual cone comprises a central ray, the aperture is transverse to the central ray, and a direction of the central ray is based on the pose of the user.

14. The system of claim 1, wherein the virtual cone comprises a proximal end, and the hardware processor is programmed to anchor the proximal end to at least one of the following locations:

a location in-between the user'"'"'s eyes, a location on a portion of a user'"'"'s arm, or a location on a user input device.

15. The system of claim 1, wherein the virtual cone comprises a distal end, and the hardware processor is programmed to scan for collisions with one or more objects within the distal end of the virtual cone.

16. The system of claim 1, wherein the virtual cone comprises a proximal end, a distal end, or a depth, and the hardware processor is programmed to anchor the proximal end, the distal end, or the depth of the virtual cone based on:

the contextual features, user input, a body gesture, a body pose, a direction of gaze, or a voice command.

17. The system of claim 1, wherein to translate the virtual cone based on the data associated with the pose of the user, the hardware processor is programmed to apply a multiplier to the pose of the user.

18. The system of claim 17, wherein the multiplier increases with distance from the user.

19. The system of claim 1, wherein to dynamically resize the aperture of the virtual cone, the hardware processor is programmed to increase a size of the aperture at a distal end of the virtual cone.

20. The system of claim 1, wherein to initiate the cone cast, the hardware processor is programmed to extend a distal end of the virtual cone until the distal end reaches a termination threshold.

21. The system of claim 20, wherein the termination threshold comprises a threshold distance or a boundary of the environment.

22. The system of claim 1, wherein the virtual cone has a depth, and the hardware processor is programmed to adjust the depth of the virtual cone based at least in part on a distance to the subgroup of objects in the FOV.

23. The system of claim 1, wherein the visual representation of the virtual cone comprises a color of the virtual cone.

24. The system of claim 23, wherein the color is based at least partly on a user preference or the environment of the user.

25. The system of claim 1, wherein the hardware processor is programmed to adjust a contrast of the visual representation of the virtual cone relative to the subgroup of objects in the FOV.

26. The system of claim 1, wherein the visual representation of at least a portion of the virtual cone comprises a visual representation of an aperture of the virtual cone, a surface of the virtual cone, or a central ray of the virtual cone.

27. The system of claim 1, wherein the objects comprise virtual objects.