Interactions with 3D virtual objects using poses and multiple-DOF controllers

• US 10,078,919 B2
• Filed: 03/29/2017
• Issued: 09/18/2018
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Hamilton, Frank, Fontaine, Marshal A., Hoover, Paul Armistead
• Primary Examiner(s): Patel, Jitesh
• Application Number:
  US15/473,444
• Publication Number:
  US 20170287225A1
• Time in Patent Office: 538 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user'"'"'s pose. The wearable system can support various user interactions with objects in the user'"'"'s environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user'"'"'s poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.

Claims

1. A system for interacting with objects for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user and permit a user interaction with virtual objects in a field of regard (FOR) of a user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the display system;

a sensor configured to acquire data associated with a pose of the user;

a hardware processor in communication with the sensor and the display system, the hardware processor programmed to;

determine a pose of the user based on the data acquired by the sensor;

initiate a cone cast on a group of virtual objects in the FOR, the cone cast comprises casting a virtual cone with an aperture in a direction based at least partly on the pose of the user, wherein the aperture has a dynamically-adjustable size;

analyze contextual information associated with the group of virtual objects in the user'"'"'s environment to calculate the dynamically-adjustable size of the aperture for the cone cast toward the group of virtual objects;

automatically update the dynamically-adjustable size of the aperture of the virtual cone based at least partly on the contextual information of the group of virtual objects; and

cause the display system to render a visual representation of the virtual cone for the cone cast.

2. The system of claim 1, wherein the contextual information comprises at least one of:

a type, a layout, a location, a size, or a density of a subgroup of virtual objects within the field of view (FOV) of the user, wherein the FOV comprises a portion of the FOR that is capable of being perceived at a given time by the user via the display system.

3. The system of claim 2, wherein the density of the subgroup of virtual objects within the FOV of the user is calculated by at least one of:

calculating a number of virtual objects in the subgroup of virtual objects;

calculating a percentage of the FOV that is covered by the subgroup of virtual objects;

or

calculating a contour map for virtual objects in the subgroup of virtual objects.

4. The system of claim 1, wherein the hardware processor is further programmed to detect a collision between the virtual cone and one or more virtual objects within the group of virtual objects in the FOR, and wherein in response to detecting the collision, the hardware processor is further programmed to present a focus indicator to the one or more virtual objects.

5. The system of claim 4, wherein the hardware processor is programmed to apply an occlusion disambiguation technique to the one or more virtual objects collided with the virtual cone to identify an occluded virtual object.

6. The system of claim 1, wherein the cone comprises a central ray and wherein the aperture is transverse to the central ray.

7. The system of claim 1, wherein the virtual cone comprises a proximal end and wherein the proximal end is anchored to at least one of the following locations:

a location in-between the user'"'"'s eyes, a location on a portion of a user'"'"'s arm, a location on a user input device, or any other location in the environment of the user.

8. The system of claim 1, wherein the hardware processor is further programmed to receive an indication from a user input device anchoring a depth of the virtual cone to a depth plane and wherein cone cast is performed on the group of virtual objects within the depth plane.

9. A method for interacting with objects for a wearable device, the method comprising:

under control of a wearable device comprising a display system and hardware processor;

receiving a selection of a target virtual object displayed to a user at a first position in a three-dimensional (3D) space;

receiving an indication to move the target virtual object from the first position, wherein the indication comprises a first amount of movement associated with the user in a first direction in the 3D space;

analyzing a distance between the target virtual object and the user in a second direction different from the first direction;

calculating a multiplier to map the first amount of movement associated with the user into a second amount of movement for the target virtual object, the multiplier calculated based at least partly on the distance between the target virtual object and the user in the second direction;

calculating the second amount of movement in the first direction for the target virtual object, the second amount of movement calculated based at least partly on the first amount of movement and the multiplier; and

causing the target virtual object to be displayed to the user at a second position, the second position based at least in part on the first position and the second amount of movement.

10. The method of claim 9, wherein the multiplier increases proportionally with an increase in the distance between the target virtual object and the user.

11. The method of claim 9, wherein the multiplier is applied to one or more of:

a position change, a speed, or an acceleration, for calculating the second amount of movement.

12. The method of claim 9, wherein the indication to move the target virtual object comprises at least one of:

an actuation of a user input device associated with the wearable device or a change in a pose of the user.

13. The method of claim 12, wherein the pose comprises one or more of:

a head pose, an eye pose, or a body pose.

14. The method of claim 9, wherein the multiplier is greater than one where the distance between the target virtual object and the user is beyond the user'"'"'s hand reach.

15. A system for interacting with objects for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view showing a target virtual object;

a hardware processor in communication with the display system, the hardware processor programmed to;

receive an indication to move the target virtual object from a first position, wherein the indication comprises a first amount of movement associated with a user in a target direction in the 3D space;

analyze a distance between the target virtual object and the user;

calculate a multiplier to map the first amount of movement associated with the user into a second amount of movement for the target virtual object, the multiplier calculated based at least partly on the distance between the target virtual object and the user;

calculate the second amount of movement in the target direction for the target virtual object, the second amount of movement calculated based at least partly on the first amount of movement and the multiplier; and

by cause the display system to display the target virtual object at a second position, the second position based at least in part on the first position and the second amount of movement.

16. The system of claim 15, wherein the indication to move the target virtual object comprises a change in a pose of a user of the wearable device or an input received from a user input device associated with the wearable device.

17. The system of claim 15, wherein the multiplier equals to one when the distance is less than a threshold distance, wherein the threshold distance equals to a hand reach of the user.

18. The system of claim 15, wherein the multiplier increases proportionally with an increase in the distance between the target virtual object and the user.

19. The system of claim 15, wherein the multiplier is applied to one or more of:

a position change, a speed, or an acceleration, for calculating the second amount of movement.

20. The system of claim 15, wherein the multiplier is greater than one where the distance between the target virtual object and the user is beyond the user'"'"'s hand reach.