Automatic placement of a virtual object in a three-dimensional space

• US 10,627,625 B2
• Filed: 08/09/2017
• Issued: 04/21/2020
• Est. Priority Date: 08/11/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Hoover, Paul Armistead, Mann, Jonathan Lawrence
• Primary Examiner(s): Beutel, William A
• Application Number:
  US15/673,135
• Publication Number:
  US 20180045963A1
• Time in Patent Office: 986 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/395 with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
• CPC Class Codes: H04N 2213/001 Constructional or mechanical details

Abstract

Augmented reality systems and methods for automatically repositioning a virtual object with respect to a destination object in a three-dimensional (3D) environment of a user are disclosed. The systems and methods can automatically attach the target virtual object to the destination object and re-orient the target virtual object based on the affordances of the virtual object or the destination object. The systems and methods can also track the movement of a user and detach the virtual object from the destination object when the user'"'"'s movement passes a threshold condition.

Claims

1. An augmented reality (AR) system for automatically repositioning a virtual object in a three-dimensional (3D) environment, the AR system comprising:

an AR display configured to present virtual content;

a hardware processor in communication with the AR display, the hardware processor programmed to;

determine a representation of a focus indicator comprising a graphical representation of a cursor, virtual cone, or ray;

display the focus indicator at a location in the 3D environment of a user associated with a current pose of the user;

identify a target virtual object in the 3D environment of the user, wherein the target virtual object is associated with at least one vector representing a first location and a first orientation;

receive an indication to replicate the target virtual object;

assign the focus indicator to the target virtual object in response to the indication to replicate the target virtual object;

display the focus indicator at the location of the target virtual object in response to assigning the focus indicator to the target virtual object;

receive an indication to attach a replica of the target virtual object to a destination object, wherein the destination object is associated with at least one vector representing a second location and a second orientation;

calculate a trajectory between the target virtual object and the destination object based at least partly on the first location, the second location, and an affordance associated with at least one of the 3D environment, the target virtual object, or the destination object, wherein the affordance comprises an attribute used to simulate interactions of the target virtual object, the destination object, or the 3D environment using a law of physics;

move the replica of the target virtual object and the assigned focus indicator along the trajectory towards the destination object;

track a current location of the replica of the target virtual object;

calculate a distance between the replica of the target virtual object and the destination object based at least partly on the current location of the target virtual object and the second location;

determine whether the distance of the replica of the target virtual object and the destination virtual object is less than a threshold distance;

automatically attach the replica of the target virtual object to the destination object and orient the replica of the target virtual object to the second orientation in response to a determination that the distance is less than or equal to the threshold distance; and

render, by the AR display, the replica of the target virtual object at the second location with the second orientation where the replica of the target virtual object is overlaid on the destination object.

2. The AR system of claim 1, wherein the hardware processor is further programmed to:

analyze affordances of at least one of the target virtual object, the destination object, or the environment; and

to automatically orient the replica of the target virtual object, the hardware processor is programmed to rotate the replica of the target virtual object to align a first normal of the target virtual object with a second normal of the destination object.

3. The AR system of claim 2, wherein the affordances comprise at least one of:

a function, an orientation, a type, a location, a shape, or a size.

4. The AR system of claim 1, wherein to automatically attach the replica of the target virtual object, the hardware processor is programmed to:

simulate an attractive force between the replica of the target virtual object and the destination object, wherein the attractive force comprises at least one of gravity, an elastic force, an adhesive force, or a magnetic attraction.

5. The AR system of claim 1, wherein to calculate the distance, the hardware processor is programmed to calculate a displacement between the current location of the target virtual object and the second location associated with the destination object.

6. The AR system of claim 5, wherein the threshold distance is zero.

7. The AR system of claim 1, wherein the indication to attach the replica of the target virtual object is determined from at least one of:

an actuation of a user input device or a pose of a user.

8. The AR system of claim 1, wherein the hardware processor is further programmed to:

receive an indication to detach the replica of the target virtual object from the destination object, wherein the indication is associated with a change in the user'"'"'s current position;

determine whether a threshold condition for detaching the replica of the target virtual object is met based at least partly on the received indication;

in response to a determination that the threshold condition is met;

detach the replica of the target virtual object from the destination object;

move the replica of the target virtual object from the second location associated with the destination object to a third location; and

render the replica of the target virtual object at the third location.

9. The AR system of claim 8, wherein in response to the determination that the threshold condition is met, the hardware processor is further programmed to:

retain the second orientation for the replica of the target virtual object while moving the replica of the target virtual object to the third location.

10. The AR system of claim 8, wherein the third location corresponds to a position of the focus indicator which corresponds to the current position of the user.

11. The AR system of claim 10, wherein the threshold condition for detaching the replica of the target virtual object from the other object comprises at least one of:

a second distance between the second location where the replica of the target virtual object is attached to the destination object and the position of the focus indicator is greater than or equal to a second threshold distance;

a speed for moving from the second location to the position of the focus indicator is greater than or equal to a threshold speed;

an acceleration for moving away from the second location is greater than or equal to a threshold acceleration;

or

a jerk for moving away from the second location is greater than or equal to a threshold jerk.

12. The AR system of claim 8, wherein the hardware processor is programmed to simulate a physical force when detaching the replica of the target virtual object from the destination object, wherein the physical force comprises at least one of a friction or an elasticity.

13. A method for automatically repositioning a virtual object in a three-dimensional (3D) environment, the method comprising:

under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to permit user interactions with objects in a 3D environment;

identifying a target virtual object in the user'"'"'s 3D environment, the target virtual object having a first position and a first orientation;

receiving an indication to replicate the target virtual object;

assigning the focus indicator to the target virtual object in response to the indication to replicate the target virtual object;

displaying the focus indicator at the location of the target virtual object in response to assigning the focus indicator to the target virtual object;

identifying parameters for positioning the replica of the target virtual object, wherein the parameters comprise at least one of a direction of movement, a speed of movement, or an acceleration of movement;

analyzing affordances associated with at least one of the 3D environment, the target virtual object, and the destination object, wherein the affordances comprise attributes used to simulate interactions of the target virtual object, the destination object, or the 3D environment using a law of physics;

calculating values of the parameters for positioning the replica of the target virtual object based on the affordances;

determining a second position and a second orientation for the target virtual object and a movement of the replica of the target virtual object based on the values of the parameters for positioning the replica of the target virtual object; and

rendering the replica of the target virtual object at the second position and the second orientation and the movement of the replica of the target virtual object for reaching the second position and the second orientation from the first position and the first orientation.

14. The method of claim 13, wherein positioning the replica of the target object comprises at least one of attaching the target object to the destination object, reorienting the target object, or detaching the target object from the destination object.

15. The method of claim 13, wherein the destination object is a physical object.

16. The method of claim 13, further comprising determining whether the indication to replicate the target virtual object meets a threshold condition;

and performing said calculating, determining, and rendering in response to a determination that the indication meets the threshold condition.

17. The method of claim 16, wherein the threshold condition comprises a distance between the target virtual object and the destination object.

18. The method of claim 13, wherein one or more physical attributes is assigned to the replica of the target virtual object, and the movement of the replica of the target virtual object is determined by simulating interactions of the target virtual object, the destination object, and the environment based on the physical attributes of the target virtual object.

19. The method of claim 18, wherein the one or more physical attributes assigned to the replica of the target virtual object comprises at least one of a mass, a size, a density, a phase, a hardness, an elasticity, or an electromagnetic attribute.