Virtual object manipulation within physical environment

• US 10,373,381 B2
• Filed: 03/30/2016
• Issued: 08/06/2019
• Est. Priority Date: 03/30/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nuernberger, Benjamin, Benko, Hrvoje, Wilson, Andrew, Ofek, Eyal
• Primary Examiner(s): Ge, Jin
• Application Number:
  US15/085,905
• Publication Number:
  US 20170287218A1
• Time in Patent Office: 1,224 Days
• Field of Search: 345633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

A method, computing device and head-mounted display device for manipulating a virtual object displayed via a display device are disclosed. In one example, image data of a physical environment comprising physical features is received. A three dimensional model of at least a portion of the environment is generated. Candidate anchor features that each correspond to one of the physical features are extracted from the image data. User input is received that manipulates the virtual object as displayed within the environment. Based on the manipulation, a correspondence between a virtual anchor feature of the virtual object and a corresponding candidate anchor feature is identified. An indication of the corresponding candidate anchor feature at its corresponding physical feature within the environment is displayed to the user.

Claims

1. A method for manipulating a virtual object displayed via a display device, the method comprising:

receiving image data of a real world physical environment comprising a plurality of physical features;

using the image data, generating a three dimensional model of at least a portion of the real world physical environment;

extracting from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment;

receiving user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;

based on the manipulation of the virtual object, identifying at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features, wherein the corresponding candidate anchor feature is located at an initial three dimensional location in the three dimensional model, and is extracted from a frame of the image data received at an initial time;

based on identifying the at least one correspondence, displaying via the display device an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment;

determining that the corresponding candidate anchor feature is not present in subsequent frames of the image data received after the initial time; and

maintaining the corresponding candidate anchor feature at the initial three dimensional location in the three dimensional model until expiration of a predetermined period of time after the initial time.

2. The method of claim 1, further comprising:

based on receiving a user selection input, aligning the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature; and

displaying the virtual object with its virtual anchor feature aligned with the corresponding candidate anchor feature.

3. The method of claim 2, wherein the corresponding physical feature is found on a physical object in the real world physical environment, the method further comprising, when the physical object moves within the real world physical environment, displaying a corresponding movement of the virtual object such that its virtual anchor feature remains aligned with the corresponding candidate anchor feature of the corresponding physical feature on the physical object.

4. The method of claim 2, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the virtual object is displayed located at a virtual object location in the real world physical environment, the method further comprising, when the physical object moves within the real world physical environment, maintaining the display of the virtual object at the virtual object location.

5. The method of claim 1, wherein the plurality of physical features comprises a three dimensional linear edge and a three dimensional planar surface.

6. The method of claim 1, wherein the corresponding physical feature is located at a non-visible location in the real world physical environment that is outside a field of view of the display device, the method further comprising displaying the indication extending into the field of view of the display device from the non-visible location.

7. The method of claim 6, further comprising, when the virtual anchor feature of the virtual object aligns with the corresponding candidate anchor feature of the corresponding physical feature located at the non-visible location, outputting a directional notification that indicates the non-visible location of the corresponding physical feature.

8. The method of claim 1, wherein the indication of the corresponding candidate anchor feature is displayed to extend beyond the corresponding physical feature.

9. The method of claim 1, further comprising generating a filtered plurality of candidate anchor features by filtering the plurality of candidate anchor features based on one or more of (1) a distance between the display device and each of the candidate anchor features, (2) a distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within a field of view of the display device, (4) a frequency of prior use of each of the candidate anchor features, and (5) user filtering input.

10. The method of claim 9, wherein identifying the at least one correspondence further comprises identifying the at least one correspondence from the filtered plurality of candidate anchor features.

11. The method of claim 1, further comprising receiving a user preselection input of a selected physical feature from the plurality of physical features, wherein the selected physical feature comprises the corresponding physical feature of the corresponding candidate anchor feature, and wherein identifying the at least one correspondence is also based on the user preselection input of the selected physical feature.

12. A computing device for manipulating a virtual object displayed via a display device, the computing device comprising:

a processor; and

a memory holding instructions executable by the processor to;

receive image data of a real world physical environment comprising a plurality of physical features;

using the image data, generate a three dimensional model of at least a portion of the real world physical environment;

extract from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment, wherein at least one of the corresponding physical feature is located at a non-visible location in the real world physical environment that is outside a field of view of the display device;

receive user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;

based on the manipulation of the virtual object, identify at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features, wherein the corresponding candidate anchor feature corresponds to the at least one corresponding physical feature that is located at the non-visible location;

based on identifying the at least one correspondence, display via the display device an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment, wherein the indication extends into the field of view of the display device from the non-visible location; and

when the virtual anchor feature of the virtual object aligns with the corresponding candidate anchor feature of the corresponding physical feature located at the non-visible location, output a directional notification that indicates the non-visible location of the corresponding physical feature.

13. The computing device of claim 12, wherein the instructions are executable by the processor to:

based on receiving a user selection input, align the virtual anchor feature of the virtual object with the corresponding candidate anchor feature of the corresponding physical feature; and

display via the display device the virtual object with its virtual anchor feature aligned with the corresponding candidate anchor feature.

14. The computing device of claim 13, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the instructions are executable by the processor to, when the physical object moves within the real world physical environment, display via the display device a corresponding movement of the virtual object such that its virtual anchor feature remains aligned with the corresponding candidate anchor feature of the corresponding physical feature on the physical object.

15. The computing device of claim 13, wherein the corresponding physical feature is found on a physical object in the real world physical environment, and the virtual object is displayed via the display device located at a virtual object location in the real world physical environment, and the instructions are executable by the processor to, when the physical object moves within the real world physical environment, maintain the display of the virtual object at the virtual object location.

16. The computing device of claim 12, wherein to identify the at least one correspondence further comprises to identify the at least one correspondence from a filtered plurality of candidate anchor features that are generated based on one or more of (1) a distance between the display device and each of the candidate anchor features, (2) a distance between the virtual object and each of the candidate anchor features, (3) whether each of the candidate anchor features is within a field of view of the display device, (4) a frequency of prior use of each of the candidate anchor features, and (5) user filtering input.

17. A head-mounted display device, comprising:

an at least partially see-through display;

a processor; and

a memory holding instructions executable by the processor to;

display a virtual object via the at least partially see-through display;

receive image data of a real world physical environment comprising a plurality of physical features;

using the image data, generate a three dimensional model of at least a portion of the real world physical environment;

extract from the image data a plurality of candidate anchor features that each correspond to one of the physical features in the real world physical environment;

receive user manipulation input that manipulates the virtual object as displayed within the real world physical environment, wherein the virtual object comprises a virtual anchor feature;

based on the manipulation of the virtual object, identify at least one correspondence between the virtual anchor feature of the virtual object and a corresponding candidate anchor feature from the plurality of candidate anchor features, wherein the corresponding candidate anchor feature is located at an initial three dimensional location in the three dimensional model, and is extracted from a frame of the image data received at an initial time;

based on identifying the at least one correspondence between the virtual anchor feature and the corresponding candidate anchor feature, display via the at least partially see-through display an indication of the corresponding candidate anchor feature at its corresponding physical feature within the real world physical environment;

determine that the corresponding candidate anchor feature is not present in subsequent frames of the image data received after the initial time; and

maintain the corresponding candidate anchor feature at the initial three dimensional location in the three dimensional model until expiration of a predetermined period of time after the initial time.