VIRTUAL REPRESENTATIONS OF REAL-WORLD OBJECTS

• US 20160189426A1
• Filed: 12/30/2014
• Published: 06/30/2016
• Est. Priority Date: 12/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kirschenbaum, Jamie Bryant, Thomas, Mike, Brown, Cameron G., Fajt, Nicholas Gervase
• Granted Patent: US 9,728,010 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume

Abstract

Methods for generating virtual proxy objects and controlling the location of the virtual proxy objects within an augmented reality environment are described. In some embodiments, a head-mounted display device (HMD) may identify a real-world object for which to generate a virtual proxy object, generate the virtual proxy object corresponding with the real-world object, and display the virtual proxy object using the HMD such that the virtual proxy object is perceived to exist within an augmented reality environment displayed to an end user of the HMD. In some cases, image processing techniques may be applied to depth images derived from a depth camera embedded within the HMD in order to identify boundary points for the real-world object and to determine the dimensions of the virtual proxy object corresponding with the real-world object.

Claims

1. An electronic device for generating and controlling a virtual proxy object within an augmented reality environment, comprising:

one or more processors, the one or more processors identify a real-world object and acquire a three-dimensional model based on the identification of the real-world object, the one or more processors determine a first maximum span of the real-world object in a first dimension and determine a second maximum span of the real-world object in a second dimension that is orthogonal to the first dimension, the one or more processors generate the virtual proxy object by scaling the three-dimensional model based on the first maximum span and the second maximum span, the one or more processors determine a location of the virtual proxy object within the augmented reality environment and render one or more images corresponding with the virtual proxy object; and

a see-through display in communication with the one or more processors, the see-through display displays the one or more images such that the virtual proxy object appears to be world-locked at the location within the augmented reality environment.

2. The electronic device of claim 1, wherein:

the one or more processors identify a first surface of the virtual proxy object and assign a snapping property to the first surface based on an orientation of the real-world object, the snapping property causes the first surface of the virtual proxy object to snap to a horizontal surface within the augmented reality environment if the first surface is within a threshold distance of the horizontal surface.

3. The electronic device of claim 2, wherein:

the first surface comprises a bottom surface of the virtual proxy object and the horizontal surface comprises a floor.

4. The electronic device of claim 1, wherein:

the one or more processors identify a first surface of the virtual proxy object and assign a snapping property to the first surface based on an orientation of the real-world object, the snapping property causes the first surface of the virtual proxy object to snap to a vertical surface within the augmented reality environment if the first surface is within a threshold distance of the vertical surface.

5. The electronic device of claim 4, wherein:

the first surface comprises a side surface of the virtual proxy model and the vertical surface comprises a wall.

6. The electronic device of claim 1, wherein:

the one or more processors acquire one or more depth images of the real-world object and determine a height, a length, and a width of the real-world object using the one or more depth images; and

the one or more processors generate the virtual proxy object using the three-dimensional model and the height, the length, and the width of the real-world object.

7. The electronic device of claim 1, wherein:

the one or more processors determine whether the virtual proxy object should be made partially transparent based on the location of the virtual proxy object within the augmented reality environment; and

the see-through display displays the one or more images such that the virtual proxy object appears to be partially transparent.

8. The electronic device of claim 1, wherein:

the three-dimensional model comprises a rectangular prism; and

the electronic device comprises a head-mounted display device.

9. A method for generating and controlling a virtual proxy object within an augmented reality environment using a mobile device, comprising:

identifying a real-world object within a field of view of the mobile device;

acquiring a three-dimensional model based on the identification of the real-world object;

determining one or more dimensions of the real-world object, the determining one or more dimensions includes determining a first maximum span of the real-world object in a first dimension and determining a second maximum span of the real-world object in a second dimension that is orthogonal to the first dimension;

generating the virtual proxy object using the three-dimensional model and the one or more dimensions;

determining a location of the virtual proxy object within the augmented reality environment; and

displaying one or more images corresponding with the virtual proxy object using the mobile device, the one or more images are displayed such that the virtual proxy object appears to be world-locked at the location within the augmented reality environment.

10. The method of claim 9, further comprising:

identifying a first surface of the virtual proxy object; and

assigning a snapping property to the first surface based on an orientation of the real-world object, the snapping property causes the first surface of the virtual proxy object to snap to a horizontal surface within the augmented reality environment if the first surface is within a threshold distance of the horizontal surface.

11. The method of claim 10, wherein:

the first surface comprises a bottom surface of the virtual proxy object and the horizontal surface comprises a floor.

12. The method of claim 9, further comprising:

identifying a first surface of the virtual proxy object; and

assigning a snapping property to the first surface based on an orientation of the real-world object, the snapping property causes the first surface of the virtual proxy object to snap to a vertical surface within the augmented reality environment if the first surface is within a threshold distance of the vertical surface.

13. The method of claim 12, wherein:

the first surface comprises a side surface of the virtual proxy model and the vertical surface comprises a wall.

14. The method of claim 9, wherein:

the determining one or more dimensions includes determining a height, a length, and a width of the real-world object; and

the generating the virtual proxy object includes generating the virtual proxy object using the three-dimensional model and the height, the length, and the width of the real-world object.

15. The method of claim 9, further comprising:

determining whether the virtual proxy object should be made partially transparent based on the location of the virtual proxy object within the augmented reality environment; and

the displaying one or more images includes displaying the one or more images such that the virtual proxy object appears to be partially transparent.

16. The method of claim 9, wherein:

the determining one or more dimensions includes determining a minimum size of the three-dimensional model that completely covers the real-world object.

17. The method of claim 9, wherein:

the three-dimensional model comprises at most six surfaces.

18. The method of claim 9, wherein:

the three-dimensional model comprises a rectangular prism.

19. The method of claim 9, wherein:

the mobile device comprises a head-mounted display device.

20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and controlling a virtual proxy object within an augmented reality environment using a head-mounted display device comprising the steps of:

acquiring a three-dimensional model associated with a real-world object;

determining one or more dimensions of the real-world object, the determining one or more dimensions includes determining a first maximum span of the real-world object in a first dimension and determining a second maximum span of the real-world object in a second dimension that is orthogonal to the first dimension;

generating the virtual proxy object using the three-dimensional model and the one or more dimensions;

determining a location of the virtual proxy object within the augmented reality environment;

detecting that the virtual proxy object occludes an object within a field of view of the head-mounted display device based on the location of the virtual proxy object within the augmented reality environment;

determining that the virtual proxy object should be made partially transparent in response to the detecting that the virtual proxy object occludes the object within the field of view of the head-mounted display device; and

displaying one or more images corresponding with the virtual proxy object using the head-mounted display device, the one or more images are displayed such that the virtual proxy object appears to be world-locked at the location within the augmented reality environment and such that the virtual proxy object appears to be partially transparent.