Realistic occlusion for a head mounted augmented reality display

• US 9,122,053 B2
• Filed: 04/10/2012
• Issued: 09/01/2015
• Est. Priority Date: 10/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Mount, Brian J., Latta, Stephen G., McCulloch, Daniel J., Lee, Kyungsuk David, Sugden, Ben J., Margolis, Jeffrey N., Perez, Kathryn Stone, Small, Sheridan Martin, Finocchio, Mark J., Crocco, Robert L. Jr.
• Primary Examiner(s): Hoang, Peter
• Application Number:
  US13/443,368
• Publication Number:
  US 20120206452A1
• Time in Patent Office: 1,239 Days
• Field of Search: 345/428
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/03 Application of parametric coding in stereophonic audio systems
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

Technology is described for providing realistic occlusion between a virtual object displayed by a head mounted, augmented reality display system and a real object visible to the user'"'"'s eyes through the display. A spatial occlusion in a user field of view of the display is typically a three dimensional occlusion determined based on a three dimensional space mapping of real and virtual objects. An occlusion interface between a real object and a virtual object can be modeled at a level of detail determined based on criteria such as distance within the field of view, display size or position with respect to a point of gaze. Technology is also described for providing three dimensional audio occlusion based on an occlusion between a real object and a virtual object in the user environment.

Claims

1. A see-through, augmented reality display device system for providing realistic occlusion comprising:

a see-through, augmented reality display having a user field of view and being supported by a support structure of the see-through, augmented reality display device;

at least one camera supported by the support structure for capturing image data and depth data for real objects in the user field of view of the see-through, augmented reality display;

one or more processors communicatively coupled to the at least one camera for receiving image and depth data including the user field of view and to track depth positional data from the user field of view;

the one or more processors for determining a spatial occlusion relationship between an occluding object and an occluded object, and determining a first level of detail criteria based on the positional data used in processing a level of occlusion for the occluding object, the occluding object and the occluded object including a real object and a virtual object, respectively, based on the image and depth data; and

the one or more processors being communicatively coupled to the see-through, augmented reality display and the one or more processors causing the see-through display to represent the spatial occlusion relationship in the display by modifying display of the virtual object and causing the see-through display to provide varied levels of occlusion as determined by the first level of detail criteria when rendering the occluding object.

2. The system of claim 1 wherein the one or more processors causing the see-through display to represent the spatial occlusion relationship in the display by modifying display of the virtual object further comprises:

determining a level of detail for generating a model of an occlusion interface between the real object and the virtual object based on a second level of detail criteria and generating a modified version of object boundary data for the virtual object based on the generated model; and

the see-through, augmented reality display displaying the virtual object based on the modified version of its object boundary data.

3. The system of claim 1 further comprising:

the one or more processors being communicatively coupled to one or more computer systems and the one or more processors and the one or more computer systems collaboratively determining a three-dimensional map of an environment of a user wearing the see-through, augmented reality display device system in real time in a common coordinate system based on image and depth data captured of the environment; and

the one or more processors and the one or more computer systems sharing at least one occlusion data set including a model for an occlusion interface in real time.

4. The system of claim 3 further comprising wherein the one or more computer systems include at least one other see-through, augmented reality display device in the environment of the user which shares one or more occlusion data sets.

5. The system of claim 1 further comprising:

earphones attached to the support structure;

the one or more processors being communicatively coupled to the earphones for controlling output of audio data; and

the one or more processors executing a three dimensional audio engine which modifies audio data for the occluded object in the spatial occlusion relationship based on one or more physical properties associated with the occluding object.

6. The system of claim 5 further comprising:

the one or more processors having access to memory for storing one or more sound occlusion models which represent at least one effect on sound directed at an object based on one or more physical properties of the object; and

the one or more processors executing a three dimensional audio engine which modifies audio data for the occluded object in the partial occlusion based on one or more physical properties associated with the occluding object further comprises;

the one or more processors executing a three dimensional audio engine modifying the audio data in accordance with the at least one effect on sound represented by the identified sound occlusion model.

7. The system of claim 6 further comprising:

a microphone attached to the support structure;

the one or more processors being communicatively coupled to the microphone and receiving audio data from the microphone; and

the one or more processors identifying the audio data as being from the real object which is the occluded object, wherein;

the identified sound occlusion model is the sound occlusion model of the virtual object which is the occluding object.

8. The system of claim 1, wherein a tolerance level of the first level of detail is based on the positional data, the tolerance level indicating upper and lower boundaries for varying the level of detail.

9. A method of providing realistic occlusion in a see-through, augmented reality display, comprising:

capturing image data and depth data for real objects using at least one camera in a user field of view of the see-through, augmented reality display;

receiving image and depth data, at one or more processors communicatively coupled to the at least one camera, including the user field of view and tracking depth positional data from the user field of view;

determining, by the one or more processors, a spatial occlusion relationship between an occluding object and an occluded object, and determining a level of detail criteria based on the positional data used in processing a level of occlusion for the occluding object, the occluding object and the occluded object including a real object and a virtual object, respectively, based on the image and depth data; and

rendering the spatial occlusion relationship in the display, by the one or more processors communicatively coupled to the see-through augmented reality display, by modifying display of the virtual object and causing the see-through display to provide varied levels of occlusion as determined by the level of detail criteria when displaying the occluding object.

10. The method of claim 9 the rendering further comprises:

determining a level of detail for generating a model of an occlusion interface between the real object and the virtual object based on a second level of detail criteria and generating a modified version of object boundary data for the virtual object based on the generated model; and

displaying, by the see-through, augmented reality display, the virtual object based on the modified version of its object boundary data.

11. The method of claim 9 further comprising:

collaboratively determining, by the one or more processors and one or more computer systems, the one or more processors being communicatively coupled to the one or more computer systems, a three-dimensional map of an environment of a user wearing the see-through, augmented reality display device system in real time in a common coordinate system based on image and depth data captured of the environment; and

sharing at least one occlusion data set including a model for an occlusion interface in real time by the one or more processors and the one or more computer systems.

12. The method of claim 11 wherein the one or more computer systems include at least one other see-through, augmented reality display device in the environment of the user which shares one or more occlusion data sets.

13. The method of claim 9 further comprising:

controlling output of audio data by the one or more processors, the one or more processors being communicatively coupled to earphones; and

executing a three dimensional audio engine, by the one or more processors, which modifies audio data for the occluded object in the spatial occlusion relationship based on one or more physical properties associated with the occluding object.

14. The method of claim 13 further comprising:

storing one or more sound occlusion models, by in a memory accessible by the one or more processors, which represent at least one effect on sound directed at an object based on one or more physical properties of the object; and

executing by the one or more processors a three dimensional audio engine which modifies audio data for the occluded object in the partial occlusion based on one or more physical properties associated with the occluding object further comprises;

executing by the one or more processors a three dimensional audio engine modifying the audio data in accordance with the at least one effect on sound represented by the identified sound occlusion model.

15. The method of claim 14 further comprising:

receiving audio data from a microphone, the one or more processors being communicatively coupled to the microphone; and

identifying by the one or more processors the audio data as being from the real object which is the occluded object, wherein

the identified sound occlusion model is the sound occlusion model of the virtual object which is the occluding object.

16. The method of claim 9, wherein a tolerance level of the first level of detail is based on the positional data, the tolerance level indicating upper and lower boundaries for varying the level of detail.

17. A see-through, augmented reality display device system for providing realistic occlusion comprising:

a see-through, augmented reality display having a user field of view and being supported by a support structure of the see-through, augmented reality display device;

at least one camera supported by the support structure for capturing image data and depth data for real objects in the user field of view of the see-through, augmented reality display;

one or more processors communicatively coupled to the at least one camera for receiving image and depth data including the user field of view;

the one or more processors for determining a spatial occlusion relationship between an occluding object and an occluded object, the occluding object and the occluded object including a real object and a virtual object, based on the image and depth data;

the one or more processors being communicatively coupled to the see-through, augmented reality display and the one or more processors causing the see-through display to represent the spatial occlusion relationship in the display by modifying display of the virtual object;

earphones attached to the support structure;

the one or more processors being communicatively coupled to the earphones for controlling output of audio data;

the one or more processors executing a three dimensional audio engine which modifies audio data for the occluded object in the spatial occlusion relationship based on one or more physical properties associated with the occluding object;

the one or more processors having access to memory for storing one or more sound occlusion models which represent at least one effect on sound directed at an object based on one or more physical properties of the object; and

the one or more processors executing a three dimensional audio engine which modifies audio data for the occluded object in the partial occlusion based on one or more physical properties associated with the occluding object further comprises;

the one or more processors executing a three dimensional audio engine modifying the audio data in accordance with the at least one effect on sound represented by the identified sound occlusion model.

18. The system of claim 17 further comprising:

a microphone attached to the support structure;

the one or more processors being communicatively coupled to the microphone and receiving audio data from the microphone; and

the one or more processors identifying the audio data as being from the real object which is the occluded object, wherein

the identified sound occlusion model is the sound occlusion model of the virtual object which is the occluding object.

19. The system of claim 17, wherein a tolerance level of the first level of detail is based on the positional data, the tolerance level indicating upper and lower boundaries for varying the level of detail.

20. The system of claim 17 wherein the one or more processors causing the see-through display to represent the spatial occlusion relationship in the display by modifying display of the virtual object further comprises:

determining a level of detail for generating a model of an occlusion interface between the real object and the virtual object based on a level of detail criteria and generating a modified version of object boundary data for the virtual object based on the generated model; and

the see-through, augmented reality display displaying the virtual object based on the modified version of its object boundary data.