User controlled real object disappearance in a mixed reality display

• US 9,255,813 B2
• Filed: 10/14/2011
• Issued: 02/09/2016
• Est. Priority Date: 10/14/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James C., Latta, Stephen G., Vaught, Benjamin I., Novak, Christopher M., Bennett, Darren
• Primary Examiner(s): Liu, Zhengxi
• Application Number:
  US13/274,136
• Publication Number:
  US 20130093788A1
• Time in Patent Office: 1,579 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

The technology causes disappearance of a real object in a field of view of a see-through, mixed reality display device system based on user disappearance criteria. Image data is tracked to the real object in the field of view of the see-through display for implementing an alteration technique on the real object causing its disappearance from the display. A real object may satisfy user disappearance criteria by being associated with subject matter that the user does not wish to see or by not satisfying relevance criteria for a current subject matter of interest to the user. In some embodiments, based on a 3D model of a location of the display device system, an alteration technique may be selected for a real object based on a visibility level associated with the position within the location. Image data for alteration may be prefetched based on a location of the display device system.

Claims

1. One or more processor-readable storage devices having instructions encoded thereon for causing one or more processors to execute a method for causing a real object to be obscured in a see-through display of a see-through, mixed reality display device system, the method comprising:

receiving metadata identifying one or more real objects in a field of view of the see-through display;

determining whether any of the one or more real objects satisfies user disappearance criteria, the user disappearance criteria determinative of whether the one or more objects should be obscured and based on a current subject matter of interest, wherein the determining comprises;

identifying any real object type associated with the current subject matter of interest based on notification of the current subject matter of interest to a user, while the user is viewing the one or more real objects through the see-through display; and

responsive to determining a first real object of the one or more real objects satisfies the user disappearance criteria, tracking image data to the first real object in the see-through display for causing the first real object to be obscured in the field of view of the see-through display by selectively controlling natural light transmissivity of each pixel of a dense grid of pixels of an opacity filter of the see-through display.

2. The one or more processor-readable storage devices of claim 1 further comprising:

detecting another see-through display device system within a predetermined distance of the see-through, mixed reality display device system;

based on being within the predetermined distance of the see-through, mixed reality display device system, receiving an identifier of a second real object which satisfies user disappearance criteria of the user wearing the other see-through display device system;

receiving image data for tracking to the second real object for an alteration technique from the other see-through display device system; and

displaying the image data tracking the second real object from a perspective of the field of view of the see-through display.

3. The one or more processor-readable storage devices of claim 1 wherein determining whether any of the one or more real objects satisfies user disappearance criteria further comprises:

identifying any real object in the field of view of the see-through display matching any identified real object type; and

determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest.

4. The one or more processor-readable storage devices of claim 3 wherein determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest further comprises:

applying a keyword search technique to the metadata of any real object identified as matching any identified real object type for determining a respective keyword relevancy score; and

identifying any real object not satisfying relevance criteria for the current subject matter of interest based on the respective keyword relevancy score.

5. The one or more processor-readable storage devices of claim 1 wherein the method further comprises:

determining a position of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display;

identifying a visibility level for each identified real object based on its respective determined position, and a predetermined visibility distance for a location of the display device; and

prioritizing each real object for disappearance based on its identified visibility level in the field of view.

6. The one or more processor-readable storage devices of claim 5 wherein the method further comprises:

determining a trajectory of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display; and

wherein identifying the visibility level for each identified real object is also based on its respective determined trajectory.

7. The one or more processor-readable storage devices of claim 5 wherein identifying the visibility level for each identified real object is also based on at least one appearance characteristic of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display.

8. The one or more processor-readable storage devices of claim 5 wherein tracking image data to the first real object in the see-through display for causing disappearance of the first real object in the field of view of the see-through display further comprises:

selecting an alteration technique for each real object to be obscured by alteration techniques indicated for the respective real object based on a priority of the visibility level identified for each of the real objects; and

applying the selected alteration technique for each real object for disappearance.

9. The one or more processor-readable storage devices of claim 1 further comprising:

checking for identification of any real object satisfying user disappearance criteria which is outside the field of view but within a predetermined visibility distance for a location of the display device system; and

prefetching any applicable disappearance image data for any identified real object satisfying user disappearance criteria which is outside the field of view but within the predetermined visibility distance for the location of the display device system.

10. The one or more processor-readable storage devices of claim 9 further comprising:

applying a location prediction method for identifying one or more subsequent locations for entry by the display device system;

responsive to identifying a subsequent location, checking for identification of any real object satisfying user disappearance criteria in the subsequent location based on stored real object metadata associated with the subsequent location; and

prefetching any applicable disappearance image data for any identified real object satisfying user disappearance criteria in the subsequent location.

11. A see-through, head mounted, mixed reality display device system for causing a real object to be obscured in a field of view of a see-through display of the display device system comprising:

an opacity filter selectively controlling natural light transmissivity of each pixel of a dense grid of pixels and blocking natural light from passing through a light guide optical element to obscure elements in the field of view;

one or more location detection sensors;

a memory for storing user disappearance criteria including at least one subject matter item to be selectively obscured from view through the see-through display;

one or more processors having access to the memory and configured to receive a notification of the at least one subject matter item while a user is viewing the real object through the see-through display, the one or more processors communicatively coupled to the one or more location detection sensors for receiving location identifier data for the display device system and for identifying one or more real objects in the field of view of the see-through display which are related to the at least one subject matter item and within a predetermined visibility distance for a location determined from the location identifier data; and

at least one image generation unit communicatively coupled to the one or more processors and optically coupled to the see-through display for tracking image data to the identified one or more real objects in the field of view of the see-through display for causing the one or more real objects to be selectively obscured from view through the see-through display by utilizing the opacity filter.

12. The system of claim 11 further comprising:

the user disappearance criteria being stored in the memory further comprising a replacement object for a real world object type associated with the at least one subject matter item.

13. The system of claim 11 wherein the user disappearance criteria includes an alteration indicator for a real world object type associated with the at least one subject matter item, the alteration indicator indicating a type of alteration to be applied to the respective real world object.

14. The system of claim 11 wherein the one or more processors are communicatively coupled to a remote computing system for accessing a three-dimensional model including metadata identifying one or more real objects in the location within a predetermined visibility distance of the see-through, mixed reality display device.

15. The system of claim 11 wherein the one or more location detection sensors comprise at least one of the following:

a WiFi device;

an infrared device;

an image capture device;

a Global Positioning System (GPS) device;

a cellular telecommunication based device; and

a RFID device.

16. The system of claim 11 wherein

the one or more processors having access to user profile data for predicting a subsequent location to a current location of the display device;

the one or more processors checking for identification of any real object satisfying user disappearance criteria in the subsequent location;

the one or more processors being communicatively coupled to a computer system in the subsequent location; and

the one or more processors prefetching image data for any identified real object satisfying user disappearance criteria in the subsequent location by scheduling downloading to the computer system in the subsequent location in time for arrival of the user.

17. A method for causing a real object to be obscured in a field of view of a see-through display of a see-through mixed reality display device system comprising:

receiving user input of a physical action identifying subject matter to be selectively obscured, while a user is viewing a real object through the see-through display, the subject matter including the real object currently in the field of view of the see-through display;

storing the subject matter to be selectively obscured as user disappearance criteria;

tracking image data to the real object to be selectively obscured in accordance with an alteration technique; and

responsive to identifying a user designated alteration technique to be applied to the real object, selecting the user designated alteration technique as the alteration technique and causing the real object to be selectively obscured from the current field of view through the see-through display by selectively controlling natural light transmissivity of each pixel of a dense grid of pixels of an opacity filter of the see-through display.

18. The method of claim 17 wherein the alteration technique includes at least one of the following:

replacement of the real object with a virtual object;

erasure of the real object by overlaying the real object with image data generated by blending image data of objects surrounding the real object;

redaction of the real object with black image data; and

obscuring of the real object by tracking blur image data to the first real object.

19. The method of claim 17 wherein the alteration technique includes erasure of the real object and the method further comprises:

tracking a position of the see-through, mixed reality display device to the real object by monitoring image data from physical environment facing cameras of the see-through, mixed reality display device system; and

responsive to detecting the see-through, mixed reality display device and the real object are within a collision distance, outputting a safety warning.

20. The method of claim 17 wherein the user input of a physical action identifying subject matter to be obscured is a gesture identified based on image data from at least one physical environment facing camera of the see-through, mixed reality display device system.