USER CONTROLLED REAL OBJECT DISAPPEARANCE IN A MIXED REALITY DISPLAY

• US 20130093788A1
• Filed: 10/14/2011
• Published: 04/18/2013
• Est. Priority Date: 10/14/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James C., Latta, Stephen G., Vaught, Benjamin I., Novak, Christopher M., Bennett, Darren
• Granted Patent: US 9,255,813 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

The technology causes disappearance of a real object in a field of view of a see-through, mixed reality display device system based on user disappearance criteria. Image data is tracked to the real object in the field of view of the see-through display for implementing an alteration technique on the real object causing its disappearance from the display. A real object may satisfy user disappearance criteria by being associated with subject matter that the user does not wish to see or by not satisfying relevance criteria for a current subject matter of interest to the user. In some embodiments, based on a 3D model of a location of the display device system, an alteration technique may be selected for a real object based on a visibility level associated with the position within the location. Image data for alteration may be prefetched based on a location of the display device system.

Claims

1. One or more processor-readable storage devices having instructions

encoded thereon for causing one or more processors to execute a method for causing disappearance of a real object in a see-through display of a see-through, mixed reality display device system, the method comprising:

receiving metadata identifying one or more real objects in a field of view of the see-through display;

determining whether any of the one or more real objects satisfies user disappearance criteria; and

responsive to determining a first real object satisfies the user disappearance criteria, tracking image data to the first real object in the see-through display for causing disappearance of the first real object in the field of view of the see-through display.

2. The one or more processor-readable storage devices of claim 1 further comprising

detecting another see-through display device system within a predetermined distance of the see-through, mixed reality display device system;

based on being within the predetermined distance of the see-through, mixed reality display device system, receiving an identifier of a second real object which satisfies user disappearance criteria of the user wearing the other see-through display device system;

receiving image data for tracking to the second real object for an alteration technique from the other see-through display device system; and

displaying the image data tracking the second real object from a perspective of the field of view of the see-through display.

3. The one or more processor-readable storage devices of claim 1 wherein determining whether any of the one or more real objects satisfies user disappearance criteria further comprises:

based on notification of a current subject matter of interest to a user, identifying any real object type associated with the current subject matter of interest;

identifying any real object in the field of view of the see-through display matching any identified real object type; and

determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest.

4. The one or more processor-readable storage devices of claim 3 wherein determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest further comprises

applying a keyword search technique to the metadata of any real object identified as matching any identified real object type for determining a respective keyword relevancy score;

and

identifying any real object not satisfying relevance criteria for the current subject matter of interest based on the respective keyword relevancy score.

5. The one or more processor-readable storage devices of claim 1 wherein the method further comprises:

determining a position of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display;

identifying a visibility level for each identified real object based on its respective determined position, and a predetermined visibility distance for a location of the display device; and

prioritizing each real object for disappearance based on its identified visibility level in the field of view.

6. The one or more processor-readable storage devices of claim 5 wherein the method further comprises:

determining a trajectory of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display; and

wherein identifying the visibility level for each identified real object is also based on its respective determined trajectory.

7. The one or more processor-readable storage devices of claim 5 wherein identifying the visibility level for each identified real object is also based on at least one appearance characteristic of each real object identified as satisfying user disappearance criteria within the field of view of the see-through display.

8. The one or more processor-readable storage devices of claim 5 wherein tracking image data to the first real object in the see-through display for causing disappearance of the first real object in the field of view of the see-through display further comprises:

selecting an alteration technique for each real object for disappearance from alteration techniques indicated for the respective real object based on a priority of the visibility level of the respective real object; and

applying the selected alteration technique for each real object for disappearance.

9. The one or more processor-readable storage devices of claim 1 further comprising:

checking for identification of any real object satisfying user disappearance criteria which is outside the field of view but within a predetermined visibility distance for a location of the display device system; and

prefetching any applicable disappearance image data for any identified real object satisfying user disappearance criteria which is outside the field of view but within the predetermined visibility distance for the location of the display device system.

10. The one or more processor-readable storage devices of claim 9 further comprising:

applying a location prediction method for identifying one or more subsequent locations for entry by the display device system;

responsive to identifying a subsequent location, checking for identification of any real object satisfying user disappearance criteria in the subsequent location based on stored real object metadata associated with the subsequent location; and

prefetching any applicable disappearance image data for any identified real object satisfying user disappearance criteria in the subsequent location.

11. A see-through, head mounted, mixed reality display device system

for causing disappearance of a real object in a field of view of a see-through display of the display device system comprising:

one or more location detection sensors;

a memory for storing user disappearance criteria including at least one subject matter item;

one or more processors having access to the memory and communicatively coupled to the one or more location detection sensors for receiving location identifier data for the display device system and for identifying one or more real objects in the field of view of the see-through display which are related to the at least one subject matter item and within a predetermined visibility distance for a location determined from the location identifier data; and

at least one image generation unit communicatively coupled to the one or more processors and optically coupled to the see-through display for tracking image data to the identified one or more real objects in the field of view of the see-through display for causing disappearance of the one or more real objects.

12. The system of claim 11 further comprising:

the user disappearance criteria being stored in the memory further comprising a replacement object for a real world object type associated with the at least one subject matter item.

13. The system of claim 11 wherein the user disappearance criteria includes an alteration indicator for a real world object type associated with the at least one subject matter item, the alteration indicator indicating a type of alteration to be applied to the respective real world object.

14. The system of claim 11 wherein

the one or more processors are communicatively coupled to a remote computing system for accessing a three-dimensional model including metadata identifying one or more real objects in the location within a predetermined visibility distance of the see-through, mixed reality display device.

15. The system of claim 11 wherein the one or more location detection

sensors comprise at least one of the following:

a WiFi device;

an infrared device;

an image capture device;

a Global Positioning System (GPS) device;

a cellular telecommunication based device; and

a RFID device.

16. The system of claim 11 wherein

the one or more processors having access to user profile data for predicting

a subsequent location to a current location of the display device;

the one or more processors checking for identification of any real object satisfying user disappearance criteria in the subsequent location;

the one or more processors being communicatively coupled to a computer system in the subsequent location; and

the one or more processors prefetching image data for any identified real object satisfying user disappearance criteria in the subsequent location by scheduling downloading to the computer system in the subsequent location in time for arrival of the user.

20. The method of claim 16 wherein the user input of a physical action

identifying subject matter for disappearance including a real object for disappearance is a gesture identified based on image data from at least one physical environment facing camera of the see-through, mixed reality display device system.

17. A method for causing disappearance of a real object in a field of view of a see-through display of a see-through mixed reality display device system comprising:

receiving user input of a physical action identifying subject matter for disappearance including a real object for disappearance which is currently in the field of view of the see-through display;

storing the subject matter for disappearance in user disappearance criteria;

tracking image data to the real object for disappearance in accordance with an alteration technique; and

responsive to identifying a user designated alteration technique to be applied to the real object for disappearance currently in the field of view, selecting the user designated alteration technique as the alteration technique.

18. The method of claim 17 wherein the alteration technique includes at

least one of the following:

replacement of the first real object with a virtual object;

erasure of the first real object by overlaying the first real object with image data generating by blending image data of objects surrounding the first real object;

redaction of the first real object with black image data; and

obscuring of the first real object by tracking blur image data to the first real object.

19. The method of claim 17 wherein the alteration technique includes

erasure of the first real object and the method further comprises:

tracking a position of the see-through, mixed reality display device to the first real object by monitoring image data from physical environment facing cameras of the see-through, mixed reality display device system; and

responsive to detecting the see-through, mixed reality display device and the first real object are within a collision distance, outputting a safety warning.