User controlled real object disappearance in a mixed reality display

• US 10,132,633 B2
• Filed: 12/29/2015
• Issued: 11/20/2018
• Est. Priority Date: 10/14/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, James C., Latta, Stephen G., Vaught, Benjamin I., Novak, Christopher M., Bennett, Darren
• Primary Examiner(s): Liu, Zhengxi
• Application Number:
  US14/982,720
• Publication Number:
  US 20160117861A1
• Time in Patent Office: 1,057 Days
• Field of Search: None
• CPC Class Codes: G01C 21/365 Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

The technology causes disappearance of a real object in a field of view of a see-through, mixed reality display device system based on user disappearance criteria. Image data is tracked to the real object in the field of view of the see-through display for implementing an alteration technique on the real object causing its disappearance from the display. A real object may satisfy user disappearance criteria by being associated with subject matter that the user does not wish to see or by not satisfying relevance criteria for a current subject matter of interest to the user. In some embodiments, based on a 3D model of a location of the display device system, an alteration technique may be selected for a real object based on a visibility level associated with the position within the location. Image data for alteration may be prefetched based on a location of the display device system.

Claims

1. One or more processor-readable storage devices having instructions encoded thereon for causing one or more processors to execute a method for causing a real object to disappear in a see-through display of a see-through, mixed reality display device system, the method comprising:

receiving metadata identifying one or more real objects in a field of view of the see-through display;

identifying any of the one or more real objects that satisfies user disappearance criteria by identifying the metadata as matching the user disappearance criteria of the one or more real objects;

determining a position of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display;

identifying a visibility level for each identified real object based on its respective determined position and a predetermined visibility distance for a location of the display device, wherein the visibility level represents at least a level of recognition and a distance range between the real object and the display device; and

responsive to identifying a first real object that satisfies the user disappearance criteria and based on the visibility level, tracking image data to the first real object in the see-through display for causing the first real object to disappear in the field of view of the see-through display.

2. The one or more processor-readable storage devices of claim 1, further comprising:

detecting another see-through display device system within a predetermined distance of the see-through, mixed reality display device system;

based on being within the predetermined distance of the see-through, mixed reality display device system, receiving an identifier of a second real object that satisfies the user disappearance criteria of the user wearing the other see-through display device system;

receiving image data for tracking to the second real object for an alteration technique from the other see-through display device system; and

displaying the image data tracking the second real object from a perspective of the field of view of the see-through display.

3. The one or more processor-readable storage devices of claim 1, wherein identifying any of the one or more real objects that satisfies the user disappearance criteria further comprises:

based on notification of a current subject matter of interest to a user, identifying any real object type associated with the current subject matter of interest;

identifying any real object in the field of view of the see-through display matching any identified real object type; and

determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest.

4. The one or more processor-readable storage devices of claim 3, wherein determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest further comprises:

applying a keyword search technique to the metadata of any real object identified as matching any identified real object type for determining a respective keyword relevancy score; and

identifying any real object not satisfying relevance criteria for the current subject matter of interest based on the respective keyword relevancy score.

5. The one or more processor-readable storage devices of claim 1, wherein the method further comprises:

prioritizing each real object for disappearance based on its identified visibility level in the field of view.

6. The one or more processor-readable storage devices of claim 5, wherein the method further comprises:

determining a trajectory of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display; and

wherein identifying the visibility level for each identified real object is also based on its respective determined trajectory.

7. The one or more processor-readable storage devices of claim 5, wherein identifying the visibility level for each identified real object is also based on at least one appearance characteristic of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display.

8. The one or more processor-readable storage devices of claim 5, wherein tracking image data to the first real object in the see-through display for causing disappearance of the first real object in the field of view of the see-through display further comprises:

selecting an alteration technique for each real object for disappearance from alteration techniques indicated for the respective real object based on a priority of the visibility level identified for each of the real objects; and

applying the selected alteration technique for each real object for disappearance.

9. The one or more processor-readable storage devices of claim 1, further comprising:

checking for identification of any real object satisfying the user disappearance criteria that is outside the field of view but within a predetermined visibility distance for a location of the display device system; and

prefetching any applicable disappearance image data for any identified real object satisfying the user disappearance criteria that is outside the field of view but within the predetermined visibility distance for the location of the display device system.

10. The one or more processor-readable storage devices of claim 9, further comprising:

applying a location prediction method for identifying one or more subsequent locations for entry by the display device system;

responsive to identifying a subsequent location, checking for identification of any real object satisfying the user disappearance criteria in the subsequent location based on stored real object metadata associated with the subsequent location; and

prefetching any applicable disappearance image data for any identified real object satisfying the user disappearance criteria in the subsequent location.

11. A see-through, head mounted, mixed reality display device system for causing a real object to disappear in a field of view of a see-through display of the display device system comprising:

one or more location detection sensors;

a memory for storing user disappearance criteria including at least one subject matter item, an indication of the relevancy of the at least one subject matter and overlay information;

one or more processors having access to the memory and communicatively coupled to the one or more location detection sensors for receiving location identifier data for the display device system and configured to identify a plurality of real objects in the field of view of the see-through display that are related to the at least one subject matter item and within a predetermined visibility distance for a location determined from the location identifier data; and

at least one image generation unit communicatively coupled to the one or more processors and optically coupled to the see-through display configured to;

track image data to the identified plurality of real objects in the field of view of the see-through display for causing a subset of the plurality of real objects to disappear, the subset comprising a number of real objects that is less than a total number of the plurality of real objects; and

when the identified plurality of real objects matches the relevancy criteria indicated in the disappearance criteria and is within a predetermined visibility distance of the location, replace the subset of the plurality of real objects with overlay information to assist a user of the see-through display in finding more relevant information.

12. The system of claim 11, further comprising:

the user disappearance criteria being stored in the memory further comprising a replacement object for a real world object type associated with the at least one subject matter item.

13. The system of claim 11, wherein the user disappearance criteria includes an alteration indicator for a real world object type associated with the at least one subject matter item, the alteration indicator indicating a type of alteration to be applied to the respective real world object.

14. The system of claim 11, wherein the one or more processors are communicatively coupled to a remote computing system for accessing a three-dimensional model including metadata identifying one or more real objects in the location within a predetermined visibility distance of the see-through, mixed reality display device.

15. The system of claim 11, wherein the one or more location detection sensors comprise at least one of the following:

a WiFi device;

an infrared device;

an image capture device;

a Global Positioning System (GPS) device;

a cellular telecommunication based device;

or

a RFID device.

16. The system of claim 11, wherein

the one or more processors having access to user profile data for predicting a subsequent location to a current location of the display device;

the one or more processors configured to check for identification of any real object satisfying the user disappearance criteria in the subsequent location;

the one or more processors being communicatively coupled to a computer system in the subsequent location; and

the one or more processors configured to prefetch image data for any identified real object satisfying the user disappearance criteria in the subsequent location by scheduling downloading to the computer system in the subsequent location in time for arrival of the user.

17. A method for causing a real object to disappear in a see-through display of a see-through, mixed reality display device system, the method comprising:

receiving metadata identifying one or more real objects in a field of view of the see-through display;

identifying any of the one or more real objects that satisfies user disappearance criteria by identifying the metadata as matching the user disappearance criteria of the one or more real objects;

determining a position of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display;

identifying a visibility level for each identified real object based on its respective determined position and a predetermined visibility distance for a location of the display device, wherein the visibility level represents at least a level of recognition and a distance range between the real object and the display device; and

responsive to identifying a first real object that satisfies the user disappearance criteria and based on the visibility level, tracking image data to the first real object in the see-through display for causing the first real object to disappear in the field of view of the see-through display.

18. The method of claim 17, further comprising:

detecting another see-through display device system within a predetermined distance of the see-through, mixed reality display device system;

based on being within the predetermined distance of the see-through, mixed reality display device system, receiving an identifier of a second real object that satisfies the user disappearance criteria of the user wearing the other see-through display device system;

receiving image data for tracking to the second real object for an alteration technique from the other see-through display device system; and

displaying the image data tracking the second real object from a perspective of the field of view of the see-through display.

19. The method of claim 17, wherein identifying any of the one or more real objects that satisfies the user disappearance criteria further comprises:

based on notification of a current subject matter of interest to a user, identifying any real object type associated with the current subject matter of interest;

identifying any real object in the field of view of the see-through display matching any identified real object type; and

determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest.

20. The method of claim 19, wherein determining whether any identified real object does not satisfy relevance criteria for the current subject matter of interest further comprises:

applying a keyword search technique to the metadata of any real object identified as matching any identified real object type for determining a respective keyword relevancy score; and

identifying any real object not satisfying relevance criteria for the current subject matter of interest based on the respective keyword relevancy score.

21. The method of claim 17, wherein the method further comprises:

prioritizing each real object for disappearance based on its identified visibility level in the field of view.

22. The method of claim 21, wherein the method further comprises:

determining a trajectory of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display; and

wherein identifying the visibility level for each identified real object is also based on its respective determined trajectory.

23. The method of claim 21, wherein identifying the visibility level for each identified real object is also based on at least one appearance characteristic of each real object identified as satisfying the user disappearance criteria within the field of view of the see-through display.

24. The method of claim 21, wherein tracking image data to the first real object in the see-through display for causing disappearance of the first real object in the field of view of the see-through display further comprises:

selecting an alteration technique for each real object for disappearance from alteration techniques indicated for the respective real object based on a priority of the visibility level identified for each of the real objects; and

applying the selected alteration technique for each real object for disappearance.

25. The method of claim 17, further comprising:

checking for identification of any real object satisfying the user disappearance criteria that is outside the field of view but within a predetermined visibility distance for a location of the display device system; and

prefetching any applicable disappearance image data for any identified real object satisfying the user disappearance criteria that is outside the field of view but within the predetermined visibility distance for the location of the display device system.

26. The method of claim 25, further comprising:

applying a location prediction method for identifying one or more subsequent locations for entry by the display device system;

responsive to identifying a subsequent location, checking for identification of any real object satisfying the user disappearance criteria in the subsequent location based on stored real object metadata associated with the subsequent location; and

prefetching any applicable disappearance image data for any identified real object satisfying the user disappearance criteria in the subsequent location.