Object holographic augmentation

• US 10,325,409 B2
• Filed: 06/16/2017
• Issued: 06/18/2019
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Costa, Carlos Fernando Faria
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US15/625,422
• Publication Number:
  US 20180365898A1
• Time in Patent Office: 732 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Devices, systems, and methods for augmenting a real-world object using a mixed reality device, involving capturing image data for a real-world object included in a physical space observed by the mixed reality device; automatically classifying the real-world object as being associated with an object classification based on the image data; automatically identifying a software application based on the real-world object being associated with the object classification; associating the software application with the real-world object; and executing the software application on the mixed reality device in association with the real-world object.

Claims

1. A mixed reality device comprising:

one or more outward facing image sensors included in the mixed reality device and configured to capture first image data for a first real-world object included in a physical space observed by the mixed reality device;

an object classifier configured to automatically classify the first real-world object as being associated with a first object classification based on the captured first image data;

an application identifier configured to automatically identify a first software application based on the first real-world object being associated with the first object classification;

a controller configured to;

store data associating the first software application with the first real-world object, and

execute the first software application on the mixed reality device in association with the first real-world object, wherein;

the outward facing image sensors are further configured to capture second image data for the physical space;

the object classifier is further configured to automatically classify a plurality of real-world objects included in the physical space as being associated with respective object classifications, based on the second image data; and

the mixed reality device further comprises a space classifier configured to automatically classify the physical space as being associated with a physical space classification based on the object classifications associated with the plurality of real-world objects.

2. The mixed reality device according to claim 1, wherein:

the first software application is configured to display a graphical virtual interface;

the mixed reality device further comprises;

a display subsystem configured to display images to a user of the mixed reality device, and

a visual renderer configured to repeatedly display the graphical virtual interface using the display subsystem so as to continue to appear in real time to be located proximate to the first real-world object while the mixed reality device moves through the physical space.

3. The mixed reality device according to claim 2, wherein the display subsystem includes a head mounted display device used to display the images.

4. The mixed reality device according to claim 3, wherein the head mounted display device includes a see-through display device configured to allow a user wearing the head mounted display device to directly view the first real world object through the see-through display device.

5. The mixed reality device of claim 1, wherein the object classifier is further configured to transmit second image data to a remote object identification system configured to automatically classify the first real-world object as being associated with the first object classification based on the second image data.

6. The mixed reality device of claim 1, wherein:

the application identifier is further configured to automatically identify, based on the first real-world object being associated with the first object classification, a plurality of software applications, wherein the first software application is included in the plurality of software applications, and the automatic identification of the first software application is performed by the automatic identification of the plurality of software applications;

the controller is further configured to;

present, to a user of the mixed reality device, the plurality of software applications as suggestions for association with the first real-world object,

receive a user selection, via the mixed reality device and in response to the presentation of the plurality of software applications, of the first software application, and

automatically download and install the first software application on the mixed reality device in response to the received selection; and

the association of the first software application with the first real-world object is stored in response to the received selection.

7. The mixed reality device of claim 1, further comprising:

a wireless receiver adapted to receive one or more wireless signals transmitted by the first real-world object,

wherein the object classifier is further configured to automatically classify the first real-world object as being associated with a first object classification based on a content of the one or more wireless signals.

8. A method for augmenting a real-world object using a mixed reality device, the method comprising:

capturing first image data for a first real-world object included in a physical space observed by the mixed reality device;

automatically classifying the first real-world object as being associated with a first object classification based on the first image data;

automatically identifying a first software application based on the first real-world object being associated with the first object classification;

associating the first software application with the first real-world object;

executing the first software application on the mixed reality device in association with the first real-world object;

automatically classifying a plurality of real-world objects included in the physical space as being associated with respective object classifications, based on second image data captured by the mixed reality device; and

automatically classifying the physical space as being associated with a physical space classification based on the object classifications associated with the plurality of real-world objects.

9. The method of claim 8, wherein:

the first software application is configured to display a graphical virtual interface; and

the method further comprises repeatedly displaying the graphical virtual interface using the mixed reality device so as to continue to appear in real time to be located proximate to the first real-world object while the mixed reality device moves through the physical space.

10. The method of claim 9, further comprising:

capturing second image data for the real-world object while the first real-world object moves through the physical space over a first period of time;

automatically determining updated positions for the first real-world object based on the second image data; and

repeatedly displaying the graphical virtual interface using the mixed reality device, based on the updated positions for the first real-world object, so as to continue to appear in real time to be located proximate to the real-world object while the real-world object moves through the physical space over the first period of time.

11. The method of claim 9, further comprising:

capturing second image data for the first real-world object while the first real-world object changes pose over a first period of time;

automatically determining updated poses for the first real-world object based on the second image data; and

repeatedly displaying the graphical virtual interface using the mixed reality device, based on the updated poses for the first real-world object, so as to continue to appear in real time to be fixed relative to a portion of the first real-world object while the first real-world object changes pose over the first period of time.

12. The method of claim 8, further comprising:

identifying an avatar comprising one or more virtual objects as being associated with the first real-world object; and

repeatedly displaying the avatar using the mixed reality device so as to continue to, in real time, superimpose the avatar on a portion of the first real-world object while the mixed reality device and/or the first real-world object move through the physical space.

13. The method of claim 8, further comprising:

automatically identifying, based on the first real-world object being associated with the first object classification, a plurality of software applications, wherein the first software application is included in the plurality of software applications, and the automatic identification of the first software application is performed by the automatic identification of the plurality of software applications;

presenting, to a user of the mixed reality device, the plurality of software applications as suggestions for association with the first real-world object;

receiving a user selection, via the mixed reality device and in response to the presentation of the plurality of software applications, of the first software application; and

automatically downloading and installing the first software application on the mixed reality device in response to the received user selection,

wherein the associating the first software application with the first real-world object occurs in response to the received selection.

14. The method of claim 8, further comprising:

automatically identifying a second software application based on the physical space being associated with the physical space classification;

associating the second software application with the physical space; and

executing the second software application on the mixed reality device in association with the physical space.

15. The method of claim 8, wherein:

the first real-world object is configured to respond to first commands received via a wired or wireless communication medium;

the first software application is configured to issue the first commands; and

the method further comprises automatically configuring the first software application to direct issued commands to the first real-world object via the wired or wireless communication medium.

16. The method of claim 8, further comprising:

in response to the automatic identification of the first software application, displaying a visual indicator using the mixed reality device so as to appear to be located proximate to the first real-world object, the visual indicator indicating to the user that the first real-world object may be augmented using one or more application programs;

receiving, via the mixed reality device, a user selection of the first real-world object while the visual indicator is being displayed; and

providing, in response to receiving the user selection, a user interface configured to receive, via the mixed reality device, a user instruction to associate the first software application with the first real-world object,

wherein the associating the first software application with the first real-world object occurs in response to receiving the user instruction.

17. A nontransitory machine readable medium including instructions which, when executed by one or more processors included in the mixed reality device, cause the mixed reality device to perform the method of claim 8.

18. A method for augmenting a real-world object using a mixed reality device, the method comprising:

capturing first image data for a first real-world object included in a physical space observed by the mixed reality device;

automatically classifying the first real-world object as being associated with a first object classification based on the first image data;

automatically identifying a first software application based on the first real-world object being associated with the first object classification;

associating the first software application with the first real-world object;

executing the first software application on the mixed reality device in association with the first real-world object; and

receiving, with a receiver included in the mixed reality device, one or more wireless signals transmitted by the first real-world object,

wherein the automatic classification of the first real-world object is further based on a content of the one or more wireless signals.