PROVIDING CONTEXTUAL PERSONAL INFORMATION BY A MIXED REALITY DEVICE

• US 20130044130A1
• Filed: 01/30/2012
• Published: 02/21/2013
• Est. Priority Date: 08/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Bennett, Darren, Markovic, Relja, Latta, Stephen G., McCulloch, Daniel J., Hastings, Ryan L., Kipman, Alex Aben-Athar, Fuller, Andrew John, Margolis, Jeffrey Neil, Perez, Kathryn Stone, Small, Sheridan Martin, Scott, Jason
• Granted Patent: US 9,153,195 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The technology provides contextual personal information by a mixed reality display device system being worn by a user. A user inputs person selection criteria, and the display system sends a request for data identifying at least one person in a location of the user who satisfy the person selection criteria to a cloud based application with access to user profile data for multiple users. Upon receiving data identifying the at least one person, the display system outputs data identifying the person if he or she is within the field of view. An identifier and a position indicator of the person in the location is output if not. Directional sensors on the display device may also be used for determining a position of the person. Cloud based executing software can identify and track the positions of people based on image and non-image data from display devices in the location.

Claims

1. One or more processor-readable storage devices having instructions encoded thereon for causing one or more software controlled processors to execute a method for providing contextual personal information by a mixed reality display device system, the method comprising:

receiving and storing person selection criteria associated with a user wearing the mixed reality display device system;

sending a request including a location of the user and the person selection criteria to a personal information service engine executing on one or more remote computer systems for a personal identification data set for each person sharing the location and satisfying the person selection criteria;

receiving at least one personal identification data set from the personal identification service engine for a person sharing the location;

determining whether the person associated with the at least one personal identification data set is in the field of view of the mixed reality display device system; and

responsive to the person associated with the at least one personal identification data set being in the field of view, outputting data which identifies the person in the field of view.

2. The one or more processor-readable storage devices of claim 1 further comprising

determining a personal interaction status for a person associated with the at least one personal identification data set based on audio and image data captured by the mixed reality display device system;

and

outputting a personal interaction status stored for the person.

3. The one or more processor-readable storage devices of claim 2 wherein the personal interaction status identifies at least one or more of the following:

none;

greeting;

conversation;

conversation related to personal information selection criteria;

conversation related to user profile data; and

gesture acknowledgement.

4. The one or more processor-readable storage devices of claim 2 wherein determining a personal interaction status for a person associated with the at least one personal identification data set based on audio and image data captured by the mixed reality display device system further comprises identifying one or more interactions from the person within a time period of the person coming into the field of view of the mixed reality display device system;

identifying one or more user physical actions within the time period of the one or more interactions while the person is an object of interest;

identifying one or more personal interaction status candidates which may apply based on the one or more interactions and the one or more user physical actions within the time period; and

selecting as a personal interaction status the candidate indicating the highest level of interaction.

5. The one or more processor-readable storage devices of claim 1 wherein outputting data which identifies the person in the field of view further comprises

displaying a visual indicator registered to the person in a display of the mixed reality display device system.

6. The one or more processor-readable storage devices of claim 1 wherein outputting data which identifies the person in the field of view further comprises

playing audio including a personal information item of the person which satisfies the person selection criteria through at least one earphone of the mixed reality, display device system.

7. The one or more processor-readable storage devices of claim 1 further comprising:

responsive to the person associated with the at least one personal identification data set not being in the field of view, displaying a personal identifier and a position within the location of the person associated with the at least one personal identification data set.

8. A method for providing contextual personal information to a user, head mounted, mixed reality, display device system comprising:

receiving a request indicating a location of a user head mounted, mixed reality display device system and person selection criteria for at least one personal identification data set for each person sharing the location and satisfying the person selection criteria;

determining whether there is a person sharing the location and satisfying the person selection criteria based on accessible user profile data;

responsive to there being a person sharing the location and satisfying the person selection criteria, generating at least one personal identification data set for the person;

determining whether the person associated with the at least one personal identification data set is currently within a field of view of the user head mounted, mixed reality, display device system;

responsive to the person not being currently within the field of view of the user display device system, determining a position of the person within the location, and sending the position in the at least one personal identification data set; and

responsive to the person being currently within the field of view of the user display device system, sending the at least one personal identification data set including a personal identifier for the person in the field of view.

9. The method of claim 8 wherein determining whether there is a person sharing the location and satisfying the person selection criteria based on accessible user profile data further comprises:

searching user profile data of one or more persons having a relationship link in the user profile data of the user for data which satisfies the person selection criteria and for which the user has been permitted access.

10. The method of claim 8 wherein the personal identifier is an object identifier unique for the location.

11. The method of claim 8 wherein determining whether the person associated with the at least one personal identification data set is currently within a field of view of the user head mounted, mixed reality, display device system further comprises

receiving image data of one or more users in the location;

assigning a unique object identifier to each of the one or more users;

tracking a position of each of the one or more users within the location based on an image mapping of the location; and

determining whether an object associated with the unique object identifier assigned to the person is within the field of view of the user display device system based on correlating image data received from at least one field of view camera of the user display device system and the position of the person in the image mapping of the location.

12. The method of claim 8 wherein determining a position of the person within the location further comprises:

determining whether there is a head mounted, mixed reality, display device system associated with the personal identifier of the person in the location; and

responsive to there being a display device system associated with the personal identifier of the person in the location,

retrieving from the display device system associated with the personal identifier field of view image data,

identifying the position of the person within the location based on image mapping data of the location and the field of view image data of the display device system associated with the personal identifier,

identifying a position of the user head mounted, mixed reality, display device system,

generating virtual data for directing the user to the person in the location, and

sending the virtual data to the user display device system.

13. The method of claim 12 further comprising:

responsive to there not being a head mounted, mixed reality, display device system associated with the personal identifier of the person in the location, determining whether there is a second head mounted, mixed reality, display device system associated with another personal identifier in the location having the person in its field of view; and

responsive to there being a second head mounted, mixed reality display device system having the person in its field of view,

retrieving field of view image data from the second display device system,

identifying the position of the person within the location based on image mapping data of the location and the field of view image data from the second display device system,

identifying a position of the user display device system,

generating virtual data for directing the user to the person in the location, and

sending the virtual data to the user display device system.

14. The method of claim 13 further comprising:

responsive to there not being a head mounted, mixed reality, display device system associated with the personal identifier of the person in the location, determining whether there is a second head mounted, mixed reality, display device system associated with another personal identifier in the location having non-image location data indicating a relative position of the person from a position of the second display device system; and

responsive to there being a second head mounted, mixed reality, display device system having the non-image location data indicating the relative position of the person from the second display device system,

identifying the position of the second display device system within the location based on image mapping data of the location and field of view image data of the second display device system,

identifying the position of the person within the location based on the relative position of the person from the second display device system and the position of the second display device system within the location,

identifying a position of the user display device system,

generating virtual data for directing the user to the person in the location, and

sending the virtual data to the user display device system.

15. The method of claim 14 wherein the non-image location data is data from one or more directional sensors on the second display device system.

16. A see-through, head mounted, mixed reality display device system for providing contextual personal information comprising:

a see-through, mixed reality display positioned by a head mounted support structure;

at least one front facing camera positioned on the support structure for capturing image data of a field of view of the see-through display;

one or more directional sensors attached to the support structure, each having a sensor position with reference to a body part of the user and transmitting an identity data set including the sensor position;

one or more software controlled processors communicatively coupled to the at least one front facing camera for receiving the image data of the field of view;

the one or more software controlled processors being communicatively coupled to a remote computer system executing a personal information service engine for sending a request with person selection criteria and a location of a user wearing the head mounted support structure, and receiving a personal identification data set of each person sharing the location and satisfying the person selection criteria; and

at least one image generation unit communicatively coupled to the one or more software controlled processors and optically coupled to the see-through, mixed reality display for tracking virtual data to a person sharing the location and satisfying the person selection criteria in a field of view of the see-through display.

17. The system of claim 16 further comprising:

a microphone supported by the support structure for capturing audio data and being communicatively coupled to the one or more software controlled processors for sending audio data;

one or more eye tracking assemblies positioned by the support structure for capturing image data of each eye and communicatively coupled to the one or more software controlled processors for sending image data of each eye;

the one or more software controlled processors identifying a person object of interest based on the image data of each eye and image data of the field of view of the see-through display; and

the one or more software controlled processors determining a personal interaction status for the person object of interest based on audio and image data captured of the person object of interest and a personal interaction rule set stored in an accessible memory.

18. The system of claim 16 further comprising:

a microphone supported by the support structure for capturing audio data and being communicatively coupled to the one or more software controlled processors for sending audio data;

the one or more software controlled processors identifying a person object of interest based on a gesture of the user recognized in image data of the field of view of the see-through display; and

the one or more software controlled processors determining a personal interaction status for the person object of interest based on audio and image data captured of the person object of interest and a personal interaction rule set stored in an accessible memory.

19. The system of claim 17 further comprising:

the one or more software controlled processors storing in an accessible memory a personal interaction status for the person satisfying the person selection criteria; and

the at least one image generation unit displaying a personal interaction status for the person satisfying the person selection criteria responsive to user input.

20. The system of claim 16 wherein the one or more directional sensors comprise at least one of the following:

a directional antenna;

a WiFi device;

an infrared device;

a Global Positioning System (GPS) device;

a cellular telecommunication based device;

a WUSB device;

a Bluetooth®

device; and

a RFID device.