Providing location occupancy analysis via a mixed reality device

• US 10,223,832 B2
• Filed: 09/25/2015
• Issued: 03/05/2019
• Est. Priority Date: 08/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Geisner, Kevin A., Bennett, Darren, Markovic, Relja, Latta, Stephen G., McCulloch, Daniel J., Scott, Jason, Hastings, Ryan L., Kipman, Alex Aben-Athar, Fuller, Andrew John, Margolis, Jeffrey Neil, Perez, Kathryn Stone, Small, Sheridan Martin
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US14/865,410
• Publication Number:
  US 20160086382A1
• Time in Patent Office: 1,257 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/002 Specific input/output arrangements not covered by G06F3/01 - G06F3/16 other optical apparatus G02B27/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 5/00 Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators image data processing or generation, in general G06T

Abstract

The technology provides contextual personal information by a mixed reality display device system being worn by a user. A user inputs person selection criteria, and the display system sends a request for data identifying at least one person in a location of the user who satisfy the person selection criteria to a cloud based application with access to user profile data for multiple users. Upon receiving data identifying the at least one person, the display system outputs data identifying the person if he or she is within the field of view. An identifier and a position indicator of the person in the location is output if not. Directional sensors on the display device may also be used for determining a position of the person. Cloud based executing software can identify and track the positions of people based on image and non-image data from display devices in the location.

Claims

1. A machine-system implemented method of determining that a first among a plurality of areas is not occupied by one or more persons of specific interest to a user of the machine system, the method comprising:

receiving person selection criteria by way of one or more mobile devices of the user where the one or more mobile devices are operatively coupled to the machine system, the received person selection criteria being usable by the machine system to identify one or more candidate persons in a pool of candidate persons satisfying specific criteria of the received person selection the criteria, the specific criteria specifying one or more characteristics, relationships, interests and experiences of the one or more candidate persons of specific interest to the user because of said specified one or more of characteristics, relationships, interests and experiences, where the machine system has access to at least one user profile of at least one of the one or more candidate persons;

automatically determining a current location of the user;

automatically identifying a first among a plurality of automatically searchable areas each capable of containing one or more persons and each capable of automated detecting of identities of one or more persons in that area, the identified first area being most or second most proximate to the determined current location of the user;

based on use of one or more persons identifying services, automatically determining whether the identified first area contains any candidate persons currently satisfying the specific criteria of the person selection criteria received from the user;

responsive to the one or more persons identifying services failing to indicate presence of any candidate person currently satisfying the specific criteria of the received person selection criteria within the identified first area, indicating to the user by way of at least one of said one or more mobile devices of the user that are operatively coupled to the machine system that the first area does not contain at least one person currently satisfying the received person selection criteria;

responsive to the one or more persons identifying services indicating presence of at least one candidate person currently satisfying the specific criteria of the received person selection criteria within the identified first area, generating a personal identification data set for the at least one candidate person;

determining whether the at least one candidate person is currently within a field of view of the machine system based at least on received field of view data; and

responsive to the at least one candidate person falling within the field of view of the machine system, outputting audio and/or visual data associated with the personal identification data set to the user.

2. The method of claim 1 wherein the used one or more persons identifying services include one or more persons locating services.

3. The method of claim 2 wherein the used one or more persons locating services include a GPS based service.

4. The method of claim 2 wherein the used one or more persons locating services include an automated features recognition service.

5. The method of claim 4 wherein the automated features recognition service uses one or more head mounted devices (HMDs) each capable of identifying its respective wearer and location of that respective wearer and each operatively coupled to a network by way of which the identity of its respective wearer can be reported.

6. The method of claim 5 wherein the one or more HMDs each have one or more outwardly facing sensors usable for automatically identifying locally disposed other persons.

7. The method of claim 6 wherein the one or more outwardly facing sensors include depth cameras usable for providing a local object recognition service.

8. The method of claim 6 wherein the one or more outwardly facing sensors include sound recognition sensors usable for providing sound-based recognition of the locally disposed other persons.

9. The method of claim 4 wherein the automated features recognition service uses one or more at location, physical reality capturing devices each operatively coupled to a network and each capable of capturing physical features of persons and objects disposed proximate to the physical reality capturing device.

10. The method of claim 1 wherein the used one or more persons identifying services do not report whether the identified first area contains or does not contain a given person currently satisfying the person selection criteria received from the user if the given person has indicated a desire to not be automatically identified.

11. The method of claim 1 wherein the specific criteria specify specific one or more of characteristics and the specific characteristics include at least one of a satisfactory candidate person'"'"'s gender, a satisfactory candidate person'"'"'s height, a satisfactory candidate person'"'"'s social attributes, a satisfactory candidate person'"'"'s political attributes, a satisfactory candidate person'"'"'s religious attributes, a satisfactory candidate person'"'"'s economic attributes and a satisfactory candidate person'"'"'s employment related attributes.

12. The method of claim 1 wherein the specific criteria specify specific one or more of relationships and the specific relationships include at least one of a satisfactory candidate person'"'"'s belonging to one or more restricted access friendship categories, having ownership of a specific kind of property, belonging to a specific organization, and belonging to a specific social media users group.

13. The method of claim 1 wherein the specific criteria specify specific one or more of interests and the specific interests include at least one of a satisfactory candidate person'"'"'s interest in one or more hobbies and a satisfactory candidate person'"'"'s interest in one or more fields of knowledge.

14. The method of claim 1 wherein the specific criteria specify specific one or more of experiences and the specific experiences include at least one of a satisfactory candidate person'"'"'s work related experiences and a satisfactory candidate person'"'"'s leisure related experiences.

15. An automated machine system for determining that a first among a plurality of areas is not occupied by one or more persons of interest, the machine system comprising:

a receiver configured to receive person selection criteria by way of one or more mobile devices of the user where the one or more mobile devices are operatively coupled to the machine system, the received person selection criteria being usable to identify one or more candidate persons in a pool of candidate persons satisfying specific criteria of the received person selection criteria, the specific criteria specifying one or more of the characteristics, relationships, interests and experiences of the one or more candidate persons of specific interest to the user because of said specified one or more characteristics, relationships, interests and experiences, where the machine system has access to at least one user provide of at least one of the one or more candidate persons;

a user locator configured to automatically determine a current location of the user;

an areas identifier configured to automatically identify a subset among a plurality of automatically searchable areas each capable of containing one or more persons and each capable of automated detecting of identities of one or more persons in that area, the identified subset including at least one of a most or second most proximate searchable area relative to the determined current location of the user;

an area occupancy determiner configured to use one or more persons identifying services and to automatically determine, based on use of the one or more persons identifying services, whether the identified subset of automatically searchable areas contains any persons satisfying the received person selection criteria received from the user;

an area occupancy reporter coupled to the area occupancy determiner and configured to report to the user by way of at least one of said one or more mobile devices of the user that the subset of automatically searchable areas does not contain any person satisfying the received person selection criteria;

a personal information engine configured to;

responsive to the one or more persons identifying services indicating presence of at least one candidate person currently satisfying the specific criteria of the received person selection criteria within the identified first area, generate a personal identification data set for the at least one candidate person; and

determine whether the at least one candidate person is currently within a field of view of the machine system based at least on received field of view data; and

an image generation unit configured to, responsive to the at least one candidate person falling within the field of view of the machine system, output audio and/or visual data associated with the personal identification data set to the user.

16. The machine system of claim 15 wherein:

the used one or more persons identifying services include one or more persons locating services;

the used one or more persons locating services include an automated features recognition service; and

the automated features recognition service uses one or more head mounted devices (HMDs) each capable of identifying its respective wearer and location of that respective wearer and each operatively coupled to a network by way of which the identity of its respective wearer can be reported.

17. The machine system of claim 16 wherein the one or more HMDs each have one or more outwardly facing sensors usable for automatically identifying locally disposed other persons.

18. The machine system of claim 17 wherein the one or more outwardly facing sensors include depth cameras usable for providing a local object recognition service.

19. The machine system of claim 15 wherein:

the used one or more persons identifying services include one or more persons locating services;

the used one or more persons locating services include an automated features recognition service; and

the automated features recognition service uses one or more at location, physical reality capturing devices each operatively coupled to a network and each capable of capturing physical features of persons and objects disposed proximate to the physical reality capturing device.

20. A computer-readable storage device operatively coupled to one or more data processing machines and storing instructions readable and executable by at least one of the data processing machines, the stored instructions being configured to cause the one or more data processing machines to carry out an automated process comprising:

receiving person selection criteria by way of one or more mobile devices of the user where the one or more mobile devices are operatively coupled to the machine system, the received person selection criteria being usable by the machine system to identify one or more candidate persons in a pool of candidate persons satisfying specific criteria of the received person selection the criteria, the specific criteria specifying one or more characteristics, relationships, interests and experiences of the one or more candidate persons of specific interest to the user because of said specified one or more of characteristics, relationships, interests and experiences, where the machine system has access to at least one user profile of at least one of the one or more candidate persons;

automatically determining a current location of the user;

automatically identifying a subset of a plurality of automatically searchable areas each capable of containing one or more persons and each capable of automated detecting of identities of one or more persons in that area, the identified subset of areas including at least one that is most or second most proximate to the user;

based on use of one or more persons identifying services, automatically determining whether the identified first area contains any candidate persons currently satisfying the specific criteria of the person selection criteria received from the user;

responsive to the one or more persons identifying services failing to indicate presence of any candidate person currently satisfying the specific criteria of the received person selection criteria within the identified first area, indicating to the user by way of at least one of said one or more mobile devices of the user that are operatively coupled to the machine system that the first area does not contain at least one person currently satisfying the received person selection criteria;

responsive to the one or more persons identifying services indicating presence of at least one candidate person currently satisfying the specific criteria of the received person selection criteria within the identified first area, generating a personal identification data set for the at least one candidate person;

determining whether the at least one candidate person is currently within a field of view of the machine system based at least on received field of view data; and

responsive to the at least one candidate person falling within the field of view of the machine system, outputting audio and/or visual data associated with the personal identification data set to the user.