Touch and social cues as inputs into a computer

• US 9,536,350 B2
• Filed: 11/29/2012
• Issued: 01/03/2017
• Est. Priority Date: 08/24/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Novak, Christopher Michael, Liu, James, Latta, Stephen, Andrews, Anton O. A., Maitlen, Craig R., Martin, Sheridan
• Primary Examiner(s): Pham, Linh K
• Assistant Examiner(s): Nicholas, Wendy
• Application Number:
  US13/689,471
• Publication Number:
  US 20130169682A1
• Time in Patent Office: 1,496 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system for automatically displaying virtual objects within a mixed reality environment is described. In some embodiments, a see-through head-mounted display device (HMD) identifies a real object (e.g., a person or book) within a field of view of the HMD, detects one or more interactions associated with real object, and automatically displays virtual objects associated with the real object if the one or more interactions involve touching or satisfy one or more social rules stored in a social rules database. The one or more social rules may be used to infer a particular social relationship by considering the distance to another person, the type of environment (e.g., at home or work), and particular physical interactions (e.g., handshakes or hugs). The virtual objects displayed on the HMD may depend on the particular social relationship inferred (e.g., a friend or acquaintance).

Claims

1. A method for automatically displaying virtual objects on a mobile device, comprising:

receiving one or more images associated with a field of view of the mobile device;

identifying a particular object located within the field of view using the one or more images, the particular object comprises a particular person;

detecting one or more interactions between a person associated with the mobile device and the particular person;

acquiring an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person;

determining whether the one or more interactions satisfy at least one social rule of one or more social rules, the at least one social rule is satisfied if the person associated with the mobile device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person, wherein the at least one social rule is satisfied if the person performs a handshake or high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting between the person and the particular person;

acquiring virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the mobile device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person; and

displaying the virtual data on the mobile device.

2. The method of claim 1, wherein:

the acquiring virtual data includes a first virtual object.

3. The method of claim 1, wherein:

the one or more interactions include the particular person speaking in the direction of the mobile device.

4. The method of claim 3, wherein:

the at least one social rule is satisfied if the person associated with the mobile device touches the particular person and the particular person is located within a particular distance of the mobile device for more than a particular period of time.

5. The method of claim 3, wherein:

the at least one social rule is satisfied if the particular person is associated with a particular privacy level; and

the determining whether the one or more interactions satisfy at least one social rule includes requesting profile information associated with the particular person, the profile information includes the particular privacy level.

6. The method of claim 3, wherein:

the detecting one or more interactions includes performing skeletal tracking; and

the one or more interactions include the person associated with the mobile device touching the particular person via a handshake, the mobile device comprises a head-mounted display device.

7. The method of claim 6, wherein:

the virtual data includes a text description displayed to the particular person.

8. An electronic device for automatically displaying virtual objects, comprising:

a memory, configured to store at least one rule of one or more rules;

one or more processors, configured to receive one or more images associated with a field of view of the electronic device, and identify a particular object located within the field of view using the one or more images, the particular object comprises a particular person, the one or more processors configured to detect one or more interactions between a person associated with the electronic device and the particular person, the one or more processors configured to acquire an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person, the one or more processors configured to determine whether the one or more interactions satisfy the at least one rule, the at least one rule is satisfied if the person associated with the electronic device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the electronic device and the particular person, wherein the at least one rule is satisfied if the person performs a handshake or a high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting, the one or more processors configured to acquire virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the electronic device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the electronic device and the particular person; and

a display, configured to display the virtual data if the one or more interactions satisfy the at least one rule.

9. The electronic device of claim 8, wherein:

the one or more interactions include the particular person speaking in the direction of the electronic device.

10. The electronic device of claim 9, wherein:

the at least one rule is satisfied if the person associated with the electronic device touches the particular person and the particular person is located within a particular distance of the electronic device for more than a particular period of time.

11. The electronic device of claim 9, wherein:

the at least one rule is satisfied if the particular person is associated with a particular privacy level.

12. The electronic device of claim 9, wherein:

the one or more processors identify the particular object located within the field of view by performing facial recognition on the particular person.

13. The electronic device of claim 12, wherein:

the virtual data includes a text description displayed to the particular person.

14. One or more hardware storage devices containing processor readable code for programming one or more processors to perform a method for automatically displaying virtual objects comprising the steps of:

receiving one or more images associated with a field of view of the mobile device;

identifying a particular object located within the field of view using the one or more images, the particular object comprises a particular person;

detecting one or more interactions between a person associated with the mobile device and the particular person;

acquiring an electronic calendar for the person, the electronic calendar includes an electronically scheduled meeting between the person and the particular person;

determining whether the one or more interactions satisfy at least one social rule of one or more social rules, the at least one social rule is satisfied if the person associated with the mobile device touches the particular person at a point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person, wherein the at least one social rule is satisfied if the person performs a handshake or high-five gesture with the particular person at the point in time coinciding with the electronically scheduled meeting between the person and the particular person;

acquiring virtual data associated with an augmented reality environment displayed to the particular person in response to detecting that the person associated with the mobile device has touched the particular person at the point in time coinciding with the electronically scheduled meeting between the person associated with the mobile device and the particular person; and

determining if the virtual data has been displayed on the mobile device within a predetermined period of time; and

displaying the virtual data on the mobile device if the virtual data has not been displayed within the predetermined period of time.

15. The one or more hardware storage devices of claim 14, wherein:

the one or more interactions include the particular person speaking in the direction of the mobile device.

16. The one or more hardware storage devices of claim 15, wherein:

the at least one social rule is satisfied if the person associated with the mobile device touches the particular person and the particular person is located within a proximity of the mobile device for more than a particular period of time.