Providing electronic communications in a physical world

• US 9,146,398 B2
• Filed: 07/12/2011
• Issued: 09/29/2015
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen G., Small, Sheridan Martin, Liu, James C., Vaught, Benjamin I., Bennett, Darren
• Primary Examiner(s): Johnson, Gerald
• Assistant Examiner(s): Kohlman, Christopher
• Application Number:
  US13/181,156
• Publication Number:
  US 20130016033A1
• Time in Patent Office: 1,540 Days
• Field of Search: 345/7, 345/8
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/147 using display panels

Abstract

Techniques are provided for displaying electronic communications using a head mounted display (HMD). Each electronic communication may be displayed to represent a physical object that indentifies it as a specific type or nature of electronic communication. Therefore, the user is able to process the electronic communications more efficiently. In some aspects, computer vision allows a user to interact with the representation of the physical objects. One embodiment includes accessing electronic communications, and determining physical objects that are representative of at least a subset of the electronic communications. A head mounted display (HMD) is instructed how to display a representation of the physical objects in this embodiment.

Claims

1. A method comprising:

learning, by an electronic system, how a user interacted at a specific location with electronic communications from specific senders;

accessing, by the electronic system, electronic communications that are addressed to the user, each of the electronic communications having a sender;

determining, by the electronic system, a present location of the user, wherein the present location is the specific location;

determining, by the electronic system based on the learning, a first subset of the electronic communications addressed to the user that are to be presented to the user at the present location and a second subset of the electronic communications addressed to the user that are not to be presented to the user at the present location;

determining, by the electronic system, a plurality of physical objects that are representative of the first subset of the electronic communications;

displaying the plurality of physical objects in a head mounted display (HMD), including displaying an image of the physical objects as being registered to a surface in the present location other than the user'"'"'s body;

receiving, by the electronic system while the image of the physical objects is being displayed as being registered to the surface in the present location, a selection of a first physical object of the plurality of physical objects whose image is to follow the user, wherein the first physical object corresponds to a first electronic communication in the first subset of the electronic communications;

displaying, in the HMD, an image of the first physical object registered to the user'"'"'s body in response to receiving the selection of the first physical object to follow the user; and

continuing to display the image of the first physical object in the HMD as registered to the user'"'"'s body after the user has left the present location.

2. The method of claim 1, further comprising:

tracking actions of the user while the user is wearing the HMD and the HMD is displaying the image of the physical objects;

determining how the user is intending to interact with the first physical object based on the actions; and

determining how to alter presentation of the first physical object on the HMD based on the actions in order accomplish the user'"'"'s intent to interact.

3. The method of claim 2, wherein:

the tracking actions of the user wearing the HMD and the determining how the user is intending to interact include;

tracking positions and movements of the user'"'"'s fingers using computer vision to determine that the user is intending to open an envelope; and

the determining how to alter presentation includes determining to show the envelope being opened.

4. The method of claim 1, wherein the determining a plurality of physical objects includes determining the first physical object based on a source of the first electronic communication.

5. The method of claim 1, further comprising:

receiving a selection of the first physical object after the user has left the present location; and

providing, by the electronic system, content of the first electronic communication to the HMD in response to the selection.

6. The method of claim 1, further comprising:

determining, by the electronic system, an arrangement of the physical objects in response to a command to sort the electronic communications; and

instructing, by the electronic system, the HMD to display the arrangement of the physical objects.

7. The method of claim 1, further comprising:

determining content within the first electronic communication of the electronic communications; and

determining the first physical object that is representative of the content of the first electronic communication.

8. The method of claim 1, further comprising:

accessing, by the electronic system, a model of the present location; and

determining, by the electronic system, an arrangement of the physical objects in the present location based on the model.

9. The method of claim 8, further comprising:

learning, by the electronic system, preferences of the user for arranging the physical objects in the present location, wherein the electronic system determines the arrangement of the physical objects in the present location based on the learned user preferences.

10. The method of claim 8, wherein the arrangement is selected from a hap-hazard arrangement and an ordered arrangement.

11. The method of claim 1, wherein continuing to display the image of the first physical object in the HMD as registered to the user'"'"'s body after the user has left the present location allows the user to take the first electronic communication with the user.

12. The method of claim 1, further comprising:

tracking, by the electronic system, actions of the user after the user has left the present location and the HMD is displaying the image of the first physical object registered to the user'"'"'s body, wherein the actions include the user manipulating the image of the first physical object;

determining, by the electronic system, that the user is intending to open the first physical object; and

providing, by the electronic system, content of the first electronic communication to the HMD in response to determining that the user is intending to open the first physical object.

13. A system comprising:

a head mounted display (HMD);

a processor in communication with the HMD; and

computer storage media having instructions stored thereon which, when executed on the processor, cause the processor to perform the following;

learn how a user interacts at a specific location with electronic communications from specific senders;

access electronic communications that are addressed to the user, each of the electronic communications having a sender, each of the electronic communications is one of a plurality of types of communication;

determine a first subset of the electronic communications that are addressed to the user that are to be presented to the user at a present location based on the learning and a second subset of the electronic communications that are addressed to the user that are not to be presented to the user at the present location based on the learning;

determine a plurality of physical objects that are representative of the electronic communications in the first subset, each of the physical objects identifies the corresponding electronic communication as one the types of communication;

display an image of the plurality of physical objects in the HMD, the image makes it appear as if the plurality of physical objects are registered to a surface in the present location other than the user'"'"'s body;

receive a selection of a first physical object of the plurality of physical objects whose image is to follow the user after the user has left the present location, wherein the first physical object corresponds to a first electronic communication in the first subset of the electronic communications;

display an image of the first physical object in the HMD registered to the user'"'"'s body in response to receiving the selection of the first physical object to follow the user after the user has left the present location; and

continue to display the image of the first physical object in the HMD registered to the user'"'"'s body after the user has left the present location.

14. The system of claim 13, wherein the system further includes an image capture device, and wherein the instructions further cause the processor to:

track actions of a the user while the user is wearing the HMD based on a depth image from the image capture device;

determine how the user is intending to interact with the first physical object that corresponds to the first electronic communication based on the actions; and

determine how to alter presentation of the first physical object on the HMD based on the actions in order accomplish the user'"'"'s intent to interact.

15. The system of claim 13, wherein the instructions further cause the processor to:

determine how to re-arrange the plurality of physical objects in response to a command to sort or index the plurality of physical objects; and

provide an updated image to the HMD to affect the re-arrangement of the plurality of physical objects.

16. Computer storage hardware having instructions stored thereon which, when executed on a processor, cause the processor to:

learn what type of electronic communication should be presented at a specific location based on a user'"'"'s behavior with respect to various types of electronic communication at that specific location;

access electronic communications having content and that are addressed to the user, each of the electronic communications is one of a plurality of types of communication;

determine a present location of the user, wherein the present location is the specific location;

based on the learning, determine a first subset of the electronic communications addressed to the user that are to be presented to the user at the present location and a second subset of the electronic communications addressed to the user that are not to be presented to the user at the present location;

generate images for representing the first subset of the electronic communications as physical objects that are representative of the respective electronic communications;

provide the images to a head mounted display (HMD) that make it appear as if the physical objects are on a surface in the present location other than the user'"'"'s body;

receive a first selection of a first physical object of the physical objects whose image is to follow the user, wherein the first physical object corresponds to a first electronic communication in the first subset of the electronic communications;

display an image of the first physical object in the HMD registered to the user'"'"'s body in response to receiving the first selection of the first physical object to follow the user; and

continue to display the image of the first physical object in the HMD registered to the user'"'"'s body after the user has left the present location;

receive a second selection of the first physical object after the user has left the present location; and

provide content of the first electronic communication to the HMD in response to the second selection.

17. The computer storage hardware of claim 16, wherein the instructions further cause the processor to:

track actions of the user while the user is wearing the HMD;

determine how the user is intending to interact with a second of the physical objects for a second of the electronic communications based on the actions;

update the images based on the actions; and

provide the updated images to the head mounted display (HMD).

18. The computer storage hardware of claim 16, wherein the instructions further cause the processor to:

receive a request to sort the electronic communications;

determine an arrangement for the physical objects to be displayed in the HMD in response to the request;

generate an additional image that contains the arrangement of the physical objects; and

provide the additional image to the HMD.

19. The computer storage hardware of claim 16, wherein the instructions that cause the processor to generate one or more images for representing the first subset of the electronic communications as physical objects that are representative of the electronic communications further cause the processor to:

determine how to render the first physical object in order to indicate a source of the first electronic communication.

20. The computer storage hardware of claim 16, wherein the instructions that cause the processor to learn further cause the processor:

learn that the user consistently ignores certain electronic communications at the specific location.