User-to-user communication enhancement with augmented reality

• US 9,990,770 B2
• Filed: 12/20/2011
• Issued: 06/05/2018
• Est. Priority Date: 12/20/2011
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Anderson, Glen J.
• Primary Examiner(s): Chauhan, Ulka
• Assistant Examiner(s): Valdez, Patrick F
• Application Number:
  US13/977,552
• Publication Number:
  US 20130293584A1
• Time in Patent Office: 2,359 Days
• Field of Search: None
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 1/7243 with interactive means for internal management of messages
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/024 Guidance services

Abstract

The enhancement of user-to-user communication with augmented reality is described. In one example, a method includes receiving virtual object data at a local device from a remote user, generating a virtual object using the received virtual object data, receiving an image at the local device from a remote image store, augmenting the received image at the local device by adding the generated virtual object to the received image, and displaying the augmented received image on the local device.

Claims

1. A method comprising:

receiving virtual message object data at a local device from a remote user, the data describing a virtual vehicle that will virtually convey a message from the remote user that is sending the message to the local device that is receiving the message;

determining a real position of the remote user;

determining a real position of the local device;

generating the virtual vehicle and its association with the message at the local device as a virtual object using the received virtual message object data;

receiving an image at the local device, the image being of an area that includes the remote user position;

augmenting the received image at the local device by adding the generated virtual object to the received image at the remote user position;

displaying the augmented image on the local device;

determining the local device position on the augmented image; and

launching the conveying of the message by the virtual vehicle at the local device in response to a user command at the local device, the conveying corresponding to the virtual vehicle with the message virtually moving across the augmented image from the remote user position toward the local device position, the conveying beginning after launching.

2. The method of claim 1, wherein receiving virtual message object data comprises receiving location data regarding the remote user position from the remote user and wherein augmenting the received image comprises adding the generated virtual object at a location in the received image corresponding to the received location data.

3. The method of claim 2, wherein augmenting the received image comprises adding the generated virtual object at locations between the location corresponding to the received location data and the local device position over time and wherein displaying the augmented image comprises displaying the virtual object in motion across the augmented image between the location corresponding to the received location data and the local device position.

4. The method of claim 1, wherein receiving an image comprises receiving a plurality of images for locations between the remote user position and the local device position and wherein augmenting the received image comprises adding the generated virtual object to the received images to display the virtual object in motion across the images in a sequence corresponding to travel between the remote user position and the local device position.

5. The method of claim 4, wherein the travel between the remote user position and the local device position corresponds to a route selected by the remote user.

6. The method of claim 4, further comprising:

receiving an alert from the remote user to look for the virtual object;

receiving user input at the local device in response to the alert; and

modifying a course of travel between the remote user position and the local device position based on the user input.

7. The method of claim 6, wherein the user input comprises direction commands.

8. The method of claim 1, wherein the virtual object corresponds to travel of the message conveyed by the virtual vehicle as it is sent from the remote user to the local device.

9. The method of claim 8, further comprising receiving user input at the local device to interact with the virtual object and displaying the interaction on the augmented image on the local device.

10. The method of claim 8, further comprising receiving an alert from the remote user to look for the sent message as it is conveyed by the virtual vehicle.

11. The method of claim 10, further comprising receiving user input at the local device in response to the alert, the user input affecting the course of travel of the sent message conveyed by the virtual vehicle from the remote user position to the local device position.

12. The method of claim 10, further comprising receiving user input at the local device in response to the alert, the user input allowing the local device to virtually capture the virtual object corresponding to the sent message.

13. The method of claim 10, further comprising:

capturing a facial expression of a user of the local device after receiving the alert;

converting the facial expression to an avatar representing the facial expression; and

sending the avatar to the remote user.

14. The method of claim 13, further comprising sending the local device position to the remote user with the avatar.

15. The method of claim 13, further comprising sending an image of the local device position to the remote user with the avatar.

16. The method of claim 1, wherein generating a virtual object comprises generating audio and haptic effects for the virtual object and wherein displaying the augmented image comprises producing audio and haptic effects associated with the virtual object.

17. A local device comprising:

a data communication module to receive virtual message object data from a remote user and to receive an image at the local device, the data describing a virtual vehicle that will virtually convey a message from the remote user to the local device, the image being of an area that includes a real position of the remote user;

a combine module to generate the virtual vehicle and its association with the message at the local device as a virtual object using the received virtual message object data and to augment the received image at the local device by adding the generated virtual object to the received image;

a location determination system to determine the real position of the remote user, to determine a real position of the local device, and to determine the position of the local device on the augmented received image;

a screen rendering module to display the augmented image on the local device and to launch the conveying of the message by the virtual vehicle of the virtual object at the local device in response to a user command at the local device, the conveying corresponding to the virtual vehicle with the message virtually moving across the augmented image from the remote user position toward the local device position, the conveying beginning after launching.

18. The local device of claim 17, wherein the combine module adds the generated virtual object at locations between the remote user position and the local device position over time and wherein the augmented image shows the virtual object in virtual motion across the augmented image between the remote user position and the local device position.

19. The local device of claim 18, further comprising a user input system of the local device to receive direction commands and wherein the combine module modifies the motion of the virtual object based on the received direction commands.

20. The local device of claim 19, further comprising:

a camera to capture a facial expression of a user of the local device; and

an avatar representation module to generate an avatar representing the captured facial expression,

wherein the data communication module sends the generated avatar to the remote user.

21. A machine-readable non-transitory medium having instruction thereon that, when operated on by the machine cause the machine to perform operations comprising:

receiving virtual message object data at a local device from a remote user, the data describing a virtual vehicle that will virtually convey a message from the remote user to the local device;

determining a real position of the remote user;

determining a real Position of the local device;

generating the virtual vehicle and its association with the message at the local device as a virtual object using the received virtual message object data;

receiving an image at the local device, the image being of an area that includes the remote user position;

augmenting the received image at the local device by adding the generated virtual object to the received image at the remote user position;

displaying the augmented image on the local device;

determining the local device position on the augmented image; and

launching the conveying of the message by the virtual vehicle at the local device in response to a user command at the local device, the conveying corresponding to the virtual vehicle with the message virtually moving across the augmented image from the remote user position toward the local device position, the conveying beginning after launching.

22. An apparatus comprising:

a communications interface to receive virtual message object data from a remote user and to receive an image at a local device, the data describing a virtual vehicle that will virtually convey a message from the remote user to the local device, the image being of an area that includes a real position of the remote user;

a processor having a combine module to generate a virtual vehicle as a virtual object and its association with the message using the received virtual message object data and to augment the received image at the local device by adding the generated virtual object to the received image;

a positioning system receiver to determine a position of the local device;

a display to display the augmented image on the local device; and

an interface to present a launch command to launch conveying the message by the virtual vehicle at the local device, the conveying corresponding to the virtual vehicle with the message virtually moving across the augmented image from the remote user position toward the local device position, the conveying beginning after launching.

23. The apparatus of claim 22, wherein the processor augments the received image by adding the generated virtual object at locations between the remote user position and the local device position over time and wherein the processor renders the virtual object in motion across the received image between the remote user position and the local device position.

24. The apparatus of claim 23, further comprising a camera to capture a facial expression of a user of the local device after receiving an alert and wherein the processor converts the facial expression to an avatar representing the facial expression and the communications interface sends the avatar to the remote user.