AUGMENTED REALITY OVERLAYS USING POSITION AND ORIENTATION TO FACILITATE INTERACTIONS BETWEEN ELECTRONIC DEVICES

• US 20150193982A1
• Filed: 01/02/2015
• Published: 07/09/2015
• Est. Priority Date: 01/03/2014
• Status: Abandoned Application

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mihelich, Patrick, Lee, Johnny
• Application Number:
  US14/588,515
• Publication Number:
  US 20150193982A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

A system includes a first electronic device and a second electronic device. The first electronic device is to display a first augmented reality overlay on imagery of a local environment captured by the first electronic device, the first augmented reality overlay including a depiction of a virtual object from a first perspective that is based on a position and orientation of the first electronic device using a three-dimensional mapping of the local environment. The second electronic device is to display a second augmented reality overlay on imagery of the local environment captured by the second electronic device, the second augmented reality overlay including a depiction of the virtual object from a perspective based on a position and orientation of the second electronic device using the three-dimensional mapping.

Claims

1. A method comprising:

determining a position and orientation of a first electronic device relative to a second electronic device based on a three-dimensional mapping of an environment of the first and second electronic devices; and

controlling an interaction between the first and second electronic devices based on the position and orientation.

2. The method of claim 1, wherein controlling an interaction between the first and second electronic devices comprises:

displaying a first augmented reality overlay at the first electronic device, the first augmented reality overlay displaying a first perspective of a virtual object in the environment based on a position and orientation of the first electronic device; and

displaying a second augmented reality overlay at the second electronic device, the second augmented reality overlay displaying a second perspective of the virtual object based on a position and orientation of the second electronic device.

3. The method of claim 2, wherein controlling an interaction between the first and second electronic devices further comprises:

receiving, at the first electronic device, user input representing a manipulation of the virtual object; and

modifying a depiction of the virtual object in the second augmented reality overlay at the second electronic device in response to the manipulation of the virtual object at the first electronic device.

4. The method of claim 3, wherein:

the manipulation of the virtual object comprises an insertion of the virtual object into the first augmented reality overlay at the first electronic device; and

modifying the depiction of the virtual object in the second augmented reality overlay comprises initiating display of the virtual object in the second augmented reality overlay.

5. The method of claim 3, further comprising:

communicating first data representing the manipulation of the virtual object from the first electronic device to a remote processing system; and

communicating a representation of the first data from the remote processing system to the second electronic device;

wherein modifying the depiction of the virtual object in the second augmented reality overlay comprises modifying the depiction of the virtual object based on the representation of the first data.

6. The method of claim 1, wherein controlling an interaction between the first and second electronic devices comprises:

receiving user input at the first electronic device, the user input representing initiation of motion of a virtual object displayed in a first augmented reality overlay at the first electronic device; and

in response to the user input;

identifying the second electronic device based on the motion of the virtual object and the position and orientation of the first electronic device relative to the second electronic device; and

controlling the first augmented reality overlay at the first electronic device and a second augmented reality overlay at the second electronic device to depict a transfer of the virtual object from the first electronic device to the second electronic device.

7. The method of claim 1, wherein controlling an interaction between the first and second electronic devices comprises:

receiving user input at the first electronic device, the user input representing a manipulation of a virtual object displayed in an augmented reality overlay at the first electronic device; and

communicating data representative the manipulation of the virtual object to a remote processing device for distribution to the second electronic device.

8. The method of claim 1, wherein controlling an interaction between the first and second electronic devices comprises:

receiving, at the first electronic device, data representing a manipulation of a virtual object displayed in an augmented reality overlay at the second electronic device; and

displaying, at the first electronic device, a representation of the virtual object in an augmented reality overlay based on the manipulation of the virtual object and based on the position and orientation of the first electronic device relative to the second electronic device.

9. The method of claim 1, wherein determining the position and orientation of the first electronic device relative to the second electronic device comprises:

determining at least one of an orientation and a position of the first electronic device in the environment based on one of;

depth information from a depth sensor; and

multiview analysis of imagery from a plurality of imaging sensors.

10. The method of claim 9, wherein determining the position and orientation of the first electronic device relative to the second electronic device further comprises:

determining at least one of an orientation or a position of the second electronic device in the environment based on one of;

depth information from a depth sensor; and

multiview analysis of imagery from a plurality of imaging sensors.

11. A method comprising:

displaying, at a first time at a first electronic device, a first augmented reality overlay on imagery of a local environment captured by the first electronic device, the first augmented reality overlay including a depiction of a virtual object from a first perspective that is based on a position and orientation of the first electronic device using a three-dimensional mapping of the local environment; and

displaying, at a second time at a second electronic device, a second augmented reality overlay on imagery of the local environment captured by the second electronic device, the second augmented reality overlay including a depiction of the virtual object from a perspective based on a position and orientation of the second electronic device using the three-dimensional mapping.

12. The method of claim 11, wherein the first time and the second time are concurrent.

13. The method of claim 12, further comprising:

transmitting first data from the first electronic device to a remote processing system, the first data representing a user manipulation of the virtual object at the first electronic device; and

receiving second data from the remote processing system at the second electronic device, the second data representative of the user manipulation of the virtual object; and

wherein the second augmented reality overlay comprises a depiction of the user manipulation of the virtual object from the perspective based on the position and orientation of the second electronic device.

14. The method of claim 11, wherein the second time is offset from the first time.

15. The method of claim 14, further comprising:

transmitting first data from the first electronic device to a remote processing system at the first time, the first data representing a user manipulation of the virtual object at the first electronic device;

storing the first data at the remote processing system;

subsequently accessing, at the second electronic device, a representation of the first data from the remote processing system at the second time; and

wherein the second augmented reality overlay comprises a depiction of the user manipulation of the virtual object from the perspective based on the position and orientation of the second electronic device.

16. The method of claim 11, further comprising:

separately determining the three-dimensional mapping of the local environment at each of the first electronic device and the second electronic device using visual telemetry.

17. The method of claim 11, further comprising:

receiving, at the first electronic device, data representative of the three-dimensional mapping of the local environment; and

determining at least one of the position and the orientation of the first electronic device in the local environment based on the three-dimensional mapping and based on visual telemetry at the first electronic device.

18. The method of claim 17, wherein the visual telemetry uses at least one of:

depth sensing using a depth sensor; and

multiview analysis using at least two imaging sensors.

19. An electronic device comprising:

a position/orientation tracking module to determine a position and orientation of the electronic device within a local environment based on a three-dimensional mapping of the local environment; and

a location-based services module coupled to the position/orientation tracking module, the location-based services module to provide an augmented reality overlay on imagery of the local environment captured by the electronic device, the augmented reality overlay including a depiction of a virtual object from a perspective that is based on the position and orientation of the electronic device.

20. The electronic device of claim 19, wherein the position/orientation tracking module is to determine at least one of the position and the orientation of the electronic device based on visual telemetry information.

21. The electronic device of claim 20, wherein the electronic device comprises:

a depth sensor coupled to the position/orientation tracking module, wherein the position/orientation tracking module is to determine the visual telemetry information using depth information from the depth sensor.

22. The electronic device of claim 20, wherein the electronic device comprises:

a plurality of imaging sensors coupled to the position/orientation tracking module, wherein the position/orientation tracking module is to determine the visual telemetry information based on multiview analysis of image information from the plurality of imaging sensors.

23. The electronic device of claim 19, further comprising:

an interface connected to a remote processing system; and

wherein the location-based services module is to receive data representing a manipulation of the virtual object by a user of another electronic device and to modify the depiction of the virtual object by the augmented reality overlay based on the manipulation of the virtual object.

24. The electronic device of claim 19, further comprising:

a first interface to couple to a remote processing system;

an interface to receive input representative of a manipulation of the virtual object by a user of the electronic device; and

wherein the location-based services module is to provide to the first interface data representative of the manipulation of the virtual object for transmission to the remote processing system for distribution to at least one other electronic device.

25. A system comprising:

a first electronic device to display a first augmented reality overlay on imagery of a local environment captured by the first electronic device, the first augmented reality overlay including a depiction of a virtual object from a first perspective that is based on a position and orientation of the first electronic device using a three-dimensional mapping of the local environment; and

a second electronic device to display a second augmented reality overlay on imagery of the local environment captured by the second electronic device, the second augmented reality overlay including a depiction of the virtual object from a perspective based on a position and orientation of the second electronic device using the three-dimensional mapping.

26. The system of claim 25, further comprising:

a remote processing system coupled to the first and second electronic devices, the remote processing system to communicate data representative of a manipulation of the virtual object by a user of the first electronic device from the first electronic device to the second electronic device; and

wherein the second electronic device is to modify the depiction of the virtual object in the second augmented reality overlay based on the data representative of the manipulation of the virtual object.

27. The system of claim 25, wherein at least one of the first electronic device and the second electronic device is to determine at least one of a position and an orientation using visual telemetry.

28. The system of claim 27, wherein the visual telemetry uses at least one of:

depth sensing using a depth sensor; and

multiview analysis using at least two imaging sensors.

29. The system of claim 25, wherein the first and second augmented reality overlays are displayed concurrently.

30. The system of claim 25, wherein the first and second augmented reality overlays are displayed at separate times.