Offloading augmented reality processing

• US 9,672,660 B2
• Filed: 02/11/2016
• Issued: 06/06/2017
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian
• Primary Examiner(s): DOAN, DUYEN MY
• Application Number:
  US15/041,437
• Publication Number:
  US 20160163112A1
• Time in Patent Office: 481 Days
• Field of Search: 709203, 709219
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A system and method for offloading augmented reality processing is described. A first sensor of a server generates a first set of sensor data corresponding to a location and an orientation of a display device. The server receives a request from the display device to offload a combination of at least one of a tracking process and a rendering process from the display device. The server generates offloaded processed data based on a combination of at least one of the first set of sensor data and a second set of sensor data. The second set of sensor data is generated by a second sensor at the display device. The server streams the offloaded processed data to the display device.

Claims

1. A server comprising:

a first sensor configured to generate a first set of sensor data corresponding to a location and an orientation of a display device;

a hardware processor configured to;

receive a request from the display device to offload a combination of at least one of a tracking process and a rendering process from the display device;

render a first number of virtual objects less than a total number of virtual objects associated with the location and the orientation of the display device, the display device being configured to render a second number of virtual objects less than the total number of virtual objects associated with the location and orientation of the display device;

generate offloaded processed data in response to the request, the offloaded processed data generated based on a combination of at least one of the first set of sensor data and a second set of sensor data, the second set of sensor data generated by a second sensor at the display device; and

stream the offloaded processed data to the display device, the display device configured to generate a visualization of the offloaded processed data.

2. The server of claim 1, wherein the first set of sensor data includes external tracking data based on the location and the orientation of the display device relative to the first sensor that is external to the display device, wherein the second set of sensor data includes internal tracking data based on a geographic location and a position of the display device.

3. The server of claim 2, wherein the processor is further configured to:

render virtual objects based on the external tracking data.

4. The server of claim 2, wherein the processor is further configured to:

render virtual objects based on the internal tracking data.

5. The server of claim 1, wherein the offloaded processed data include results from a combination of the tracking process and the rendering process.

6. The server of claim 1, wherein the first number of virtual objects is associated with a central portion of the display of the display device, and wherein the second number of virtual objects is associated with a peripheral portion of the display of the display device.

7. The server of claim 2, wherein the offloaded processed data includes the external tracking data, the display device is configured to render virtual objects based on the external tracking data received from the server, and the virtual objects displayed in the display of the display device are relative to predefined references of a physical environment local to the user.

8. The server of claim 4, wherein the display device is configured to adjust the visualization of the offloaded processed data using updated internal tracking data from the display device, the updated internal tracking data being more recent than the internal tracking data, a position of the virtual objects in the display of the display device being relative to predefined references of physical environment local to the user.

9. The server of claim 1, wherein the location includes a geographic location determined based on wireless data generated by the display device or triangulated from the predefined references of the physical environment, wherein the orientation is determined based on gyroscope data from the display device or is externally determined based on a three-dimensional camera sensor, and wherein the display device comprises the display in a mobile communication device hand held by the user or a transparent display mounted to a head of the user.

15. The computer-method of claim 1, wherein the first number of virtual objects is associated with a central portion of the display of the display device, and wherein the second number of virtual objects is associated with a peripheral portion of the display of the display device.

10. A computer-method comprising:

generating, with a first sensor of a server, a first set of sensor data corresponding to a location and an orientation of a display device, the first sensor external to the display device;

receiving, at the server, a request from the display device to offload a combination of at least one of a tracking process and a rendering process from the display device;

generating offloaded processed data in response to the request, the offloaded processed data generated based on a combination of at least one of the first set of sensor data and a second set of sensor data, the second set of sensor data generated by a second sensor at the display device;

streaming the offloaded processed data to the display device, the display device configured to generate a visualization of the offloaded processed data; and

rendering, at the server, a first number of virtual objects less than a total number of virtual objects associated with the location and the orientation of the display device, the display device being configured to render a second number of virtual objects less than the total number of virtual objects associated with the location and orientation of the display device.

11. The computer-method of claim 10, wherein the first set of sensor data includes external tracking data based on the location and the orientation of the display device relative to the first sensor that is external to the display device, wherein the second set of sensor data includes internal tracking data based on a geographic location and a position of the display device.

12. The computer-method of claim 11, further comprising:

rendering virtual objects based on the external tracking data.

13. The computer-method of claim 11, further comprising:

rendering virtual objects based on the internal tracking data.

14. The computer-method of claim 10, wherein the offloaded processed data include results from a combination of the tracking process and the rendering process.

16. The computer-method of claim 11, wherein the offloaded processed data includes the external tracking data, the display device is configured to render virtual objects based on the external tracking data received from the server, and the virtual objects displayed in the display of the display device are relative to predefined references of a physical environment local to the user.

17. The computer-method of claim 13, wherein the display device is configured to adjust the visualization of the offloaded processed data using updated internal tracking data from the display device, the updated internal tracking data being more recent than the internal tracking data, a position of the virtual objects in the display of the display device being relative to predefined references of physical environment local to the user.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

generating, with a first sensor of a server, a first set of sensor data corresponding to a location and an orientation of a display device, the first sensor external to the display device;

receiving, at the server, a request from the display device to offload a combination of at least one of a tracking process and a rendering process from the display device;

generating offloaded processed data in response to the request, the offloaded processed data generated based on a combination of at least one of the first set of sensor data and a second set of sensor data, the second set of sensor data generated by a second sensor at the display device;

streaming the offloaded processed data to the display device, the display device configured to generate a visualization of the offloaded processed data; and

rendering, at the server, a first number of virtual objects less than a total number of virtual objects associated with the location and the orientation of the display device, the display device being configured to render a second number of virtual objects less than the total number of virtual objects associated with the location and orientation of the display device.