Adaptive Cloud Offloading of Mobile Augmented Reality

• US 20190197339A1
• Filed: 12/21/2017
• Published: 06/27/2019
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Han, Bo, Gopalakrishnan, Vijay, Hao, Shuai
• Granted Patent: US 10,565,464 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 9/45512 Command shells
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4424 Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04
• CPC Class Codes: H04N 21/44245 Monitoring the upstream path of the transmission network, e.g. its availability, bandwidth
• CPC Class Codes: H04N 21/6377 directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

Concepts and technologies for adaptive cloud offloading of mobile augmented reality are provided herein. In an embodiment, a method can include receiving, by an augmented reality system, an acquired image frame captured by an acquisition device. The acquired image frame can indicate a plurality of acquired image frame parameters. The method can include determining, by the augmented reality system, a plurality of augmented reality process instances. The method can include determining a plurality of local feature extraction time estimates based on the plurality of acquired image frame parameters, where a local feature extraction time estimate is created for each of the plurality of the augmented reality process instances. The method can include obtaining a network condition measurement, and generating a plurality of offload commands based on the network condition measurement and at least one of the plurality of local feature extraction time estimates.

Claims

1. An augmented reality system comprising:

a processor; and

a memory that stores computer-executable instructions that, in response to execution by the processor, cause the processor to perform operations comprising;

receiving an acquired image frame captured by an acquisition device, wherein the acquired image frame indicates a plurality of acquired image frame parameters,

determining a plurality of augmented reality process instances corresponding to different process modules that can be executed locally on the augmented reality system or can be executed remotely via an augmented reality cloud server to provide feature point extraction for the acquired image frame,

determining a plurality of local feature extraction time estimates for the plurality of augmented reality process instances based on the plurality of acquired image frame parameters, wherein a respective local feature extraction time estimate is determined for each of the plurality of augmented reality process instances, wherein each of the plurality of local feature extraction time estimates provides an estimated amount of time for locally executing a respective augmented reality process instance on the augmented reality system to extract feature points from the acquired image frame,

obtaining a network condition measurement,

generating an offload command for each of the plurality of augmented reality process instances such that a plurality of offload commands are generated for the acquired image frame, wherein generating each of the plurality of offload commands is based on the network condition measurement and at least one of the plurality of local feature extraction time estimates, and

selecting one of the plurality of offload commands to activate a respective augmented reality process instance.

2. The augmented reality system of claim 1, wherein generating the plurality of offload commands is based on determining whether a dynamic acquired image frame parameter associated with the acquired image frame exceeds an offload time differential threshold for each of the plurality of augmented reality process instances, and wherein determining the plurality of augmented reality process instances comprises retrieving augmented reality process identifiers corresponding to the plurality of augmented reality process instances.

3. The augmented reality system of claim 2, wherein each offload command is associated with a respective one of the plurality of augmented reality process instances.

4. The augmented reality system of claim 3, wherein each offload command includes one of a command to offload the acquired image frame as a raw file to a cloud server, or a command to offload feature points from the acquired image frame to the cloud server.

5. The augmented reality system of claim 4, wherein the command to offload the feature points from the acquired image frame to the cloud server is based on determining that the dynamic acquired image frame parameter exceeds the offload time differential threshold corresponding to one of the plurality of augmented reality process instances.

6. The augmented reality system of claim 4, wherein the feature points are generated from the acquired image frame via implementing an augmented reality process instance of the plurality of augmented reality process instances associated with one of the augmented reality process identifiers.

7. The augmented reality system of claim 2, wherein the operations further comprise activating one of the plurality of offload commands for the acquired image frame.

8. A method comprising:

receiving, by an augmented reality system, an acquired image frame captured by an acquisition device, wherein the acquired image frame indicates a plurality of acquired image frame parameters;

determining, by the augmented reality system, a plurality of augmented reality process instances corresponding to different process modules that can be executed locally on the augmented reality system or can be executed remotely via an augmented reality cloud server to provide feature point extraction for the acquired image frame;

determining, by the augmented reality system, a plurality of local feature extraction time estimates for the plurality of augmented reality process instances based on the plurality of acquired image frame parameters, wherein a local feature extraction time estimate is determined for each of the plurality of augmented reality process instances, wherein each of the plurality of local feature extraction time estimates provides an estimated amount of time for locally executing a respective augmented reality process instance on the augmented reality system to extract feature points from the acquired image frame;

obtaining, by the augmented reality system, a network condition measurement;

generating, by the augmented reality system, an offload command for each of the plurality of augmented reality process instances such that a plurality of offload commands are generated for the acquired image frame, wherein generating each of the plurality of offload commands is based on the network condition measurement and at least one of the plurality of local feature extraction time estimates; and

selecting, by the augmented reality system, one of the plurality of offload commands to activate a respective augmented reality process instance.

9. The method of claim 8, wherein generating the plurality of offload commands is based on determining whether a dynamic acquired image frame parameter associated with the acquired image frame exceeds an offload time differential threshold for each of the plurality of augmented reality process instances, and wherein determining the plurality of augmented reality process instances comprises retrieving augmented reality process identifiers corresponding to the plurality of augmented reality process instances.

10. The method of claim 9, wherein each offload command is associated with a respective one of the plurality of augmented reality process instances.

11. The method of claim 10, wherein each offload command includes one of a command to offload the acquired image frame as a raw file to a cloud server, or a command to offload feature points from the acquired image frame to the cloud server.

12. The method of claim 11, wherein the command to offload the feature points from the acquired image frame to the cloud server is based on determining that the dynamic acquired image frame parameter exceeds the offload time differential threshold corresponding to one of the plurality of augmented reality process instances.

13. The method of claim 12, wherein the feature points are generated from the acquired image frame via implementing an augmented reality process instance associated with one of the augmented reality process identifiers.

14. The method of claim 9, further comprising activating, by the augmented reality system, one of the plurality of offload commands for the acquired image frame.

15. A computer storage medium having computer-executable instructions stored thereon that, in response to execution by a processor, cause the processor to perform operations comprising:

receiving an acquired image frame captured by an acquisition device, wherein the acquired image frame indicates a plurality of acquired image frame parameters;

determining a plurality of augmented reality process instances corresponding to different process modules that can be executed locally on the augmented reality system or can be executed remotely via an augmented reality cloud server to provide feature point extraction for the acquired image frame;

determining a plurality of local feature extraction time estimates for the plurality of augmented reality process instances based on the plurality of acquired image frame parameters, wherein a respective local feature extraction time estimate is determined for each of the plurality of augmented reality process instances, wherein each of the plurality of local feature extraction time estimates provides an estimated amount of time for locally executing a respective augmented reality process instance on the augmented reality system to extract feature points from the acquired image frame;

obtaining a network condition measurement;

generating an offload command for each of the plurality of augmented reality process instances such that a plurality of offload commands are generated for the acquired image frame, wherein generating each of the plurality of offload commands is based on the network condition measurement and at least one of the plurality of local feature extraction time estimates; and

selecting one of the plurality of offload commands to activate a respective augmented reality process instance.

16. The computer storage medium of claim 15, wherein generating the plurality of offload commands is based on determining whether a dynamic acquired image frame parameter associated with the acquired image frame exceeds an offload time differential threshold for each of the plurality of augmented reality process instances, and wherein determining the plurality of augmented reality process instances comprises retrieving augmented reality process identifiers corresponding to the plurality of augmented reality process instances.

17. The computer storage medium of claim 16, wherein each offload command is associated with a respective one of the plurality of augmented reality process instances.

18. The computer storage medium of claim 17, wherein each offload command includes one of a command to offload the acquired image frame as a raw file to a cloud server, or a command to offload feature points from the acquired image frame to the cloud server.

19. The computer storage medium of claim 18, wherein the command to offload the feature points from the acquired image frame to the cloud server is based on determining that the dynamic acquired image frame parameter exceeds the offload time differential threshold corresponding to one of the plurality of augmented reality process instances.

20. The computer storage medium of claim 16, wherein the operations further comprise activating one of the plurality of offload commands for the acquired image frame.