Adaptable Framework for Cloud Assisted Augmented Reality

• US 20120243732A1
• Filed: 09/19/2011
• Published: 09/27/2012
• Est. Priority Date: 09/20/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Swaminathan, Ashwin, Jiang, Bolan, Chari, Murali R., Diaz Spindola, Serafin, Baheti, Pawan Kumar, Narayanan, Vidya, Sharma, Piyush
• Granted Patent: US 9,495,760 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A mobile platform efficiently processes sensor data, including image data, using distributed processing in which latency sensitive operations are performed on the mobile platform, while latency insensitive, but computationally intensive operations are performed on a remote server. The mobile platform acquires sensor data, such as image data, and determines whether there is a trigger event to transmit the sensor data to the server. The trigger event may be a change in the sensor data relative to previously acquired sensor data, e.g., a scene change in an image. When a change is present, the sensor data may be transmitted to the server for processing. The server processes the sensor data and returns information related to the sensor data, such as identification of an object in an image or a reference image or model. The mobile platform may then perform reference based tracking using the identified object or reference image or model.

Claims

1. A method comprising:

acquiring sensor data using a mobile platform;

determining whether there is a trigger event comprising a change in the sensor data relative to previously acquired sensor data;

transmitting the sensor data to a server when there is the trigger event; and

receiving information related to the sensor data from the server.

2. The method of claim 1, wherein the sensor data comprises a captured image of an object.

3. The method of claim 2, the method further comprising determining a quality of the captured image before transmitting the sensor data to the server, wherein the sensor data is transmitted to the server only if the quality of the captured image is better than a threshold.

4. The method of claim 3, wherein determining the quality of the captured image comprises at least one of analyzing a degree of sharpness of the captured image, analyzing a number of detected corners in the captured image;

and using statistics derived from the image with a learning classifier.

5. The method of claim 2, further comprising rendering augmentation with respect to the object based on the information related to the sensor data received from the server.

6. The method of claim 2, wherein the information related to the sensor data comprises an identification of the object.

7. The method of claim 2, wherein the captured image comprises a plurality of objects and the information related to the sensor data comprises an identification of the plurality of objects.

8. The method of claim 7, further comprising:

obtaining poses for each of the plurality of objects with respect to the mobile platform; and

tracking each of the plurality of objects using the poses and the information related to the sensor data.

9. The method of claim 2, further comprising:

obtaining a pose of the mobile platform with respect to the object; and

tracking the object using the pose and the information related to the sensor data.

10. The method of claim 9, wherein the information related to the sensor data comprises a reference image of the object, and wherein obtaining the pose comprises receiving from the server a first pose based on the captured image and the reference image.

11. The method of claim 10, further comprising performing reference-free tracking of the object until the first pose is received from the server.

12. The method of claim 10, further comprising:

acquiring a second captured image of the object when the first pose is received from the server;

tracking the object between the captured image and the second captured image to determine an incremental change; and

using the incremental change and the first pose to obtain the pose of the mobile platform with respect to the object.

13. The method of claim 10, further comprising:

acquiring a second captured image of the object;

detecting the object in the second captured image using the reference image;

using the object detected in the second captured image and the reference image to obtain the pose of the mobile platform with respect to the object; and

using the pose to initialize reference based tracking of the object.

14. The method of claim 2, wherein the information related to the sensor data comprises at least one of the following:

a two dimensional (2D) model of the object, a three dimensional (3D) model of the object, augmentation information, saliency information about the object, and information related to object matching.

15. The method of claim 2, wherein determining whether there is the trigger event comprises determining whether there is a scene change in the captured image with respect to a previous captured image.

16. The method of claim 15, wherein determining whether there is the scene change comprises:

determining a first change metric using the captured image and the previous captured image;

determining a second change metric using the captured image and a second previous captured image from a previous trigger event;

generating a histogram change metric for the captured image; and

using the first change metric, the second change metric and the histogram change metric to determine the scene change.

17. The method of claim 2, wherein the information related to the sensor data comprises an object identification, the method further comprising:

acquiring additional captured images of the object;

identifying the object in the additional captured images using the object identification;

generating a tracking mask for the additional captured images based on the object identification, the tracking mask indicating regions in the additional captured images where the object is identified;

using the tracking mask with the additional captured images of the object to identify remaining regions of the additional captured images; and

detecting trigger events comprising scene changes in the remaining regions of the additional captured images.

18. The method of claim 1, wherein the sensor data includes one or more of the following:

image data, motion sensor data, position data, barcode recognition, text detection results, and contextual information.

19. The method of claim 18, wherein the contextual information includes one or more of the following:

user behavior, user preferences, location, information about the user, time of day, and lighting quality.

20. The method of claim 1, wherein the sensor data comprises an image of a face and the information received from the server comprises an identity associated with the face.

21. The method of claim 1, wherein the sensor data comprises a plurality of images of an object captured with a camera at different positions and a coarse estimate of a pose of the camera with respect to the object, and the information received from the server comprises at least one of a refinement of the pose and a three-dimensional model of the object.

22. A mobile platform comprising:

a sensor adapted to acquire sensor data;

a wireless transceiver;

a processor coupled to the sensor and the wireless transceiver, the processor adapted to acquire sensor data via the sensor, to determine whether there is a trigger event comprising a change in the sensor data relative to previously acquired sensor data, to transmit via the wireless transceiver the sensor data to an external processor when the trigger event is present, and to receive information related to the sensor data from the external processor via the wireless transceiver.

23. The mobile platform of claim 22, wherein the sensor is a camera and the sensor data comprises a captured image of an object.

24. The mobile platform of claim 23, wherein the processor is further adapted to determine a quality of the captured image before the sensor data is transmitted to the external processor, wherein the sensor data is transmitted to the external processor only if the quality of the captured image is better than a threshold.

25. The mobile platform of claim 24, wherein the processor is adapted to determine the quality of the captured image by being adapted to perform at least one of a degree of sharpness analysis of the captured image, an analysis of a number of detected corners in the captured image;

and processing of a learning classifier with statistics derived from the image.

26. The mobile platform of claim 23, wherein the processor is further adapted to render augmentation with respect to the object based on the information related to the sensor data received via the wireless transceiver.

27. The mobile platform of claim 23, wherein the information related to the sensor data comprises an identification of the object.

28. The mobile platform of claim 23, wherein the captured image comprises a plurality of objects and the information related to the sensor data comprises an identification of the plurality of objects.

29. The mobile platform of claim 28, wherein the processor is further adapted to obtain poses for each of the plurality of objects with respect to the mobile platform, and track each of the plurality of objects using the poses and the information related to the sensor data.

30. The mobile platform of claim 23, wherein the processor is further adapted to obtain a pose of the mobile platform with respect to the object and to track the object using the pose and the information related to the sensor data.

31. The mobile platform of claim 30, wherein the information related to the sensor data comprises a reference image of the object, and wherein the processor is adapted to receive from the external processor a first pose based on the captured image and the reference image.

32. The mobile platform of claim 31, wherein the processor is further adapted to perform reference-free tracking of the object until the first pose is received from the server.

33. The mobile platform of claim 31, wherein the processor is further adapted to acquire a second captured image of the object when the first pose is received from the external processor, to track the object between the captured image and the second captured image to determine an incremental change, to use the incremental change and the first pose to obtain the pose of the mobile platform with respect to the object.

34. The mobile platform of claim 31, wherein the processor is further adapted to acquire a second captured image of the object, to detect the object in the second captured image using the reference image, to use the object detected in the second captured image and the reference image to obtain the pose of the mobile platform with respect to the object, and to use the pose to initialize reference based tracking of the object.

35. The mobile platform of claim 23, wherein the processor is further adapted to receive from the external processor via the wireless transmitter at least one of the following:

a two dimensional (2D) model of the object, a three dimensional (3D) model of the object, augmentation information, saliency information about the object, and information related to object matching.

36. The mobile platform of claim 23, wherein the processor is adapted to determine whether there is the trigger event by being adapted to determine whether there is a scene change in the captured image with respect to a previous captured image.

37. The mobile platform of claim 36, wherein the processor is adapted to determine whether there is the scene change by being adapted to determine a first change metric using the captured image and the previous captured image, to determine a second change metric using the captured image and a second previous captured image from a previous trigger event, to generate a histogram change metric for the captured image, and to use the first change metric, the second change metric and the histogram change metric to determine the scene change.

38. The mobile platform of claim 23, wherein the information related to the sensor data comprises an object identification, wherein the processor is further adapted to acquire additional captured images of the object, to identify the object in the additional captured images using the object identification, to generate a tracking mask for the additional captured images based on the object identification, the tracking mask indicating regions in the additional captured images where the object is identified, to use the tracking mask with the additional captured images of the object to identify remaining regions of the additional captured images, and to detect trigger events comprising scene changes in the remaining regions of the additional captured images.

39. The mobile platform of claim 22, wherein the sensor data includes one or more of the following:

image data, motion sensor data, position data, barcode recognition, text detection results, and contextual information.

40. The mobile platform of claim 39, wherein the contextual information includes one or more of the following:

user behavior, user preferences, location, information about the user, time of day, and lighting quality.

41. The mobile platform of claim 22, wherein the sensor comprises a camera and the sensor data comprises an image of a face and the information received via the wireless transceiver comprises an identity associated with the face.

42. The mobile platform of claim 22, wherein the sensor comprises a camera and the sensor data comprises a plurality of images of an object captured with the camera at different positions and a coarse estimate of a pose of the camera with respect to the object, and the information received from the server comprises at least one of a refinement of the pose and a three-dimensional model of the object.

43. A mobile platform comprising:

means for acquiring sensor data;

means for determining whether there is a trigger event comprising a change in the sensor data relative to previously acquired sensor data;

means for transmitting the sensor data to a server when there is the trigger event; and

means for receiving information related to the sensor data from the server.

44. The mobile platform of claim 43, wherein the means for acquiring sensor data is a camera and the sensor data is a captured image of an object and wherein the information related to the sensor data comprises a reference image of the object, the mobile platform further comprising:

means for obtaining a pose of the mobile platform with respect to the object; and

means for tracking the object using the pose and the reference image of the object.

45. The mobile platform of claim 43, wherein the means for acquiring sensor data is a camera and the sensor data is a captured image of an object, wherein the means for determining whether there is the trigger event comprises means for determining whether there is a scene change in the captured image with respect to a previous captured image.

46. A non-transitory computer-readable medium including program code stored thereon, comprising:

program code to acquire sensor data;

program code to determine whether there is a trigger event comprising a change in the sensor data relative to previously acquired sensor data;

program code to transmit the sensor data to an external processor when the trigger event is present, and

program code to receive information related to the sensor data from the external processor.

47. The non-transitory computer-readable medium of claim 46, wherein the sensor data is a captured image of an object and information related to the sensor data comprises a reference image of the object, the non-transitory computer-readable medium further comprising:

program code to obtain a pose with respect to the object; and

program code to track the object using the pose and the reference image of the object.

48. The non-transitory computer-readable medium of claim 46, wherein the sensor data is a captured image of an object and wherein the program code to determine whether there is the trigger event comprises program code to determine whether there is a scene change in the captured image with respect to a previous captured image.