Server-assisted object recognition and tracking for mobile devices

• US 9,424,255 B2
• Filed: 11/04/2011
• Issued: 08/23/2016
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Pengelly, Robert, Cosman, Stephen, Brown, Shawn M.
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Wang, Carol
• Application Number:
  US13/289,598
• Publication Number:
  US 20130114849A1
• Time in Patent Office: 1,754 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

Exemplary embodiments for performing server-assisted object recognition and tracking are disclosed herein. For example, in certain embodiments of the disclosed technology, one or more objects are efficiently recognized and tracked on a mobile device by using a remote server that can provide high capacity computing and storage resources. With the benefit of high-speed image processing on a remote server and high-bandwidth communication networks connecting the mobile device and the remote server, it is possible to identify an object and to track changes in the object'"'"'s characteristics or location, so that a user experiences seamless, real-time tracking.

Claims

1. A method, comprising:

capturing digital image data using a camera of a mobile device;

performing preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

tracking the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

receiving from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

refining tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

2. The method of claim 1, further comprising repeating the acts of capturing, transmitting, receiving, and tracking.

3. The method of claim 1, wherein the preliminary object detection comprises a coarse-grained detection of text objects in the image data or a speculative detection of areas of interest in the image data.

4. The method of claim 1, further comprising refining the preliminary object detection based at least in part on the received object data.

5. The method of claim 4, wherein the refining the preliminary object detection involves at least one of:

eliminating one or more objects recognized through the preliminary object detection; and

receiving a translated text object associated with text in the preliminary object detection data.

6. The method of claim 1, further comprising:

capturing one or more additional instances of digital image data using the camera; and

tracking changes in a location of the recognized object in the one or more additional instances of digital image data, the tracking being performed based at least in part on the object data received from the remote server.

7. The method of claim 6, wherein the tracking further comprises updating the data for the object reference frame for the one or more additional instances of digital image data;

and

wherein the method further comprises displaying the one or more additional instances of digital image data on the display of the mobile electronic device together with the updated object reference frame for the recognized object.

8. The method of claim 1, wherein the transmitting and the receiving reduces a computational burden on the mobile device relative to performing object recognition only on the mobile device itself.

9. The method of claim 1, further comprising transmitting sensor data to the remote server in addition to the at least a portion of the image data.

10. The method of claim 9, wherein the sensor data further comprises one or more of data from an accelerometer associated with the mobile device, data from a compass associated with the mobile device, data from a gyroscope associated with the mobile device, or data associated with global positioning system coordinates for the mobile device.

11. The method of claim 9, wherein the object data comprises data predicting a location of the recognized object based at least in part on the transmitted sensor data.

12. One or more computer-readable storage media storing computer-executable instructions which when executed by a computing device cause the computing device to perform a method for processing image data, the stored instructions comprising:

instructions to capture digital image data using a camera of a mobile device;

instructions to perform preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

instructions to track the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

instructions to receive from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

instructions to refine tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

13. A system, comprising:

a memory or storage device storing a program; and

at least one processor that executes the program, wherein said execution of the program causes the at least one processor to;

capture digital image data using a camera of a mobile device;

perform preliminary object detection on at least a portion of the image data to recognize an object and generate preliminary object detection data for the recognized object;

track the recognized object using the preliminary object detection data, while transmitting the at least a portion of the image data to a remote server for confirmation of the recognized object and refinement of the preliminary object detection data;

receive from the remote server object data for the recognized object in the transmitted image data, the object data including data for an object reference frame for rendering on a display of the mobile device; and

refine tracking of the recognized object based on the received object data and the preliminary object detection data, wherein the tracking commences on the mobile device using the preliminary object detection data and continues after receiving the object data from the remote server using both the preliminary object detection data and the received object data.

14. The system of claim 13, wherein said execution of the program causes the at least one processor to:

repeat the acts of capturing, transmitting, receiving, and tracking.

15. The system of claim 13, wherein the preliminary object detection comprises a coarse-grained detection of text objects in the image data or a speculative detection of areas of interest in the image data.

16. The system of claim 13, wherein said execution of the program causes the at least one processor to refine the preliminary object detection based at least in part on the received object data, wherein the refining the preliminary object detection involves at least one of:

eliminating one or more objects recognized through the preliminary object detection; and

receiving a translated text object associated with text in the preliminary object detection data.

17. The system of claim 13, wherein said execution of the program causes the at least one processor to:

capture one or more additional instances of digital image data using the camera; and

track changes in a location of the recognized object in the one or more additional instances of digital image data, the tracking being performed based at least in part on the object data received from the remote server.

18. The system of claim 17, wherein the tracking further causes the at least one processor to update the data for the object reference frame for the one or more additional instances of digital image data;

and

wherein said execution of the program causes the processing unit to displaying the one or more additional instances of digital image data on the display of the mobile electronic device together with the updated object reference frame for the recognized object.

19. The system of claim 13, wherein the transmitting and the receiving reduces a computational burden on the mobile device relative to performing object recognition only on the mobile device itself.

20. The system of claim 13, wherein said execution of the program causes the at least one processor to transmit sensor data to the remote server in addition to the at least a portion of the image data;

wherein the sensor data further comprises one or more of data from an accelerometer associated with the mobile device, data from a compass associated with the mobile device, data from a gyroscope associated with the mobile device, or data associated with global positioning system coordinates for the mobile device; and

wherein the object data comprises data predicting a location of the recognized object based at least in part on the transmitted sensor data.