Hybrid optical character recognition

• US 9,305,227 B1
• Filed: 12/23/2013
• Issued: 04/05/2016
• Est. Priority Date: 12/23/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Nambiar, Rakesh Madhavan, Jahagirdar, Sonjeev, Benitez, Matias Omar Gregorio, Jia, Junxiong, Cole, Matthew Joseph, Ramos, David Paul
• Primary Examiner(s): Fitzpatrick, Atiba O
• Assistant Examiner(s): Potts, Ryan P
• Application Number:
  US14/139,752
• Time in Patent Office: 834 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 10/95 structured as a network, e.g. client-server architectures
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks

Abstract

Embodiments of the subject technology provide for a hybrid OCR approach which combines server and device side processing that can offset disadvantages of performing OCR solely on the server side or the device side. More specifically, the subject technology utilizes image characteristics such as glyph details and image quality measurements to opportunistically schedule OCR processing on the mobile device and/or server. In this regard, text extracted by a “faster” OCR engine (e.g., one with less latency) is displayed to a user, which is then updated by the result of a more accurate OCR engine (e.g., an OCR engine provided by the server). This approach allows factoring in additional parameters such as network latency and user preference for making scheduling decisions. Thus, the subject technology may provide significant gains in terms of reduced latency and increased accuracy by implementing one or more techniques associated with this hybrid OCR approach.

Claims

1. A computer-implemented method comprising:

generating an image at a mobile computing device using a camera;

determining features corresponding to a first word and a second word of text in the image;

generating, on the mobile computing device, mobile optical character recognition (OCR) data including mobile OCR results associated with the first word by performing OCR on the features associated with the first word;

determining an OCR latency of the mobile computing device;

determining an OCR accuracy of the mobile computing device;

sending the image to a remote device to perform remote OCR on the image;

causing the mobile OCR results to be displayed on the mobile computing device, including causing a first textual output associated with the first word to be displayed;

receiving remote OCR data, wherein the remote OCR data includes remote OCR results from the remote device associated with the second word;

determining differences between the mobile OCR data and the remote OCR data;

generating hybrid OCR results based on the differences by merging the mobile OCR data and the remote OCR data, including generating hybrid OCR results that include a second textual output associated with the second word and wherein the generating occurs based on at least one of the OCR latency being less than a threshold amount of time or the OCR accuracy being less than an accuracy threshold; and

causing the hybrid OCR results to be displayed on the mobile computing device, including causing the first textual output to be displayed with the second textual output.

2. The computer-implemented method of claim 1, wherein merging the mobile OCR data and the remote OCR data comprises adding the second textual output from the remote OCR results to the first textual output from the mobile OCR results to create a textual list of outputs to be displayed.

3. The computer-implemented method of claim 1, wherein merging the mobile OCR data and the remote OCR data comprises:

comparing a first mobile OCR result associated with the first word from the mobile OCR results to a second remote OCR result associated with the first word from the remote OCR results to determine letters from the second remote OCR result with higher confidence scores than letters of the first mobile OCR result; and

replacing the letters of the first mobile OCR result from the mobile OCR results with the letters of the second remote OCR result with the higher confidence scores to create an updated textual output associated with the first word to be displayed.

4. A system comprising:

at least one processor; and

a memory device including instructions that, when executed by the at least one processor, cause the at least one processor to;

receive an image at the system;

determine an OCR latency of the system;

determine an OCR accuracy of the system;

send the image to a remote device to perform remote optical character recognition (OCR) on the image;

generate mobile OCR data, wherein the mobile OCR data comprises mobile OCR results;

receive remote OCR data, wherein the remote OCR data includes remote OCR results from the remote device;

determining differences between the mobile OCR data and the remote OCR data;

generate hybrid OCR results based at least in part on;

the differences,

the mobile OCR data,

the remote OCR data, and

at least one of the OCR latency being less than a threshold amount of time or the OCR accuracy being less than an accuracy threshold; and

cause the hybrid OCR results to be displayed.

5. The system of claim 4, wherein to generate the OCR data, the instructions further cause the at least one processor to:

determine features corresponding to text in the image; and

perform OCR on the features of the image to provide the mobile OCR data including the mobile OCR results.

6. The system of claim 4, wherein merging the mobile OCR data and the remote OCR data comprises:

adding a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with a second word from the remote OCR results to create a list of OCR results to be displayed.

7. The system of claim 4, wherein merging the mobile OCR data and the remote OCR data comprises:

comparing a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with the first word from the remote OCR results to determine letters from the second remote OCR result with higher confidence scores than letters of the first mobile OCR result; and

replacing the letters of the first mobile OCR result from the mobile OCR results with the letters of the second remote OCR result with the higher confidence scores to create an updated OCR result associated with the first word to be displayed.

8. The system of claim 4, wherein merging the mobile OCR data and the remote OCR data comprises:

comparing a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with a second word from the remote OCR results to determine whether the second remote OCR result associated with the second word has a higher confidence score than the first mobile OCR result associated with the first word; and

replacing the first word of the first mobile OCR result from the mobile OCR results with the second word of the second remote OCR result with the higher confidence score to create an updated OCR result to be displayed.

9. The system of claim 4, wherein the memory device includes further instructions that cause the at least one processor to:

before sending the image to the remote device to perform OCR, determine whether operational factors are satisfied.

10. The system of claim 9, wherein to determine whether the operational factors are satisfied further causes the at least one processor to:

determine whether an indication of network strength is above a threshold amount of network strength at the system, wherein the indication of network strength represents an amount of signal strength from the system to a network;

responsive to determining that the indication of network strength is above the threshold amount of network strength, determine whether a battery level of the system is above a threshold amount of battery level;

responsive to determining that the battery level of the system is above the threshold amount of battery level, determine whether a user preference indicating a preference for solely performing OCR on the system or solely performing OCR on the remote device exists; and

responsive to determining that the user preference indicating the preference for solely performing OCR on the system or solely performing OCR on the remote device exists, indicate that the operational factors are satisfied.

11. The system of claim 10, wherein the instructions further cause the at least one processor to:

responsive to determining that the indication of network strength is below the threshold amount of network strength, perform OCR on the image at the system.

12. The system of claim 10, wherein the instructions further cause the at least one processor to:

responsive to determining that the battery level of the system is below the threshold amount of battery level, send the image to the remote device to perform OCR on the image.

13. A non-transitory computer-readable medium including instructions stored therein that, when executed by at least one computing device, cause the at least one computing device to:

receive an image at the at least one computing device;

determine an OCR latency of the at least one computing device;

determine an OCR accuracy of the at least one computing device;

send the image to a remote device to perform remote optical character recognition (OCR) on the image;

generate mobile OCR data, wherein the mobile OCR data comprises mobile OCR results;

receive remote OCR data, wherein the remote OCR data includes remote OCR results from the remote device;

determining differences between the mobile OCR data and the remote OCR data;

generate hybrid OCR results based at least in part on;

the differences,

the mobile OCR data,

the remote OCR data, and

at least one of the OCR latency being less than a threshold amount of time or the OCR accuracy being less than an accuracy threshold; and

cause the hybrid OCR results to be displayed.

14. The non-transitory computer-readable medium of claim 13, wherein to generate the OCR data, the instructions further cause the at least one computing device to:

determine features corresponding to text in the image; and

perform OCR on the features of the image to provide the mobile OCR data including the mobile OCR results.

15. The non-transitory computer-readable medium of claim 13, wherein merging the mobile OCR data and the remote OCR data comprises:

adding a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with a second word from the remote OCR results to create a list of OCR results to be displayed.

16. The non-transitory computer-readable medium of claim 13, wherein merging the mobile OCR data and the remote OCR data comprises:

comparing a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with the first word from the remote OCR results to determine letters from the second remote OCR result with higher confidence scores than letters of the first mobile OCR result; and

replacing the letters of the first mobile OCR result from the mobile OCR results with the letters of the second remote OCR result with the higher confidence scores to create an updated OCR result associated with the first word to be displayed.

17. The non-transitory computer-readable medium of claim 13, wherein merging the mobile OCR data and the remote OCR data comprises:

comparing a first mobile OCR result associated with a first word from the mobile OCR results to a second remote OCR result associated with a second word from the remote OCR results to determine whether the second remote OCR result associated with the second word has a higher confidence score than the first mobile OCR result associated with the first word; and

replacing the first word of the first mobile OCR result from the mobile OCR results with the second word of the second remote OCR result with the higher confidence score to create an updated OCR result to be displayed.

18. The non-transitory computer-readable medium of claim 13, wherein the instructions further cause the at least one computing device to:

before sending the image to the remote device to perform OCR, determine whether operational factors are satisfied.

19. The non-transitory computer-readable medium of claim 18, wherein to determine whether the operational factors are satisfied further causes the at least one computing device to:

determine whether an indication of network strength is above a threshold amount of network strength at the at least one computing device, wherein the indication of network strength represents an amount of signal strength from the at least one computing device to a network;

responsive to determining that the indication of network strength is above the threshold amount of network strength, determine whether a battery level of the at least one computing device is above a threshold amount of battery level;

responsive to determining that the battery level of the at least one computing device is above the threshold amount of battery level, determine whether a user preference indicating a preference for solely performing OCR on the at least one computing device or solely performing OCR on the remote device exists; and

responsive to determining that the user preference indicating the preference for solely performing OCR on the at least one computing device or solely performing OCR on the remote device exists, indicate that the operational factors are satisfied.

20. The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the at least one computing device to:

responsive to determining that the indication of network strength is below the threshold amount of network strength, perform OCR on the image at the at least one computing device.