Optical receipt processing

• US 10,229,314 B1
• Filed: 09/30/2016
• Issued: 03/12/2019
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Groupon Incorporated
• Original Assignee: Groupon Incorporated
• Inventors: Mitchell, Stephen Clark, Melnichuk, Pavel
• Primary Examiner(s): Niu, Feng
• Application Number:
  US15/281,517
• Time in Patent Office: 893 Days
• Field of Search: None
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06Q 20/047 using payment protocols involving electronic receipts
• CPC Class Codes: G06V 2201/09 Recognition of logos
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/19113 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Techniques for providing improved optical character recognition (OCR) for receipts are discussed herein. Some embodiments may provide for a system including one or more servers configured to perform receipt image cleanup, logo identification, and text extraction. The image cleanup may include transforming image data of the receipt by using image parameters values that optimize the logo identification, and performing logo identification using a comparison of the image data with training logos associated with merchants. When a merchant is identified, a second image clean up may be performed by using image parameter values optimized for text extraction. A receipt structure may be used to categorize the extracted text. Improved OCR accuracy is also achieved by applying on format rules of the receipt structure to the extracted text.

Claims

1. A system, comprising:

one or more servers configured to;

receive a first transformed image of a receipt from a consumer device, wherein the first transformed image is created by programmatically transforming image data captured by a camera using first image parameter values optimized for logo detection;

determine a training logo associated with a merchant;

perform a logo identification based on determining whether the first transformed image includes a logo corresponding with the training logo;

subsequent to determining that the first transformed image includes a logo that corresponds with the training logo associated with the merchant;

determine second image parameter values optimized for text extraction;

create a second transformed image by programmatically transforming the image data using the second image parameter values; and

perform a text extraction using the second transformed image to create receipt text data.

2. The system of claim 1, wherein the one or more servers are further configured to determine the second image parameter values optimized for the text extraction based on the one or more servers being configured to:

generate test images of a second receipt using different combinations of image parameter values;

determine keywords of the second receipt;

determine text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determine the second image parameter values based on the text extraction scores.

3. The system of claim 1, wherein the one or more servers are further configured to:

determine whether the receipt text data has been successfully created based on the text extraction; and

in response to determining that the receipt text data fails to be successfully created based on the text extraction, send a request for a third transformed image of the receipt to the consumer device, wherein the third transformed image is created by programmatically transforming second image data captured by the camera.

4. The system of claim 1 further comprising:

the consumer device configured to;

create the first transformed image by programmatically transforming the image data using the first image parameter values; and

send the first transformed image to the one or more servers.

5. The system of claim 1, wherein the one or more servers are further configured to determine the first image parameter values optimized for logo detection based on being configured to:

generate test images of a second receipt using different combinations of image parameter values;

determine keywords of the second receipt;

determine text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determine the second image parameter values based on the text extraction scores.

6. The system of claim 1, wherein the one or more servers are further configured to, in response to determining that the first transformed image fails to include a logo corresponding with the training logo, send a request for a third transformed image of the receipt to the consumer device, wherein the third transformed image is created by programmatically transforming second image data captured by the camera.

7. The system of claim 1, wherein the one or more servers configured to perform the logo identification includes the one or more servers being configured to perform a point-by-point comparison of the first transformed image with the training logo.

8. The system of claim 1, wherein the one or more servers are further configured to provide the first image parameter values to the consumer devices prior to receiving the first transformed image from the consumer device.

9. A method comprising:

receiving a first transformed image of a receipt from a consumer device, wherein the first transformed image is created by programmatically transforming image data captured by a camera using first image parameter values optimized for logo detection;

determining a training logo associated with a merchant;

performing a logo identification based on determining whether the first transformed image includes a logo corresponding with the training logo;

subsequent to determining that the first transformed image includes a logo that corresponds with the training logo associated with the merchant;

determining second image parameter values optimized for text extraction;

creating a second transformed image by programmatically transforming the image data using the second image parameter values; and

performing a text extraction using the second transformed image to create receipt text data.

10. The method of claim 9, wherein determining the second image parameter values optimized for the text extraction comprises:

generating test images of a second receipt using different combinations of image parameter values;

determining keywords of the second receipt;

determining text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determining the second image parameter values based on the text extraction scores.

11. The method of claim 9, further comprising:

determining whether the receipt text data has been successfully created based on the text extraction; and

in response to determining that the receipt text data fails to be successfully created based on the text extraction, sending a request for a third transformed image of the receipt to the consumer device, wherein the third transformed image is created by programmatically transforming second image data captured by the camera.

12. The method of claim 9, wherein the consumer device is configured to:

create the first transformed image by programmatically transforming the image data using the first image parameter values; and

send the first transformed image to the one or more servers.

13. The method of claim 9, wherein determining the first image parameter values optimized for logo detection comprises:

generating test images of a second receipt using different combinations of image parameter values;

determining keywords of the second receipt;

determining text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determining the second image parameter values based on the text extraction scores.

14. The method of claim 9, further comprising in response to determining that the first transformed image fails to include a logo corresponding with the training logo, sending a request for a third transformed image of the receipt to the consumer device, wherein the third transformed image is created by programmatically transforming second image data captured by the camera.

15. The method of claim 9, further comprising providing the first image parameter values to the consumer devices prior to receiving the first transformed image from the consumer device.

16. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instruction stored therein, the computer-executable program code instructions comprising program code instructions configured to:

receive a first transformed image of a receipt from a consumer device, wherein the first transformed image is created by programmatically transforming image data captured by a camera using first image parameter values optimized for logo detection;

determine a training logo associated with a merchant;

perform a logo identification based on determining whether the first transformed image includes a logo corresponding with the training logo;

subsequent to determining that the first transformed image includes a logo that corresponds with the training logo associated with the merchant;

determine second image parameter values optimized for text extraction;

create a second transformed image by programmatically transforming the image data using the second image parameter values; and

perform a text extraction using the second transformed image to create receipt text data.

17. The computer program product of claim 16, wherein determining the second image parameter values optimized for the text extraction comprises:

generating test images of a second receipt using different combinations of image parameter values;

determining keywords of the second receipt;

determining text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determining the second image parameter values based on the text extraction scores.

18. The computer program product of claim 16, wherein the computer-executable program code instructions comprising program code instructions further configured to:

determine whether the receipt text data has been successfully created based on the text extraction; and

in response to determining that the receipt text data fails to be successfully created based on the text extraction, send a request for a third transformed image of the receipt to the consumer device, wherein the third transformed image is created by programmatically transforming second image data captured by the camera.

19. The computer program product of claim 16, wherein the consumer device is configured to:

create the first transformed image by programmatically transforming the image data using the first image parameter values; and

send the first transformed image to the one or more servers.

20. The computer program product of claim 16, wherein determining the first image parameter values optimized for logo detection comprises:

generating test images of a second receipt using different combinations of image parameter values;

determining keywords of the second receipt;

determining text extraction scores based on comparisons of extracted text from the test images with the keywords; and

determining the second image parameter values based on the text extraction scores.