Gift card recognition using a camera

• US 9,430,766 B1
• Filed: 12/09/2014
• Issued: 08/30/2016
• Est. Priority Date: 12/09/2014
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Kraft, Adam Wiggen
• Primary Examiner(s): Gudorf, Laura
• Application Number:
  US14/565,245
• Time in Patent Office: 630 Days
• Field of Search: 235/380
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06K 19/06 characterised by the kind of the digital marking, e.g. shape, nature, code
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 20/3276 using a pictured code, e.g. barcode or QR-code, being read by the M-device
• CPC Class Codes: G06Q 20/342 Cards defining paid or billed services or quantities
• CPC Class Codes: G06Q 20/3433 the counter having monetary units
• CPC Class Codes: G06Q 20/352 Contactless payments by cards
• CPC Class Codes: G06Q 20/387 Payment using discounts or coupons relating to marketing G06Q30/02
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/15 Cutting or merging image elements, e.g. region growing, watershed or clustering-based techniques
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Various embodiments describe systems and methods enable a computing device of a user to capture an image of a gift card, or other such monetary device containing a code, with a camera or otherwise receive an image of that gift card. The computing device can be configured to recognize codes, such as digit claim codes, of the gift card by using one or more image processing, computer vision, and/or machine learning algorithms. After a successful detection and verification of a claim code, money or funds deposited in, or otherwise available from, an account associated with the gift card can be utilized, such as applied to a purchase or deposited into the user'"'"'s account. In many instances, a user interface (UI) can be provided on the computing device for the user to use to capture an image of a gift card and redeem the funds from the corresponding card.

Claims

1. A system, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

provide a user interface for a user of the gift card to capture an image of the gift card, the gift card containing a plurality of claim code characters;

receive the image of the gift card;

binarize the image;

determine a plurality of character patches of binarized image;

group the plurality of character patches along one or more baselines;

classify grouped character patches into a first set of candidate claim code characters according to one or more machine learning algorithms;

determine whether the plurality of claim code characters are found in the first set of candidate claim code characters;

in response to the plurality of claim code characters not being found in the first set of candidate claim code characters, rotate the grouped character patches along the one or more baselines to determine one or more new baselines;

determine top and bottom positions of the grouped character patches to perform a first close cut based at least upon X-gradient information of the grouped character patches;

perform a second close cut on the grouped character patches based at least upon Y-gradient information of the grouped character patches;

classify the grouped character patches into a second set of candidate claim code characters according to the one or more machine learning algorithms;

determine the plurality of claim code characters from the second set of candidate claim code characters; and

in response to a successful verification of the plurality of claim code characters, enable funds associated with the gift card to be applied or accessed.

2. The system of claim 1, wherein one of the plurality of claim code characters is a checksum character, and the instructions when executed further cause the system to:

verify the plurality of claim code characters based at least upon the checksum character.

3. The system of claim 1, wherein the instructions when executed further cause the system to:

assign, using a trained classifier, a confidence score to each of the first set of candidate claim code characters and the second set of candidate claim code characters; and

in response to a specific candidate claim code character having a corresponding confidence score below a threshold confidence score, remove the specific candidate claim code character in determining the plurality of claim code characters.

4. The system of claim 1, wherein the instructions when executed further cause the system to:

pre-process the image to determine at least one of;

a type of the gift card, a location of the plurality of claim code characters, or regions in the image having a high possibility to find the plurality of claim code characters.

5. A computer-implemented method, comprising:

receiving an image of a gift card, the gift card containing a plurality of claim code characters;

binarizing the image;

determining a plurality of character patches of binarized image;

grouping the plurality of character patches along one or more baselines;

classifying grouped character patches into a first set of candidate claim code characters according to one or more machine learning algorithms;

determining whether the plurality of claim code characters are found in the first set of candidate claim code characters;

in response to the plurality of claim code characters not being found in the first set of candidate claim code characters, rotating the grouped character patches along the one or more baselines to determine one or more new baselines;

determining top and bottom positions of the grouped character patches to perform a first close cut based at least upon X-gradient information of the grouped character patches;

performing a second close cut on the grouped character patches based at least upon Y-gradient information of the grouped character patches;

classifying the grouped character patches into a second set of candidate claim code characters according to the one or more machine learning algorithms; and

determining the plurality of claim code characters from the second set of candidate claim code characters; and

in response to a successful verification of the plurality of claim code characters, enabling funds associated with the gift card to be applied or accessed.

6. The computer-implemented method of claim 5, further comprising:

collecting images of claim codes of gift cards captured under at least one of different lighting, positions or noises, wherein the images of claim codes have at least one of different fonts, heights, thicknesses, spacing or sizes; and

training a classifier using collected images of claim codes based at least upon the one or more machine learning algorithms.

7. The computer-implemented method of claim 6, wherein the one or more machine learning algorithms include at least one of machine-learning models comprising:

a neural network model, support vector machine based model, Bayesian statistics, case-based reasoning, decision tree, inductive logic programming, Gaussian process regression, linear regression, group method of data handling, learning automata, random forest, ensembles of classifiers, ordinal classification, and conditional random field.

8. The computer-implemented method of claim 7, further comprising:

in response to a new set of claim code samples being collected, processing the new set of samples to readjust and refine parameter weights of the neural network model.

9. The computer-implemented method of claim 5, further comprising:

pre-processing the image to determine at least one of;

a type of the gift card, a location of the plurality of claim code characters, or regions in the image having a high possibility to find the plurality of claim code characters.

10. The computer-implemented method of claim 9, further comprising:

providing an indication on a user interface for a user of the gift card to zoom in on a specific region of the gift card based upon results of pre-processing the image.

11. The computer-implemented method of claim 5, further comprising:

verifying the plurality of claim code characters based at least upon a checksum character, where the checksum character is one of the plurality of claim code characters.

12. The computer-implemented method of claim 5, further comprising:

receiving a second image containing the plurality of claim code characters of the gift card;

determine a second plurality of claim code characters based on the second image; and

comparing the plurality of claim code characters with the second plurality of claim code characters.

13. The computer-implemented method of claim 5, wherein the image is binarized by using one or more binarization methods including an Otsu method, contrast adaptive method, light-weight text image processing method, noise-attribute-thresholding method, threshold selection method, Niblack method, Sauvola method, and Christian method.

14. The computer-implemented method of claim 13, further comprising:

determining spaces between the first set of candidate claim code characters; and

determining one or more claim code boundaries by identifying specific characters, or by identifying specific gaps between adjacent candidate claim code characters greater than a threshold value.

15. The computer-implemented method of claim 5, wherein performing the second close cut further comprises:

determining one or more clean-cut regions of the binarized image, wherein the one or more clean-cut regions have a subset of the grouped character patches satisfying a predetermined condition;

determining a first range of width for grouped character patches in the first subset;

determining a second subset of grouped character patches, grouped character patches in the second subset having a second range of width within the first range of width, wherein any grouped character patches in the second subset does not satisfy the predetermined condition; and

performing the second close cut on the first subset of grouped character patches and the second subset of grouped character patches.

16. The computer-implemented method of claim 5, wherein the first close cut and the second close cut are performed according to one or more algorithms including:

white space and pitch, vertical projection analysis, connected component analysis, and landmarks.

17. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to:

receive an image of a gift card, the gift card containing a plurality of claim code characters;

binarize the image;

determine a plurality of character patches of binarized image;

group the plurality of character patches along one or more baselines;

classify grouped character patches into a first set of candidate claim code characters according to one or more machine learning algorithms;

determine whether the plurality of claim code characters are found in the first set of candidate claim code characters;

in response to the plurality of claim code characters not being found in the first set of candidate claim code characters, rotate the grouped character patches long the one or more baselines to determine one or more new baselines;

determine top and bottom positions of the grouped character patches to perform a first close cut based at least upon X-gradient information of the grouped character patches;

perform a second close cut on the grouped character patches based at least upon Y-gradient information of the grouped character patches;

classify the grouped character patches into a second set of candidate claim code characters according to the one or more machine learning algorithms; and

determine the plurality of claim code characters from the second set of candidate claim code characters; and

in response to a successful verification of the plurality of claim code characters, enable funds associated with the gift card to be applied or accessed.

18. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

process background pixels and white space in the image to identify homogeneous regions by using one or more algorithms including;

X-Y tree, pixel-based projection profile, connected component-based projection profile, white space tracing, white space thinning and Doctrum method; and

determine the X-gradient information and Y-gradient information based at least upon the homogeneous regions.

19. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

perform the first close cut and the second close cut according to one or more algorithms including;

white space and pitch, vertical projection analysis, connected component analysis, and landmarks.

20. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

pre-process the image to determine at least one of;

a type of the gift card, a location of the plurality of claim code characters, or regions in the image having a high possibility to find the plurality of claim code characters.