Automatic learning in a merchandise checkout system with visual recognition

• US 20100217678A1
• Filed: 02/09/2010
• Published: 08/26/2010
• Est. Priority Date: 02/09/2009
• Status: Active Grant

• Current Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Original Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Inventors: Goncalves, Luis F.
• Granted Patent: US 8,494,909 B2
• Time in Patent Office: Days
• US Class Current: 705/22
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration

Abstract

A merchandise checkout system for ringing up merchandise in a grocery or other retail facility is disclosed. The merchandise checkout system includes visual sensors for acquiring images of an object to be rung up at checkout; a database configured to retain a visual model and UPC code of known items; and a processor configured to maintain and update the database with information about new merchandise presented for checkout. The visual model preferably includes at least one image of each item of merchandise in the retail store as well as geometric point features extracted from images of the merchandise. The processor in the exemplary embodiment is configured to: (a) compare the image of the object acquired at checkout to the visual model of the items in the database; (b) recognize the object based on the overall appearance of the packaging; (c) automatically ring up the price of the object at the POS; and (d) automatically add the acquired image to the database if the image of the object is new, i.e., it does not match the visual model of any of the items recorded in the database, thereby enabling the checkout system to dynamically and automatically build up its database of visual models.

Claims

1. A merchandise checkout system operably coupled to a point of sale (POS) system and a barcode scanner for reading a universal product code (UPC) of an object, the merchandise check out system comprising:

one or more imagers for acquiring an image of the object;

a data store configured to retain records for a plurality of items, each record comprising a visual model of one of the plurality of items and the UPC of the item;

the visual model comprising;

a) at least one image of the item; and

b) one or more geometric point features extracted from the one or more images of the item;

a processor configured to;

a) compare the image of the object to the visual model of at least one of the plurality of items;

b) recognize the object from the plurality of items based on the image of the object; and

c) automatically add the acquired image to the data store if the image of the object does not match the visual model of any of the plurality of items.

2. The merchandise checkout system of claim 1, wherein the processor is further configured to automatically ring up a price of the identified object at the POS.

3. The merchandise checkout system of claim 1, wherein the geometric point features are scale-invariant features.

4. The merchandise checkout system of claim 3, wherein the scale-invariant features are scale-invariant feature transform (SIFT) features.

5. The merchandise checkout system of claim 1, wherein the processor is further configured to:

extract geometric point features from the image of the object;

compare the extracted geometric point features with the geometric point features of one or more visual models;

determine if the object matches the item associated with any of the plurality of items based on the comparison;

whereby the object is identified based on the image of the object.

6. The merchandise checkout system of claim 1, wherein the processor is further configured to transmit an alert if the UPC read from the object does not match a UPC associated with the identified object.

7. The merchandise checkout system of claim 6, wherein the alert requires a manual override to be cleared.

8. The merchandise checkout system of claim 1, wherein the each visual model of an item comprises a plurality of images of an item, each image corresponding to a side of the item.

9. The merchandise checkout system of claim 1, wherein the at least one image of each visual model consists of an image of a single side of an item.

10. The merchandise checkout system of claim 9, wherein the image of a single side of the item is composed of a plurality of images stitched together.

11. The merchandise checkout system of claim 10, wherein the stitched image is stitched based on alignment of a plurality of corresponding geometric point features.

12. The merchandise checkout system of claim 1, wherein each of the plurality of records is configured to have a state selected from the group consisting of:

accumulation mode, live mode, and retired mode.

13. The merchandise checkout system of claim 1, wherein a record is configured to transition from an accumulation mode to a live mode if the visual model of the record has not been updated after a predetermined number of recognitions of the object based on that visual model.

14. The merchandise checkout system of claim 1, wherein a record is configured to transition from a live mode to a retired mode if the visual model of the record has not been recognized after a predetermined number of observations of the item having the same UPC as the record.

15. Merchandise check out system comprising:

one or more barcode scanners for reading the UPC if visible;

one or more imagers for acquiring images of items;

a data store configured to retain models of a plurality of items, each model configured to associate a UPC of an item with one or more images of the item;

a processor configured to create a new model by;

a) associating the one or more acquired images of the item with the UPC of the item if there is no model of the item with images; and

b) associating one or more acquired images with the UPC of the item if the acquired images do not match the images of an existing model for the UPC of the item.