Systems and methods for merchandise checkout

• US 8,267,316 B2
• Filed: 08/22/2006
• Issued: 09/18/2012
• Est. Priority Date: 02/27/2004
• Status: Active Grant

• Current Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Original Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Inventors: Ostrowski, Jim, Goncalves, Luis, Cremean, Michael, Simonini, Alex, Hudnut, Alec
• Primary Examiner(s): Le, Thien M.
• Assistant Examiner(s): KIM, TAE W
• Application Number:
  US11/466,371
• Publication Number:
  US 20060283943A1
• Time in Patent Office: 2,219 Days
• Field of Search: 235/383
• US Class Current: 235/383
• CPC Class Codes: A47F 9/045 Handling of baskets or shopping trolleys at check-out counters, e.g. unloading, checking
• CPC Class Codes: G07F 7/02 by keys or other credit registering devices
• CPC Class Codes: G07G 1/0036 Checkout procedures
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G07G 1/0081 the reader being a portable scanner or data reader G07G1/009 takes precedence
• CPC Class Codes: G07G 3/00 Alarm indicators, e.g. bells
• CPC Class Codes: G07G 3/003 Anti-theft control theft alarms G08B13/00
• CPC Class Codes: G08B 13/1961 Movement detection not involving frame subtraction, e.g. motion detection on the basis of luminance changes in the image
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream

Abstract

Systems and methods for recognizing and identifying items located on the lower shelf of a shopping cart in a checkout lane of a retail store environment for the purpose of reducing or preventing loss or fraud and increasing the efficiency of a checkout process. The system includes one or more visual sensors that can take images of items and a computer system that receives the images from the one or more visual sensors and automatically identifies the items. The system can be trained to recognize the items using images taken of the items. The system relies on matching visual features from training images to match against features extracted from images taken at the checkout lane. Using the scale-invariant feature transformation (SIFT) method, for example, the system can compare the visual features of the images to the features stored in a database to find one or more matches, where the found one or more matches are used to identify the items.

Claims

1. A system for checking out merchandise, comprising:

at least one visual sensor for capturing an image of an object on a cart, wherein the at least one visual sensor is in a mounted position at a checkout counter and wherein the visual sensor has, from its mounted position, a field of view directed to a checkout lane to capture the image of the object when the cart is in the checkout lane; and

a subsystem coupled to the at least one visual sensor, wherein the subsystem is configured to;

extract a plurality of visual features comprising feature descriptors from the image of the object on the cart;

compare each of the plurality of feature descriptors of the extracted visual features to a plurality of feature descriptors of visual features associated with a plurality of known objects, and

identify a plurality of matching visual descriptors to find a match between the object on the cart and one of the plurality of known objects.

2. The system of claim 1, wherein the at least one visual sensor is a digital camera with a charge-coupled-device (CCD) imager, a complementary metal-oxide semiconductor (CMOS) imager, an infrared imager, or any combination thereof.

3. The system of claim 1, wherein the subsystem comprises:

a checkout subsystem configured to receive visual data from the at least one visual sensor,

a server configured to receive visual data from the checkout subsystem, recognize the object, and send match data to the checkout subsystem; and

an object database configured to store the plurality of visual features associated with the plurality of known objects.

4. The system of claim 3, wherein the object database is spread over a plurality of storage devices connected via a network.

5. The system of claim 3, wherein the checkout subsystem is coupled to one or more input devices, each of the one or more input devices including a barcode scanner, a scale, a keyboard, a keypad, a touch screen, a card reader or any combination thereof.

6. The system of claim 3, wherein the checkout subsystem comprises a checkout terminal used .by a cashier or a self-service checkout terminal.

7. The system of claim 1, wherein the plurality of extracted visual features and the plurality of visual features associated with the plurality of known objects are scale-invariant feature transform (SIFT) features.

8. The system of claim 1, wherein the cart comprises a shopping cart.

9. The system of claim 8, wherein the shopping cart comprises a bottom of basket, and wherein the at least one visual sensor comprises one or more cameras directed to the bottom of basket when the cart is in a checkout lane.

10. The system of claim 3, wherein the checkout subsystem is coupled at least one input device, wherein the input device is selected from the group consisting of:

a barcode scanner, a scale, a keyboard, a keypad, a touch screen, a card reader or a combination thereof.

11. A system for checking out merchandise in a shopping cart, comprising:

at least one visual sensor for capturing an image of the merchandise in the shopping cart, wherein the at least one visual sensor is in a mounted position at a checkout counter and wherein the visual sensor has, from its mounted position, a field of view directed to a checkout lane to capture the image of the merchandise when the shopping cart is in the checkout lane;

a checkout subsystem adapted to receive visual data from the at least one visual sensor; and

a server adapted to;

receive the visual data from the checkout subsystem,

extract a plurality of visual features comprising feature descriptors from the image of the merchandise in the shopping cart,

recognize the merchandise based on comparing the feature descriptors of the extracted visual features to feature descriptors of visual features associated with known objects to find a match between the merchandise in the shopping cart and one of the known objects.

12. A non-transitory computer readable medium in a merchandise checkout system embodying program code with instructions for recognizing an object, the merchandise checkout system including at least one visual sensor, said non-transitory computer readable medium comprising:

program code for receiving visual image data of an object on a cart, the visual image data comprising one or more visual features of the object, wherein the at least one visual sensor is in a mounted position at a checkout counter and wherein the visual sensor has, from its mounted position, a field of view directed to a checkout lane to capture an image of the object when the cart is in the checkout lane;

program code for comparing the visual features of the object with visual features of a plurality of known objects to find a set of matches;

program code for identifying the object on the cart as one of the plurality of known object based on the set of matches; and

program code for sending a recognition alert to a checkout terminal.

13. A system for checking out merchandise, comprising:

at least one visual sensor for capturing an image of one or more objects on a cart, wherein the at least one visual sensor is in a mounted position at a checkout counter and wherein the visual sensor has, from its mounted position, a field of view directed to a checkout lane to capture the image of the object(s) when the cart is in the checkout lane; and

a subsystem coupled to the at least one visual sensor and configured to detect and recognize the object from a plurality of known objects by analyzing the image of the one or more objects on the cart using a scale-invariant feature transform (SIFT) to extract visual features from the image of the one or more objects on the cart.

14. A method for checking out merchandise, the method comprising:

capturing an image of one or more objects on a shopping cart, wherein the image is captured from a visual sensor in a mounted position in proximity to a checkout counter and wherein the visual sensor has, from its mounted position, a field of view directed to a checkout lane to capture the image of the object(s) when the shopping cart is in the checkout lane;

extracting a plurality of features comprising scale-invariant feature descriptors from the image of the objects on the shopping cart; and

comparing the plurality of scale-invariant feature descriptors to a plurality of known features comprising scale-invariant feature descriptors associated with a plurality of known objects;

identifying one or more matches between the plurality of scale-invariant feature descriptors and the plurality of known scale-invariant feature descriptors; and

identifying each of the one or more objects on the shopping cart based on the one or more matches.

15. The method of claim 14, wherein the each of the plurality of extracted scale-invariant features is a scale-invariant feature transform (SIFT) feature.