System and method for detecting generic items in image sequence

• US 9,064,161 B1
• Filed: 06/08/2007
• Issued: 06/23/2015
• Est. Priority Date: 06/08/2007
• Status: Active Grant

• Current Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Original Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Inventors: Boman, Robert, Goncalves, Luis, Ostrowski, James
• Primary Examiner(s): Lee, Michael G
• Assistant Examiner(s): TARDIF, DAVID P
• Application Number:
  US11/811,211
• Time in Patent Office: 2,937 Days
• Field of Search: 235/383
• US Class Current: 1/1
• CPC Class Codes: G06K 7/00 Methods or arrangements for sensing record carriers, e.g. for reading patterns methods or arrangements for marking the record carrier in digital fashion G06K1/00; pattern recognition G06F18/00; arrangements for image or video recognition or understanding G06V10/00; character recognition, recognising digital ink or document-oriented image-based pattern recognition G06V30/00
• CPC Class Codes: G06K 7/01 Details
• CPC Class Codes: G06Q 30/00 Commerce
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G07G 1/00 Cash registers alarm indicators G07G3/00
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G07G 3/003 Anti-theft control theft alarms G08B13/00

Abstract

A system and method for detecting the presence of known or unknown objects based on visual features is disclosed. In the preferred embodiment, the system is a checkout system for detecting items of merchandise on a shopping cart. The merchandise checkout system preferably includes a feature extractor for extracting visual features from a plurality of images; a motion detector configured to detect one or more groups of the visual features present in at least two of the plurality of images; a classifier to classify each of said groups of the visual features based on one or more classification criteria, wherein each of the one or more parameters is associated with one of said groups of visual features; and an alarm configured to generate an alert if the one or more parameters for any of said groups of the visual features satisfy one or more classification criteria.

Claims

1. A merchandise checkout system for detecting a merchandise item from sequentially captured images, the system comprising:

a feature extractor for extracting visual features from the sequentially captured images, the extracted visual features characterized by feature vectors;

a feature comparison module operable to compare visual features extracted from a first image of the sequentially captured images to visual features extracted from a second image of the sequentially captured images to identify a group of extracted visual features that match between the first and second images;

a classifier operable to measure, for the group of extracted visual features, a classification parameter so as to generate a parameter measurement that collectively characterizes the group of extracted visual features for comparison of the parameter measurement to an associated classification criterion, the associated classification criterion defining a first classification representing presence of merchandise items and a second classification representing presence of non-merchandise items, and the classifier further operable to classify the group of extracted visual features into either the first classification or the second classification based on whether the parameter measurement satisfies the associated classification criterion, wherein the associated classification criterion includes a threshold amount of entropy of an area occupied by the group of extracted visual features; and

an alarm configured to generate an alert in response to the classifier classifying the group of extracted visual features as belonging to the first classification and thereby indicating the presence of the merchandise item.

2. The system of claim 1, wherein the sequentially captured images comprise images of a moving structure.

3. The system of claim 2, wherein the moving structure is a shopping cart.

4. The system of claim 3, wherein the shopping cart comprises one or more items of merchandise thereon.

5. The system of claim 1, wherein the visual features extracted from the sequentially captured images comprise scale-invariant features.

6. The system of claim 5, wherein the visual features extracted from sequentially captured images comprise scale-invariant feature transform (SIFT) features.

7. The system of claim 1, wherein the classification parameter corresponds to one or both of a translation motion parameter and a rotational motion parameter.

8. The system of claim 7, wherein the translation motion parameter comprises a speed parameter and direction parameter.

9. The system of claim 1, wherein the classification parameter is selected from the group consisting of:

a number of tracked features parameter, a group entropy parameter, a time of image capture parameter, an aspect ratio parameter, an elapsed time parameter, an edge score parameter, and combinations thereof.

10. The system of claim 1, wherein the classification parameter corresponds to one or more codebook histogram parameters.

11. The system of claim 1, wherein the sequentially captured images from which visual features are extracted are selected from a video sequence.

12. The system of claim 1, wherein the classifier is selected from the group consisting of:

one or more thresholding conditions, a linear classifier, a 3-D Motion Estimation classifier, a Nearest Neighbor classifier, a Neural Network classifier, a Vector Quantization classifier, and combinations thereof.

13. The system of claim 1, further comprising a motion estimator configured to estimate, based on an affine transformation, motion of the group of extracted visual features identified by the feature comparison module.

14. The system of claim 1, further comprising a motion estimator configured to estimate, based on a homography transformation, motion of the group of extracted visual features identified by the feature comparison module.

15. The system of claim 1, wherein the alarm is further configured to:

temporarily suspends a purchase transaction, or

generate an audio or visual alert without temporarily suspending the purchase transaction.

16. The system of claim 1, wherein the parameter measurement comprises an amount of horizontal motion of the group, wherein the associated classification criterion includes a threshold of horizontal motion, and the classifier is configured to classify the group into the first classification in response to the amount exceeding the threshold.

17. The system of claim 1, wherein the parameter measurement comprises an amount of vertical motion of the group, wherein the associated classification criterion includes a threshold of vertical motion, and the classifier is configured to classify the group into the first classification in response to the amount being less than the threshold.

18. The system of claim 1, wherein the criterion includes a threshold of rotation.

19. The system of claim 1, wherein the criterion includes threshold number of matching features.

20. The system of claim 1, wherein the criterion includes an aspect ratio of structural members of a shopping cart.

21. The system of claim 1, wherein the criterion includes an expected period between capturing of the first and second images.

22. The system of claim 1, wherein the criterion includes an expected period since a start of a checkout transaction process.

23. The system of claim 1, wherein the criterion includes a threshold edge score.

24. A method of detecting a merchandise item from sequentially captured images, the method comprising:

extracting visual features from the sequentially captured images, the extracted visual features characterized by feature vectors;

comparing visual features extracted from a first image of the sequentially captured images to visual features extracted from a second image of the sequentially captured images to identify a group of extracted visual features that match between the first and second images;

measuring one or both motion parameters or appearance parameters so as to generate a parameter measurement that collectively characterizes the group of extracted visual features for comparison of the parameter measurement to an associated classification criterion, the associated classification criterion defining a first classification representing presence of merchandise items and a second classification representing presence of non-merchandise items, wherein the associated classification criterion includes a threshold amount of entropy of an area occupied by the group of extracted visual features;

classifying the group of extracted visual features into either the first classification or the second classification based on whether the parameter measurement satisfies the associated classification criterion; and

generating an alert if the one or both motion parameters or appearance parameters for the group of extracted visual features satisfy the associated classification criterion.

25. The method of claim 24, further comprising using an affine transformation to estimate motion of the group of extracted visual features.

26. The method of claim 24, using a homography transformation to estimate motion of the group of extracted visual features.