Systems and methods for merchandise checkout

• US 20050189411A1
• Filed: 12/27/2004
• Published: 09/01/2005
• Est. Priority Date: 02/27/2004
• Status: Active Grant

• Current Assignee: Datalogic Adc, Inc. (Datalogic S.p.A.)
• Original Assignee: Evolution Robotics Inc. (iRobot Corporation)
• Inventors: Cremean, Michael, Hudnut, Alec, Goncalves, Luis, Ostrowski, Jim, Simonini, Alex
• Granted Patent: US 7,100,824 B2
• Time in Patent Office: Days
• US Class Current: 235/383
• CPC Class Codes: A47F 9/045 Handling of baskets or shopping trolleys at check-out counters, e.g. unloading, checking
• CPC Class Codes: G07F 7/02 by keys or other credit registering devices
• CPC Class Codes: G07G 1/0036 Checkout procedures
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G07G 1/0081 the reader being a portable scanner or data reader G07G1/009 takes precedence
• CPC Class Codes: G07G 3/00 Alarm indicators, e.g. bells
• CPC Class Codes: G07G 3/003 Anti-theft control theft alarms G08B13/00
• CPC Class Codes: G08B 13/1961 Movement detection not involving frame subtraction, e.g. motion detection on the basis of luminance changes in the image
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream

Abstract

Systems and methods for recognizing and identifying items located on the lower shelf of a shopping cart in a checkout lane of a retail store environment for the purpose of reducing or preventing loss or fraud and increasing the efficiency of a checkout process. The system includes one or more visual sensors that can take images of items and a computer system that receives the images from the one or more visual sensors and automatically identifies the items. The system can be trained to recognize the items using images taken of the items. The system relies on matching visual features from training images to match against features extracted from images taken at the checkout lane. Using the scale-invariant feature transformation (SIFT) method, for example, the system can compare the visual features of the images to the features stored in a database to find one or more matches, where the found one or more matches are used to identify the items.

Claims

1. A system for checking out a merchandise, comprising:

at least one visual sensor for capturing an image of an object on a moveable structure; and

a subsystem coupled to the at least one visual sensor and configured to detect and recognize the object by analyzing the image.

2. The system of claim 1, wherein the at least one visual sensor is a digital camera with a charge-coupled-device (CCD) imager, a complementary metal-oxide semiconductor (CMOS) imager, an infrared imager, or any combination thereof.

3. The system of claim 1, wherein the subsystem comprises:

a checkout subsystem for receiving visual data from the at least one visual sensor and analyzing the visual data;

a server for receiving analyzed visual data from the checkout subsystem, recognizing the object and sending match data to the checkout subsystem; and

an Object Database coupled to the server and configured to store one or more objects to recognize.

4. The method of claim 3, wherein the subsystem further comprises a Log Data Storage coupled to the server and configured to store the match data.

5. The system of claim 4, wherein the Log Data Storage comprises an Output Table comprising an object identification (ID) field, a view ID field, a camera ID field, an image field and a timestamp field.

6. The system of claim 3, wherein the Object Database comprises a Feature Table comprising an object ID field, a view name field, an object name field, a view ID field, a feature ID field, a feature coordinates field and a feature descriptor field.

7. The system of claim 6, wherein the Object Database further comprises an Object Recognition Table comprising a feature descriptor field, an object ID field, a view ID field and a feature ID field.

8. The system of claim 3, wherein the Object Database is contained in a single storage device.

9. The system of claim 3, wherein the Object Database is spread over a plurality of storage devices connected via a network.

10. The system of claim 3, further comprising:

a feature extractor interposed between the at least one visual sensor and the checkout subsystem and configured to receive the visual data from the at least one visual sensor, analyze the visual data and send the analyzed visual data to the checkout subsystem.

11. The system of claim 3, wherein the checkout subsystem is coupled to one or more input devices, each of the one or more input devices including a barcode scanner, a scale, a keyboard, a keypad, a touch screen, a card reader or any combination thereof.

12. The system of claim 3, wherein the checkout subsystem is a checkout terminal used by a casher or a self-service checkout terminal.

13. The system of claim 1, further comprising a network communication device for connecting the checkout subsystem to a local area network or a wide area network.

14. The system of claim 1, wherein the subsystem comprises:

a checkout subsystem for receiving analyzed visual data from the at least one visual sensor;

a feature extractor;

a server for receiving the analyzed visual data from the checkout subsystem, recognizing the object and sending match analyzed data to the checkout subsystem; and

an Object Database coupled to the server and configured to store one or more objects to recognize.

15. The system of claim 14, wherein the subsystem further comprises:

a Log Data Storage coupled to the server and configured to store the match data.

16. A system for checking out a merchandise, comprising:

at least one visual sensor for capturing an image of an object in a moveable structure;

a checkout subsystem for receiving visual data from the at least one visual sensor and analyzing the visual data;

a server for receiving analyzed visual data from the checkout system, recognizing the object and sending match data to the checkout subsystem; and

an Object Database coupled to the server and configured to store one or more objects to recognize.

17. The system of claim 16, further comprising a Log Data Storage that is coupled to the server and configured to store the match data and comprises an Output Table.

18. The system of claim 17, wherein the Output Table comprises an object identification (ID) field, a view ID field, a camera ID field, an image field and a timestamp field.

19. The system of claim 16, wherein the at least one visual sensor is a digital camera with a charge-coupled-device (CCD) imager, a complementary metal-oxide semiconductor (CMOS) imager, an infrared imager, or any combination thereof.

20. The system of claim 16, wherein the Object Database comprises:

a Feature Table comprising an object ID field, a view ID field, a feature ID field, a feature coordinates field, a view name field, an object name field and a feature descriptor field; and

an Object Recognition Table comprising a feature descriptor field, an object ID field, a view ID field and a feature ID field.

21. The system of claim 16, wherein the Object Database is contained in a single storage device.

22. The system of claim 16, wherein the Object Database is spread over a plurality of storage devices connected via a network.

23. The system of claim 16, further comprising:

a feature extractor interposed between the at least one visual sensor and the checkout subsystem and configured to receive the visual data from the at least one visual sensor, analyze the visual data and send the analyzed visual data to the checkout subsystem.

24. The system of claim 16, wherein the checkout subsystem is coupled to one or more input devices, each of the one or more input devices including a barcode scanner, a scale, a keyboard, a keypad, a touch screen, a card reader or any combination thereof, and wherein the checkout subsystem is a checkout terminal used by a casher or a self-service checkout terminal.

25. The system of claim 16, further comprising a network communication device for connecting the checkout subsystem to a local area network or a wide area network, the network communication device including a network interface card, a modem or an infrared port.

26. A system for checking out a merchandise, comprising:

at least one visual sensor for capturing an image of an object on a moveable structure;

a checkout subsystem;

a computer for receiving visual data from the at least one visual sensor, sending match data to the checkout subsystem and receiving transaction data from the checkout subsystem;

a server for receiving log data from the checkout subsystem and providing database information to the computer; and

an Object Database coupled to the server and configured to store one or more objects to recognize.

27. The system of claim 26, further comprising:

a Log Data Storage coupled to the server and configured to store the match data.

28. The system of claim 27, wherein the server comprises a supervisor application for managing the Object Database and the Log Data Storage.

29. The system of claim 27, wherein the Log Data Storage comprises:

an Output Table comprising an object identification (ID) field, a view ID field, a camera ID field, an image field and a timestamp field.

30. The system of claim 26, wherein the Object Database comprises:

a Feature Table comprising an object ID field, a view ID field, a feature ID field, a view name field, an object name field, a feature coordinates field and a feature descriptor field.

31. The system of claim 30, wherein the Object Database further comprises:

an Object Recognition Table comprising a feature descriptor field, an object ID field, a view ID field and a feature ID field.

32. The system of claim 26, wherein the Object Database is contained in a single storage device.

33. The system of claim 26, wherein the Object Database is spread over a plurality of storage devices connected via a network.

34. The system of claim 26, wherein the checkout subsystem communicates with the computer via an interface, the interface including a hardwired connection.

35. The system of claim 26, wherein the checkout subsystem communicates with the computer via an interface, the interface including a wireless connection.

36. The system of claim 26, wherein the computer analyzes the visual data to extract features and compares the features with the database information to generate the match data.

37. The system of claim 26, wherein the checkout subsystem is coupled to one or more input devices, each of the one or more input devices including a barcode scanner, a scale, a keyboard, a keypad, a touch screen, a card reader or any combination thereof.

38. The system of claim 26, wherein the checkout subsystem is a checkout terminal used by a casher attended POS.

39. The system of claim 26, wherein the checkout subsystem is a checkout terminal used by a self-service checkout POS.

40. A system for checking out a merchandise, comprising:

at least one visual sensor for capturing an image of an object in a shopping cart;

a checkout subsystem;

a computer for receiving visual data from the at least one visual sensor, sending match data to the checkout subsystem and receiving transaction data from the checkout subsystem;

a server for receiving log data from the checkout subsystem and providing database information to the computer;

an Object Database coupled to the server and configured to store one or more objects to recognize, the Object Database comprising a Feature Table, and an Object Recognition Table; and

a Log Data Storage coupled to the server and configured to store the match data, the Log Data Storage comprising an Output Table.

41. A system for checking out merchandise in a shopping cart, comprising:

a checkout lane;

at least one visual sensor for capturing an image of the merchandise;

a checkout subsystem for receiving visual data from the at least one visual sensor and analyzing the visual data;

a server for receiving analyzed visual data from the checkout system, recognizing the merchandise and sending match data to the checkout subsystem; and

an Object Database coupled to the server and configured to store one or more objects to recognize, the Object Database including;

a Feature Table; and

an Object Recognition Table.

42. The system of claim 41, further comprising:

a Log Data Storage coupled to the server and configured to store the match data, the Log Data Storage including an Output Table.

43. The system of claim 41, wherein the Object Database includes information stored in an object identification (ID) field, an object name field, a view ID field, a view name field, a feature ID field, a feature coordinates field, a feature descriptor field and a feature descriptor field, and wherein the Log Data Storage includes information stored an object identification (ID) field, a view ID field, a camera ID field, an image field and a timestamp field.

44. A database, comprising:

a Feature Table comprising an object ID field, a view ID field, a feature ID field, a feature coordinates field, an object name field, a view name field, and a feature descriptor field.

45. The database of claim 44, further comprising:

an Object Recognition Table comprising a feature descriptor field, an object ID field, a view ID field and a feature ID field.

46. A database, comprising:

an Output Table comprising an object identification (ID) field, a view ID field, a camera ID field, an image field and a timestamp field.

47. A method of checking out merchandise, comprising:

(a) receiving visual image data of an object;

(b) comparing the visual image data with data stored in a database to find a set of matches;

(c) determining if the set of matches is found; and

(d) sending a recognition alert.

48. The method of claim 47, further comprising:

analyzing the visual image data to extract one or more visual features.

49. The method of claim 48, wherein the step of comparing is based on a scale invariant feature transform (SIFT) method.

50. The method of claim 48, wherein the step of comparing comprises:

finding a match for each of the one or more features.

51. The method of claim 50, wherein the step of comparing further comprises:

associating a quality measure with the match.

52. The method of claim 51, wherein the step of comparing further comprises:

if the associated quality measure exceeds a predetermined threshold, including the match in the set of matches.

53. The method of claim 51, wherein the quality measure is a match confidence that ranges from 0 to 100%.

54. The method of claim 51, wherein the step of comparing further comprises:

selecting a particular match associated with a highest quality measure.

55. The method of claim 54, wherein the step of comparing comprises a step of including the particular match in the set of matches.

56. The method of claim 48, further comprising, prior to the step of sending a recognition alert:

computing a statistical probability that each of the one or more visual features can be recognized.

57. The method of claim 47, further comprising, prior to the step of sending a recognition alert:

(e) checking if each element of the set of matches is reliable.

58. The method of claim 57, further comprising:

(f) if all elements of the set of matches are unreliable, repeating the steps (a)-(e).

59. The method of claim 57, wherein the step of checking comprises:

recognizing each element of the set of matches for a plurality of process cycles.

60. The method of claim 57, wherein the step of receiving an image comprises:

capturing a plurality of images.

61. The method of claim 60, wherein the step of receiving an image further comprises:

comparing two consecutive ones of the plurality of images to detect a motion; and

if the motion is detected, taking later one of the two consecutive images.

62. A computer readable medium embodying program code with instructions for recognizing an object, said computer readable medium comprising:

program code for receiving a visual image data of the object;

program code for comparing the visual image data with data stored in a database to find a set of matches;

program code for determining if the set of matches is found; and

program code for sending a recognition alert.

63. The computer readable medium of claim 62, further comprising:

program code for analyzing the visual image data to extract one or more visual features.

64. The computer readable medium of claim 62, further comprising:

program code for checking if each element of the set of matches is reliable.

65. The compute readable medium of claim 62, further comprising:

program code for repeating operation of the program code for receiving a visual image data to the program code for sending a recognition alert.

66. A method of checking out a merchandise, comprising:

(a) receiving visual image data of an object;

(b) comparing the visual image data with data stored in a database to find a set of matches;

(c) determining if the set of matches is found;

(d) if the set of matches is not found, repeating the steps (a)-(c);

(e) checking if each element of the set of matches is reliable;

(f) if all elements of the set of matches are unreliable, repeating the steps (a)-(e); and

(g) sending match data.

67. The method of claim 66, further comprising the step of analyzing the visual image data to extract one or more visual features.

68. The method of claim 67, wherein the step of checking comprises:

computing a statistical probability that each of the one or more visual features can be recognized.

69. The method of claim 66, wherein the step of checking comprises:

recognizing each element of the set of matches for a plurality of process cycles.

70. A computer readable medium embodying program code with instructions for recognizing an object, said computer readable medium comprising:

program code for receiving a visual image data of the object;

program code for comparing the visual image data with data stored in a database to find a set of matches;

program code for determining if the set of matches is found;

program code for checking if each element of the set of matches is reliable;

program code for sending a recognition alert; and

program code for repeating operation of the program code for receiving visual image data to the program code for sending a recognition alert.

71. The computer readable medium of claim 70, further comprising program code for analyzing the visual image data to extract one or more visual features.

72. A method for training a system for recognizing an object, said method comprising:

(a) receiving a visual image of the object;

(b) receiving data associated with the visual image;

(c) storing the visual image and the data in a data storage;

(d) determining if there is additional image to capture; and

(e) running a training subroutine.

73. The method of claim 72, further comprising, prior to the step of running a training subroutine:

if the determination in step (d) is positive, repeating the steps (a)-(d).

74. The method of claim 72, further comprising, after the step of running a training subroutine:

deleting the visual image from the data storage.

75. The method of claim 72, wherein the object is not recognized by the system for a predetermined period of time.

76. The method of claim 72, wherein the data include a distance between a visual sensor and the object at a time of image capture, a name of the object, a view name, an object ID, a view ID, a unique identifier, a text string associated with the object, a name of a computer file associated with the visual image, a UPC of the object and a flag indicating that the object is a high security-risk item or any combination thereof.

77. The method of claim 72, wherein the visual image and data are electronically sent by a manufacturer of the object.

78. The method of claim 72, wherein the step of running a training subroutine comprises:

selecting an untrained visual image;

analyzing the untrained visual image to extract one or more features from the untrained visual image; and

saving the one or more features in a database.

79. A computer readable medium embodying program code with instructions for training a system for recognizing an object, said computer readable medium comprising:

program code for receiving a visual image of the object;

program code for receiving data associated with the visual image;

program code for storing the visual image and the data in a data storage;

program code for determining if there is additional image to capture; and

program code for running a training subroutine.

80. The computer readable medium of claim 79, further comprising:

program code for repeating operation of the program code for receiving a visual image to the program code for determining if there is additional image to capture.

81. The computer readable medium of claim 79, further comprising:

program code for deleting the visual image from the data storage.

82. The computer readable medium of claim 79, further comprising:

program code for selecting an untrained visual image;

program code for analyzing the untrained visual image to extract one or more features from the untrained visual image; and

program code for saving the one or more features in a database.