Detecting primitive events at checkout

• US 9,299,229 B2
• Filed: 11/29/2008
• Issued: 03/29/2016
• Est. Priority Date: 10/31/2008
• Status: Active Grant

• Current Assignee: Toshiba Global Commerce Solutions Holdings Corporation (Toshiba Corporation)
• Original Assignee: Toshiba Global Commerce Solutions Holdings Corporation (Toshiba Corporation)
• Inventors: Bobbitt, Russell Patrick, Fan, Quanfu, Hampapur, Arun, Kjeldsen, Frederik, Pankanti, Sharathchandra Umapathirao, Yanagawa, Akira, Zhai, Yun
• Primary Examiner(s): RUBIN, BLAKE J
• Application Number:
  US12/325,177
• Publication Number:
  US 20100134624A1
• Time in Patent Office: 2,677 Days
• Field of Search: 348/150
• US Class Current: 1/1
• CPC Class Codes: G06Q 20/20 Point-of-sale [POS] network systems
• CPC Class Codes: G06Q 20/203 Inventory monitoring
• CPC Class Codes: G06Q 20/208 Input by product or record sensing, e.g. weighing or scanner processing
• CPC Class Codes: G06Q 20/4016 involving fraud or risk level assessment in transaction processing
• CPC Class Codes: G07F 19/20 Automatic teller machines [ATMs]
• CPC Class Codes: G07F 19/207 Surveillance aspects at ATMs
• CPC Class Codes: G07G 1/0054 with control of supplementary check-parameters, e.g. weight or number of articles
• CPC Class Codes: G07G 1/0063 with means for detecting the geometric dimensions of the article of which the code is read, such as its size or height, for the verification of the registration
• CPC Class Codes: G07G 3/00 Alarm indicators, e.g. bells
• CPC Class Codes: G07G 3/003 Anti-theft control theft alarms G08B13/00
• CPC Class Codes: G07G 3/006 False operation
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion

Abstract

Techniques for detecting one or more events are provided. The techniques include identifying one or more segments in a video sequence as one or more candidates for one or more events by a temporal ordering of the one or more candidates, and analyzing one or more motion patterns of the one or more candidates to detect the one or more events.

Claims

1. A method for detecting one or more events, comprising:

generating a motion graph from a video sequence captured at a point-of-sale (POS) system, the motion graph representing human motion in the video sequence;

detecting two consecutive scan events at the POS system by identifying a first reference motion pattern in the motion graph;

identifying a segment of the video sequence between the two consecutive scan events; and

upon identifying the video sequence segment, analyzing a portion of the motion graph corresponding to the video sequence segment to detect at least one of a pick-up event and a drop event by identifying a second reference motion pattern in the motion graph.

2. The method of claim 1,

wherein the first motion pattern corresponds to one or more motion peaks in the motion graph.

3. The method of claim 1, wherein analyzing the video sequence segment to detect at least one of a pick-up event and a drop event comprises:

detecting the second motion pattern using a portion of the motion graph corresponding to the video sequence segment, wherein the second motion pattern corresponds to a motion valley flanked by two motion peaks in the motion graph.

4. The method of claim 1, wherein the motion graph comprises three portions each associated with three regions of interest included within the video sequence, each region of interest corresponding to the scan, pick-up, and drop events, respectively,

wherein monitoring the video sequence to detect two consecutive scan events comprises identifying the two consecutive scan events by monitoring the portion of the motion graph associated with the region of interest of the scan event,

wherein analyzing the video sequence segment comprises analyzing the portion of the motion graph corresponding to at least one of the pick-up event and drop event.

5. The method of claim 1, wherein the pick-up event comprises a cashier picking up an item from a lead-in portion of the POS system, the scan event comprises the cashier at least one of reading the item via a scanner and weighing the item, and the drop event comprises the cashier placing the item onto a take-away portion of the POS system.

6. The method of claim 1, wherein generating the motion graph based on the human motion comprises counting one or more motion pixels obtained by frame differencing the video sequence.

7. The method of claim 1, further comprising, upon detecting the pick-up or drop event, combining one or more of the consecutive scan events and the pick-up or drop event according to one or more temporal ordering constraints to validate a transaction process of the POS system.

8. The method of claim 1, wherein the two consecutive scan events are detected by determining whether a cashier'"'"'s motion exceeds a threshold of motion.

9. A computer program product comprising a computer readable device having computer readable program code for detecting one or more events, said computer program product including:

computer readable program code for generating a motion graph from a video sequence captured at a point-of-sale (POS) system, the motion graph representing human motion in the video sequence;

computer readable program code for detecting two consecutive scan events at the POS system by identifying a first reference motion pattern in the motion graph;

computer readable program code for identifying a segment of the video sequence between the two consecutive scan events; and

computer readable program code for, upon identifying the video sequence segment, analyzing a portion of the motion graph corresponding to the video sequence segment to detect at least one of a pick-up event and a drop event at the POS system by identifying a second reference motion pattern in the motion graph.

10. The computer program product of claim 9,

wherein the first motion pattern corresponds to one or more motion peaks in the motion graph.

11. The computer program product of claim 9, wherein analyzing the video sequence segment to detect at least one of a pick-up event and a drop event comprises:

computer readable program code for detecting the second motion pattern using a portion of the motion graph corresponding to the video sequence segment, wherein the second motion pattern corresponds to a motion valley flanked by two motion peaks in the motion graph.

12. The computer program product of claim 9, wherein the motion graph comprises three portions each associated with three regions of interest included within the video sequence, each region of interest corresponding to the scan, pick-up, and drop events, respectively,

wherein monitoring the video sequence to detect two consecutive scan events comprises identifying the two consecutive scan events by monitoring the portion of the motion graph associated with the region of interest of the scan event,

wherein analyzing the video sequence segment comprises analyzing the portion of the motion graph corresponding to at least one of the pick-up event and drop event.

13. The computer program product of claim 9, wherein the pick-up event comprises a cashier picking up an item from a lead-in portion of the POS system, the scan event comprises the cashier at least one of reading the item via a scanner and weighing the item, and the drop event comprises the cashier placing the item onto a take-away portion of the POS system.

14. The computer program product of claim 9, further comprising, computer readable program code for, upon detecting the pick-up or drop event, combining one or more of the consecutive scan events and the pick-up or drop event according to one or more temporal ordering constraints to validate a transaction process of the POS system.

15. A system for detecting one or more events, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data, comprising;

generating a motion graph from a video sequence captured at a point-of-sale (POS) system, the motion graph representing human motion in the video sequence;

detecting two consecutive scan events at the POS system by identifying a first reference motion pattern in the motion graph;

identifying a segment of the video sequence between the two consecutive scan events; and

upon identifying the video sequence segment, analyzing a portion of the motion graph corresponding to the video sequence segment to detect at least one of a pick-up event and a drop event by identifying a second reference motion pattern in the motion graph.

16. The system of claim 15, wherein

wherein the first motion pattern corresponds to one or more motion peaks in the motion graph.

17. The system of claim 15, wherein analyzing the video sequence segment to detect at least one of a pick-up event and a drop event comprises:

detecting the second motion pattern using a portion of the motion graph corresponding to the video sequence segment, wherein the second motion pattern corresponds to a motion valley flanked by two motion peaks in the motion graph.

18. The system of claim 15, wherein the motion graph comprises three portions each associated with three regions of interest included within the video sequence, each region of interest corresponding to the scan, pick-up, and drop events, respectively,

wherein monitoring the video sequence to detect two consecutive scan events comprises identifying the two consecutive scan events by monitoring the portion of the motion graph associated with the region of interest of the scan event,

wherein analyzing the video sequence segment comprises analyzing the portion of the motion graph corresponding to at least one of the pick-up event and drop event.

19. The system of claim 15, wherein the pick-up event comprises a cashier picking up an item from a lead-in portion of the POS system, the scan event comprises the cashier at least one of reading the item via a scanner and weighing the item, and the drop event comprises the cashier placing the item onto a take-away portion of the POS system.

20. The system of claim 15, the operation further comprising, upon detecting the pick-up or drop event, combining one or more of the consecutive scan events and the pick-up or drop event according to one or more temporal ordering constraints to validate a transaction process of the POS system.

21. An apparatus for detecting one or more events, said apparatus comprising:

means for generating a motion graph from a video sequence captured at a point-of-sale (POS) system, the motion graph representing human motion in the video sequence;

means for detecting two consecutive scan events at the POS system by identifying a first reference motion pattern in the motion graph;

means for identifying a segment of the video sequence between the two consecutive scan events; and

means for, upon identifying the video sequence segment, analyzing a portion of the motion graph corresponding to the video sequence segment to detect at least one of a pick-up event and a drop event by identifying a second reference motion pattern in the motion graph.