Method and apparatus for detecting movement patterns at a self-service checkout terminal

• US 6,236,736 B1
• Filed: 02/06/1998
• Issued: 05/22/2001
• Est. Priority Date: 02/07/1997
• Status: Expired due to Fees

• Current Assignee: NCR Voyix Corporation
• Original Assignee: NCR Corporation
• Inventors: Kim, Junhoy, Crabtree, Ralph N., Huang, Jianzhong
• Primary Examiner(s): Au, Amelia M.
• Assistant Examiner(s): Wu, Jingge
• Application Number:
  US09/020,322
• Time in Patent Office: 1,201 Days
• Field of Search: 382/103, 382/104, 382/107, 382/115, 382/100, 235/383, 348/135, 348/155, 348/157, 186/61
• US Class Current: 382/103
• CPC Class Codes: G01S 3/7865 using correlation of the live video image with a stored image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G07G 1/0054 with control of supplementary check-parameters, e.g. weight or number of articles
• CPC Class Codes: G07G 3/006 False operation
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19652 Systems using zones in a single scene defined for different treatment, e.g. outer zone gives pre-alarm, inner zone gives alarm
• CPC Class Codes: G08B 13/1968 Interfaces for setting up or customising the system

Abstract

A system and method for detecting hand and item movement patterns comprising a video camera positioned to view a scene which includes therein a scanner for scanning items, wherein the video camera generates a sequence of video frames representing activity in the scene, processing means coupled to the video camera, the processing means performing steps of identifying regions of a video frame representing a hand; and tracking hand movement with respect to the scanner over a plurality of video frames. Event information descriptive of user activity at the self-service checkout terminal is generated based on the tracking information.

Claims

1. A system for detecting movement patterns with respect to at least one hand of an individual and an item in the individual'"'"'s hand to be purchased at a checkout terminal, comprising:

(a) a video camera positioned to view a scene which includes therein a scanner for scanning items;

(b) frame grabber means coupled to the video camera for generating a stream of video frames representing activity in the scene;

(c) processor means coupled to the frame grabber means, the processor means performing steps of;

(1) detecting whether the hand in a video frame;

(2) tracking hand movement with respect to the scanner over a plurality of video frames and generating tracking information representative thereof including associated X and Y Cartesian coordinates for the location of the hand and the height and width of the hand;

(3) eliminating tracking information corresponding to the detected hand;

(4) determining the presence of the item to be purchased in the detected hand; and

(5) generating event information descriptive of user activity at the checkout terminal based on the tracking information.

2. The system of claim 1, and further comprising storage means, wherein the processor means stores the tracking information and/or event information in the storage means.

3. The system of claim 1, wherein the processor means detects one or more hands by:

(i) generating regions clusters for a video frame which are hypothesis of a hand; and

(ii) evaluating each region cluster with hand model information and generating a confidence value that represents a likelihood that the region cluster represents a hand.

4. The system of claim 3, wherein the processor means generates the region clusters by periodically generating a background image representing background image information for the scene, comparing information for a video frame with the background image to generate regions, and forming region clusters by representing the regions into color regions, and combining one or more color regions based on predetermined criteria.

5. The system of claim 4, wherein the processor means discards those regions which are highly unlikely to contain a hand, prior to regimentation of the regions.

6. The system of claim 4, wherein the processor means forms a hypothesis list of regions clusters in order of highest likelihood of representing a hand.

7. The system of claim 6, wherein the processor means evaluates each region cluster in the hypothesis list by comparing it with hand model information and generating a confidence value representing a degree of match with the hand model information, and wherein a region cluster determined to have confidence value which exceeds a minimum confidence threshold is added to a hand list.

8. The system of claim 7, wherein the processor means is programmed to determine whether an item is associated with a region cluster in the hand list.

9. The system of claim 8, wherein the processor means tracks hand movement by:

(i) determining a measure of correspondence between region clusters in consecutive video frames that are determined to represent a hand;

(ii) constructing hand tracks based on the measure of correspondence of region dusters determined between video frames.

10. The system of claim 1, wherein the processor generates event information for at least one of the following events:

(a) an item has passed over the scanner, including its direction;

(b) an empty hand has passed over the scanner, (c) an item passed over a scanned product zone in the self-service checkout terminal;

(d) an empty hand passed over scanned product zone;

(e) an item was placed in the scanned product zone;

(f) an item was removed from the scanned product zone; and

(g) an item was passed over the scanner and placed in the scanned product zone.

11. A method for detecting movement patterns with respect to at least one hand of an individual and an item in the individual'"'"'s hand to be purchased in a scene of activity at a checkout terminal having a scanner, based on a sequence of video frames representing activity in the scene, comprising steps of:

(a) detecting whether the hand in a video frame;

(b) tracking hand movement with respect to the scanner over a plurality of video frames and generating tracking information representative thereof including associated X and Y Cartesian coordinates for the location of the hand and the height and width of the hand;

(c) eliminating tracking information corresponding to the detected hand;

(d) determining the presence of the item to be purchased in the detected hand; and

(e) generating event information descriptive of user activity at the checkout terminal based on the tracking information.

12. The method of claim 11, and further comprising-the step of storing the tracking information and/or event information.

13. The method of claim 11, wherein the step of detecting one or more hands in a video frame comprises steps of:

(i) generating regions dusters for a video frame which are hypothesis of a hand; and

(ii) evaluating each region cluster with hand model information and generating a confidence value that represents a likelihood that the region cluster represents a hand.

14. The method of claim 13, wherein the step of generating the region clusters comprises steps of periodically generating a background image representing background image information for the scene, comparing image information for a video frame with the background image to generate regions, and forming region clusters by resegmenting the regions into color regions, and combining one or more color regions based on predetermined criteria.

15. The method of claim 14, and further comprising the step of discarding those regions which are highly unlikely to contain a hand, prior to resegmentation of the regions.

16. The method of claim 14, and further comprising the step of forming a hypothesis list of regions clusters in order of highest likelihood of representing a hand.

17. The method of claim 16, wherein the step of evaluating comprises evaluating each region cluster in the hypothesis list by comparing it with hand model information and generating a confidence value representing a degree of match with the hand model information, and wherein a region duster determined to have confidence value which exceeds a confidence threshold is added to a hand list.

18. The method of claim 17, and further comprising the step of determining whether an item is associated with a region cluster in the hand list.

19. The method of claim 18, wherein the step of tracking hand movement comprises steps of:

(i) determining a measure of correspondence between region clusters in consecutive video frames that are determined to represent a hand;

(ii) constructing hand tracks based on the measure of correspondence of region clusters determined between video frames.

20. The method of claim 11, wherein step of generating event information comprises generating information for at least one of the following events:

(a) an item has passed over the scanner, including its direction, (b) an empty hand has passed over the scanner;

(c) an item passed over a scanned product zone in the self-service checkout terminal;

(d) an empty hand passed over scanned product zone;

(e) an item was placed in the scanned product zone;

(f) an item was removed from the scanned product zone, and (g) an item was passed over the scanner and placed in the scanned product zone.

21. The method of claim 11, wherein the step of determining the presence of the item to be purchased in the detected hand comprises steps of:

(a) calculating an average product size for the item over time;

(b) comparing the calculated average product size with a minimum product size threshold; and

(c) determining from the comparison whether the item to be purchased in the detected hand.